如何在使用Hadoop時解決NoClassDefFoundError？

Question

我越來越

Exception in thread "main" java.lang.NoClassDefFoundError: com/linkedin/camus/etl/IEtlKey.

在運行命令：

hadoop jar camus-etl-kafka-0.1.0-SNAPSHOT.jar 
com.linkedin.camus.etl.kafka.CamusJob -P camus.properties 

我得到下面的異常..

2016-04-27 11:34:04.622 java[13567:351959] Unable to load realm mapping info from SCDynamicStore 
[NativeCodeLoader] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
Exception in thread "main" java.lang.NoClassDefFoundError: com/linkedin/camus/etl/IEtlKey 
    at com.linkedin.camus.etl.kafka.CamusJob.run(CamusJob.java:252) 
    at com.linkedin.camus.etl.kafka.CamusJob.run(CamusJob.java:235) 
    at com.linkedin.camus.etl.kafka.CamusJob.run(CamusJob.java:691) 
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) 
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) 
    at com.linkedin.camus.etl.kafka.CamusJob.main(CamusJob.java:646) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:606) 
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221) 
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136) 
Caused by: java.lang.ClassNotFoundException: com.linkedin.camus.etl.IEtlKey 

我已經包含在類路徑camus-example-0.1.0-SNAPSHOT-shaded.jar。

請讓我知道如果我失去了一些東西。

由於提前

Soumyajit

Answer 1

你應該嘗試包括camus-api你可以找到這個LinkedIn's previous generation Kafka to HDFS pipeline頁面上，因爲缺少類包含在這個包，你可以看到here。

注意加繆可能需要的其他傳遞依賴。

此外，以確保課程將在類路徑中找到，當你在命令行中使用hadoop jar，您可以添加libjars命令行選項，如Using the libjars option with Hadoop報道：

$ export LIBJARS=/path/jar1,/path/jar2 
$ hadoop jar my-example.jar com.example.MyTool -libjars ${LIBJARS} -mytoolopt value 

可能知道Camus將會被Gobblin所取代：

Camus is being phased out and replaced by Gobblin. For those using or interested in Camus, we suggest taking a look at Gobblin.

For instructions on Migrating from Camus to Gobblin , please take a look at Camus Gobblin Migration .