spark java.lang.AbstractMethodError在cloudera上失敗

Question

下面是我的POM文件。我對Cloudera的5.10

星火計劃的火花運行期間寫的火花與AWS流室壁運動

<dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-streaming_2.10</artifactId> 
     <version>1.6.0</version> 
    </dependency> 

    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-core_2.10</artifactId> 
     <version>1.6.0</version> 
    </dependency> 
    <dependency> 
     <groupId>com.amazonaws</groupId> 
     <artifactId>amazon-kinesis-client</artifactId> 
     <version>1.6.1</version> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-streaming-kinesis-asl_2.10</artifactId> 
     <version>1.6.0</version> 
    </dependency> 

我面對異常下面

17/04/27 05:34:04 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 58.0 (TID 179, hadoop1.local, executor 5): java.lang.AbstractMethodError 
at org.apache.spark.Logging$class.log(Logging.scala:50) 
at org.apache.spark.streaming.kinesis.KinesisCheckpointer.log(KinesisCheckpointer.scala:39) 
at org.apache.spark.Logging$class.logDebug(Logging.scala:62) 
at org.apache.spark.streaming.kinesis.KinesisCheckpointer.logDebug(KinesisCheckpointer.scala:39) 
at org.apache.spark.streaming.kinesis.KinesisCheckpointer.startCheckpointerThread(KinesisCheckpointer.scala:119) 
at org.apache.spark.streaming.kinesis.KinesisCheckpointer.<init>(KinesisCheckpointer.scala:50) 
at org.apache.spark.streaming.kinesis.KinesisReceiver.onStart(KinesisReceiver.scala:149) 
at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:148) 
at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:130) 
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:575) 
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:565) 
at org.apache.spark.SparkContext$$anonfun$38.apply(SparkContext.scala:2000) 
at org.apache.spark.SparkContext$$anonfun$38.apply(SparkContext.scala:2000) 
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) 
at org.apache.spark.scheduler.Task.run(Task.scala:89) 
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745) 

這個運行在EMR4.4完全正常不過CDH失敗。任何建議

Answer 1

根本的問題似乎是使用org.apache.spark.Logging的：

注：請不要使用星火之外此類。 它被用作內部工具。 在將來的版本中，這可能會被更改或刪除。

http://spark.apache.org/docs/1.6.2/api/java/org/apache/spark/Logging.html

如https://issues.apache.org/jira/browse/SPARK-9307提到這是固定在2.0.0。