Spark Streaming Kafka java.lang.ClassNotFoundException：org.apache.kafka.common.serialization.StringDeserializer

Question

我正在使用與Kafka集成的Spark流，當我在本地模式下從我的IDE運行流應用程序時，一切都可以作爲魅力。但是當我將其提交到羣集我保持具有以下錯誤：

java.lang.ClassNotFoundException: org.apache.kafka.common.serialization.StringDeserializer

我使用SBT組件來構建我的項目。

我SBT是這樣的：

libraryDependencies ++= Seq(
    "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0" % Provided, 
    "org.apache.spark" % "spark-core_2.11" % "2.2.0" % Provided, 
    "org.apache.spark" % "spark-streaming_2.11" % "2.2.0" % Provided, 
    "org.marc4j" % "marc4j" % "2.8.2", 
    "net.sf.saxon" % "Saxon-HE" % "9.7.0-20" 
) 


run in Compile := Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run)).evaluated 


mainClass in assembly := Some("EstimatorStreamingApp") 

我也試過用--package選項

嘗試1

--packages org.apache.spark:spark-streaming-kafka-0-10_2.11:2.2.0 

嘗試2

--packages org.apache.spark:spark-streaming-kafka-0-10-assembly_2.11:2.2.0 

所有沒有成功。有沒有人有什麼建議

Answer 1

您需要從卡夫卡的依賴去掉「規定」的標誌，因爲它不是與星火提供開箱即用的依賴：

libraryDependencies ++= Seq(
    "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0", 
    "org.apache.spark" % "spark-core_2.11" % "2.2.0" % Provided, 
    "org.apache.spark" % "spark-streaming_2.11" % "2.2.0" % Provided, 
    "org.marc4j" % "marc4j" % "2.8.2", 
    "net.sf.saxon" % "Saxon-HE" % "9.7.0-20" 
)