我是否缺少任何導入？

Question

在運行

> spark-submit --class "TwitterPopularTags" --master local[2]  >/home/raja/begin/target/scala-2.11/simple-project_2.11-1.0.jar 

> "Exception in thread "main" java.lang.NoClassDefFoundError:      >org/apache/spark/Logging 
>... 
>Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging 
>... 
>" 

我SBT是好的：乾淨，重新加載，編譯，打包，但是當我跑上述火花提交，我得到了上面的世界著名的錯誤。

>My simple.sbt: 

> name := "Simple Project" 
> version := "1.0" 
> scalaVersion := "2.11.6" 

>libraryDependencies ++= Seq(
> "org.apache.spark" %% "spark-core" % "2.0.1", 
> "org.apache.spark" %% "spark-streaming" % "2.0.1", 
> "org.apache.spark" %% "spark-streaming-twitter" % "1.6.2", 
> "com.google.code.gson" % "gson" % "2.7", 
> "org.twitter4j" % "twitter4j-core" % "4.0.4", 
> "org.twitter4j" % "twitter4j-stream" % "4.0.4", 
>"org.apache.logging.log4j" % "log4j-slf4j-impl" % "2.7" 
>) 

>resolvers += "Maven Central" at "https://repo1.maven.org/maven2/" 
>resolvers += "Akka Repository" at "http://repo.akka.io/releases/" 

我進口：

>import org.apache.spark.streaming.StreamingContext._ 
>import org.apache.spark.streaming.dstream.DStream 
>import org.apache.spark.streaming.twitter._ 
>import org.apache.spark.streaming.{Seconds, StreamingContext} 
>import org.apache.spark.{SparkConf, SparkContext} 

>import org.slf4j.{Logger, LoggerFactory} 
>import org.slf4j.impl.StaticLoggerBinder 
>import org.apache.log4j.{Level, LogManager, PropertyConfigurator} 
>import org.apache.log4j.Logger 
>import twitter4j.auth.OAuthAuthorization 
>import twitter4j.conf.ConfigurationBuilder 
>import twitter4j.Twitter 
>import twitter4j.Status 
>import twitter4j.auth.Authorization 
>import twitter4j.TwitterFactory 

請不要告訴我改版本。我配置了log4j.properties，屬性但沒有機會。 SLF4J還是Logback我不得不看？我試圖從maven那裏獲得該版本的火星核心jar，但沒有機會。

任何人都可以針點我在這裏

感謝， 拉賈

Answer 1

您有不正確的依賴關係：

• 星火2.0
• 星火Twitter的連接器1.6

Twitter的連接器已在Spark 2.0中刪除。您可以使用Apache Bahir或將Spark降級到1.6。