build.sbt：如何添加火花依賴

Question

你好我想下載spark-core，spark-streaming，twitter4j和spark-streaming-twitter在下面的build.sbt文件：

name := "hello" 

version := "1.0" 

scalaVersion := "2.11.8" 

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1" 
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1" 

libraryDependencies ++= Seq(
    "org.twitter4j" % "twitter4j-core" % "3.0.3", 
    "org.twitter4j" % "twitter4j-stream" % "3.0.3" 
) 

libraryDependencies += "org.apache.spark" % "spark-streaming-twitter_2.10" % "0.9.0-incubating" 

我只是把這個libraryDependencies在線等我不確定要使用哪些版本等。

有人可以請向我解釋我應該如何解決這個.sbt文件。我花了幾個小時試圖弄清楚，但沒有任何建議起作用。我安裝scala通過自制軟件，我的版本2.11.8

我所有的錯誤都是關於：

Modules were resolved with conflicting cross-version suffixes. 

Answer 1

的問題是，你混合斯卡拉2.11和2.10的文物。您有：

scalaVersion := "2.11.8" 

然後：

libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1" 

當被要求2.10神器。您還混合星火版本，而不是使用一個一致的版本：

// spark 1.6.1 
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1" 

// spark 1.4.1 
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1" 

// spark 0.9.0-incubating 
libraryDependencies += "org.apache.spark" % "spark-streaming-twitter_2.10" % "0.9.0-incubating" 

這裏是一個build.sbt能解決兩個問題：

name := "hello" 

version := "1.0" 

scalaVersion := "2.11.8" 

val sparkVersion = "1.6.1" 

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % sparkVersion, 
    "org.apache.spark" %% "spark-streaming" % sparkVersion, 
    "org.apache.spark" %% "spark-streaming-twitter" % sparkVersion 
) 

你也不需要手動添加twitter4j的依賴，因爲它們是由spark-streaming-twitter傳遞添加。