我試圖建立一個超級罐子,所以我可以部署我的星火計劃這樣做:星火2 SBT裝配重複數據刪除錯誤斯卡拉2.11.8
運行:
sbt assembly
它輸出了很多錯誤的:
[error] deduplicate: different file contents found in the following:
[error] /Users/samibadawi/.ivy2/cache/commons-collections/commons-collections/jars/commons-collections-3.2.1.jar:org/apache/commons/collections/FastHashMap$CollectionView$CollectionViewIterator.class
[error] /Users/samibadawi/.ivy2/cache/commons-beanutils/commons-beanutils/jars/commons-beanutils-1.7.0.jar:org/apache/commons/collections/FastHashMap$CollectionView$CollectionViewIterator.class
的有關斯卡拉2.10問題的答案沒有工作: spark + sbt-assembly: "deduplicate: different file contents found in the following"
經過多次黑客攻擊之後,我收到了一個hello world項目,沒有使用以下build.sbt文件進行編譯的有用代碼:
這似乎是隨機進入排除和進入合併策略的內容。有沒有更簡單更系統的方法來做到這一點?
(除了使用: 「org.apache.spark」 %% 「火花核」 %sparkVersion% 「提供」, 在這種情況下,不存在部署的依賴關係。)
build.sbt摘錄:
import sbtassembly.AssemblyPlugin._
//Define dependencies. These ones are only required for Test and Integration Test scopes.
libraryDependencies ++= Seq(
("org.apache.spark" %% "spark-core" % sparkVersion).
exclude("commons-beanutils", "commons-beanutils-core").
exclude("commons-collections", "commons-collections").
exclude("commons-logging", "commons-logging").
exclude("com.esotericsoftware.minlog", "minlog").
exclude("com.codahale.metrics", "metrics-core").
exclude("aopalliance","aopalliance")
,
"org.scalatest" %% "scalatest" % "2.2.4" % "test,it"
)
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last
case PathList("javax", "inject", xs @ _*) => MergeStrategy.last
case PathList("javax", "activation", xs @ _*) => MergeStrategy.last
case PathList("org", "apache", xs @ _*) => MergeStrategy.last
case PathList("com", "google", xs @ _*) => MergeStrategy.last
case PathList("com", "esotericsoftware", xs @ _*) => MergeStrategy.last
case PathList("com", "codahale", xs @ _*) => MergeStrategy.last
case PathList("com", "yammer", xs @ _*) => MergeStrategy.last
case "about.html" => MergeStrategy.rename
case "META-INF/ECLIPSEF.RSA" => MergeStrategy.last
case "META-INF/mailcap" => MergeStrategy.last
case "META-INF/mimetypes.default" => MergeStrategy.last
case "plugin.properties" => MergeStrategy.last
case "log4j.properties" => MergeStrategy.last
case x => old(x)
}
}
Project.inConfig(Test)(assemblySettings)