2017-01-30 44 views
0

我試圖建立一個超級罐子,所以我可以部署我的星火計劃這樣做:星火2 SBT裝配重複數據刪除錯誤斯卡拉2.11.8

運行:

sbt assembly 

它輸出了很多錯誤的:

[error] deduplicate: different file contents found in the following: 
[error] /Users/samibadawi/.ivy2/cache/commons-collections/commons-collections/jars/commons-collections-3.2.1.jar:org/apache/commons/collections/FastHashMap$CollectionView$CollectionViewIterator.class 
[error] /Users/samibadawi/.ivy2/cache/commons-beanutils/commons-beanutils/jars/commons-beanutils-1.7.0.jar:org/apache/commons/collections/FastHashMap$CollectionView$CollectionViewIterator.class 

的有關斯卡拉2.10問題的答案沒有工作: spark + sbt-assembly: "deduplicate: different file contents found in the following"

經過多次黑客攻擊之後,我收到了一個hello world項目,沒有使用以下build.sbt文件進行編譯的有用代碼:

這似乎是隨機進入排除和進入合併策略的內容。有沒有更簡單更系統的方法來做到這一點?

(除了使用: 「org.apache.spark」 %% 「火花核」 %sparkVersion% 「提供」, 在這種情況下,不存在部署的依賴關係。)

build.sbt摘錄:

import sbtassembly.AssemblyPlugin._ 

//Define dependencies. These ones are only required for Test and Integration Test scopes. 
libraryDependencies ++= Seq(
    ("org.apache.spark" %% "spark-core" % sparkVersion). 
    exclude("commons-beanutils", "commons-beanutils-core"). 
    exclude("commons-collections", "commons-collections"). 
    exclude("commons-logging", "commons-logging"). 
    exclude("com.esotericsoftware.minlog", "minlog"). 
    exclude("com.codahale.metrics", "metrics-core"). 
    exclude("aopalliance","aopalliance") 
    , 
    "org.scalatest" %% "scalatest" % "2.2.4" % "test,it" 
) 

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) => 
    { 
    case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last 
    case PathList("javax", "inject", xs @ _*) => MergeStrategy.last 
    case PathList("javax", "activation", xs @ _*) => MergeStrategy.last 
    case PathList("org", "apache", xs @ _*) => MergeStrategy.last 
    case PathList("com", "google", xs @ _*) => MergeStrategy.last 
    case PathList("com", "esotericsoftware", xs @ _*) => MergeStrategy.last 
    case PathList("com", "codahale", xs @ _*) => MergeStrategy.last 
    case PathList("com", "yammer", xs @ _*) => MergeStrategy.last 
    case "about.html" => MergeStrategy.rename 
    case "META-INF/ECLIPSEF.RSA" => MergeStrategy.last 
    case "META-INF/mailcap" => MergeStrategy.last 
    case "META-INF/mimetypes.default" => MergeStrategy.last 
    case "plugin.properties" => MergeStrategy.last 
    case "log4j.properties" => MergeStrategy.last 
    case x => old(x) 
    } 
} 

Project.inConfig(Test)(assemblySettings) 

回答

0

確實有點多線索的錯誤,並作出了我的真正的程序工作了build.sbt:

一個問題,我是Postgres的一個罐子版本的重複問題。 我解決了這個註釋掉這些依賴關係:

// "org.postgresql" % "postgresql" % "9.4.1212", //Small gap between Doobie and Spark dependency 
// "org.postgis" % "postgis-jdbc" % "1.3.3", //Causes conflicts 

我還沒有開始使用PostGIS中,它有PostgreSQL相關8.3-603.jdbc4.jar的依賴

我不得不取出Postgres的直接依賴。

從工作build.sbt:

val doobieVersion = "0.4.1" 

libraryDependencies ++= Seq(
    "ch.qos.logback" % "logback-classic" % "1.0.13", //comment and warning go away 
    "ch.qos.logback" % "logback-core" % "1.0.13", 
    "com.citymaps" % "tile-library" % "1.4", 
    "com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.7.2", 
    "com.github.scopt" %% "scopt" % "3.5.0", 
    "com.typesafe.play" %% "play-json" % "2.5.9", 
    "org.apache.spark" %% "spark-core" % sparkVersion % "provided", 
    "org.apache.spark" %% "spark-sql" % sparkVersion % "provided", 
    "org.apache.spark" %% "spark-mllib" % sparkVersion % "provided", 
    "graphframes" % "graphframes" % "0.3.0-spark2.0-s_2.11", 
    "org.clapper" %% "grizzled-slf4j" % "1.3.0", 
// "org.postgresql" % "postgresql" % "9.4.1212", //Small gap between Doobie and Spark dependency 
// "org.postgis" % "postgis-jdbc" % "1.3.3", //Causes conflicts 
    "org.scalatest" %% "scalatest" % "3.0.0" % "test" withSources() withJavadoc(), 
    "org.spire-math" %% "spire" % "0.11.0", 
    "org.tpolecat" %% "doobie-core-cats" % doobieVersion, 
    "org.tpolecat" %% "doobie-postgres-cats" % doobieVersion 
) 

運行

sbt clean 

在這之後停止工作。 原來,postgis-jdbc有一個衝突,最後一個版本是2.2.1,但是正常Maven倉庫中可用的最後一個版本是1.3.3,並且依賴於舊的Postgres驅動程序jar。

看了很多回購,找不到postgis-jdbc 2.2.1。

下載2.2.1版本 https://github.com/postgis/postgis-java

本次發佈版本設置爲2.2.2SNAPSHOT。因此,在pom.xml和jdbc/pom.xml中更改版本號

使用此命令構建jar。它是挑剔的Maven版本:

/usr/local/Cellar/maven/3.3.9/bin/mvn install 

現在包括這個依賴

resolvers ++= Seq(
    Resolver.mavenLocal 

"net.postgis" % "postgis-jdbc" % "2.2.1", 

和運行

sbt assembly 

它終於奏效。