2017-09-15 68 views
0

我有一個使用breeze的spark代碼。我可以選擇我的項目的微風版本:

$ gradle dependencies | grep breeze 
    | | +--- org.scalanlp:breeze_2.11:0.12 
    | | | +--- org.scalanlp:breeze-macros_2.11:0.12 
    +--- org.scalanlp:breeze_2.11:0.12 (*) 
    | | +--- org.scalanlp:breeze_2.11:0.12 
    | | | +--- org.scalanlp:breeze-macros_2.11:0.12 
    +--- org.scalanlp:breeze_2.11:0.12 (*) 
    | | +--- org.scalanlp:breeze_2.11:0.12 
    | | | +--- org.scalanlp:breeze-macros_2.11:0.12 
    +--- org.scalanlp:breeze_2.11:0.12 (*) 
| | | +--- org.scalanlp:breeze_2.11:0.12 
| | | | +--- org.scalanlp:breeze-macros_2.11:0.12 
| +--- org.scalanlp:breeze_2.11:0.12 (*) 
| | | +--- org.scalanlp:breeze_2.11:0.12 
| | | | +--- org.scalanlp:breeze-macros_2.11:0.12 
| +--- org.scalanlp:breeze_2.11:0.12 (*) 

spark中包含的breeze版本是0.12。我可以看到這個尋找火花罐子目錄:

spark-2.1.1-bin-hadoop2.4$ find . -name *.jar | grep breeze 
./jars/breeze_2.11-0.12.jar 
./jars/breeze-macros_2.11-0.12.jar 

但是,當我提交作業火花(甚至地方)我得到這個錯誤:使用

java.lang.NoSuchMethodError: breeze.linalg.tile$.tile_DM_Impl2(Lscala/reflect/ClassTag;Lbreeze/storage/Zero;Lbreeze/generic/UFunc$InPlaceImpl2;)Lbreeze/generic/UFunc$UImpl2; 
    at mypackage.MyClass.calcOne(MyClass.scala:51) 
    at mypackage.MyClass$$anonfun$1.apply(MyClass.scala:36) 
    at mypackage.MyClass$$anonfun$1.apply(MyClass.scala:35) 
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) 
    at scala.collection.Iterator$class.foreach(Iterator.scala:893) 
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) 
    at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157) 
    at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1336) 
    at scala.collection.TraversableOnce$class.fold(TraversableOnce.scala:212) 
    at scala.collection.AbstractIterator.fold(Iterator.scala:1336) 
    at org.apache.spark.rdd.RDD$$anonfun$fold$1$$anonfun$20.apply(RDD.scala:1044) 

命令行:

spark-2.1.1-bin-hadoop2.4/bin/spark-submit --class my.Main myjar.jar 

回答

0

發現了問題:

SPARK_HOME environme nt變量被設置爲一箇舊的火花版本。

所以斌/火花級在這個其他的路徑正在尋找罐子依賴