星火之間不同的行爲火花提交和火花外殼

林星火1.3.1（在Ubuntu 14.04）stand alone，SBT 0.13.10，並試圖執行以下腳本：星火之間不同的行爲火花提交和火花外殼

package co.some.sheker 
import java.sql.Date 
import org.apache.spark.{SparkContext, SparkConf} 
import SparkContext._ 
import org.apache.spark.sql.{Row, SQLContext} 
import com.datastax.spark.connector._ 
import java.sql._ 
import org.apache.spark.sql._ 
import org.apache.spark.sql.cassandra.CassandraSQLContext 
import java.io.PushbackReader 
import java.lang.{ StringBuilder => JavaStringBuilder } 
import java.io.StringReader 
import com.datastax.spark.connector.cql.CassandraConnector 
import org.joda.time.{DateTimeConstants} 

case class TableKey(key1: String, key2: String) 

object myclass{ 
    def main(args: scala.Array[String]) { 
    val conf = ... 
    val sc = new SparkContext(conf) 
    val sqlContext = new SQLContext(sc) 
    val csc = new CassandraSQLContext(sc) 
    val data_x = csc.sql("select distinct key1, key2 from keyspace.table where key1 = 'sheker'").map(row => (row(0).toString, row(1).toString)) 
    println("Done cross mapping") 
    val snapshotsFiltered = data_x.map(x => TableKey(x._1,x._2)).joinWithCassandraTable("keyspace", "table") 
    println("Done join") 
    val jsons = snapshotsFiltered.map(_._2.getString("json")) 
... 

    sc.stop() 
    println("Done.") 
} 
}

使用：

/home/user/spark-1.3.1/bin/spark-submit --master spark://1.1.1.1:7077 --driver-class-path /home/user/spark-cassandra-connector-java-assembly-1.3.1-FAT.jar --properties-file prop.conf --class "myclass" "myjar.jar"

的prop.conf文件是：

spark.cassandra.connection.host myhost 
spark.serializer org.apache.spark.serializer.KryoSerializer 
spark.eventLog.enabled true 
spark.eventLog.dir /var/tmp/eventLog 
spark.executor.extraClassPath /home/ubuntu/spark-cassandra-connector-java-assembly-1.3.1-FAT.jar

我得到這個例外：

Done cross mapping 
Exception in thread "main" java.lang.NoSuchMethodError: com.datastax.spark.connector.mapper.ColumnMapper$.defaultColumnMapper(Lscala/reflect/ClassTag;Lscala/reflect/api/TypeTags$TypeTag;)Lcom/datastax/spark/connector/mapper/ColumnMapper; 
    at co.crowdx.aggregation.CassandraToElasticTransformater$.main(CassandraToElasticTransformater.scala:79) 
    at co.crowdx.aggregation.CassandraToElasticTransformater.main(CassandraToElasticTransformater.scala) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:606) 
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569) 
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166) 
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189) 
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110) 
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 
Done Sending Signal aggregation job to Spark

和奇怪的部分是，當我試圖在外殼的運行從腳本 - 命令做工精細。即時通訊使用：

/home/user/spark-1.3.1/bin/spark-shell --master spark://1.1.1.1:7077 --driver-class-path /home/ubuntu/spark-cassandra-connector-java-assembly-1.3.1-FAT.jar --properties-file prop.conf

的Build.scala文件：

import sbt._ 
import Keys._ 
import sbtassembly.Plugin._ 
import AssemblyKeys._ 

object AggregationsBuild extends Build { 
    lazy val buildSettings = Defaults.defaultSettings ++ Seq(
version := "1.0.0", 
organization := "co.sheker", 
scalaVersion := "2.10.4" 
) 

lazy val app = Project(
"geo-aggregations", 
file("."), 
settings = buildSettings ++ assemblySettings ++ Seq(
    parallelExecution in Test := false, 
    libraryDependencies ++= Seq(
    "com.datastax.spark" %% "spark-cassandra-connector" % "1.2.1", 
    // spark will already be on classpath when using spark-submit. 
    // marked as provided, so that it isn't included in assembly. 
    "org.apache.spark" %% "spark-core" % "1.2.1" % "provided", 
    "org.apache.spark" %% "spark-catalyst" % "1.2.1" % "provided", 
    "org.apache.spark" %% "spark-sql" % "1.2.1" % "provided", 
    "org.scalatest" %% "scalatest" % "2.1.5" % "test", 
    "org.postgresql" % "postgresql" % "9.4-1201-jdbc41", 
"com.github.nscala-time" %% "nscala-time" % "2.4.0", 
"org.elasticsearch" % "elasticsearch-hadoop" % "2.2.0" % "provided" 
), 
    resolvers += "conjars.org" at "http://conjars.org/repo", 
resolvers += "clojars" at "https://clojars.org/repo" 
) 
) 
}

有什麼不對？爲什麼它在提交時失敗，但不在shell中？

來源

2016-05-31 Rada

你是如何創建提交的jar？你需要做一個裝配，而不是一個包裝。 – Reactormonk

@Reactormonk - 是的，我做「sbt程序集」給我的「myjar.jar」文件 – Rada

你說你使用的是Spark 1.3，但你的版本包含Spark 1.2.1的依賴關係。你檢查了嗎？我相信你的火花驅動程序的版本與你應用程序中的版本不同，導致你得到的錯誤。 – eliasah

你說，你正在使用的火花1.3，但你的版本包含火花1.2.1依賴。

就像我在註釋中說，我相信你的火花驅動程序的版本是從一個在你的應用程序，它會導致你得到的錯誤不同。

來源

2016-05-31 12:44:32 eliasah

星火之間不同的行爲火花提交和火花外殼

回答

相關問題