火花1.3.1 SQL庫：異常在線程 「主」 java.lang.NoSuchMethodError：org.apache.spark.sql.SQLContext.implicits（）

Question

下面的代碼在Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.SQLContext.implicits()Lorg/apache/spark/sql/SQLContext$implicits$

import org.apache.spark.SparkConf 
import org.apache.spark.SparkContext 
import org.apache.spark.SparkContext._ 
import org.apache.log4j.Logger 
import org.apache.log4j.Level 

object Small { 

    def main(args: Array[String]) { 
    Logger.getLogger("org.apache.spark").setLevel(Level.WARN) 
    Logger.getLogger("org.eclipse.jetty.server").setLevel(Level.OFF) 
    // set up environment 
    val conf = new SparkConf() 
     .setMaster("local[1]") 
     .setAppName("Small") 
     .set("spark.executor.memory", "2g") 
    val sc = new SparkContext(conf) 

    val sqlContext = new org.apache.spark.sql.SQLContext(sc) 
    import sqlContext.implicits._ 
    val df = sc.parallelize(Array((1,30),(2,10),(3,20),(1,10), (2,30))).toDF("books","readers") 
    df.show 

項目結果是建有SBT：

name := "Small" 

version := "1.0" 

scalaVersion := "2.10.4" 

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.3.1" 

libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.3.1" 

我運行這個與提交腳本：

#!/bin/sh 
/home/test/usr/spark-1.1.0/bin/spark-submit \ 
--class Small \ 
--master local[*] \ 
--driver-memory 2g \ 
/home/test/wks/Pairs/target/scala-2.10/small_2.10-1.0.jar 

任何想法？

SBT編譯並打包此代碼。然而，當我嘗試使用sbt run運行這段代碼，我得到另一個異常：[error] (run-main-0) scala.reflect.internal.MissingRequirementError: class org.apache.spark.sql.catalyst.ScalaReflection in JavaMirror with java.net.URLClassLoader@210ce673 of type class java.net.URLClassLoader with classpath [file:/home/test/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.10.4.jar, ...

有沒有什麼辦法讓sbt run包括所有依賴斯卡拉啓動程序時？

Answer 1

「/home/test/usr/spark-1.1.0/bin/spark-submit」 您的編譯版本是1.3.1，它與運行時版本不同。 版本1.1.0的SqlContext沒有定義「對象隱含」。