Spark作業服務器中的Spark作業「java.lang.NoClassDefFoundError：org/apache/spark/sql/SQLContext」的錯誤

我使用IntelliJ創建了一個spark作業，並且我希望它由spark作業服務器加載並運行。爲此，我按照此鏈接中的步驟操作：http://github.com/ooyala/spark-jobserver 而我的火花版本是1.4.0。Spark作業服務器中的Spark作業「java.lang.NoClassDefFoundError：org/apache/spark/sql/SQLContext」的錯誤

這是我的項目中Scala代碼：

import org.apache.spark.sql.DataFrame 
import org.apache.spark.sql.hive.HiveContext 

import org.apache.spark.{SparkConf, SparkContext} 

import scala.collection.mutable.ArrayBuffer 
//spark job server 
import com.typesafe.config.{Config, ConfigFactory} 
import scala.util.Try 
import spark.jobserver.SparkJob 
import spark.jobserver.SparkJobValidation 
import spark.jobserver.SparkJobValid 
import spark.jobserver.SparkJobInvalid 

class hiveSparkRest extends SparkJob { 
    var idCard:String ="" 


    def main(args: Array[String]): Unit = { 


    val sc = new SparkContext("local[4]", "SmartApp") 
    val config = ConfigFactory.parseString("") 

    val results = runJob(sc, config) 
    println("Result is " + results) 


    enterTimesMax(sc, hiveContext) 

    } 


    override def validate(sc: SparkContext, config: Config): SparkJobValidation = { 
    Try(config.getString("input.string")) 
     .map(x => SparkJobValid) 
     .getOrElse(SparkJobInvalid("No input.string config param")) 
    } 

    override def runJob(sc: SparkContext,config: Config): Any = { 
    idCard = config.getString("input.string") 
    enterTimesMax(sc, hiveContext) 
    } 

    def enterTimesMax(sc:SparkContext,hiveContext:HiveContext): Unit = { 
    val hiveContext = new HiveContext(sc) 
    hiveContext.sql("use default") 

    val sqlUrl = "select max(num) from (select idcard,count(1) as num from passenger group by idcard)as t" 

    val idCardArray = hiveContext.sql(sqlUrl).collect() 


    } 
}

但是，當我執行它，我得到捲曲：（52）空從服務器與此錯誤火花作業服務器回覆：

> job-server[ERROR] Uncaught error from thread [JobServer-akka.actor.default-dispatcher-12] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[JobServer] 
job-server[ERROR] java.lang.NoClassDefFoundError: org/apache/spark/sql/SQLContext 
job-server[ERROR] at java.lang.ClassLoader.defineClass1(Native Method) 
job-server[ERROR] at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) 
job-server[ERROR] at java.lang.ClassLoader.defineClass(ClassLoader.java:615) 
job-server[ERROR] at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) 
job-server[ERROR] at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) 
job-server[ERROR] at java.net.URLClassLoader.access$000(URLClassLoader.java:58) 
job-server[ERROR] at java.net.URLClassLoader$1.run(URLClassLoader.java:197) 
job-server[ERROR] at java.security.AccessController.doPrivileged(Native Method) 
job-server[ERROR] at java.net.URLClassLoader.findClass(URLClassLoader.java:190) 
job-server[ERROR] at java.lang.ClassLoader.loadClass(ClassLoader.java:306) 
job-server[ERROR] at java.lang.ClassLoader.loadClass(ClassLoader.java:247) 
job-server[ERROR] at sql.hiveSparkRest.shadePassenger(hiveSparkRest.scala:62) 
job-server[ERROR] at sql.hiveSparkRest.runJob(hiveSparkRest.scala:56) 
job-server[ERROR] at spark.jobserver.JobManagerActor$$anonfun$spark$jobserver$JobManagerActor$$getJobFuture$4.apply(JobManagerActor.scala:222) 
job-server[ERROR] at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) 
job-server[ERROR] at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) 
job-server[ERROR] at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42) 
job-server[ERROR] at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) 
job-server[ERROR] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) 
job-server[ERROR] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) 
job-server[ERROR] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) 
job-server[ERROR] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 
job-server[ERROR] Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SQLContext 
job-server[ERROR] at java.net.URLClassLoader$1.run(URLClassLoader.java:202) 
job-server[ERROR] at java.security.AccessController.doPrivileged(Native Method) 
job-server[ERROR] at java.net.URLClassLoader.findClass(URLClassLoader.java:190) 
job-server[ERROR] at java.lang.ClassLoader.loadClass(ClassLoader.java:306) 
job-server[ERROR] at java.lang.ClassLoader.loadClass(ClassLoader.java:247) 
job-server[ERROR] ... 22 more 
job-server ... finished with exit code 255

似乎HiveContext是由spark jar文件支持的spark-assembly-1.4.0-hadoop1.0.4.jar。

來源

2016-01-26 Robin

你確定你已經添加[火花SQL]（http://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.10/1.4.0）JAR依賴？ –

@FelipeAlmeida Thx的幫助！哪個spark-sql JAR文件是你的意思？我認爲Spark類中的spark-assembly-1.4.0-hadoop1.0.4.jar支持類SQLContext。 – Robin

你可以在我給你的鏈接上找到這個罐子。您需要將它添加到您的依賴文件（pom.xml，build.sbt或同等文件）。 –

我不認爲ooyala回購是主要的。在維護的repo中，下面的鏈接顯示了使用HiveContext的測試工作。對於SparkHiveJob特質，您需要jobserver-extras jar。

https://github.com/spark-jobserver/spark-jobserver/blob/master/job-server-extras/src/spark.jobserver/HiveTestJob.scala

來源

2016-01-26 05:20:52 noorul

感謝Norul的幫助。我會嘗試。 – Robin

似乎答案是正確的！但是，當我運行我的應用程序（我的斯卡拉類延伸SparkSqlJob），我得到了以下的回答： { 「狀態」：「ERROR」，「結果」：「無效的作業類型的上下文中，」 } 你能幫我解決這個問題嗎？我提出了一個新問題來跟蹤它，謝謝！ http://stackoverflow.com/questions/35032545/the-error-invalid-job-type-for-this-context-in-spark-sql-job-with-spark-jobse – Robin

Spark作業服務器中的Spark作業「java.lang.NoClassDefFoundError：org/apache/spark/sql/SQLContext」的錯誤

回答

相關問題