Spark java.io.InvalidClassException：org.apache.spark.unsafe.types.UTF8String;本地類不兼容

Question

我試圖在遠程集羣上運行我的Spark應用程序，並且出現序列化錯誤。 Scala和Spark版本是一樣的。我被困在這一點上。羣集上

火花殼-version：

root@a913008dd071:/usr/local/spark-2.1.1# ./bin/spark-submit --version 
Welcome to 
     ____    __ 
    /__/__ ___ _____/ /__ 
    _\ \/ _ \/ _ `/ __/ '_/ 
    /___/ .__/\_,_/_/ /_/\_\ version 2.1.1 
     /_/ 

Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_131 
Branch 
Compiled by user jenkins on 2017-04-25T23:51:10Z 
Revision 
Url 
Type --help for more information. 

build.sbt

import sbt.ExclusionRule 

name := "hxfa" 
version := "1.0" 
scalaVersion := "2.11.8" 

val elasticVersion = "5.4.1" 

resolvers += "Spark Packages" at "https://dl.bintray.com/spark-packages/maven/" 
resolvers += "Additional spark packages" at "https://dl.bintray.com/sbcd90/org.apache.spark" 
resolvers += "Apache HBase" at "https://repository.apache.org/content/repositories/releases" 
resolvers += "Thrift" at "http://people.apache.org/~rawson/repo/" 
resolvers += "Spring Plugins" at "http://repo.spring.io/plugins-release/" 

/* Dependencies */ 
libraryDependencies ++= Seq(
    // Framework and configuration 
    "org.springframework.boot" % "spring-boot-starter-web" % "1.5.4.RELEASE", 
    "org.hibernate" % "hibernate-validator" % "5.2.4.Final", 

    /* Serializations */ 
    "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7", 
    "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7", 
    "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.8.7", 
    "com.esotericsoftware" % "kryo" % "4.0.0", 


    // Spark and utilities 
    "org.apache.spark" %% "spark-core" % "2.1.0", 
    "org.apache.spark" %% "spark-sql" % "2.1.0" , 
    "org.apache.spark" %% "spark-mllib" % "2.1.0" , 
    "graphframes" % "graphframes" % "0.5.0-spark2.1-s_2.11", 


    // Spark connectors 
    "org.elasticsearch" % "elasticsearch-spark-20_2.11" % elasticVersion, 
    "org.mongodb.spark" % "mongo-spark-connector_2.11" % "2.0.0", 


    //JDBC 
    "mysql" % "mysql-connector-java" % "5.1.35", 

    // HBase 
    "org.apache.hbase" % "hbase" % "1.2.4", 
    "org.apache.hbase" % "hbase-client" % "1.2.4", 
    "org.apache.hbase" % "hbase-common" % "1.2.4", 

    // OrientDB 
    "com.orientechnologies" % "orientdb-graphdb" % "2.2.20" 

).map(_.excludeAll(ExclusionRule("org.slf4j", "slf4j-log4j12"), ExclusionRule("log4j", "log4j"))) 

libraryDependencies ++= Seq(
    "org.apache.hbase" % "hbase-server" % "1.2.4" 
).map(_.excludeAll(
    ExclusionRule("com.sun.jersey", "jersey-server"), 
    ExclusionRule("tomcat"), 
    ExclusionRule("log4j", "log4j") 
)) 


/* Assembly  */ 

mainClass in assembly := Some("com.x.x.hello.app.HX") 
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false, includeDependency = false) 

assemblyMergeStrategy in assembly := { 
    case PathList("META-INF", xs@_*) => MergeStrategy.discard 
    case x => MergeStrategy.first 
} 

堆棧跟蹤：

java.io.InvalidClassException：org.apache.spark .unsafe.types.UTF8String;本地類不兼容：stream classdesc serialVersionUID = -2992553500466442037，本地類serialVersionUID = -5670082246090726217 位於java.io.ObjectStreamClass.initNonProxy（ObjectStreamClass.java:616）〜[na：1.8.0_45] 位於java.io.ObjectInputStream.readNonProxyDesc （ObjectInputStream.java:1843）〜[na：1.8.0_45] at java.io.ObjectInputStream.readClassDesc（ObjectInputStream.java:1713）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream .java：2000）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java ：2245）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject（ObjectInputStream.java:422）〜[na：1.8.0_45] at scala.collection.immutable.List $ SerializationProxy.readObject（List.scala：479）〜[scala-library-2.11。 8.jar：1.0.0-M1] at sun.reflect.GeneratedMethodAccessor4.invoke（Unknown Source）〜[na：na] at sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）〜[na：1.8 .0_45] at java.lang.reflect.Method.invoke（Method.java:498）〜[na：1.8.0_45] at java.io.ObjectStreamClass.invokeReadObject（ObjectStreamClass.java:1058）〜[na：1.8 .0_45] at java .io.ObjectInputStream.readSerialData（ObjectInputStream.java:2136）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] at java.io .ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream .readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject0 （ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream.re在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45]上的adSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject0 ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject（ObjectInputStream.java:422）〜[na：1.8.0_45] at scala.collection.immutable.List $ SerializationProxy。 readObject（List.scala：479）〜[scala-library-2.11.8.jar：1.0.0-M1] at sun.reflect.GeneratedMethodAccessor4.invoke（Unknown Source）〜[na：na] at sun.reflect .DelegatingMethodAccessorImpl。invoke（DelegatingMethodAccessorImpl.java:43）〜[na：1.8.0_45] at java.lang.reflect.Method.invoke（Method.java:498）〜[na：1.8.0_45] at java.io.ObjectStreamClass。 invokeReadObject（ObjectStreamClass.java:1058）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2136）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ ObjectInputStream.java:2027）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream。 java：2245）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java： 2027 ）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜 [Na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na ：1.8.0_45] at java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8 （ObjectInputStream.java:2027）〜[na：1.8.0_45]。[0_45] ] at java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] at java。 io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io. ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] 在java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java在java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] .io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] at java.io .ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream .readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] at java.io.ObjectInput Stream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream。 readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject0（ ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject（ObjectInputStream.java:422）〜[na：1.8.0_45] at scala.collection.immutable.List $ SerializationProxy。 readObject（List.scala：479）〜[scala-library-2.11.8.jar：1.0.0-M1] at sun.reflect.GeneratedMethodAccessor4.invoke（Unknown Source）〜[na：na] at sun.reflect .DelegatingMethodAccessorImpl。invoke（DelegatingMethodAccessorImpl.java:43）〜[na：1.8.0_45] at java.lang.reflect.Method.invoke（Method.java:498）〜[na：1.8.0_45] at java.io.ObjectStreamClass。 invokeReadObject（ObjectStreamClass.java:1058）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2136）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ ObjectInputStream.java:2027）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream。 java：2245）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java： 2027 ）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜 [Na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na ：1.8.0_45] at java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject（ObjectInputStream.java:422）〜[na：1.8 .0_45] at org.apache.spark.serializer.JavaDeserializationStream.readObject（JavaSerializer.scala：75）〜[spark-core_2.11-2.1.0.jar：2.1.0] at org.apache.spark.serializer .JavaSerializerInstance.deserialize（j avaSerializer.scala：114）〜[spark-core_2.11-2.1.0.jar：2.1.0] at org.apache.spark.scheduler.ResultTask.runTask（ResultTask.scala：80）〜[spark-core_2。 11-2.1.0.jar：2.1.0] at org.apache.spark.scheduler.Task.run（Task.scala：99）〜[spark-core_2.11-2.1.0.jar：2.1.0] at org.apache.spark.executor.Executor $ TaskRunner.run（Executor.scala：322）〜[spark-core_2.11-2.1.0.jar：2.1.0] at java.util.concurrent.ThreadPoolExecutor。 runWorker（ThreadPoolExecutor.java:1142）〜[na：1.8.0_45] at java.util.concurrent.ThreadPoolExecutor $ Worker.run（ThreadPoolExecutor.java:617）〜[na：1.8.0_45] at java.lang。 Thread.run（Thread.java:748）〜[na：1.8.0_45]

Answer 1

spark-subm它--version顯示它的包的spark和scala版本，而不是你的系統，而你的sbt正在使用你的系統的scala版本。所以

請更改

"org.apache.spark" %% "spark-core" % "2.1.0", 
    "org.apache.spark" %% "spark-sql" % "2.1.0" , 
    "org.apache.spark" %% "spark-mllib" % "2.1.0" , 

到

"org.apache.spark" % "spark-core_2.11" % "2.1.1", 
    "org.apache.spark" % "spark-sql_2.11" % "2.1.1" , 
    "org.apache.spark" % "spark-mllib_2.11" % "2.1.1" , 

如果它沒有幫助，請與您的系統Scala的版本，你是如何提交應用程序和遠程機器的更新你的問題斯卡拉和火花版本。