1
我正在嘗試解決使用Java的HashMaps的Spark序列化問題。 我指的是鏈接Save Spark Dataframe into Elasticsearch - Can’t handle type exception。無法轉換爲org.apache.spark.serializer.Serializer
現在我打了以下問題:
java.lang.ClassCastException: com.spark.util.umf.MyKryoRegistrator cannot be cast to org.apache.spark.serializer.Serializer at org.apache.spark.SparkEnv$.create(SparkEnv.scala:259) at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:163) at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:267) at org.apache.spark.SparkContext.(SparkContext.scala:270) at org.apache.spark.api.java.JavaSparkContext.JavaSparkContext.scala:61) at com.spark.util.umf.MyMain.main(MyMain.java:46) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:480) 15/10/16 01:47:22 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: com.spark.util.umf.MyKryoRegistrator cannot be cast to org.apache.spark.serializer.Serializer)
創建我Kryo
registrator如下:
import java.io.Serializable;
import org.apache.spark.serializer.KryoRegistrator;
import com.esotericsoftware.kryo.Kryo;
public class MyKryoRegistrator implements KryoRegistrator, Serializable {
@Override
public void registerClasses(Kryo kryo) {
// Product POJO associated to a product Row from the DataFrame
kryo.register(MyRecord.class);
}
}
主要方法:
public static void main(String args[]){
SparkConf sConf= new SparkConf().setAppName("SparkTestJob");
sConf.set("spark.driver.allowMultipleContexts", "true");
//Kryo kryo = new Kryo();;
//kryo.setDefaultSerializer(MyRecord.class);
//my.registerClasses(kryo);
sConf.set("spark.serializer","com.spark.util.umf.MyKryoRegistrator");
[...]
}
'main'在哪裏定義?請發佈完整的代碼,因爲它看起來像'MyKryoRegistrator'是一個內部類,如果是這樣'嘗試'公共靜態類MyKryoRegistrator' ... – earcam
好的。這是我的錯誤。 我用過「sConf.set(」spark.serializer「,」com.spark.util.umf.MyKryoRegistrator「);」它應該像 sConf.set(「spark.kryo.registrator」,「com.spark.util.umf.MyKryoRegistrator」); – Harisyam
你可以把評論作爲你自己問題的答案;-) – earcam