找不到價值類串行：「org.apache.hadoop.hbase.client.Result」

Question

我想讀出的數據HBase的，並保存爲一個sequenceFile，但得到

java.io.IOException: Could not find a serializer for the Value class: 'org.apache.hadoop.hbase.client.Result'. Please ensure that the configuration 'io.serializations' is properly configured, if you're usingcustom serialization. 

錯誤。

我看到兩個類似的帖子：

hadoop writables NotSerializableException with Apache Spark API

和

Spark HBase Join Error: object not serializable class: org.apache.hadoop.hbase.client.Result

下面這兩個職位，我註冊了凱洛班，三個班，但仍沒有運氣。

這是我的計劃：

 String tableName = "validatorTableSample"; 
     System.out.println("Start indexing hbase: " + tableName); 
     SparkConf sparkConf = new SparkConf().setAppName("HBaseRead"); 
     Class[] classes = {org.apache.hadoop.io.LongWritable.class, org.apache.hadoop.io.Text.class, org.apache.hadoop.hbase.client.Result.class}; 
     sparkConf.registerKryoClasses(classes); 
     JavaSparkContext sc = new JavaSparkContext(sparkConf); 
     Configuration conf = HBaseConfiguration.create(); 
     conf.set(TableInputFormat.INPUT_TABLE, tableName); 
//  conf.setStrings("io.serializations", 
//   conf.get("io.serializations"), 
//   MutationSerialization.class.getName(), 
//   ResultSerialization.class.getName()); 
     conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer"); 

     JavaPairRDD<ImmutableBytesWritable, Result> hBasePairRDD = sc.newAPIHadoopRDD(
      conf, 
      TableInputFormat.class, 
      ImmutableBytesWritable.class, 
      Result.class); 

     hBasePairRDD.saveAsNewAPIHadoopFile("/tmp/tempOutputPath", ImmutableBytesWritable.class, Result.class, SequenceFileOutputFormat.class); 
     System.out.println("Finished readFromHbaseAndSaveAsSequenceFile() ........."); 

這裏的錯誤堆棧跟蹤：

java.io.IOException: Could not find a serializer for the Value class: 'org.apache.hadoop.hbase.client.Result'. Please ensure that the configuration 'io.serializations' is properly configured, if you're usingcustom serialization. 
    at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1254) 
    at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1156) 
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273) 
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:530) 
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64) 
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75) 
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1112) 
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1095) 
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) 
    at org.apache.spark.scheduler.Task.run(Task.scala:86) 
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    at java.lang.Thread.run(Thread.java:745) 
17/05/25 10:58:38 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.io.IOException: Could not find a serializer for the Value class: 'org.apache.hadoop.hbase.client.Result'. Please ensure that the configuration 'io.serializations' is properly configured, if you're usingcustom serialization. 
    at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1254) 
    at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1156) 
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273) 
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:530) 
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64) 
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75) 
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1112) 
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1095) 
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) 
    at org.apache.spark.scheduler.Task.run(Task.scala:86) 
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    at java.lang.Thread.run(Thread.java:745) 

17/05/25 10:58:38 ERROR TaskSetManager: Task 0 in stage 0.0 failed 1 times; aborting job 

Answer 1

這裏需要的是什麼，使工作

因爲我們使用HBase的存儲我們的數據，這reducer將其結果輸出到HBase表，Hadoop告訴我們他不知道如何序列化我們的數據。這就是爲什麼我們需要幫助它。在setUp中設置io.serializations變量

conf.setStrings("io.serializations", new String[]{hbaseConf.get("io.serializations"), MutationSerialization.class.getName(), ResultSerialization.class.getName()});