在Hadoop中實現自定義Writable？

我在Hadoop中定義了一個自定義的Writable類，但是在運行我的程序時Hadoop給了我下面的錯誤信息。在Hadoop中實現自定義Writable？

java.lang.RuntimeException: java.lang.NullPointerException 
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) 
at org.apache.hadoop.io.SortedMapWritable.readFields(SortedMapWritable.java:180) 
at EquivalenceClsAggValue.readFields(EquivalenceClsAggValue.java:82) 
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67) 
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40) 
at org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:1282) 
at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:1222) 
at org.apache.hadoop.mapred.Task$CombineValuesIterator.next(Task.java:1301) 
at Mondrian$Combine.reduce(Mondrian.java:119) 
at Mondrian$Combine.reduce(Mondrian.java:1) 
at org.apache.hadoop.mapred.Task$OldCombinerRunner.combine(Task.java:1442) 
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1436) 
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1298) 
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437) 
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) 
at org.apache.hadoop.mapred.Child$4.run(Child.java:255) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:415) 
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136) 
at org.apache.hadoop.mapred.Child.main(Child.java:249)

產生的原因：在java.util.concurrent.ConcurrentHashMap.hash（ConcurrentHashMap.java:332）顯示java.lang.NullPointerException ....

EquivalenceClsAggValue是可寫I類的名稱已經定義，這是我的班級：

public class EquivalenceClsAggValue implements WritableComparable<EquivalenceClsAggValue>{ 

public ArrayList<SortedMapWritable> aggValues; 
public EquivalenceClsAggValue(){   

    aggValues = new ArrayList<SortedMapWritable>(); 
} 
@Override 
public void readFields(DataInput arg0) throws IOException { 

    int size = arg0.readInt(); 

    for (int i=0;i<size;i++){ 
     SortedMapWritable tmp = new SortedMapWritable(); 
     tmp.readFields(arg0); 
     aggValues.add(tmp); 
    }  
} 

@Override 
public void write(DataOutput arg0) throws IOException { 

    //write the size first 
    arg0.write(aggValues.size()); 

    //write each element 
    for (SortedMapWritable s:aggValues){ 
     s.write(arg0); 
    } 

}

我想知道什麼是問題的根源。

來源

2013-01-10 H.Z.

看起來像一個錯誤在你write(DataOutput)方法：

@Override 
public void write(DataOutput arg0) throws IOException { 
    //write the size first 
    // arg0.write(aggValues.size()); // here you're writing an int as a byte 

    // try this instead: 
    arg0.writeInt(aggValues.size()); // actually write int as an int 

    //..

看API文檔的DataOutput.write(int) VS DataOutput.writeInt(int)

我還修訂閱讀字段使用您的SortedMapWritable TMP局部變量的創建ReflectionUtils.newInstance()：

@Override 
public void readFields(DataInput arg0) throws IOException { 

    int size = arg0.readInt(); 

    for (int i=0;i<size;i++){ 
    SortedMapWritable tmp = ReflectionUtils.newInstance(
     SortedMapWritable.class, getConf()); 
    tmp.readFields(arg0); 
    aggValues.add(tmp); 
    }  
}

注意這個工作，你也會nee d修改您類簽名以擴展Configurable（這樣的Hadoop將注入Configuration對象時最初創建的對象）：

public class EquivalenceClsAggValue 
      extends Configured 
      implements WritableComparable<EquivalenceClsAggValue> {

來源

2013-01-10 02:28:49

感謝。它解決了我的問題。 –

在Hadoop中實現自定義Writable？

回答

相關問題