Hadoop 0.20.2 reducer在迭代值時拋出ArrayIndexOutOfBoundsException

我對hadoop相當陌生，但是我一直在閱讀「Hadoop：權威指南」，所以我想我對基本概念有所瞭解。Hadoop 0.20.2 reducer在迭代值時拋出ArrayIndexOutOfBoundsException

我用的Hadoop 0.20.2運行一個相當簡單的工作，但我得到以下異常：

java.lang.ArrayIndexOutOfBoundsException: 4096 
     at java.io.ByteArrayInputStream.read(ByteArrayInputStream.java:127) 
     at java.io.DataInputStream.readInt(DataInputStream.java:373) 
     at com.convertro.mapreduce.WritableHit.readFields(Unknown Source) 
     at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeseria 
lizer.deserialize(WritableSerialization.java:67) 
     at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeseria 
lizer.deserialize(WritableSerialization.java:40) 
     at org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext. 
java:116) 
     at org.apache.hadoop.mapreduce.ReduceContext$ValueIterator.next(ReduceCo 
ntext.java:163) 
     at com.convertro.mapreduce.HitConvertingIterable$HitConvertingIterator.n 
ext(HitConvertingIterable.java:35) 
     at com.convertro.mapreduce.HitConvertingIterable$HitConvertingIterator.n 
ext(HitConvertingIterable.java:1) 
     at com.convertro.naive.NaiveHitReducer.reduce(Unknown Source) 
     at com.convertro.mapreduce.HitReducer.reduce(Unknown Source) 
     at com.convertro.mapreduce.HitReducer.reduce(Unknown Source) 
     at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176) 
     at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:566 
) 
     at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408) 
     at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:2

這種情況發生在WritableHit類（輸入值減少相）的閱讀過程中。下面是WritableHit類代碼：

public class WritableHit implements WritableComparable<WritableHit> { 

    private Hit hit; 

    public WritableHit() { 
     this(null); 
    } 

    public WritableHit(Hit hit) { 
     this.hit = hit; 
    } 

    @Override 
    public void readFields(DataInput input) throws IOException { 
     String clientName = input.readUTF(); 
     String clientSiteId = input.readUTF(); 
     String eventUniqueId = input.readUTF(); 
     String eventValue = input.readUTF(); 
     String pageRequested = input.readUTF(); 
     String refererUrl = input.readUTF(); 
     String uniqueHitId = input.readUTF(); 
     String userAgent = input.readUTF(); 
     String userIdentifier = input.readUTF(); 
     String userIp = input.readUTF(); 
     int timestamp = input.readInt(); 
     int version = input.readInt(); 

     hit = new Hit(version, uniqueHitId, clientName, clientSiteId, timestamp, userIdentifier, 
       userIp, pageRequested, refererUrl, userAgent, eventUniqueId, eventValue); 
    } 

    @Override 
    public void write(DataOutput output) throws IOException { 
     output.writeUTF(hit.getClientName()); 
     output.writeUTF(hit.getClientSiteId()); 
     output.writeUTF(hit.getEventUniqueId()); 
     output.writeUTF(hit.getEventValue()); 
     output.writeUTF(hit.getPageRequested()); 
     output.writeUTF(hit.getRefererUrl()); 
     output.writeUTF(hit.getUniqueHitId()); 
     output.writeUTF(hit.getUserAgent()); 
     output.writeUTF(hit.getUserIdentifier()); 
     output.writeUTF(hit.getUserIp()); 
     output.write(hit.getTimestamp()); 
     output.write(hit.getVersion()); 
    } 

    public Hit getHit() { 
     return hit; 
    } 

    @Override 
    public int compareTo(WritableHit o) { 
     return hit.getUniqueHitId().compareTo(o.getHit().getUniqueHitId()); 
    } 
    }

任何幫助將不勝感激。

謝謝

來源

2011-10-01 Amir Tuval

我想通了。

顯然，當你實現一個可寫的對象，你應該使用writeInt方法，而不是寫方法。

一旦我做到了，它就像一個魅力。

來源

2011-10-02 05:29:49

既然你設法解決你自己的問題（做得好！），你應該把這個標記爲「被接受」的答案。 –

Hadoop 0.20.2 reducer在迭代值時拋出ArrayIndexOutOfBoundsException

回答

相關問題