Reducer可迭代的值似乎在Java MapReduce中不一致

Question

我在reduce函數中有下面的代碼。當我嘗試使用CollectionUtils.addAll製作淺拷貝時，副本不成功;所有的項目都會有LAST項目的引用，而不是迭代器中的其他項目。

下面是從我減速代碼：

public void reduce(Text key, Iterable<ArrayListWritable<Writable>> values, Context context) 
    throws IOException, InterruptedException { 
    ArrayList<ArrayListWritable<Writable>> listOfWordPairs = new ArrayList<ArrayListWritable<Writable>>(); 

    // CollectionUtils.addAll(listOfWordPairs, values.iterator()); 
    // listOfWordPairs seems to all be the last item in the iterator 

    Iterator<ArrayListWritable<Writable>> iter = values.iterator(); 

    // Manually do the copy 
    while (iter.hasNext()) { 
     // listOfWordPairs.add(iter.next()); 
     //Same behaviour as CollectionUtils.addAll() 

     listOfWordPairs.add(new ArrayListWritable<Writable>(iter.next())); 
     //Only working way to do it -> deep copy :(
     } 
    } 

任何人有爲什麼發生這種情況的任何想法？我可以看到，如果MR以這種方式實現它，它可以節省大量的內存，但似乎有一些魔法會在這裏實現。我是新來的MR所以希望這個問題是不是太傻了

這裏是我的地圖代碼的人誰是感興趣

@Override 
     public void map(LongWritable key, Text value, Context context) 
      throws IOException, InterruptedException { 
      Map<String, HMapStFW> stripes = new HashMap<>(); 

      List<String> tokens = Tokenizer.tokenize(value.toString()); 

      if (tokens.size() < 2) return; 
      context.getCounter(StripesPmiEnums.TOTALENTRIES).increment(tokens.size()); 

      for (int i = 0; i < tokens.size() && i<40; i++) { 
      for (int j = 0;j<tokens.size() && j<40;j++){ 
       if (j == i) 
        continue; 
       //Make Stripe if doesn't exist 
       if (!stripes.containsKey(tokens.get(i))){ 
        HMapStFW newStripe = new HMapStFW(); 
        stripes.put(tokens.get(i), newStripe); 
       } 

       HMapStFW stripe = stripes.get(tokens.get(i)); 
       if (stripe.containsKey(tokens.get(j))){ 
        stripe.put(tokens.get(j), stripe.get(tokens.get(j))+1.0f); 
       }else{ 
        stripe.put(tokens.get(j), 1.0f); 
       } 
      } 
      } 

      for (String word1 : stripes.keySet()) { 
      TEXT.set(word1); 
      context.write(TEXT, stripes.get(word1)); 
      } 
     } 

而且還ArrayListWritable可在這裏 https://github.com/lintool/tools/blob/master/lintools-datatypes/src/main/java/tl/lin/data/array/ArrayListWritable.java

Answer 1

這是因爲迭代器在縮減器中工作不同。簡短的回答，您可以在您遍歷迭代器

while (iter.hasNext()) { 
    //this is correct 
    listOfWordPairs.add(new ArrayListWritable<Writable>(iter.next())); 

    } 
} 

看看下面的鏈接克隆的對象，這是很好的解釋

https://cornercases.wordpress.com/2011/08/18/hadoop-object-reuse-pitfall-all-my-reducer-values-are-the-same/