MapReduce中的全局變量或屬性？

Question

我希望能夠在MR作業的地圖階段設置某種變量或標誌，我可以在作業完成後檢查。我認爲最好的方式來證明什麼，我想用一些代碼：P.S我使用Hadoop 2.2.0

public class MRJob { 

    public static class MapperTest 
     extends Mapper<Object, Text, Text, IntWritable>{ 


    public void map(Object key, Text value, Context context 
        ) throws IOException, InterruptedException { 
     //Do some computation to get new value and key 
     ... 
     //Check if new value equal to some condition e.g if(value < 1) set global variable to true 

     context.write(newKey, newValue); 
    } 
    } 

    public static void main(String[] args) throws Exception { 
    Configuration conf = new Configuration(); 

    Job job = Job.getInstance(new Configuration(), "word_count"); 
    //set job configs 

    job.waitForCompletion(true); 

    //Here I want to be able to check if my global variable has been set to true by any one of the mappers 

    } 
} 

Answer 1

使用Counter對這一問題。

public static enum UpdateCounter { 
    UPDATED 
} 

@Override 
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { 

    if(value < 1) { 
     context.getCounter(UpdateCounter.UPDATED).increment(1); 
    } 

    context.write(newKey, newValue); 
} 

作業後，您可以檢查：

Configuration conf = new Configuration(); 

Job job = Job.getInstance(new Configuration(), "word_count"); 
//set job configs 

job.waitForCompletion(true); 
long counter = job.getCounters().findCounter(UpdateCounter.UPDATED).getValue(); 

if(counter > 0) 
    // some mapper has seen the condition