3
我有一個reduce函數,我想在處理一些'n'鍵後暫停reduce函數。我已經設置了一個計數器來增加每個鍵,並且條件是從reduce函數返回。在條件下停止減少Hadoop中的函數
下面是代碼
public class wordcount {
public static class Map extends Mapper<LongWritable, Text, IntWritable, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
private IntWritable leng=new IntWritable();
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
String lword=tokenizer.nextToken();
leng.set(lword.length());
context.write(leng, one);
}
}
}
public static class Reduce extends Reducer<IntWritable, IntWritable, IntWritable, IntWritable> {
int count=0;
public void reduce(IntWritable key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
count++;
}
context.write(key, new IntWritable(sum));
if(count>19) return;
}
}
有,我做到這一點任何其他方式。
熊,如果你有一個以上的減速,你不能處理的處理,按鍵內部限制。例如,如果你想在10個鍵後停止,但你有2個reducer,那麼你最終總共會處理20個鍵。無論您從哪裏開始工作,您都需要從外部控制此限制。 – Quetzalcoatl 2013-03-21 10:16:21
我正在使用一個reducer來實現我所需的top n鍵的條件。謝謝你的提示。 – Skandy 2013-03-21 15:45:02