Hadoop返回映射器的輸出而不是reducer

Question

我正在處理這個hadoop代碼，但無法弄清楚爲什麼它不會生成reducer輸出，而是它完全輸出mapper的結果。我已經玩了很長時間的代碼，測試不同的輸出，但沒有運氣。

我的自定義映射：

​​3210

用戶自己定製的減速機：

/* Reducer Class */ 
public static class UserReducer extends Reducer<Text, IntWritable, Text, IntWritable> { 
private IntWritable result = new IntWritable(); 

public void reduce(Text key, Iterable<Text> values, Context context) 
      throws IOException, InterruptedException { 
    int sum = 0; 
    for (Text val : values) { 
     sum += 1; //val.get(); 
    } 
    result.set(0); 
    context.write(key, result); 
    } 
} 

主程序的身體：

Job job = new Job(conf, "User Popular Categories"); 
job.setJarByClass(popularCategories.class); 
job.setMapperClass(UserMapper.class); 
job.setCombinerClass(UserReducer.class); 
job.setReducerClass(UserReducer.class); 

job.setMapOutputKeyClass(Text.class); 
job.setMapOutputValueClass(Text.class); 

job.setOutputKeyClass(Text.class); 
job.setOutputValueClass(IntWritable.class); 
job.setNumReduceTasks(2); 

FileInputFormat.addInputPath(job, new Path(otherArgs[0])); 
FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); 

System.exit(job.waitForCompletion(true) ? 0 : 1); 

和輸出文件：/用戶/ hduser /輸出/一部分-R-00000

0001be1731ee7d1c519bc7e87110c9eb880cb396 This is a test 
0001bfa0c494c01f9f8c141c476c11bb4625a746 This is a test 
0002bd9c3d654698bb514194c4f4171ad6992266 This is a test 
000433e0ef411c2cb8ee1727002d6ba15fe9426b This is a test 
00051f5350f4d9f3f4f5ba181b0a66d749b161ee This is a test 
00066c85bf96469b905e2fb148095448797b2368 This is a test 
0007b1a0334de785b3189b67bb73276d602fb7d4 This is a test 
0007d018861d588e99e834fc29ca76a523b20e35 This is a test 
000992b67ed22d2707ba65046d523ce66dfcfcb8 This is a test 
000ad93a0819e2cbd7f0193e1d1ec481a0241b44 This is a test 

Answer 1

不過我很驚訝怎麼上面的代碼塊爲你工作。像在Hadoop (java) change the type of Mapper output values關於Mapper的另一個問題中，您應該在這裏得到例外。

看來輸出是Mapper而不是Reducer。你確定文件名嗎？

/user/hduser/output/part-r-00000 

instead of 

/user/hduser/output/part-m-00000 

映射器輸出應該減速輸入。

public static class UserMapper extends Mapper<Object, Text, Text, Text> { 

寫入輸出密鑰作爲Text和輸出值作爲Text。

你Reducer定義爲

public static class UserReducer extends Reducer<Text, IntWritable, Text, IntWritable> { 

裝置輸入鍵是文本（正確的）but value is wrongly made as IntWritable (It should be Text)

更改聲明

public static class UserReducer extends Reducer<Text, Text, Text, IntWritable> { 

並相應地設置在Driver程序的參數。