HADOOP - 1.2.1穩定的字數統計示例

Question

我正在通過hadoop 1.2.1的字數統計示例。但是事情一定會發生變化，因爲我似乎無法使其發揮作用。

這裏是我降低等級：

public static class Reduce extends Reducer<WritableComparable, Writable, WritableComparable, Writable> { 

    public void reduce(WritableComparable key, 
         Iterator<Writable> values, 
         OutputCollector<WritableComparable, NullWritable> output, 
         Reporter reporter) throws IOException { 

     output.collect(key, NullWritable.get()); 

    } 

} 

而我的主要功能：

public static void main(String[] args) throws Exception { 

    JobConf jobConf = new JobConf(MapDemo.class); 

    jobConf.setNumMapTasks(10); 
    jobConf.setNumReduceTasks(1); 

    jobConf.setJobName("MapDemo"); 

    jobConf.setOutputKeyClass(Text.class); 
    jobConf.setOutputValueClass(NullWritable.class); 

    jobConf.setMapperClass(Map.class); 
    jobConf.setReducerClass(Reduce.class); 

    jobConf.setInputFormat(TextInputFormat.class); 
    jobConf.setOutputFormat(TextOutputFormat.class); 

    FileInputFormat.setInputPaths(jobConf, new Path(args[0])); 
    FileOutputFormat.setOutputPath(jobConf, new Path(args[1])); 

    JobClient.runJob(jobConf); 
} 

我的IDE告訴我有錯誤，Maven的佐證：

[ERROR] COMPILATION ERROR : 
[INFO] ------------------------------------------------------------- 
[ERROR] com/example/mapreduce/MapDemo.java:[71,16] method setReducerClass in class org.apache.hadoop.mapred.JobConf cannot be applied to given types; 
required: java.lang.Class<? extends org.apache.hadoop.mapred.Reducer> 
found: java.lang.Class<com.example.mapreduce.MapDemo.Reduce> 
reason: actual argument java.lang.Class<com.example.mapreduce.MapDemo.Reduce> cannot be converted to java.lang.Class<? extends org.apache.hadoop.mapred.Reducer> by method invocation conversion 
[INFO] 1 error 
[INFO] ------------------------------------------------------------- 
[INFO] ------------------------------------------------------------------------ 
[INFO] BUILD FAILURE 
[INFO] ------------------------------------------------------------------------ 
[INFO] Total time: 1.679s 
[INFO] Finished at: Mon Sep 16 09:23:08 PDT 2013 
[INFO] Final Memory: 17M/202M 
[INFO] ------------------------------------------------------------------------ 
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project inventory: Compilation failure 
[ERROR] com/example/mapreduce/MapDemo.java:[71,16] method setReducerClass in class org.apache.hadoop.mapred.JobConf cannot be applied to given types; 
[ERROR] required: java.lang.Class<? extends org.apache.hadoop.mapred.Reducer> 
[ERROR] found: java.lang.Class<com.example.mapreduce.MapDemo.Reduce> 

我相信網上的數字例子已經過時了1.2.1。我該如何解決？有沒有人有鏈接到一個工作1.2.1字數java源？

Answer 1

您追蹤哪個鏈接？我從來沒有見過這樣的WC。但是，無論你使用的是舊API，它們肯定是過時的。我懷疑你是否正確地遵守了它。

這應該工作：

public class WordCount { 
    /** 
    * The map class of WordCount. 
    */ 
    public static class TokenCounterMapper extends 
      Mapper<Object, Text, Text, IntWritable> { 

     private final static IntWritable one = new IntWritable(1); 
     private Text word = new Text(); 

     public void map(Object key, Text value, Context context) 
       throws IOException, InterruptedException {    

      StringTokenizer itr = new StringTokenizer(value.toString()); 
      while (itr.hasMoreTokens()) { 
       word.set(itr.nextToken()); 
       context.write(word, one); 
      } 
     } 
    } 

    /** 
    * The reducer class of WordCount 
    */ 
    public static class TokenCounterReducer extends 
      Reducer<Text, IntWritable, Text, IntWritable> { 
     public void reduce(Text key, Iterable<IntWritable> values, 
       Context context) throws IOException, InterruptedException { 
      int sum = 0; 
      for (IntWritable value : values) { 
       sum += value.get(); 
      } 
      context.write(key, new IntWritable(sum)); 
     } 
    } 

    /** 
    * The main entry point. 
    */ 
    public static void main(String[] args) throws Exception { 
     Configuration conf = new Configuration(); 
     conf.addResource(new Path("/Users/miqbal1/hadoop-eco/hadoop-1.1.2/conf/core-site.xml")); 
     conf.addResource(new Path("/Users/miqbal1/hadoop-eco/hadoop-1.1.2/conf/hdfs-site.xml")); 
     conf.set("fs.default.name", "hdfs://localhost:9000"); 
     conf.set("mapred.job.tracker", "localhost:9001"); 
     Job job = new Job(conf, "WordCount"); 
     job.setJarByClass(WordCount.class); 
     job.setMapperClass(TokenCounterMapper.class); 
     job.setReducerClass(TokenCounterReducer.class); 
     job.setNumReduceTasks(2); 
     job.setOutputKeyClass(Text.class); 
     job.setOutputValueClass(IntWritable.class); 
     FileInputFormat.addInputPath(job, new Path("/inputs/demo.txt")); 
     FileOutputFormat.setOutputPath(job, new Path("/outputs/1111223")); 
     System.exit(job.waitForCompletion(true) ? 0 : 1); 
    } 
} 

一些意見，使：

• 您沒有產生任何數，我可以看到NullWritable得到從減速發出。它只會發射密鑰而沒有任何計數。
• 使用適當的類型爲您的輸入和輸出鍵/值。使用新API。它更乾淨更好。

Answer 2

該行建議你混合的Hadoop mapred和的MapReduce API版本了。

actual argument java.lang.Class<com.example.mapreduce.MapDemo.Reduce> cannot be converted to java.lang.Class<? extends org.apache.hadoop.mapred.Reducer> by method invocation conversion 

可以使用mapred或mapreduce API。