錯誤 - MapReduce中的Hadoop字數統計程序

Question

我是Hadoop的新手，因此如果這看起來像個傻問題，那麼請原諒我。

我正在我下面的MapReduce程序並得到以下錯誤：

java.lang.Exception的：產生java.io.IOException：從地圖鍵類型不匹配：預計org.apache.hadoop.io.Text ，收到org.apache.hadoop.io.LongWritable at org.apache.hadoop.mapred.LocalJobRunner $ Job.run（LocalJobRunner.java:354） 引起來自：java.io.IOException：類型與映射關鍵字不匹配：預計org.apache.hadoop.io.Text，收到org.apache.hadoop.io.LongWritable 在org.apache.hadoop.mapred.MapTask $ MapOutputBuffer.collect（MapTask.java:1019）

任何幫助APPR eciated。

公共類字計數{

// Mapper Class 
public static class MapperClass extends Mapper<Object, Text, Text, IntWritable>{ 


    private final static IntWritable one = new IntWritable(1); 
    private Text word = new Text(); 

    // Mapper method defined 
    public void mapperMethod(Object key,Text lineContent,Context context){ 
     try{ 
     StringTokenizer strToken = new StringTokenizer(lineContent.toString()); 
     //Iterating through the line 
     while(strToken.hasMoreTokens()){ 
      word.set(strToken.nextToken()); 
      try{ 
      context.write(word, one); 
      } 
      catch(Exception e){ 
       System.err.println(new Date()+" ---> Cannot write data to hadoop in Mapper."); 
       e.printStackTrace(); 
      } 
     } 
    } 
    catch(Exception ex){ 
     ex.printStackTrace(); 
    } 
    } 
} 
// Reducer Class 
public static class ReducerClass extends Reducer<Text, IntWritable, Text, IntWritable>{ 

    private IntWritable result = new IntWritable(); 

    //Reducer method 
    public void reduce(Text key,Iterable<IntWritable> values,Context context){ 
     try{ 
     int sum=0; 
     for(IntWritable itr : values){ 
      sum+=itr.get(); 
     } 
     result.set(sum); 
     try { 
      context.write(key,result); 
     } catch (Exception e) { 
      System.err.println(new Date()+" ---> Error while sending data to Hadoop in Reducer"); 
      e.printStackTrace(); 
     } 

    } 
    catch (Exception err){ 
     err.printStackTrace(); 

    } 
} 

} 


public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException { 
    try{ 
    Configuration conf = new Configuration(); 
    String [] arguments = new GenericOptionsParser(conf, args).getRemainingArgs(); 
    if(arguments.length!=2){ 
     System.err.println("Enter both and input and output location."); 
     System.exit(1); 
    } 
    Job job = new Job(conf,"Simple Word Count"); 

    job.setJarByClass(WordCount.class); 
    job.setMapperClass(MapperClass.class); 
    job.setReducerClass(ReducerClass.class); 


    job.setOutputKeyClass(Text.class); 
    job.setOutputValueClass(IntWritable.class); 



    FileInputFormat.addInputPath(job, new Path(arguments[0])); 
    FileOutputFormat.setOutputPath(job, new Path(arguments[1])); 

    System.exit(job.waitForCompletion(true) ? 0 : 1); 
} 
    catch(Exception e){ 

    } 
} 

}

Answer 1

你需要重寫地圖方法的映射器類，而不是你有一個新的方法。 因爲你沒有map方法覆蓋你的程序，可以歸結爲你的錯誤歸結爲一個只減少工作。 Reducer獲得LongWritable，Text的輸入，但是你已經聲明瞭Intwritable和text作爲輸入。

希望這解釋。

public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> { 
     private final static IntWritable one = new IntWritable(1); 
     private Text word = new Text(); 

     public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException { 
     String line = value.toString(); 
     StringTokenizer tokenizer = new StringTokenizer(line); 
      while (tokenizer.hasMoreTokens()) { 
     word.set(tokenizer.nextToken()); 
      output.collect(word, one); 
      } 
     } 
     }