Map減少作業生成空輸出文件

程序正在生成空輸出文件。任何人都可以請建議我我哪裏錯了。任何幫助將不勝感激。我試圖把job.setNumReduceTask（0），因爲我不使用reducer但仍輸出文件是空的。Map減少作業生成空輸出文件

public static class PrizeDisMapper extends Mapper<LongWritable, Text, Text, Pair>{ 
int rating = 0; 
Text CustID; 
IntWritable r; 
Text MovieID; 
public void map(LongWritable key, Text line, Context context 
       ) throws IOException, InterruptedException { 
     String line1 = line.toString(); 
     String [] fields = line1.split(":"); 
     if(fields.length > 1) 
      { 
      String Movieid = fields[0]; 
      String line2 = fields[1]; 
      String [] splitline = line2.split(","); 
      String Custid = splitline[0]; 
      int rate = Integer.parseInt(splitline[1]); 
      r = new IntWritable(rate); 
      CustID = new Text(Custid); 
      MovieID = new Text(Movieid); 
      Pair P = new Pair(); 
      context.write(MovieID,P); 
      } 
      else 
      { 
      return; 
      } 
    } 
} 

public static class IntSumReducer extends Reducer<Text,Pair,Text,Pair> { 
private IntWritable result = new IntWritable(); 
public void reduce(Text key, Iterable<Pair> values, 
        Context context 
        ) throws IOException, InterruptedException { 
    for (Pair val : values) { 
    context.write(key, val); 
    } 
    } 

    public class Pair implements Writable 
    { 
    String key; 
    int value; 
    public void write(DataOutput out) throws IOException { 
    out.writeInt(value); 
    out.writeChars(key); 
    } 
    public void readFields(DataInput in) throws IOException { 
    key = in.readUTF(); 
    value = in.readInt(); 
    } 
    public void setVal(String aKey, int aValue) 
    { 
    key = aKey; 
     value = aValue; 
    }

主要類：

public static void main(String[] args) throws Exception { 
Configuration conf = new Configuration(); 
String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs(); 
if (otherArgs.length != 2) { 
    System.err.println("Usage: wordcount <in> <out>"); 
    System.exit(2); 
} 
Job job = new Job(conf, "word count"); 
job.setJarByClass(WordCount.class); 
job.setMapperClass(TokenizerMapper.class); 
job.setCombinerClass(IntSumReducer.class); 
job.setReducerClass(IntSumReducer.class); 
job.setInputFormatClass (TextInputFormat.class); 
FileInputFormat.addInputPath(job, new Path(otherArgs[0])); 
FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); 
job.setOutputKeyClass(Text.class); 
job.setOutputValueClass(Pair.class); 
System.exit(job.waitForCompletion(true) ? 0 : 1);

感謝@Pathmanaban Palsamy和@克里斯Gerken您的建議。我根據您的建議修改了代碼，但仍然獲取空輸出文件。任何人都可以請我建議我在我的主類輸入和輸出配置。我是否需要在輸入中指定Pair類到mapper &如何？

來源

2014-11-04 Manvi

我猜作爲

public void reduce(Text key, Iterable<Pair> values, 
       Context context 
       ) throws IOException, InterruptedException

您獲得通過一個可迭代（對象從中可以得到一個迭代器），您可以使用它遍歷所有的都是值的降低方法應該聲明映射到給定的密鑰。

來源

2014-11-04 04:47:47

右鍵dignosis（+1）。看看http://stackoverflow.com/questions/26710866/probelm-with-mapreduce-wordcount-program-output-is-same-as-the-input-file，OP搞砸了2種方法。 – blackSmith 2014-11-04 06:55:47

由於不需要減速，我懷疑下面一行

Pair P = new Pair(); context.write(MovieID,P);

對空將是問題。也請您檢查您給正確keyclass和valueclass您的驅動程序類像

job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(Pair.class);

來源

2014-11-04 08:17:59

Map減少作業生成空輸出文件

回答

相關問題