Mapreduce MultipleOutputs錯誤

Question

我想將mapreduce作業的輸出存儲在兩個不同的目錄中。 儘管我的代碼被設計爲在不同的目錄中存儲相同的輸出。

下面

public class WordCountMain { 


public static void main(String[] args) throws Exception { 

Configuration conf = new Configuration(); 

Job myhadoopJob = new Job(conf); 

myhadoopJob.setJarByClass(WordCountMain.class); 
myhadoopJob.setJobName("WORD COUNT JOB"); 
FileInputFormat.addInputPath(myhadoopJob, new Path(args[0])); 

myhadoopJob.setMapperClass(WordCountMapper.class); 
myhadoopJob.setReducerClass(WordCountReducer.class);  
myhadoopJob.setInputFormatClass(TextInputFormat.class); 
myhadoopJob.setOutputFormatClass(TextOutputFormat.class); 

myhadoopJob.setMapOutputKeyClass(Text.class); 
myhadoopJob.setMapOutputValueClass(IntWritable.class); 

myhadoopJob.setOutputKeyClass(Text.class); 
myhadoopJob.setOutputValueClass(IntWritable.class); 

MultipleOutputs.addNamedOutput(myhadoopJob, "output1", TextOutputFormat.class, Text.class, IntWritable.class); 
MultipleOutputs.addNamedOutput(myhadoopJob, "output2", TextOutputFormat.class, Text.class, IntWritable.class); 
FileOutputFormat.setOutputPath(myhadoopJob, new Path(args[1])); 




System.exit(myhadoopJob.waitForCompletion(true) ? 0 : 1); 



} 

我Driver類代碼}

我的映射器代碼

public class WordCountMapper extends Mapper<LongWritable, Text, Text,  IntWritable> 

{ 





@Override 
protected void map(LongWritable key, Text value, Context context)throws IOException, InterruptedException { 

String line = value.toString(); 
String word =null; 

StringTokenizer st = new StringTokenizer(line,","); 


while(st.hasMoreTokens()) 
{ 
word= st.nextToken(); 



context.write(new Text(word), new IntWritable(1)); 




} 



} 

}

我減速器代號低於

public class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> 

{ 

MultipleOutputs mout =null; 

protected void reduce(Text key, Iterable<IntWritable> values, Context context)throws IOException, InterruptedException { 


int count=0; 
int num =0; 



    Iterator<IntWritable> ie =values.iterator(); 

    while(ie.hasNext()) 
    { 
     num = ie.next().get();//1 
     count= count+num; 

    } 
mout.write("output1", key, new IntWritable(count)); 

mout.write("output2", key, new IntWritable(count)); 

@Override 
protected void setup(org.apache.hadoop.mapreduce.Reducer.Context context) 
     throws IOException, InterruptedException { 
    // TODO Auto-generated method stub 
    super.setup(context); 

    mout = new MultipleOutputs<Text, IntWritable>(context); 
} 




} 

@Override 
protected void setup(org.apache.hadoop.mapreduce.Reducer.Context context) 
     throws IOException, InterruptedException { 

    super.setup(context); 

    mout = new MultipleOutputs<Text, IntWritable>(context); 
} 

}

我只是給在輸出目錄減少方法本身

但是當我運行使用下面的命令，這MapReduce工作，它什麼都不做。甚至Mapreduce也沒有開始。只是一個空白，並保持閒置。

hadoop jar WordCountMain.jar /user/cloudera/inputfiles/words.txt /user/cloudera/outputfiles/mapreduce/multipleoutputs 

有人能解釋我出了什麼問題？如何解決這個我的代碼

實際上發生的事情是有不同的名稱，兩個輸出文件都存儲在/用戶/ Cloudera公司/ outputfiles/MapReduce的/ multipleoutputs 。

但我需要的是將輸出文件存儲在不同的目錄中。

在豬，我們可以通過給不同的目錄

兩個STORE語句中使用怎樣實現的MapReduce

Answer 1

你可以嘗試關閉多個輸出對象在減速清理方法相同。