集成Map-Reduce與Oozie的錯誤

當我嘗試使用oozie在mapreduce中運行一個字數prg時，它只是讀取輸入記錄並顯示它。我猜它甚至沒有調用我的映射器和reducer類。因爲我使用新的API，所以在workflow.xml中也包含了new-api屬性標記。集成Map-Reduce與Oozie的錯誤

的map-reduce片段：

public class WordCount { 

    public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> { 
    private final static IntWritable one = new IntWritable(1); 
    private Text word = new Text(); 

    public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { 
     String line = value.toString(); 
     StringTokenizer tokenizer = new StringTokenizer(line); 
     while (tokenizer.hasMoreTokens()) { 
      word.set(tokenizer.nextToken()); 
      context.write(word, one); 
     } 
}

}

public static class Reduce extends Reducer<Text,IntWritable,Text,IntWritable> { 

    public void reduce(Text key, Iterable<IntWritable> values, Context context) 
     throws IOException, InterruptedException { 
     int sum = 0; 
     for (IntWritable val : values) { 
      sum += val.get(); 
     } 
      context.write(key, new IntWritable(sum)); 
    }

}

我workflow.xml：

<?xml version="1.0" encoding="UTF-8"?> 
    <workflow-app xmlns='uri:oozie:workflow:0.1' name="wordcount"> 
    <start to="wc-node" /> 
    <action name="wc-node"> 
    <map-reduce> 
     <job-tracker>${jobTracker}</job-tracker> 
     <name-node>${nameNode}</name-node> 
     <prepare> 
      <delete path="${nameNode}/user/${wf:user()}/${wordcountRoot}/output- data/${outputDir}"/> 
     </prepare> 

     <configuration> 

      <property> 
       <name>mapred.mapper.new-api</name> 
       <value>true</value> 
      </property> 

      <property> 
       <name>mapred.reducer.new-api</name> 
       <value>true</value> 
       </property> 

      <property> 
       <name>mapreduce.map.class</name> 
       <value>WordCount.Map</value> 
      </property> 

      <property> 
       <name>mapreduce.reduce.class</name> 
       <value>WordCount.Reduce</value> 
      </property> 

      <property> 
       <name>mapred.output.key.class</name> 
       <value>org.apache.hadoop.io.Text</value> 
      </property> 

      <property> 
       <name>mapred.output.value.class</name> 
       <value>org.apache.hadoop.io.IntWritable</value> 
      </property> 

      <property> 
       <name>mapred.map.tasks</name> 
       <value>1</value> 
      </property> 

      <property> 
       <name>mapred.input.dir</name> 
       <value>/user/${wf:user()}/${wordcountRoot}/input-data</value> 
      </property> 
      <property> 
       <name>mapred.output.dir</name> 
       <value>/user/${wf:user()}/${wordcountRoot}/output-data/${outputDir}</value> 
      </property> 

      <property> 
       <name>mapred.job.queue.name</name> 
       <value>${queueName}</value> 
      </property> 

      <property> 
      <name>mapreduce.job.acl-view-job</name> 
      <value>*</value> 
      </property> 

      <property> 
       <name>oozie.launcher.mapreduce.job.acl-view-job</name> 
       <value>*</value> 
      </property> 

     </configuration> 

    </map-reduce> 

    <ok to="end" /> 
    <error to="fail" /> 
</action> 

<kill name="fail"> 
    <message>Map/Reduce failed</message> 
</kill> 
<end name="end" />

我把這個鏈接https://cwiki.apache.org/OOZIE/map-reduce-cookbook.html引用了，但還是沒有運氣。如果any1遇到過這個問題，請指導我哪裏出錯。

在此先感謝。

來源

2013-02-04 Shash

解決的問題...... 在使用新的MapReduce API..we需要加前綴「$」符號映射器和減速機類名稱：

<property> 
<name>mapreduce.map.class</name> 
<value>oozie.WordCount$Map</value> 
</property> 
<property> 
<name>mapreduce.reduce.class</name> 
<value>oozie.WordCount$Reduce</value> 
</property>

來源

2013-10-04 07:42:49 Shash

集成Map-Reduce與Oozie的錯誤

回答

相關問題