Ozzie hadoop操作錯誤：類型不匹配

我在通過OOzie運行Hadoop作業時出現問題。我有以下配置運行的作業正常：Ozzie hadoop操作錯誤：類型不匹配

public int run(String[] args) throws Exception { 
     // TODO Auto-generated method stub 
     Job job = new Job(getConf(), "GCAnalyzer"); 

     job.setJarByClass(GCMapper.class); 

     job.setMapperClass(getMapperClass()); 
     job.setReducerClass(getReducerClass()); 

     job.setOutputKeyClass(Text.class); 
     job.setOutputValueClass(NullWritable.class); 

     job.setInputFormatClass(TextInputFormat.class); 
     job.setOutputFormatClass(TextOutputFormat.class); 

     FileInputFormat.addInputPath(job, new Path(args[0])); 
     FileOutputFormat.setOutputPath(job, new Path(args[1])); 

     job.setNumReduceTasks(1); 

     return job.waitForCompletion(true) ? 0 : 1; 
    } 

    public static void main(String args[]) throws Exception { 
     int res = ToolRunner.run(new Configuration(), new GCMapper(), args); 
     System.exit(res); 
    }

當我運行具有以下Oozie的配置相同的工作，我得到錯誤：

<workflow-app xmlns="uri:oozie:workflow:0.2" name="log_analysis"> 
<start to="gclogs" /> 
<action name="gclogs"> 

<map-reduce> 
    <job-tracker>${jobTracker}</job-tracker> 
    <name-node>${nameNode}</name-node> 
    <prepare> 
    <delete path="${gcouput}"/> 
    </prepare> 

    <configuration> 

<property> 
    <name>mapred.mapper.new-api</name> 
    <value>true</value> 
    </property> 

<property> 
    <name>mapred.reducer.new-api</name> 
    <value>true</value> 
    </property> 

<property> 
<name>mapred.map.class</name> 
<value>${gcmapper}</value> 
</property> 

<property> 
<name>mapreduce.reduce.class</name> 
<value>${gcreducer}</value> 
</property> 



<property> 
<name>mapred.mapoutput.key.class</name> 
<value>org.apache.hadoop.io.Text</value> 
</property> 

<property> 
<name>mapred.mapoutput.value.class</name> 
<value>org.apache.hadoop.io.NullWritable</value> 
</property> 

<property> 
<name>mapreduce.inputformat.class</name> 
<value>org.apache.hadoop.mapreduce.lib.input.TextInputFormat</value> 
</property> 


<property> 
    <name>mapreduce.outputformat.class</name> 
    <value>org.apache.hadoop.mapreduce.lib.output.TextOutputFormat</value> 
    </property> 



<property> 
<name>mapred.input.dir</name> 
<value>${gcinput}</value> 
</property> 

<property> 
<name>mapred.output.dir</name> 
<value>${gcoutput}</value> 
</property> 

<property> 
    <name>mapred.reduce.tasks</name> 
    <value>1</value> 
    </property> 

<property> 
    <name>mapred.job.queue.name</name> 
    <value>default</value> 
    </property> 


</configuration> 
</map-reduce> 
<ok to="end" /> 
<error to="fail" /> 
</action> 

<kill name="fail"> 
    <message>Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> 
    </kill> 
    <end name="end" /> 

</workflow-app>

工作屬性文件如下：

oozie.wf.application.path=${nameNode}/user/svc_hdp/ozzie/apps/log_analyzer 
oozie.libpath=${nameNode}/user/svc_hdp/oozie/apps/share-lib 

nameNode=hdfs://path 
jobTracker=path 
queueName=default 


# GCMApred 
gcinput=${nameNode}/user/a.ntimbadi/inputgc 
gcoutput=${nameNode}/user/a.ntimbadi/ouputgc 
gcmapper=edu.*******.hadoop.log.GCMapper$CasMap 
gcreducer=org.apache.hadoop.mapreduce.Reducer

錯誤如下：

java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable

我確定我的hadoop代碼很好，因爲它在我手動運行時運行。我的oozie配置有問題。請讓我知道是否有明顯的錯誤。

來源

2013-08-05 Nemil A Timbadia

Ohk我在這裏得到了錯誤： mapred.map.class應該是mapreduce.map.class

來源

2013-08-06 15:59:01

Ozzie hadoop操作錯誤：類型不匹配

回答

相關問題