在Oozie中爲Avro密鑰定義架構

我是新來的地圖reduce和Avro。我的項目基本上只具有映射函數，它在文本數據和輸出數據的Avro和我已經宣佈我的映射是這樣的：在Oozie中爲Avro密鑰定義架構

public class AvroMapper extends Mapper(LongWritable, Text, AvroKey<CharSequence>, NullWritable)

我有麻煩設置模式在Oozie的工作流程關鍵。我Oozie的文件配置爲：

<property> 
    <name>mapred.output.key.class</name> 
    <value>org.apache.avro.mapred.NullWriatable</value> 
</property> 
<property> 
    <name>mapred.mapoutput.key.class</name> 
    <value>org.apache.avro.mapred.AvroKey</value> 
</property> 
<property> 
    <name>mapred.mapoutput.value.class</name> 
    <value>org.apache.avro.mapred.NullWritable</value> 
</property> 
<property> 
<name>mapred.output.key.comparator.class</name> 
<value>org.apache.avro.mapred.AvroKeyComparator</value> 
</property> 
<property> 
    <name>avro.schema.output.key</name> 
    <value>{my JSON schema}</value> 
</property> 
<property> 
<name>mapreduce.inputformat.class</name> 
<value>org.apache.hadoop.mapreduce.lib.input.TextInputFormat</value> 
</property> 
<property> 
    <name>mapreduce.outputformat.class</name> 
     <value>org.apache.avro.mapreduce.AvroKeyOutputFormat</value> 
    </property>

，但它仍然拋出：

java.lang.NullPointerException 
at org.apache.avro.mapred.Pair.getKeySchema(Pair.java:68) 
at org.apache.avro.mapred.AvroKeyComparator.setConf(AvroKeyComparator.java:39) 
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) 
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) 
at org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:818) 
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:836) 
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:376) 
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:85) 
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:584) 
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:656) 
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) 
at org.apache.hadoop.mapred.Child$4.run(Child.java:268) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:396) 
at org.apache.ha...

請指導我我要去哪裏錯了..

來源

2014-02-12 user1467245

使用AvroMapper和AvroReducer類來代替。這對我來說更容易。請記住在這種情況下使用Pair類和模式。

無論如何，Avro的Oozie配置不是微不足道的。爲了節省您一些時間，這裏是我對AvroMapper和AvroReducer的配置：

<?xml version="1.0" encoding="UTF-8" standalone="no"?> 
<configuration> 
    <property> 
     <name>avro.input.schema</name> 
     <value>{"type":"record","name":"Pair","namespace":"org.apache.avro.mapred","fields":[... your fields ...]}</value> 
    </property> 
    <property> 
     <name>avro.output.schema</name> 
     <value>{"type":"record","name":"Pair","namespace":"org.apache.avro.mapred","fields":[... your fields ...]}</value> 
    </property> 
    <property> 
     <name>avro.mapper</name> 
     <value>your.mapper.class.Name</value> 
    </property> 
    <property> 
     <name>avro.reducer</name> 
     <value>your.reducer.class.Name</value> 
    </property> 
    <property> 
     <name>mapred.output.key.comparator.class</name> 
     <value>org.apache.avro.mapred.AvroKeyComparator</value> 
    </property> 
    <property> 
     <name>mapred.reducer.class</name> 
     <value>org.apache.avro.mapred.HadoopReducer</value> 
    </property> 
    <property> 
     <name>mapred.output.format.class</name> 
     <value>org.apache.avro.mapred.AvroOutputFormat</value> 
    </property> 
    <property> 
     <name>mapred.mapper.class</name> 
     <value>org.apache.avro.mapred.HadoopMapper</value> 
    </property> 
    <property> 
     <name>mapred.input.format.class</name> 
     <value>org.apache.avro.mapred.AvroInputFormat</value> 
    </property> 
    <property> 
     <name>mapred.output.key.class</name> 
     <value>org.apache.avro.mapred.AvroWrapper</value> 
    </property> 
    <property> 
     <name>mapred.mapoutput.value.class</name> 
     <value>org.apache.avro.mapred.AvroValue</value> 
    </property> 
    <property> 
     <name>io.serializations</name> 
     <value>org.apache.hadoop.io.serializer.WritableSerialization,org.apache.avro.mapred.AvroSerialization</value> 
    </property> 
    <property> 
     <name>mapred.mapoutput.key.class</name> 
     <value>org.apache.avro.mapred.AvroKey</value> 
    </property> 
</configuration>

來源

2014-02-13 09:05:49

在Oozie中爲Avro密鑰定義架構

回答

相關問題