從POJO到Avro記錄的通用轉換

我正在尋找一種方法將POJO以通用方式轉換爲avro對象。實施應該對POJO級別的任何變化都很有效。我已經實現了它，但明確填寫了avro記錄（請參見下面的示例）。從POJO到Avro記錄的通用轉換

有沒有辦法擺脫硬編碼的字段名稱，只需填寫對象的AVO記錄？反射是唯一的方法，還是具有開箱即用的功能？

import java.util.Date; 
import java.util.HashMap; 
import java.util.Map; 

import org.apache.avro.Schema; 
import org.apache.avro.generic.GenericData.Record; 
import org.apache.avro.reflect.ReflectData; 

public class PojoToAvroExample { 

    static class PojoParent { 
     public final Map<String, String> aMap = new HashMap<String, String>(); 
     public final Map<String, Integer> anotherMap = new HashMap<String, Integer>(); 
    } 

    static class Pojo extends PojoParent { 
     public String uid; 
     public Date eventTime; 
    } 

    static Pojo createPojo() { 
     Pojo foo = new Pojo(); 
     foo.uid = "123"; 
     foo.eventTime = new Date(); 
     foo.aMap.put("key", "val"); 
     foo.anotherMap.put("key", 42); 
     return foo; 
    } 

    public static void main(String[] args) { 
     // extract the avro schema corresponding to Pojo class 
     Schema schema = ReflectData.get().getSchema(Pojo.class); 
     System.out.println("extracted avro schema: " + schema); 
     // create avro record corresponding to schema 
     Record avroRecord = new Record(schema); 
     System.out.println("corresponding empty avro record: " + avroRecord); 

     Pojo foo = createPojo(); 
     // TODO: to be replaced by generic variant: 
     // something like avroRecord.importValuesFrom(foo); 
     avroRecord.put("uid", foo.uid); 
     avroRecord.put("eventTime", foo.eventTime); 
     avroRecord.put("aMap", foo.aMap); 
     avroRecord.put("anotherMap", foo.anotherMap); 
     System.out.println("expected avro record: " + avroRecord); 
    } 
}

來源

2015-07-03 fab

爲什麼不使用[Avro公司的ReflectDatumWriter（HTTP：//計算器.com/questions/11866466/using-apache-avro-reflect）來序列化POJO？ –

我在hadoop上下文中使用avro。對於序列化我想使用AvroParquetOutputFormat – fab

一個低效的方法會有[ReflectDatumWriter寫入POJO到字節，然後GenericDatumReader讀取字節到GenericRecord]（http://stackoverflow.com/questions/26435299/write-pojos-to-parquet -file-使用反射）。 –

-1

我自己需要這樣的東西。你需要的庫是avro JAR文件，但奇怪的是，似乎沒有辦法從avro-tools命令行調用它。

調用它爲：的Java GenerateSchemaFromPOJO com.example.pojo.Person Person.java

import java.io.FileWriter; 
import java.io.IOException; 
import java.io.Writer; 

import org.apache.avro.Schema; 

import com.fasterxml.jackson.databind.ObjectMapper; 
import com.fasterxml.jackson.dataformat.avro.AvroFactory; 
import com.fasterxml.jackson.dataformat.avro.AvroSchema; 
import com.fasterxml.jackson.dataformat.avro.schema.AvroSchemaGenerator; 
import com.fasterxml.jackson.dataformat.avro.schema.VisitorFormatWrapperImpl; 

public class GenerateSchemaFromPOJO { 

    public static void main(String[] args) { 
     String className = null; 
     String outputFile = null; 
     Writer outputWriter = null; 
     try { 
      if(args.length != 2) { 
       System.out.println("Usage: java " + GenerateSchemaFromPOJO.class.getCanonicalName() + " classname output-schema-file.json"); 
       System.exit(1); 
      } 
      className = args[0]; 
      outputFile = args[1]; 

      Class<?> clazz = Class.forName(className); 

      AvroFactory avroFactory = new AvroFactory(); 
      ObjectMapper mapper = new ObjectMapper(avroFactory); 

      AvroSchemaGenerator gen = new AvroSchemaGenerator(); 
      mapper.acceptJsonFormatVisitor(clazz, gen); 
      AvroSchema schemaWrapper = gen.getGeneratedSchema(); 

      Schema avroSchema = schemaWrapper.getAvroSchema(); 
      String asJson = avroSchema.toString(true); 

      outputWriter = new FileWriter(outputFile); 
      outputWriter.write(asJson); 
     } catch (Exception ex) { 
      System.err.println("caught " + ex); 
      ex.printStackTrace(); 
      System.exit(1); 
     } finally { 
      if(outputWriter != null) { 
       try { 
        outputWriter.close(); 
       } catch (IOException e) { 
        System.err.println("Caught " + e + " while trying to close outputWriter to " + outputFile);; 
        e.printStackTrace(); 
       } 
      } 
     } 
    } 
}

來源

2016-04-08 21:11:41 timbaileyjones

據我的理解，你的代碼爲給定的'clazz'生成avro模式。這不是，我在這個問題上所要求的。我在'ReflectData.get（）。getSchema（Pojo.class）'''中執行相同的操作。我正在尋找一種方法來替代'avroRecord.put（...，...）;'與一個通用的變體 – fab

下面是你使用Spring通用的方法來轉換

public static <V> byte[] toBytesGeneric(final V v, final Class<V> cls) { 
     final ByteArrayOutputStream bout = new ByteArrayOutputStream(); 
     final Schema schema = ReflectData.get().getSchema(cls); 
     final DatumWriter<V> writer = new ReflectDatumWriter<V>(schema); 
     final BinaryEncoder binEncoder = EncoderFactory.get().binaryEncoder(bout, null); 
     try { 
      writer.write(v, binEncoder); 
      binEncoder.flush(); 
     } catch (final Exception e) { 
      throw new RuntimeException(e); 
     } 


     return bout.toByteArray(); 
    } 

public static void main(String[] args) { 
    PojoClass pojoObject = new PojoClass(); 
    toBytesGeneric(pojoObject, PojoClass.class); 
}

來源

2016-10-12 03:22:54 big

？我用Spring特性爲這個構建了一個映射器。但它也可以通過原始反射太utils的構建映射：

import org.apache.avro.Schema; 
import org.apache.avro.generic.GenericData; 
import org.apache.avro.reflect.ReflectData; 
import org.springframework.beans.PropertyAccessorFactory; 
import org.springframework.util.Assert; 

public class GenericRecordMapper { 

    public static GenericData.Record mapObjectToRecord(Object object) { 
     Assert.notNull(object, "object must not be null"); 
     final Schema schema = ReflectData.get().getSchema(object.getClass()); 
     final GenericData.Record record = new GenericData.Record(schema); 
     schema.getFields().forEach(r -> record.put(r.name(), PropertyAccessorFactory.forDirectFieldAccess(object).getPropertyValue(r.name()))); 
     return record; 
    } 

    public static <T> T mapRecordToObject(GenericData.Record record, T object) { 
     Assert.notNull(record, "record must not be null"); 
     Assert.notNull(object, "object must not be null"); 
     final Schema schema = ReflectData.get().getSchema(object.getClass()); 
     Assert.isTrue(schema.getFields().equals(record.getSchema().getFields()), "Schema fields didn't match"); 
     record.getSchema().getFields().forEach(d -> PropertyAccessorFactory.forDirectFieldAccess(object).setPropertyValue(d.name(), record.get(d.name()) == null ? record.get(d.name()) : record.get(d.name()).toString())); 
     return object; 
    } 

}

隨着該映射器可以生成一個GenericData.Record可以很容易地序列化到Avro的。當你反序列化的Avro字節組，你可以用它來反序列化記錄重建一個POJO：

序列化

byte[] serialized = avroSerializer.serialize("topic", GenericRecordMapper.mapObjectToRecord(yourPojo));

反序列化

GenericData.Record deserialized = (GenericData.Record) avroDeserializer.deserialize("topic", serialized); 

YourPojo yourPojo = GenericRecordMapper.mapRecordToObject(deserialized, new YourPojo());

來源

2017-03-23 08:28:43 TranceMaster

從POJO到Avro記錄的通用轉換

回答

相關問題