的Avro模式不兌現落後了兼容性

我有這樣的Avro架構的Avro模式不兌現落後了兼容性

{ 
"namespace": "xx.xxxx.xxxxx.xxxxx", 
"type": "record", 
"name": "MyPayLoad", 
"fields": [ 
    {"name": "filed1", "type": "string"}, 
    {"name": "filed2",  "type": "long"}, 
    {"name": "filed3", "type": "boolean"}, 
    { 
      "name" : "metrics", 
      "type": 
      { 
      "type" : "array", 
      "items": 
      { 
       "name": "MyRecord", 
       "type": "record", 
       "fields" : 
        [       
         {"name": "min", "type": "long"}, 
         {"name": "max", "type": "long"}, 
         {"name": "sum", "type": "long"}, 
         {"name": "count", "type": "long"} 
        ] 
      } 
      } 
    } 
    ] 
}

這裏是我們用它來解析

public static final MyPayLoad parseBinaryPayload(byte[] payload) { 
     DatumReader<MyPayLoad> payloadReader = new SpecificDatumReader<>(MyPayLoad.class); 
     Decoder decoder = DecoderFactory.get().binaryDecoder(payload, null); 
     MyPayLoad myPayLoad = null; 
     try { 
      myPayLoad = payloadReader.read(null, decoder); 
     } catch (IOException e) { 
      logger.log(Level.SEVERE, e.getMessage(), e); 
     } 

     return myPayLoad; 
    }

現在我想數據的代碼再添加一個字段INT模式使模式如下所示

{ 
"namespace": "xx.xxxx.xxxxx.xxxxx", 
"type": "record", 
"name": "MyPayLoad", 
"fields": [ 
    {"name": "filed1", "type": "string"}, 
    {"name": "filed2",  "type": "long"}, 
    {"name": "filed3", "type": "boolean"}, 
    { 
      "name" : "metrics", 
      "type": 
      { 
      "type" : "array", 
      "items": 
      { 
       "name": "MyRecord", 
       "type": "record", 
       "fields" : 
        [       
         {"name": "min", "type": "long"}, 
         {"name": "max", "type": "long"}, 
         {"name": "sum", "type": "long"}, 
         {"name": "count", "type": "long"} 
        ] 
      } 
      } 
    } 
    {"name": "agentType", "type": ["null", "string"], "default": "APP_AGENT"} 
    ] 
}

注意已添加的字段以及默認值已定義。問題是，如果我們收到這是使用舊的模式我得到這個錯誤

java.io.EOFException: null 
    at org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:473) ~[avro-1.7.4.jar:1.7.4] 
    at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:128) ~[avro-1.7.4.jar:1.7.4] 
    at org.apache.avro.io.BinaryDecoder.readIndex(BinaryDecoder.java:423) ~[avro-1.7.4.jar:1.7.4] 
    at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) ~[avro-1.7.4.jar:1.7.4] 
    at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) ~[avro-1.7.4.jar:1.7.4] 
    at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) ~[avro-1.7.4.jar:1.7.4] 
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:152) ~[avro-1.7.4.jar:1.7.4] 
    at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:177) ~[avro-1.7.4.jar:1.7.4] 
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:148) ~[avro-1.7.4.jar:1.7.4] 
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:139) ~[avro-1.7.4.jar:1.7.4] 
    at com.appdynamics.blitz.shared.util.XXXXXXXXXXXXX.parseBinaryPayload(BlitzAvroSharedUtil.java:38) ~[blitz-shared.jar:na]

我從this文檔明白，這本來應該是向下兼容的，但不知何故，似乎並沒有被寫入數據案件。任何想法我做錯了什麼？

來源

2016-01-12 Raghvendra Singh

有，我可以在你的架構看到兩個可能的問題

我的默認值似乎總是有工作作爲空指定這一點，你需要設置

"default": null

同樣在您的架構中，您確實忘記在數組和新字段之間添加一個（字段分隔符）。因此，嘗試改變你的模式作爲

{ "namespace": "xx.xxxx.xxxxx.xxxxx", "type": "record", "name": "MyPayLoad", "fields": [ {"name": "filed1", "type": "string"}, {"name": "filed2", "type": "long"}, {"name": "filed3", "type": "boolean"}, { "name" : "metrics", "type": { "type" : "array", "items": { "name": "MyRecord", "type": "record", "fields" : [ {"name": "min", "type": "long"}, {"name": "max", "type": "long"}, {"name": "sum", "type": "long"}, {"name": "count", "type": "long"} ] } } }, {"name": "agentType", "type": ["null", "string"], "default":null} ] }

來源

2016-02-02 09:06:12

最後我得到這個工作。我需要在SpecificDatumReader 給兩個模式所以我修改解析像這樣的地方我的讀者通過新舊架構都和它的工作就像一個魅力

public static final MyPayLoad parseBinaryPayload(byte[] payload) { 
     DatumReader<MyPayLoad> payloadReader = new SpecificDatumReader<>(SCHEMA_V1, SCHEMA_V2); 
     Decoder decoder = DecoderFactory.get().binaryDecoder(payload, null); 
     MyPayLoad myPayLoad = null; 
     try { 
      myPayLoad = payloadReader.read(null, decoder); 
     } catch (IOException e) { 
      logger.log(Level.SEVERE, e.getMessage(), e); 
     } 

     return myPayLoad; 
    }

來源

2016-02-02 21:53:18

我面對這個確切的情況。嘗試使用較新的模式讀取舊數據時，寫入的數據失敗。較新的模式只有一個附加域和聯合和默認設置。「type」：[「null」，「string」]，「doc」：「」，「default」：null

儘管設置了默認值，但null在讀取時不會自動填充。作者和讀者模式都需要在閱讀過程中提供。我的理解是向後兼容，它應該能夠支持較新的列而不需要舊的模式。

來源

2016-11-15 07:40:17 Radha

的Avro模式不兌現落後了兼容性

回答

相關問題