在python中讀取avro文件時出錯

Question

我在Python中成功安裝了Apache Avro。然後，我嘗試按照以下說明將Avro文件讀入Python。

https://avro.apache.org/docs/1.8.1/gettingstartedpython.html 

我在一個已經被設置爲Python中正確路徑的目錄中有一堆Avros。這裏是我的代碼：

import avro.schema 
from avro.datafile import DataFileReader, DataFileWriter 
from avro.io import DatumReader, DatumWriter 

reader = DataFileReader(open("part-00000-of-01733.avro", "r"), DatumReader()) 
for user in reader: 
    print (user) 
reader.close() 

但是它返回此錯誤：

Traceback (most recent call last): 
    File "I:\DJ data\read avro.py", line 5, in <module> 
    reader = DataFileReader(open("part-00000-of-01733.avro", "r"), DatumReader()) 
    File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\datafile.py", line 349, in __init__ 
    self._read_header() 
    File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\datafile.py", line 459, in _read_header 
    META_SCHEMA, META_SCHEMA, self.raw_decoder) 
    File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 525, in read_data 
    return self.read_record(writer_schema, reader_schema, decoder) 
    File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg \avro\io.py", line 725, in read_record 
    field_val = self.read_data(field.type, readers_field.type, decoder) 
    File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 515, in read_data 
    return self.read_fixed(writer_schema, reader_schema, decoder) 
    File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 568, in read_fixed 
    return decoder.read(writer_schema.size) 
    File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 170, in read 
    input_bytes = self.reader.read(n) 
    File "I:\Program Files\lib\encodings\cp1252.py", line 23, in decode 
    return codecs.charmap_decode(input,self.errors,decoding_table)[0] 
UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 863: character maps to <undefined> 

我確實知道，在指令的例子，一個模式是首先創建。但什麼是avsc文件？我應該如何創建它以及我的情況下的相應模式？ 理想情況下，我想將Avro文件讀入Python，並將其保存爲Python中的磁盤或dataframe/list類型的csv格式以供進一步分析。我使用Python 3在Windows 7

EDITED 我試圖斯特凡的代碼，並返回一個新的錯誤

Traceback (most recent call last): 
    File "I:\DJ data\read avro.py", line 5, in <module> 
    reader = DataFileReader(open("part-00000-of-01733.avro", "rb"), DatumReader()) 
    File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\datafile.py", line 352, in __init__ 
    self.codec = self.GetMeta('avro.codec').decode('utf-8') 
AttributeError: 'NoneType' object has no attribute 'decode' 

EDITED2：斯特凡的代碼在大多數情況下，但有時這樣會導致Asse田這樣

Traceback (most recent call last): 
File "I:\DJ data\read avro.py", line 42, in <module> 
for user in reader: 
File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\datafile.py", line 522, in __next__ 
datum = self.datum_reader.read(self.datum_decoder) 
File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 480, in read 
return self.read_data(self.writer_schema, self.reader_schema, decoder) 
File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 525, in read_data 
return self.read_record(writer_schema, reader_schema, decoder) 
File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 725, in read_record 
field_val = self.read_data(field.type, readers_field.type, decoder) 
File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 523, in read_data 
return self.read_union(writer_schema, reader_schema, decoder) 
File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 689, in read_union 
return self.read_data(selected_writer_schema, reader_schema, decoder) 
File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 493, in read_data 
return self.read_data(writer_schema, s, decoder) 
File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 503, in read_data 
return decoder.read_utf8() 
File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 248, in read_utf8 
input_bytes = self.read_bytes() 
File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 241, in read_bytes 
return self.read(nbytes) 
File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 171, in read 
assert (len(input_bytes) == n), input_bytes 
AssertionError: b'BlackRock Group\n\n17 December 2015\n\nFORM 8.3\n\nPUBLIC OPENING POSITION DISCLOSURE/DEALING DISCLOSURE BY\n\nA PERSON WITH INTERESTS IN RELEVANT SECURITIES REPRESENTING 1% OR MORE\n\nRule 8.3 of the Takeover Code (the "Code") \n\n\n 1.   KEY INFORMATION \n \n (a) Full name of discloser:                  BlackRock, Inc. \n------------------------------------------------------------------------------------------------- ----------------- \n (b) Owner or controller of interests and short positions disclosed, if diffe 

Answer 1

你使用Windows和Python 3

• 默認情況下爲Python 3 open以文本模式打開文件。這意味着，當進一步的讀取操作發生時，Python會嘗試將文件的內容從一些字符集解碼爲Unicode。

• 您沒有指定默認字符集，所以Python會嘗試解碼內容，就好像這些內容是使用charmap（在Windows上默認）編碼的一樣。

• 明顯的Avro的文件沒有在字符表編碼，解碼失敗，異常

• 據我記得，Avro的標題反正是二進制內容......不是文本的（不知道這一點） 。所以也許首先你應該儘量不要將文件開放解碼：

reader = DataFileReader(open("part-00000-of-01733.avro", 'rb'), DatumReader())

（注意'rb'，二進制模式）

編輯：對於接下來的問題（AttributeError的），你被一個已知的bug在1.8.1中未被修復。直到下一個版本出來了，你可能只是這樣做：

import avro.schema 
from avro.datafile import DataFileReader, DataFileWriter, VALID_CODECS, SCHEMA_KEY 
from avro.io import DatumReader, DatumWriter 
from avro import io as avro_io 


class MyDataFileReader(DataFileReader): 
    def __init__(self, reader, datum_reader): 
     """Initializes a new data file reader. 

     Args: 
      reader: Open file to read from. 
      datum_reader: Avro datum reader. 
     """ 
     self._reader = reader 
     self._raw_decoder = avro_io.BinaryDecoder(reader) 
     self._datum_decoder = None # Maybe reset at every block. 
     self._datum_reader = datum_reader 

     # read the header: magic, meta, sync 
     self._read_header() 

     # ensure codec is valid 
     avro_codec_raw = self.GetMeta('avro.codec') 
     if avro_codec_raw is None: 
      self.codec = "null" 
     else: 
      self.codec = avro_codec_raw.decode('utf-8') 
     if self.codec not in VALID_CODECS: 
      raise DataFileException('Unknown codec: %s.' % self.codec) 

     self._file_length = self._GetInputFileLength() 

     # get ready to read 
     self._block_count = 0 
     self.datum_reader.writer_schema = (
      schema.Parse(self.GetMeta(SCHEMA_KEY).decode('utf-8'))) 


reader = MyDataFileReader(open("part-00000-of-01733.avro", "r"), DatumReader()) 
for user in reader: 
    print (user) 
reader.close() 

這是非常奇怪的是，這種愚蠢的錯誤可以去釋放，雖然，這不是一個符號的代碼成熟！