帶有流式處理和緩存請求的ElementTree.iterparse拋出ParseError

Question

我有一個Flask應用程序，它從url中檢索XML文檔並對其進行處理。我使用redis的requests_cache來避免額外的請求，ElementTree.iterparse遍歷流媒體內容。下面是我的代碼示例（相同的結果，無論從開發服務器和交互式解釋時）：

>>> import requests, requests_cache 
>>> import xml.etree.ElementTree as ET 
>>> requests_cache.install_cache('test', backend='redis', expire_after=300) 
>>> url = 'http://myanimelist.net/malappinfo.php?u=doomcat55&status=all&type=anime' 
>>> response = requests.get(url, stream=True) 
>>> for event, node in ET.iterparse(response.raw): 
...  print(node.tag) 

運行上面的代碼中，一旦拋出一個ParseError：

Traceback (most recent call last): 
    File "<stdin>", line 1, in <module> 
    File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/xml/etree/ElementTree.py", line 1301, in __next__ 
    self._root = self._parser._close_and_return_root() 
    File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/xml/etree/ElementTree.py", line 1236, in _close_and_return_root 
    root = self._parser.close() 
xml.etree.ElementTree.ParseError: no element found: line 1, column 0 

但是，運行完全相同的在緩存過期之前再次執行代碼實際上會打印預期的結果！ XML解析如何僅在第一次失敗，我該如何解決它？

---

編輯： 如果它是有幫助的，我已經注意到，運行相同的代碼，而無需在不同的ParseError緩存結果：

Traceback (most recent call last): 
    File "<stdin>", line 1, in <module> 
    File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/xml/etree/ElementTree.py", line 1289, in __next__ 
    for event in self._parser.read_events(): 
    File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/xml/etree/ElementTree.py", line 1272, in read_events 
    raise event 
    File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/xml/etree/ElementTree.py", line 1230, in feed 
    self._parser.feed(data) 
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 1, column 0 

Answer 1

我可以告訴你，失敗的原因爲這兩種情況下，對於後者則是因爲數據是gzip壓縮你第一次叫生，當你讀了第二次，然後將數據解壓縮無論發生什麼情況：如果打印線

：

for line in response.raw: 
    print(line) 

你看：

�=V���H�������mqn˫+i�������UȣT����F,�-§�ߓ+���G�o~�����7�C�M{�3D����೺C����ݣ�i�����SD�݌.N�&�HF�I�֎�9���J�ķ����s�*H�@$p�o���Ĕ�Y��v�����8}I,��`�cy�����gE�� �!��B� &|(^���jo�?�^,���H���^~p��a���׫��j� 

����a۱Yk<qba�RN6�����l�/�W����{/��߸�G 

X�LxH��哫 .���g(�MQ ����Y�q��:&��>s�M�d4�v|��ܓ��k��A17� 

然後解壓：

import zlib 
def decomp(raw): 
    decompressor = zlib.decompressobj(zlib.MAX_WBITS | 16) 
    for line in raw: 
     yield decompressor.decompress(line) 

for line in decomp(response.raw): 
    print(line) 

你看減壓原理：

<?xml version="1.0" encoding="UTF-8"?> 
<myanimelist><myinfo><user_id>4731313</user_id><user_name>Doomcat55</user_name><user_watching>3</user_watching><user_completed>120</user_completed><user_onhold>8</user_onhold><user_dropped>41</user_dropped><user_plantowatch>2</user_plantowatch><user_days_spent_watching>27.83</user_days_spent_watching></myinfo><anime><series_animedb_id>64</series_animedb_id><series_title>Rozen Maiden</series_title><series_synonyms>; Rozen Maiden</series_synonyms><series_type>1</series_type><series_episodes>12</series_episodes><series_status>2</series_status><series_start>2004-10-08</series_start><series_end>2004-12-24</series_end><series_image>http://cdn.myanimelist.net/images/anime/2/15728.jpg</series_image> 
.................................. 

現在緩存之後，如果我們看幾個字節：

response.raw.read(39) 

你看，我們得到的解壓縮數據：

<?xml version="1.0" encoding="UTF-8"?> 

忘記緩存和傳遞response.raw到iterparse給出：

raise e 
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 1, column 0 

因爲它不能處理gzip壓縮數據。

而且使用在第一次運行以下，同時緩存：

for line in response.raw: 
    print(line) 

給我：

ValueError: I/O operation on closed file. 

這是因爲緩存已經消耗的數據所以實際上什麼也沒有所以不確定是否真正有可能使用raw與緩存，因爲數據被消耗且文件句柄關閉。

如果使用lxml.fromstringlist：

import requests, requests_cache 
import lxml.etree as et 
requests_cache.install_cache() 

def lazy(resp): 
    for line in resp.iter_content(): 
     yield line 

url = 'http://myanimelist.net/malappinfo.php?u=doomcat55&status=all&type=anime' 

response = requests.get(url, stream=True) 

for node in et.fromstringlist(lazy(response)): 
    print(node)