Python etag /上次修改不起作用;如何獲取最新的rss

Question

我想寫一個python程序，它將抓取並顯示自上次程序運行以來的任何rss更新。我正在使用feedparser並嘗試使用etags，並按here on SO所述進行修改，但我的測試腳本似乎不工作。

import feedparser 
rsslist=["http://skottieyoung.tumblr.com/rss","http://mrjakeparker.com/feed/"] 
for feed in rsslist: 
print('--------'+feed+'-------') 
d=feedparser.parse(feed) 
print(len(d.entries)) 
if (len(d.entries) > 0): 
    etag=d.feed.get('etag','') 
    modified=d.get('modified',d.get('updated',d.entries[0].get('published','no modified,update or published fields present in rss'))) 

    d2=feedparser.parse(feed,modified) 
    if (len(d2.entries) > 0): 
     etag2=d2.feed.get('etag','') 
     modified2=d2.get('updated',d.entries[0].get('published','')) 

    if (d2==d): #ideally we would never see this bc etags/last modified would prevent unnecessarily downloading what we all ready have. 
     print("Arrg these are the same") 

我真的不知道，如果RSS/XML技術已經從我一直在使用網上引用更改或是否有我的代碼有問題。

無論我在尋找一個最佳的解決方案來高效地使用rss提要。就目前而言，我正在尋求最大限度地減少帶寬浪費，比如使用最後修改的字段和etags字段。

在此先感謝。

Answer 1

你的問題是你正在通過最後修改日期來代替etag。 etag是parse()方法的第二個參數，modified是第三個參數。

相反的：

d2=feedparser.parse(feed,modified) 

務必：

d2=feedparser.parse(feed,modified=modified) 

源代碼服用後一看，它看起來像經過etag或modified到parse()功能的唯一的事情確實是發給服務器適當的頭部，這樣如果沒有任何變化，服務器可以返回一個空的響應。如果服務器不支持這個，那麼服務器將只返回完整的RSS提要。我將修改代碼以檢查每個條目的日期，而忽略一個具有比在先前的請求最大日期小的日期：

import feedparser 
rsslist=["http://skottieyoung.tumblr.com/rss", "http://mrjakeparker.com/feed/"] 

def feed_modified_date(feed): 
    # this is the last-modified value in the response header 
    # do not confuse this with the time that is in each feed as the server 
    # may be using a different timezone for last-resposne headers than it 
    # uses for the publish date 

    modified = feed.get('modified') 
    if modified is not None: 
     return modified 

    return None 

def max_entry_date(feed): 
    entry_pub_dates = (e.get('published_parsed') for e in feed.entries) 
    entry_pub_dates = tuple(e for e in entry_pub_dates if e is not None) 

    if len(entry_pub_dates) > 0: 
     return max(entry_pub_dates)  

    return None 

def entries_with_dates_after(feed, date): 
    response = [] 

    for entry in feed.entries: 
     if entry.get('published_parsed') > date: 
      response.append(entry) 

    return response    

for feed_url in rsslist: 
    print('--------%s-------' % feed_url) 
    d = feedparser.parse(feed_url) 
    print('feed length %i' % len(d.entries)) 

    if len(d.entries) > 0: 
     etag = d.feed.get('etag', None) 
     modified = feed_modified_date(d) 
     print('modified at %s' % modified) 

     d2 = feedparser.parse(feed_url, etag=etag, modified=modified) 
     print('second feed length %i' % len(d2.entries)) 
     if len(d2.entries) > 0: 
      print("server does not support etags or there are new entries") 
      # perhaps the server does not support etags or last-modified 
      # filter entries ourself 

      prev_max_date = max_entry_date(d) 

      entries = entries_with_dates_after(d2, prev_max_date) 

      print('%i new entries' % len(entries)) 
     else: 
      print('there are no entries') 

這將產生：

--------http://skottieyoung.tumblr.com/rss------- 
feed length 20 
modified at None 
second feed length 20 
server does not support etags or there are new entries 
0 new entries 
--------http://mrjakeparker.com/feed/------- 
feed length 10 
modified at Wed, 07 Nov 2012 19:27:48 GMT 
second feed length 0 
there are no entries