Python龍捲風httplib重試

Question

我使用tornado異常地使用HTTP代理來獲取許多網頁。所以，我的許多feth'es完成與錯誤（我的代理是不可靠的）。我想用另一個代理立即重試它們。下面是例子：

from tornado import ioloop 
from tornado import httpclient 

def handle_request(response): 
    if response.error: 
     print "Error:", response.error 
     // HERE i want to put my retry with another proxy 
    else: 
     print response.body 
    ioloop.IOLoop.instance().stop() 

http_client = httpclient.AsyncHTTPClient() 
http_client.fetch("http://www.google.com/", handle_request) 
ioloop.IOLoop.instance().start() 

但如何我可以我添加新的handle_request電流環取事件？另外，我如何將變量傳遞到handle_request（列出所有我的代理）。

Answer 1

你問兩個問題 -

我會考慮使用諧音http://docs.python.org/library/functools.html#partial-objects

from functools import partial 

PROXIES = [A, B, C, D] # As appropriate 
... 
def handle_request(proxies, response): 
    if ...BAD RESPONSE...: 
     return http_client.fetch(response.request.url, partial(handle_request, proxies[1:])) 
    # Now handle the case that you have a good result or you're out of proxies 

http_client.fetch("http://www.google.com/", partial(handle_request, PROXIES[:])) 

當然的另一種選擇是讓一個對象。

class ProxyRequest(object): 
    PROXIES = [A, B, C] 

    def __init__(self, url): 
      self.url = url 
      self.proxies = self.PROXIES[:] 
      self.fetch() 

    def fetch(self): 
      p, self.proxies = self.proxies[0], self.proxies[1:] 

      http_client.fetch(self.url, self.handle, proxy=p) 

    def handle(self, response): 
      if response.error: 
       if self.proxies: 
        return self.fetch() 
       else: 
        ...error case... 

      ...stop the ioloop if you want...