動態設置scrapy請求回電

Question

我正在使用scrapy。我想旋轉代理每個請求的基礎上，並從api得到一個代理，我已經返回一個代理。我的計劃是要對API的請求，得到一個代理，然後用它的基礎上設置代理：

http://stackoverflow.com/questions/39430454/making-request-to-api-from-within-scrapy-function 

我有以下幾點：

class ContactSpider(Spider): 
    name = "contact" 

def parse(self, response): 

.... 
     PR = Request(
     'my_api' 
     headers=self.headers, 
     meta={'newrequest': Request(url_to_scrape, headers=self.headers),}, 
     callback=self.parse_PR 
    ) 
    yield PR 


def parse_PR(self, response): 
    newrequest = response.meta['newrequest'] 
    proxy_data = response.body 
    newrequest.meta['proxy'] = 'http://'+proxy_data 
    newrequest.replace(url = 'http://ipinfo.io/ip') #TESTING 
    newrequest.replace(callback= self.form_output) #TESTING 

    yield newrequest 

def form_output(self, response): 
    open_in_browser(response) 

，但我發現：

Traceback (most recent call last): 
    File "C:\twisted\internet\defer.py", line 1126, in _inlineCallbacks 
    result = result.throwExceptionIntoGenerator(g) 
    File "C:\twisted\python\failure.py", line 389, in throwExceptionIntoGenerator 
    return g.throw(self.type, self.value, self.tb) 
    File "C:\scrapy\core\downloader\middleware.py", line 43, in process_request 
    defer.returnValue((yield download_func(request=request,spider=spider))) 
    File "C:\scrapy\utils\defer.py", line 45, in mustbe_deferred 
    result = f(*args, **kw) 
    File "C:\scrapy\core\downloader\handlers\__init__.py", line 65, in download_request 
    return handler.download_request(request, spider) 
    File "C:\scrapy\core\downloader\handlers\http11.py", line 60, in download_request 
    return agent.download_request(request) 
    File "C:\scrapy\core\downloader\handlers\http11.py", line 255, in download_request 
    agent = self._get_agent(request, timeout) 
    File "C:\scrapy\core\downloader\handlers\http11.py", line 235, in _get_agent 
    _, _, proxyHost, proxyPort, proxyParams = _parse(proxy) 
    File "C:\scrapy\core\downloader\webclient.py", line 37, in _parse 
    return _parsed_url_args(parsed) 
    File "C:\scrapy\core\downloader\webclient.py", line 20, in _parsed_url_args 
    host = b(parsed.hostname) 
    File "C:\scrapy\core\downloader\webclient.py", line 17, in <lambda> 
    b = lambda s: to_bytes(s, encoding='ascii') 
    File "C:\scrapy\utils\python.py", line 117, in to_bytes 
    'object, got %s' % type(text).__name__) 
TypeError: to_bytes must receive a unicode, str or bytes object, got NoneType 

我做錯了什麼？

Answer 1

Stacktrace信息建議Scrapy遇到請求對象，其url爲None，預期它是字符串類型。

這兩行代碼中的：預期

newrequest.replace(url = 'http://ipinfo.io/ip') #TESTING 
newrequest.replace(callback= self.form_output) #TESTING 

是行不通的，因爲方法Request.replace返回一個新的實例而不是通過修改就地原始請求。

您需要像這樣：

newrequest = newrequest.replace(url = 'http://ipinfo.io/ip') #TESTING 
newrequest = newrequest.replace(callback= self.form_output) #TESTING 

或者乾脆：

newrequest = newrequest.replace(
    url='http://ipinfo.io/ip', 
    callback=self.form_output 
)