Scrapy：如何編寫UserAgentMiddleware？

Question

我想寫一個UserAgentMiddleware爲scrapy，
的文檔說：

中間件，讓蜘蛛覆蓋缺省的用戶代理。 爲了使蜘蛛覆蓋默認的用戶代理，必須設置其user_agent屬性。

文檔： https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#module-scrapy.downloadermiddlewares.useragent

但沒有一個例子，我不知道怎樣寫。
有什麼建議嗎？

Answer 1

你看看它在安裝scrapy路徑

/Users/tarun.lalwani/.virtualenvs/project/lib/python3.6/site-packages/scrapy/downloadermiddlewares/useragent.py

「」「設置用戶代理每蜘蛛頭或使用從設置的默認值」「」

from scrapy import signals 


class UserAgentMiddleware(object): 
    """This middleware allows spiders to override the user_agent""" 

    def __init__(self, user_agent='Scrapy'): 
     self.user_agent = user_agent 

    @classmethod 
    def from_crawler(cls, crawler): 
     o = cls(crawler.settings['USER_AGENT']) 
     crawler.signals.connect(o.spider_opened, signal=signals.spider_opened) 
     return o 

    def spider_opened(self, spider): 
     self.user_agent = getattr(spider, 'user_agent', self.user_agent) 

    def process_request(self, request, spider): 
     if self.user_agent: 
      request.headers.setdefault(b'User-Agent', self.user_agent) 

你可以看到下面的例子設置隨機用戶代理

https://github.com/alecxe/scrapy-fake-useragent/blob/master/scrapy_fake_useragent/middleware.py