Scrapy LinkExtractor - 限制每個URL抓取的頁面數

Question

我想限制Scrapy中CrawlSpider中每個URL抓取的頁面數量。我有一個start_urls列表，並且我想對每個網址中正在爬網的數字設置限制。一旦達到限制，蜘蛛應該移動到下一個start_url。

我知道有關於設置的DEPTH_LIMIT參數，但這不是我正在尋找的。

任何幫助將是有用的。

這裏是我目前擁有的代碼：

我試圖實現這個

class MySpider(CrawlSpider): 
    name = 'test' 
    allowed_domains = domainvarwebsite 
    start_urls = httpvarwebsite 

    rules = [Rule(LinkExtractor(), 
      callback='parse_item', 
      follow=True) 
      ] 

    def parse_item(self, response): 
     #here I parse and yield the items I am interested in. 

編輯，但我得到exceptions.SyntaxError: invalid syntax (filter_domain.py, line 20)。關於發生了什麼的任何想法？

再次感謝。

filter_domain.py

import urlparse 
from collections import defaultdict 
from scrapy.exceptions import IgnoreRequest 

class FilterDomainbyLimitMiddleware(object): 
def __init__(self, domains_to_filter): 
    self.domains_to_filter = domains_to_filter 
    self.counter = defaultdict(int) 

@classmethod 
def from_crawler(cls, crawler): 
    settings = crawler.settings 
    spider_name = crawler.spider.name 
    max_to_filter = settings.get('MAX_TO_FILTER') 
    o = cls(max_to_filter) 
    return o 

def process_request(self, request, spider): 
    parsed_url = urlparse.urlparse(request.url) 
    (LINE 20:) if self.counter.get(parsed_url.netloc, 0) < self.max_to_filter[parsed_url.netloc]): 
     self.counter[parsed_url.netloc] += 1 
    else: 
     raise IgnoreRequest() 

settings.py

MAX_TO_FILTER = 30 

DOWNLOADER_MIDDLEWARES = { 
    'myproject.filter_domain.FilterDomainbyLimitMiddleware' :400, 

} 

Answer 1

Scrapy不直接提供，但你可以創建一個自定義的中間件，像這樣：

import urlparse 
from collections import defaultdict 
from scrapy.exceptions import IgnoreRequest 

class FilterDomainbyLimitMiddleware(object): 
    def __init__(self, domains_to_filter): 
     self.domains_to_filter = domains_to_filter 
     self.counter = defaultdict(int) 

    @classmethod 
    def from_crawler(cls, crawler): 
     settings = crawler.settings 
     spider_name = crawler.spider.name 
     domains_to_filter = settings.get('DOMAINS_TO_FILTER') 
     o = cls(domains_to_filter) 
     return o 

    def process_request(self, request, spider): 
     parsed_url = urlparse.urlparse(request.url) 
     if parsed_url.netloc in self.domains_to_filter: 
      if self.counter.get(parsed_url.netloc, 0) < self.domains_to_filter[parsed_url.netloc]): 
       self.counter[parsed_url.netloc] += 1 
      else: 
       raise IgnoreRequest() 

和設置聲明DOMAINS_TO_FILTER這樣的：

DOMAINS_TO_FILTER = { 
    'mydomain': 5 
} 

爲只接受來自該域的5項要求。還請記住在設置中啓用中間件，如指定的here