如果我不知道什麼時候蜘蛛會完成，何時返回一個物品？

Question

所以我的蜘蛛需要一個網站列表，它通過start_requests爬過每個網站，其中yieldrequest通過item作爲meta。

然後，蜘蛛探索單個網站的所有內部鏈接，並收集到item的所有外部鏈接。問題是，我不知道什麼時候蜘蛛完成爬取所有的內部鏈接，所以我不能yield和item。

class WebsiteSpider(scrapy.Spider): 
    name = "web" 

    def start_requests(self): 
     filename = "websites.csv" 
     requests = [] 
     try: 
      with open(filename, 'r') as csv_file: 
       reader = csv.reader(csv_file) 
       header = next(reader) 
       for row in reader: 
        seed_url = row[1].strip() 
        item = Links(base_url=seed_url, on_list=[]) 
        request = Request(seed_url, callback=self.parse_seed) 
        request.meta['item'] = item 
        requests.append(request) 
      return requests 
     except IOError: 
      raise scrapy.exceptions.CloseSpider("A list of websites are needed") 

    def parse_seed(self, response): 
     item = response.meta['item'] 
     netloc = urlparse(item['base_url']).netloc 
     external_le = LinkExtractor(deny_domains=netloc) 
     external_links = external_le.extract_links(response) 
     for external_link in external_links: 
      item['on_list'].append(external_link) 

     internal_le = LinkExtractor(allow_domains=netloc) 
     internal_links = internal_le.extract_links(response) 

     for internal_link in internal_links: 
      request = Request(internal_link, callback=self.parse_seed) 
      request.meta['item'] = item 
      yield request 

Answer 1

的start_requests方法需要得到Request對象。您不需要返回請求列表，但僅在準備就緒時產生請求，這是可行的，因爲scrapy請求是異步的。

同樣的情況發生在物品上，只要你認爲物品已經準備就緒，你只需要放棄你的物品，我會建議你的情況來檢查是否沒有更多的internal_links來放棄物品，或者你也可以許多項目，只要你想，然後再檢查哪一個是最後（或者一個具有更多數據）：

class WebsiteSpider(scrapy.Spider): 
    name = "web" 

    def start_requests(self): 
     filename = "websites.csv" 
     requests = [] 
     try: 
      with open(filename, 'r') as csv_file: 
       reader = csv.reader(csv_file) 
       header = next(reader) 
       for row in reader: 
        seed_url = row[1].strip() 
        item = Links(base_url=seed_url, on_list=[]) 
        yield Request(seed_url, callback=self.parse_seed, meta = {'item'=item}) 
     except IOError: 
      raise scrapy.exceptions.CloseSpider("A list of websites are needed") 

    def parse_seed(self, response): 
     item = response.meta['item'] 
     netloc = urlparse(item['base_url']).netloc 
     external_le = LinkExtractor(deny_domains=netloc) 
     external_links = external_le.extract_links(response) 
     for external_link in external_links: 
      item['on_list'].append(external_link) 

     internal_le = LinkExtractor(allow_domains=netloc) 
     internal_links = internal_le.extract_links(response) 
     if internal_links: 
      for internal_link in internal_links: 
       request = Request(internal_link, callback=self.parse_seed) 
       request.meta['item'] = item 
       yield request 
     else: 
      yield item 

你可以做的另一件事情是創建一個extension做你需要的spider_closed方法什麼和做什麼無論你想知道什麼時候蜘蛛結束。