嘗試/探測與scapy的請求

我想知道是否有可能使用scrapy請求在進行頁面的實際處理之前檢查url的有效性（url不是事先知道的，而是它們的不同模式出現可能會被測試）。失敗的示例代碼如下。（所使用的變量reties爲簡單起見，該測試條件也可以像s.th if response.code != 200 ）嘗試/探測與scapy的請求

代碼失敗，因爲在第二回調的端部（parse_page_2）控制不返回到第一回調（ parse_page_1），即使發出新的請求，其回調爲parse_page_1。這是爲什麼發生？我知道基於urllib2的解決方案表示爲here，只是檢查是否可以嚴格在scrapy環境中完成。

import scrapy 

from scrapy.contrib.spiders import CrawlSpider, Rule 
from scrapy.contrib.linkextractors import LinkExtractor 
from scrapy.http import Request 

class MySpider(CrawlSpider): 
    name = 'alexa' 
    allowed_domains = ['alexa.com'] 
    start_urls = ['http://www.alexa.com'] 
    retries = 0 
    rules = (
     # Extract links matching 'category.php' (but not matching 'subsection.php') 
     # and follow links from them (since no callback means follow=True by default). 
     # Rule(LinkExtractor(allow=('topsites',))), 

     # Extract links matching 'item.php' and parse them with the spider's method parse_item 
     Rule(LinkExtractor(allow=('topsites',)), callback='parse_page1'), 
    ) 

    def parse_page1(self, response): 
     if self.retries < 5: 
      self.retries += 1 
      print 'Retries in 1: ', self.retries 
      return scrapy.Request("http://www.alexa.com/siteieekeknfo/google.com", 
           meta={'dont_merge_cookies': True, 
           'dont_redirect': False, 
           "handle_httpstatus_list": [301, 302, 303, 404]}, 
           callback=self.parse_page2) 
     else: 
      print "Finished in 1" 

    def parse_page2(self, response): 
     if self.retries < 5: 
      self.retries += 1 
      print 'Retries in 2: ', self.retries 
      return scrapy.Request("http://www.alexa.com/siteieekeknfo/google.com", 
           meta={'dont_merge_cookies': True, 
           'dont_redirect': False, 
           "handle_httpstatus_list": [301, 302, 303, 404]}, 
           callback=self.parse_page1) 
     else: 
      print "Finished in 2"

將抓取結果粘貼here。

來源

2015-10-06 pkaramol

你可能使用了錯誤的蜘蛛。嘗試蜘蛛，蜘蛛和直接從start_requests產量，因爲你已經知道你的網址。 http://scrapy.readthedocs.org/zh/latest/topics/spiders.html#scrapy-spider – digenishjkl

遞歸回調似乎工作：

import random 


from scrapy.contrib.spiders import CrawlSpider, Rule 
from scrapy.contrib.linkextractors import LinkExtractor 
from scrapy.http import Request 

class MySpider(CrawlSpider): 
    name = 'alexa' 
    allowed_domains = ['alexa.com'] 
    start_urls = ['http://www.alexa.com']  
    rules = (
     Rule(LinkExtractor(allow=('topsites',)), callback='parse_page1'), 
    ) 

    _retries = 0 

    _random_urls = [ 
     'http://www.alexa.com/probablydoesnotexist', 
     'http://www.alexa.com/neitherdoesthis', 
     'http://www.alexa.com/siteinfo/google.com' 
    ] 

    def parse_page1(self, response): 
     print "Got status: ", response.status 
     if self._retries == 0 or response.status != 200: 
      self._retries += 1 
      print 'Retries in 1: ', self._retries 
      return Request(random.choice(self._random_urls), 
           meta={"handle_httpstatus_list": [301, 302, 303, 404]}, 
           callback=self.parse_page1) 
     else: 
      print "Exiting"

來源

2015-10-09 16:03:55 pkaramol

嘗試/探測與scapy的請求

回答

相關問題