回調函數不會被調用在Scrapy重定向

後，我有我創建如下 -回調函數不會被調用在Scrapy重定向

from scrapy.selector import Selector 
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor 
from scrapy.contrib.spiders import CrawlSpider, Rule 
from sandbox.items import SandboxItem 

class SandboxCrawlSpider(CrawlSpider): 
    name = 'sandbox_crawl' 
    allowed_domains = ['amazonaws.com'] 
    start_urls = ['http://www.amazonaws.com/'] 
    rules = (
     Rule(SgmlLinkExtractor(), callback=('parse_item'), follow=True), 
    ) 

    def parse_item(self, response): 
     sel = Selector(response) 
     i = SandboxItem() 
     print response.url 

     return i

這裏的問題是，域I已經允許，amazonaws.com最小的履帶，重定向到aws.amazon.com。

重定向後，爬網程序會爬過頁面，但從不調用回調函數。輸出是一樣的東西

2014-03-05 15:50:56+0530 [sandbox_crawl] DEBUG: Redirecting (301) to <GET http://aws.amazon.com> from <GET http://www.amazonaws.com/> 
2014-03-05 15:50:58+0530 [sandbox_crawl] DEBUG: Crawled (200) <GET http://aws.amazon.com> (referer: None) 
2014-03-05 15:50:58+0530 [sandbox_crawl] DEBUG: Filtered offsite request to 'aws.amazon.com': <GET http://aws.amazon.com/> 
2014-03-05 15:50:58+0530 [sandbox_crawl] DEBUG: Filtered offsite request to 'portal.aws.amazon.com': <GET https://portal.aws.amazon.com/gp/aws/developer/registration/index.html> 
2014-03-05 15:50:58+0530 [sandbox_crawl] DEBUG: Filtered offsite request to 'console.aws.amazon.com': <GET https://console.aws.amazon.com/> 
2014-03-05 15:50:58+0530 [sandbox_crawl] DEBUG: Filtered offsite request to 'www.youtube.com': <GET http://www.youtube.com/embed/mZ5H8sn_2ZI?autoplay=1&hd=1&rel=0> 
2014-03-05 15:50:58+0530 [sandbox_crawl] DEBUG: Filtered offsite request to 'youtube.com': <GET http://youtube.com/embed/jOhbTAU4OPI?autoplay=1&hd=1&rel=0> 
2014-03-05 15:50:58+0530 [sandbox_crawl] DEBUG: Filtered offsite request to 'www.powerof60.com': <GET http://www.powerof60.com/?00N500000026nJd=BA_AWSHP_IntelAWS_Generic&sc_icampaign=ha_en_intel_power_of_60_ed&sc_icampaigntype=partners&sc_ichannel=ha&sc_icountry=us&sc_ipage=homepage&sc_iplace=editorial_r3_right_banner&utm_campaign=BA_AWSHP_IntelAWS&utm_content=GenericAWS&utm_medium=banner&utm_source=AWSHP> 
2014-03-05 15:50:58+0530 [sandbox_crawl] DEBUG: Filtered offsite request to 'aws.typepad.com': <GET http://aws.typepad.com/> 
2014-03-05 15:50:58+0530 [sandbox_crawl] DEBUG: Filtered offsite request to 'phx.corporate-ir.net': <GET http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-InfoReq> 
2014-03-05 15:50:58+0530 [sandbox_crawl] INFO: Closing spider (finished)

所看到的parse_item永遠不會被調用.'print response.url」沒有效果，沒有做任何其他語句的功能。蜘蛛有沒有錯誤？

來源

2014-03-05 Airmine

只需添加"aws.amazon.com"到allowed_domains

allowed_domains = ['amazonaws.com', 'aws.amazon.com']

來源

2014-03-05 10:56:45 warvariuc

是。這是一個選項。但是，如果列出包含「gmail.com」和「amazonaws.com」等網域的網址，我希望抓取工具自動解析重定向的網址，而無需手動干預。這是一個靜態的情況。在我的程序中，我從文件中讀取允許的域列表。 – Airmine

嘗試禁用[OffsiteMiddleware]（http://doc.scrapy.org/zh/latest/topics/spider-middleware.html#scrapy.contrib.spidermiddleware.offsite.OffsiteMiddleware） – warvariuc

添加dont_filter = true來要求，但這並不能最終解決問題。

這樣的：

Request('http://example.org/', callback = self.func, dont_filter=True)

來源

2015-06-08 04:00:48 1a1a11a

回調函數不會被調用在Scrapy重定向

回答

相關問題