Scrapy沒有聽取拒絕規則

由於某種原因，scrapy正在解析我的拒絕規則中的URL數據：我從包含/ browse /，/ search /，/ ip /的URL獲取解析數據。
我不確定這會出錯。Scrapy沒有聽取拒絕規則

請指教，謝謝！請在下面找到我的代碼：

from scrapy.selector import HtmlXPathSelector 
from scrapy.contrib.spiders import CrawlSpider, Rule 
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor 

from wallspider.items import Website 


class mydomainSpider(CrawlSpider): 
    name = "tp" 
    allowed_domains = ["www.mydomain.com"] 
    start_urls = ["http://www.mydomain.com",] 

    """/tp/ page type to crawl""" 

    rules = (Rule (SgmlLinkExtractor(allow=('/tp/',), 
     deny=(
      'browse/', 
      'browse-ng.do?', 
      'search-ng.do?', 
      'facet=', 
      'ip/', 
      'page/' 
      'search/', 
      '/[1-9]$', 
      '(bti=)[1-9]+(?:\.[1-9]*)?', 
      '(sort_by=)[a-zA-Z]', 
      '(sort_by=)[1-9]+(?:\.[1-9]*)?', 
      '(ic=32_)[1-9]+(?:\.[1-9]*)?', 
      '(ic=60_)[0-9]+(?:\.[0-9]*)?', 
      '(search_sort=)[1-9]+(?:\.[1-9]*)?',) 
      ,) 
    , callback="parse_items", follow= True), 
    ) 

    def parse_items(self, response): 
     hxs = HtmlXPathSelector(response) 
     sites = hxs.select('//html') 
     items = [] 

     for site in sites: 
      item = Website() 
      item['referer'] = response.request.headers.get('Referer') 
      item['url'] = response.url 
      item['title'] = site.xpath('/html/head/title/text()').extract() 
      item['description'] = site.select('//meta[@name="Description"]/@content').extract() 
      items.append(item) 

     return items

我的控制檯日誌，其掠/ IP /頁?:

2013-12-11 11:21:43-0800 [tp] DEBUG: Crawled (200) <GET http://www.mydomain.com/ip/1104329> (referer: http://www.mydomain.com/tp/john-duigan) 
2013-12-11 11:21:43-0800 [tp] DEBUG: Scraped from <200 http://www.mydomain.com/ip/1104329> 
    {'description': [u'Shop Low Prices on: Molly (Widescreen) : Movies'], 
    'referer': 'http://www.mydomain.com/tp/john-duigan', 
    'title': [u'Molly (Widescreen): Movies : mydomain.com '], 
    'url': 'http://www.mydomain.com/ip/1104329'} 
2013-12-11 11:21:43-0800 [tp] DEBUG: Redirecting (302) to <GET http://www.mydomain.com/ip/17371019> from <GET http://www.mydomain.com/tp/jon-furmanski> 
2013-12-11 11:21:43-0800 [tp] DEBUG: Redirecting (302) to <GET http://www.mydomain.com/ip/17371019> from <GET http://www.mydomain.com/tp/taylor-byrd> 
2013-12-11 11:21:43-0800 [tp] DEBUG: Redirecting (302) to <GET http://www.mydomain.com/ip/17371019> from <GET http://www.mydomain.com/tp/greg-byers> 
2013-12-11 11:21:43-0800 [tp] DEBUG: Redirecting (302) to <GET http://www.mydomain.com/ip/17371019> from <GET http://www.mydomain.com/tp/tom-bowker> 
2013-12-11 11:21:43-0800 [tp] DEBUG: Crawled (200) <GET http://www.mydomain.com/ip/21152221> (referer: http://www.mydomain.com/tp/peter-levin) 
2013-12-11 11:21:43-0800 [tp] DEBUG: Scraped from <200 http://www.mydomain.com/ip/21152221> 
    {'description': [u'Shop Low Prices on: Marva Collins Story (1981) : Video on Demand by VUDU'], 
    'referer': 'http://www.mydomain.com/tp/peter-levin', 
    'title': [u'Marva Collins Story (1981): Video on Demand by VUDU : mydomain.com '], 
    'url': 'http://www.mydomain.com/ip/21152221'}

來源

2013-12-11 Jason Youk

似乎沒有什麼不對您的代碼。在'parse_items'方法中打印'response.url'並驗證被拒絕的URL是否被獲取。 – flyer

你可以在運行蜘蛛時發佈你的控制檯日誌嗎？（如果你想替換域名） –

@pault。控制檯日誌很長，但我希望提供的是足夠的嗎？ –

我發現了什麼是錯的一部分，該頁面被重定向到一個頁面類型這是我的拒絕規則。謝謝你的幫助！我很感激！

來源

2013-12-11 19:17:31

從網頁中提取鏈接時，適用SgmlLinkExtractor的規則。在你的情況下，你的一些.../tp/...請求正被重定向到.../ip/...頁面。

Redirecting (302) to <GET http://www.mydomain.com/ip/17371019> from <GET http://www.mydomain.com/tp/tom-bowker>

allow和deny模式確實沒有應用到網址重定向後。

您可以禁用通過設置REDIRECT_ENABLED到False完全以下的重定向（見RedirectMiddleware）

來源

2013-12-13 11:49:48

真棒謝謝！我想我會改變方向。 –

Scrapy沒有聽取拒絕規則

回答

相關問題