在Scrapy中創建可編輯的CrawlSpider規則

Question

我一直在嘗試創建一個簡單的Scrapy CrawlSpider腳本，該腳本可以很容易地進行更改，但我無法弄清楚如何讓鏈接提取器規則正常工作。

這裏是我的代碼：

class LernaSpider(CrawlSpider): 
"""Our ad-hoc spider""" 

name = "lerna" 

def __init__(self, url, allow_follow='.*', deny_follow='', allow_extraction='.*', deny_extraction=''): 
    parsed_url = urlparse(url) 
    domain = str(parsed_url.netloc) 
    self.allowed_domains = [domain] 
    self.start_urls = [url] 
    self.rules = (
     # Extract links 
     # and follow links from them (since no callback means follow=True by default). 
     Rule(SgmlLinkExtractor(allow=(allow_follow,), deny=(deny_follow,))), 

     # Extract links and parse them with the spider's method parse_item 
     Rule(SgmlLinkExtractor(allow=(allow_extraction,), deny=(deny_extraction,)), callback='parse_item'), 
    ) 

    super(LernaSpider, self).__init__() 

def parse_item(self, response): 

    print 'Crawling... %s' % response.url 
    # more stuff here 

我有這樣的代碼，但我從來沒有得到過允許/拒絕規則，以正常工作，我真的不知道爲什麼。是否留下空弦使它拒絕一切？我認爲，因爲這是一個RE，如果我輸入'。*'或其他什麼，它只會做一個全面的否定。

任何幫助，將不勝感激。

Answer 1

你是在自己實例化蜘蛛嗎？是這樣的：

spider = LernaSpider('http://example.com') 

，否則如果你是從你使用的URL作爲構造函數的第一個參數不正確的命令行運行$scrapy crawl lerna（應該是名字），你也不會再傳遞到超。也許試試這個：

class LernaSpider(CrawlSpider): 
    """Our ad-hoc spider""" 

    name = "lerna" 

    def __init__(self, name=None, url=url, allow_follow='.*', deny_follow='', allow_extraction='.*', deny_extraction='', **kw): 
     parsed_url = urlparse(url) 
     domain = str(parsed_url.netloc) 
     self.allowed_domains = [domain] 
     self.start_urls = [url] 
     self.rules = (
      # Extract links 
      # and follow links from them (since no callback means follow=True by default). 
      Rule(SgmlLinkExtractor(allow=allow_follow, deny=deny_follow)), 

      # Extract links and parse them with the spider's method parse_item 
      Rule(SgmlLinkExtractor(allow=allow_extraction, deny=deny_extraction), callback='parse_item'), 
     ) 
     super(LernaSpider, self).__init__(name, **kw) 

    def parse_item(self, response): 
     print 'Crawling... %s' % response.url 
     # more stuff here 

正則表達式的東西看起來很好：空值允許所有和拒絕沒有。