Scrapy鏈接提取器否認以前刮過的鏈接

Question

我正在構建使用CrawlSpider類的抓取工具。我懷疑鏈接提取器一遍又一遍地循環相同的鏈接。有沒有辦法限制鏈接提取和拒絕已被刮掉的鏈接？這可以在拒絕輸入中沒有正則表達式的情況下完成嗎？

My Rules look like this: 

{

rules = (
    #Rule(SgmlLinkExtractor((allow='profile')), follow=True), 
    Rule(SgmlLinkExtractor(deny='feedback\.html'),callback='parse_item', follow=True), 
    ) 

} 
And my parse_item is: 

{

def parse_item(self, response): 
    hxs = HtmlXPathSelector(response) 
    element = hxs.select('//table[@id="profilehead"]/tr/td/a/@href').extract() 
    try: 
     open('urls.txt', 'a').write(element[0]) 
     open('urls.txt', 'a').write('\n') 
    except IndexError: 
     # Site doesn't have link to another website 
     pass 

}

Answer 1

我覺得scrapy不遵守已訪問過的鏈接。但是如果你想限制某一部分沒有被跟蹤，那麼你可以嘗試這樣的事情

restrict_xpaths=('//a[starts-with(@title,"Next ")]')),

http://doc.scrapy.org/en/latest/topics/link-extractors.html