Scrapy蜘蛛不爬行

我想測試scrapy CrawlSpider，但我不明白它爲什麼不爬行。它應該做的是爬行維基百科的數學頁面只有一個深度級別，並返回每個爬行頁面的標題。我錯過了什麼？非常感謝幫助！Scrapy蜘蛛不爬行

from scrapy.spiders import CrawlSpider, Rule 
from scrapy.linkextractors import LinkExtractor 
from scrapy.selector import Selector 
from Beurs.items import WikiItem 

class WikiSpider(CrawlSpider): 
    name = 'WikiSpider' 
    allowed_domains = ['wikipedia.org'] 
    start_urls = ["http://en.wikipedia.org/wiki/Mathematics"] 

    Rules = (
     Rule(LinkExtractor(restrict_xpaths=('//div[@class="mw-body"]//a/@href'))), 
     Rule(LinkExtractor(allow=("http://en.wikipedia.org/wiki/",)),  callback='parse_item', follow=True),   
     ) 


def parse_item(self, response): 
    sel = Selector(response) 
    rows = sel.xpath('//span[@class="innhold"]/table/tr') 
    items = [] 

     for row in rows[1:]: 
      item = WikiItem() 
      item['agent'] = row.xpath('./td[1]/a/text()|./td[1]/text()').extract() 
      item['org'] = row.xpath('./td[2]/text()').extract() 
      item['link'] = row.xpath('./td[1]/a/@href').extract() 
      item['produkt'] = row.xpath('./td[3]/text()').extract() 
     items.append(item) 
     return items

設置：

BOT_NAME = 'Beurs' 

SPIDER_MODULES = ['Beurs.spiders'] 
NEWSPIDER_MODULE = 'Beurs.spiders' 
DOWNLOAD_HANDLERS = { 
    's3': None, 
} 
DEPTH_LIMIT = 1

和日誌：

C:\Users\Jan Willem\Anaconda\Beurs>scrapy crawl BeursSpider 
2015-11-07 15:14:36 [scrapy] INFO: Scrapy 1.0.3 started (bot: Beurs) 
2015-11-07 15:14:36 [scrapy] INFO: Optional features available: ssl, http11, boto 
2015-11-07 15:14:36 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'Beurs.spiders', 'SPIDER_MODULES': ['Beurs.spiders'], 'DEPTH_LIMIT': 1, 'BOT_NAME': 'Beurs'} 
2015-11-07 15:14:36 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState 
2015-11-07 15:14:36 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats 
2015-11-07 15:14:36 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware 
2015-11-07 15:14:36 [scrapy] INFO: Enabled item pipelines: 
2015-11-07 15:14:36 [scrapy] INFO: Spider opened 
2015-11-07 15:14:36 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
2015-11-07 15:14:36 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023 
2015-11-07 15:14:36 [scrapy] DEBUG: Redirecting (301) to <GET https://en.wikipedia.org/wiki/Mathematics> from <GET http://en.wikipedia.org/wiki/Mathematics> 
2015-11-07 15:14:37 [scrapy] DEBUG: Crawled (200) <GET https://en.wikipedia.org/wiki/Mathematics> (referer: None) 
2015-11-07 15:14:37 [scrapy] INFO: Closing spider (finished) 
2015-11-07 15:14:37 [scrapy] INFO: Dumping Scrapy stats: 
{'downloader/request_bytes': 530, 
'downloader/request_count': 2, 
'downloader/request_method_count/GET': 2, 
'downloader/response_bytes': 60393, 
'downloader/response_count': 2, 
'downloader/response_status_count/200': 1, 
'downloader/response_status_count/301': 1, 
'finish_reason': 'finished', 
'finish_time': datetime.datetime(2015, 11, 7, 14, 14, 37, 274000), 
'log_count/DEBUG': 3, 
'log_count/INFO': 7, 
'response_received_count': 1, 
'scheduler/dequeued': 2, 
'scheduler/dequeued/memory': 2, 
'scheduler/enqueued': 2, 
'scheduler/enqueued/memory': 2, 
'start_time': datetime.datetime(2015, 11, 7, 14, 14, 36, 852000)} 
2015-11-07 15:14:37 [scrapy] INFO: Spider closed (finished)

來源

2015-11-07 Argali

所以我改變了代碼的解析部分Dup的步驟的一個（見下文），但我仍然得到相同的日誌抓取0頁（0頁/分鐘），刮0項（0項/分鐘）。有人知道我能做什麼？ – Argali

所以基本上你的正則表達式是不完全正確和你的Xpath需要一些調整。我想下面的代碼符合您的要求，請嘗試一下，讓我知道如果你需要更多的幫助：

def parse_item(self, response): 
    sel = Selector(response) 
    rows = sel.xpath('//span[@class="innhold"]/table/tr') 
    items = [] 

    for row in rows[1:]: 
     item = SasItem() 
     item['agent'] = row.xpath('./td[1]/a/text()|./td[1]/text()').extract() 
     item['org'] = row.xpath('./td[2]/text()').extract() 
     item['link'] = row.xpath('./td[1]/a/@href').extract() 
     item['produkt'] = row.xpath('./td[3]/text()').extract() 
     items.append(item) 
    return items

來源

2015-11-07 14:31:59

感謝您的快速響應！我嘗試了你的調整，但不幸的是沒有運行。還有以下問題：爬行0頁（0頁/分鐘），刮0項（0項/分鐘）。它看起來像爬行本身根本沒有發生，我不明白爲什麼。有什麼建議麼？ – Argali

Scrapy蜘蛛不爬行

回答

相關問題