Scrapy返回 「爬0頁」

Question

我在Windows 7和我下面這個教程：

https://realpython.com/blog/python/web-scraping-with-scrapy-and-mongodb/

這是我的代碼：

from scrapy import Spider 
from scrapy.selector import Selector 
from stack.items import StackItem 


class StackSpider(Spider): 
    name = "stack" 
    allowed_domains = ["stackoverflow.com"] 
    start_urls = [ 
     "http://stackoverflow.com/questions?pagesize=50&sort=newest", 
    ] 

    def parse(self, response): 
     questions = Selector(response).xpath('//div[@class="summary"]/h3') 

     for question in questions: 
      item = StackItem() 
      item['title'] = question.xpath(
       'a[@class="question-hyperlink"]/text()').extract()[0] 
      item['url'] = question.xpath(
       'a[@class="question-hyperlink"]/@href').extract()[0] 
      yield item 

而這正是CMD返回

2015-10-21 20:28:19 [scrapy] INFO: Scrapy 1.0.3 started (bot: stack) 
2015-10-21 20:28:19 [scrapy] INFO: Optional features available: ssl, http11 
2015-10-21 20:28:19 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'stack.spiders', 'SPIDER_MODULES': ['stack.spiders'], 'BOT_NAME': 'stack'} 
2015-10-21 20:28:19 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState 
2015-10-21 20:28:19 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats 
2015-10-21 20:28:19 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware 
2015-10-21 20:28:19 [scrapy] INFO: Enabled item pipelines: 
2015-10-21 20:28:19 [scrapy] INFO: Spider opened 
2015-10-21 20:28:19 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
2015-10-21 20:28:19 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023 
2015-10-21 20:28:19 [scrapy] DEBUG: Crawled (200) <GET http://stackoverflow.com/questions?pagesize=50&sort=newest> (referer: None) 
2015-10-21 20:28:20 [scrapy] ERROR: Spider error processing <GET http://stackoverflow.com/questions?pagesize=50&sort=newest> (referer: None) 
Traceback (most recent call last): 
    File "C:\Python27\lib\site-packages\scrapy\utils\defer.py", line 102, in iter_errback 
    yield next(it) 
    File "C:\Python27\lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 28, in process_spider_output 
    for x in result: 
    File "C:\Python27\lib\site-packages\scrapy\spidermiddlewares\referer.py", line 22, in <genexpr> 
    return (_set_referer(r) for r in result or()) 
    File "C:\Python27\lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 37, in <genexpr> 
    return (r for r in result or() if _filter(r)) 
    File "C:\Python27\lib\site-packages\scrapy\spidermiddlewares\depth.py", line 54, in <genexpr> 
    return (r for r in result or() if _filter(r)) 
    File "C:\Users\unska\Desktop\python\blogspot\stack\stack\spiders\stack_spider.py", line 21, in parse 
    'a[@class="question-hyperlink"]/text()').extract()[0] 
    File "C:\Python27\lib\site-packages\scrapy\item.py", line 63, in __setitem__ 
    (self.__class__.__name__, key)) 
KeyError: 'StackItem does not support field: title' 
2015-10-21 20:28:20 [scrapy] INFO: Closing spider (finished) 
2015-10-21 20:28:20 [scrapy] INFO: Dumping Scrapy stats: 
{'downloader/request_bytes': 248, 
'downloader/request_count': 1, 
'downloader/request_method_count/GET': 1, 
'downloader/response_bytes': 29059, 
'downloader/response_count': 1, 
'downloader/response_status_count/200': 1, 
'finish_reason': 'finished', 
'finish_time': datetime.datetime(2015, 10, 21, 17, 28, 20, 95000), 
'log_count/DEBUG': 2, 
'log_count/ERROR': 1, 
'log_count/INFO': 7, 
'response_received_count': 1, 
'scheduler/dequeued': 1, 
'scheduler/dequeued/memory': 1, 
'scheduler/enqueued': 1, 
'scheduler/enqueued/memory': 1, 
'spider_exceptions/KeyError': 1, 
'start_time': datetime.datetime(2015, 10, 21, 17, 28, 19, 585000)} 
2015-10-21 20:28:20 [scrapy] INFO: Spider closed (finished) 

這裏似乎是什麼問題？

PS。

獲得「您的文章似乎大多代碼」，所以我寫的東西在這裏亂...

Answer 1

您收到此錯誤：

KeyError: 'StackItem does not support field: title' 

而你正試圖分配給一個項目，不存在一個字段：

item['title'] = question.xpath(
      'a[@class="question-hyperlink"]/text()').extract()[0] 

你StackItem從stack.items不包含字段title。標題不會作爲StackItem類中的屬性出現。只需在StackItem中添加title即可。