爲什麼我在Scrapy中獲取KeyError？

我使用內部芹菜Scrapy蜘蛛，我隨機收到這種錯誤的爲什麼我在Scrapy中獲取KeyError？

Unhandled Error 
Traceback (most recent call last): 
    File "/usr/lib/python2.7/site-packages/twisted/internet/base.py", line 428, in fireEvent 
     DeferredList(beforeResults).addCallback(self._continueFiring) 
    File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 321, in addCallback 
     callbackKeywords=kw) 
    File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 310, in addCallbacks 
     self._runCallbacks() 
    File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks 
     current.result = callback(current.result, *args, **kw) 
    --- <exception caught here> --- 
    File "/usr/lib/python2.7/site-packages/twisted/internet/base.py", line 441, in _continueFiring 
     callable(*args, **kwargs) 
    File "/usr/lib/python2.7/site-packages/twisted/internet/base.py", line 667, in disconnectAll 
     selectables = self.removeAll() 
    File "/usr/lib/python2.7/site-packages/twisted/internet/epollreactor.py", line 191, in removeAll 
     [self._selectables[fd] for fd in self._reads], 
    exceptions.KeyError: 94

數量從外殼變爲情況下（94可能是另一種情況下，97等）

我使用：

celery==3.1.19 
Django==1.9.4 
Scrapy==1.3.0

我這是怎麼運行Scrapy裏面芹菜：

from billiard import Process 
from scrapy.crawler import CrawlerProcess 
from scrapy.utils.project import get_project_settings 

class MyCrawlerScript(Process): 
    def __init__(self, **kwargs): 
     Process.__init__(self) 
     settings = get_project_settings('my_scraper') 
     self.crawler = CrawlerProcess(settings) 
     self.spider_name = kwargs.get('spider_name') 
     self.kwargs = kwargs 

    def run(self): 
     self.crawler.crawl(self.spider_name, qwargs=self.kwargs) 
     self.crawler.start() 

def my_crawl_manager(**kwargs): 
    crawler = MyCrawlerScript(**kwargs) 
    crawler.start() 
    crawler.join()

裏面芹菜任務，我呼籲：

my_crawl_manager(spider_name='my_spider', url='www.google.com/any-url-here')

請任何想法，爲什麼發生這種情況？

來源

2017-02-20 mou55

我有幾乎相同的確切錯誤。任何解決方案？ –

我曾經有過這個問題。

檢查您是否有空文件__init__.py文件在spiders文件夾或。它應該在那裏。

來源

2017-02-20 12:15:51 Umair

謝謝你，我已經檢查了，並且我沒有在代碼 – mou55

中缺少\ _ \ _init \ _ \ _。py爲什麼__init__.py是一個多數？我不忍受這個問題，但我很好奇... btw我沒有__init__文件那裏 –

@AmiHollander在這裏閱讀答案https://stackoverflow.com/questions/448271/what-is-init-py-for – Umair

爲什麼我在Scrapy中獲取KeyError？

回答

相關問題