爲什麼我無法在Srapy中恢復抓取？

我試圖恢復我已經進行抓取（後來試圖恢復）使用命令：爲什麼我無法在Srapy中恢復抓取？

scrapy crawl somespider -s JOBDIR=crawls/somespider-1

但沒有恢復，而是顯示以下日誌輸出：

2013-07-17 12:36:57+0530 [scrapy] INFO: Scrapy 0.16.5 started (bot: thesentientspider) 
2013-07-17 12:36:58+0530 [scrapy] DEBUG: Enabled extensions: AutoThrottle, LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState 
2013-07-17 12:36:59+0530 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, RandomUserAgentMiddleWare, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMiddleware, ChunkedTransferMiddleware, DownloaderStats 
2013-07-17 12:36:59+0530 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware 
2013-07-17 12:36:59+0530 [scrapy] DEBUG: Enabled item pipelines: MongoDBPipeline 
2013-07-17 12:36:59+0530 [zomatoSpider] INFO: Spider opened 
2013-07-17 12:36:59+0530 [zomatoSpider] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
2013-07-17 12:36:59+0530 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6033 
2013-07-17 12:36:59+0530 [scrapy] DEBUG: Web service listening on 0.0.0.0:6090 
2013-07-17 12:36:59+0530 [zomatoSpider] DEBUG: Redirecting (301) to <GET http://www.zomato.com/hyderabad/restaurants> from <GET http://www.zomato.com/hyderabad/restaurants/> 
2013-07-17 12:37:00+0530 [zomatoSpider] DEBUG: Crawled (200) <GET http://www.zomato.com/hyderabad/restaurants> (referer: None) 
2013-07-17 12:37:00+0530 [zomatoSpider] DEBUG: slot: www.zomato.com | conc: 1 | delay: 1000 ms | latency: 283 ms | size:158792 bytes 
2013-07-17 12:37:00+0530 [scrapy] DEBUG: Next page URL: http://www.zomato.com/hyderabad/restaurants?page=2 
2013-07-17 12:37:00+0530 [zomatoSpider] INFO: Closing spider (finished) 
2013-07-17 12:37:00+0530 [zomatoSpider] INFO: Dumping Scrapy stats: 
    {'downloader/request_bytes': 619, 
    'downloader/request_count': 2, 
    'downloader/request_method_count/GET': 2, 
    'downloader/response_bytes': 23308, 
    'downloader/response_count': 2, 
    'downloader/response_status_count/200': 1, 
    'downloader/response_status_count/301': 1, 
    'finish_reason': 'finished', 
    'finish_time': datetime.datetime(2013, 7, 17, 7, 7, 0, 496989), 
    'log_count/DEBUG': 10, 
    'log_count/INFO': 4, 
    'request_depth_max': 1, 
    'response_received_count': 1, 
    'scheduler/dequeued': 2, 
    'scheduler/dequeued/disk': 2, 
    'scheduler/enqueued': 2, 
    'scheduler/enqueued/disk': 2, 
    'start_time': datetime.datetime(2013, 7, 17, 7, 6, 59, 463810)} 
2013-07-17 12:37:00+0530 [zomatoSpider] INFO: Spider closed (finished)

這是我的蜘蛛代碼（如果我沒有弄錯的話，這些請求看起來是可序列化的）。設置：http://pastebin.com/CUsf4sTJ 蜘蛛：http://pastebin.com/at98Qhjh

我在做什麼錯？任何方式我可以挽救爬行？

來源

2013-07-17 paari

@alecxe：任何想法？ – paari

可能是目錄權限問題，請嘗試在「crawls」目錄中設置777。 – alecxe

它是否進入54-55行的'parse_restaurant_details'？ –

您從BaseSpider繼承，它只是抓取start_urls。你應該繼承CrawlSpider（scrapy.contrib.spiders.CrawlSpider）。

來源

2013-07-19 02:43:48

是的，聽起來是對的......我不得不放入額外的邏輯來存儲並從最後一個起始URL恢復。 – paari

爲什麼我無法在Srapy中恢復抓取？

回答

相關問題