Scrapy start_urls不被解析

我剛剛更新scrapy到0.22，現在所面臨的幾個問題。Scrapy start_urls不被解析

我使用的是基本的蜘蛛，只是簡單地想請求一個START_URL和看到解析（個體經營，響應）函數的結果。

class wildberries(Spider): 

    name = "wildberries" 
    allowed_domains = ['wildberries.ru'] 
    start_urls = ['http://www.wildberries.ru/'] 

    def parse(self, response): 
     hxs   = HtmlXPathSelector(response) 
     print "Hello World", response.url 
     maincats = hxs.select('//div[@class="tomenu"]/ul[@class="green_menu"]/li/a/@href').extract() 
     print maincats

我從執行蜘蛛得到的唯一輸出是：

[[email protected] krauler]$ scrapy crawl wildberries 
2014-02-25 12:46:19+0400 [scrapy] INFO: Scrapy 0.22.2 started (bot: scrapybot) 
2014-02-25 12:46:19+0400 [scrapy] INFO: Optional features available: ssl, http11 
2014-02-25 12:46:19+0400 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE':  'krauler.spiders', 'LOG_STDOUT': True, 'SPIDER_MODULES': ['krauler.spiders'], 'USER_AGENT': 'Mozilla/5.0 (Windows NT 6.1; WOW64)', 'CONCURRENT_REQUESTS': 1}

我不知道爲什麼我的START_URL不會被解析，因爲我沒有得到任何解析響應（）我假定該函數甚至沒有被調用。對建議感到高興，如果有任何問題發生，我很樂意提供進一步的代碼。

來源

2014-02-25 KangLi

，正如另外：我通過outcommenting LOG_STDOUT固定的問題=在我的設置文件中爲True。仍然不知道爲什麼scrapy0.22無法應付它。 – KangLi

縮進看起來怪怪的。我不知道這是否是因爲StackOverflow的什麼的，但嘗試

from scrapy.spider import Spider 
from scrapy.selector import Selector 


class wildberries(Spider): 
    name = "wildberries" 
    allowed_domains = ['wildberries.ru'] 
    start_urls  = ['http://www.wildberries.ru/'] 

    def parse(self, response): 
     selector = Selector(response) 
     print "Hello World", response.url 
     maincats = selector.xpath('//div[@class="topmenu"]/ul[@class="green_menu"]/li/a/@href').extract() 
     print maincats

我得到當「正確」的輸出：

[email protected]:~/tmp$ scrapy runspider wildberries.py 
2014-02-25 10:34:44+0100 [scrapy] INFO: Scrapy 0.23.0 started (bot: scrapybot) 
2014-02-25 10:34:44+0100 [scrapy] INFO: Optional features available: ssl, http11, boto, django 
2014-02-25 10:34:44+0100 [scrapy] INFO: Overridden settings: {} 
2014-02-25 10:34:44+0100 [scrapy] INFO: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState 
2014-02-25 10:34:44+0100 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats 
2014-02-25 10:34:44+0100 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware 
2014-02-25 10:34:44+0100 [scrapy] INFO: Enabled item pipelines: 
2014-02-25 10:34:44+0100 [wildberries] INFO: Spider opened 
2014-02-25 10:34:44+0100 [wildberries] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
2014-02-25 10:34:44+0100 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023 
2014-02-25 10:34:44+0100 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080 
2014-02-25 10:34:44+0100 [wildberries] DEBUG: Crawled (200) <GET http://www.wildberries.ru/> (referer: None) 
Hello World http://www.wildberries.ru/ 
[u'http://www.wildberries.ru/', u'http://www.wildberries.ru/catalog/1/women.aspx', u'http://www.wildberries.ru/catalog/2/men.aspx', u'http://www.wildberries.ru/catalog/3/child.aspx', u'http://www.wildberries.ru/catalog/413/women.aspx', u'http://www.wildberries.ru/catalog/492/women.aspx', u'http://www.wildberries.ru/catalog/524/women.aspx', u'http://www.wildberries.ru/catalog/2467/page.aspx', u'http://www.wildberries.ru/catalog/4263/page.aspx', u'http://www.wildberries.ru/catalog/181/women.aspx'] 
2014-02-25 10:34:44+0100 [wildberries] INFO: Closing spider (finished) 
2014-02-25 10:34:44+0100 [wildberries] INFO: Dumping Scrapy stats: 
    {'downloader/request_bytes': 217, 
    'downloader/request_count': 1, 
    'downloader/request_method_count/GET': 1, 
    'downloader/response_bytes': 15623, 
    'downloader/response_count': 1, 
    'downloader/response_status_count/200': 1, 
    'finish_reason': 'finished', 
    'finish_time': datetime.datetime(2014, 2, 25, 9, 34, 44, 519186), 
    'log_count/DEBUG': 3, 
    'log_count/INFO': 7, 
    'response_received_count': 1, 
    'scheduler/dequeued': 1, 
    'scheduler/dequeued/memory': 1, 
    'scheduler/enqueued': 1, 
    'scheduler/enqueued/memory': 1, 
    'start_time': datetime.datetime(2014, 2, 25, 9, 34, 44, 143133)} 
2014-02-25 10:34:44+0100 [wildberries] INFO: Spider closed (finished)

來源

2014-02-25 09:38:31

保羅你好，首先感謝所有。我很高興代碼是正確的。但是，我仍然無法讓蜘蛛跑步。要看看我的scrapy設置。 – KangLi

Scrapy start_urls不被解析

回答

相關問題