0
我正在製作一個4chan刮板從線程下載圖像。 一切工作正常，我能夠刮圖像鏈接。 圖像管道似乎也在工作，這是我得到的響應。找不到從Scrapy python框架下載的圖像
[email protected]:~/Work/ScrapyTests/FourChan/FourChan$ sudo scrapy crawl imageSpider
2016-01-23 13:45:31 [scrapy] INFO: Scrapy 1.0.4 started (bot: FourChan)
2016-01-23 13:45:31 [scrapy] INFO: Optional features available: ssl, http11
2016-01-23 13:45:31 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'FourChan.spiders', 'SPIDER_MODULES': ['FourChan.spiders'], 'BOT_NAME': 'FourChan'}
2016-01-23 13:45:31 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState
2016-01-23 13:45:32 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2016-01-23 13:45:32 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-01-23 13:45:32 [scrapy] INFO: Enabled item pipelines: ImagesPipeline
2016-01-23 13:45:32 [scrapy] INFO: Spider opened
2016-01-23 13:45:32 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-01-23 13:45:32 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-01-23 13:45:33 [scrapy] DEBUG: Crawled (200) <GET http://boards.4chan.org/a/thread/136492097/would-you-still-stay-with-you-waifu-if-she-was> (referer: None)
2016-01-23 13:45:33 [scrapy] DEBUG: File (uptodate): Downloaded image from <GET http://i.4cdn.org/a/1453536312441.jpg> referred in <None>
2016-01-23 13:45:33 [scrapy] DEBUG: Scraped from <200 http://boards.4chan.org/a/thread/136492097/would-you-still-stay-with-you-waifu-if-she-was>
{'fileName': u'1453536312441.jpg',
'image_urls': [u'http://i.4cdn.org/a/1453536312441.jpg']}
2016-01-23 13:45:33 [scrapy] INFO: Closing spider (finished)
2016-01-23 13:45:33 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 279,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 5639,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'file_count': 1,
'file_status_count/uptodate': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2016, 1, 23, 8, 15, 33, 468893),
'item_scraped_count': 1,
'log_count/DEBUG': 4,
'log_count/INFO': 7,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2016, 1, 23, 8, 15, 32, 44913)}
2016-01-23 13:45:33 [scrapy] INFO: Spider closed (finished)
這裏是圖像流水線從settings.py文件
ITEM_PIPELINES = {'scrapy.pipelines.images.ImagesPipeline': 1}
IMAGES_STORE = '/Work/ScrapyTests/FourChan/FourChan/downloads/'
設置過程下來之後，我去檢查，如果圖像被下載，在給定的目錄，但該文件夾爲空。 有人請看看。
這裏是主要參考蜘蛛代碼
import scrapy
import FourChan.items as items
class DmozSpider(scrapy.Spider):
name = "imageSpider"
allowed_domains = ["4chan.org"]
# thread = raw_input("Enter thread link: ")
start_urls = [
# thread
"http://boards.4chan.org/a/thread/136492097/would-you-still-stay-with-you-waifu-if-she-was",
]
def parse(self, response):
page = response
item = items.FourchanItem()
testCss = page.selector.css('.fileThumb::attr(href)').extract()
for test in testCss:
url = "http:"+test
# url = test[2:]
filename = url.split("/")[-1]
# print filename
item['fileName'] = filename
item['image_urls'] = [url] #This needs to be a list, in order for the image pipeline to work, can't be a string
yield item