在Scrapy中獲得http.response對象的最簡單方法

Question

我是Scrapy的新手，我試圖將「獲取網頁內容」轉換爲響應對象（如果我正確理解的話）。

我跟隨http://doc.scrapy.org/en/latest/topics/selectors.html, 然而，它在scrapy外殼工作。我想直接在Python代碼中工作。

我寫的代碼報廢http://doc.scrapy.org/en/latest/_static/selectors-sample1.html

import scrapy 
from scrapy.http import HtmlResponse 
URL = 'http://doc.scrapy.org/en/latest/_static/selectors-sample1.html' 
response = HtmlResponse(url=URL)  
print response.selector.xpath('//title/text()') 

和輸出

>> [] 

爲什麼我不能獲得標題正確的值？看起來HtmlResponse（）不是從網上下載數據......爲什麼？我該如何修復！

非常感謝！

帽

Answer 1

你的陳述

response = HtmlResponse(url=URL) 

只有建立a "local scope" HtmlResponse object，一個空體。它不會下載任何東西，尤其是不在http://doc.scrapy.org/en/latest/_static/selectors-sample1.html處的資源。在Scrapy中，您通常不會自己創建HtmlResponse對象，您可以讓Scrapy框架爲您構建它們，當它完成處理您提供的Request實例時，例如， Request(url='http://doc.scrapy.org/en/latest/_static/selectors-sample1.html')

如果你正在嘗試Scrapy，我建議你scrapy shell玩：使用fetch('http://someurl')交互式shell裏面，你可以觸發下載（並獲得「真正的」 Response對象一起工作）：

$ scrapy shell 
2016-06-14 10:59:31 [scrapy] INFO: Scrapy 1.1.0 started (bot: scrapybot) 
(...) 
[s] Available Scrapy objects: 
[s] crawler <scrapy.crawler.Crawler object at 0x7f1a6591d588> 
[s] item  {} 
[s] settings <scrapy.settings.Settings object at 0x7f1a6ce290f0> 
[s] Useful shortcuts: 
[s] shelp()   Shell help (print this help) 
[s] fetch(req_or_url) Fetch request (or URL) and update local objects 
[s] view(response) View response in a browser 
>>> fetch('http://doc.scrapy.org/en/latest/_static/selectors-sample1.html') 
2016-06-14 10:59:51 [scrapy] INFO: Spider opened 
2016-06-14 10:59:51 [scrapy] DEBUG: Crawled (200) <GET http://doc.scrapy.org/en/latest/_static/selectors-sample1.html> (referer: None) 
>>> response.xpath('//title/text()').extract() 
['Example website'] 

外殼之外，實際下載數據，您需要：

• 子scrapy.Spider，
• 定義網址，在那裏乞討從，
• 和編寫回調方法下載對下載的數據進行工作，包裹在裏面Response對象得到傳遞給他們

一個很簡單的例子（在調用，比如文件，test.py：

import scrapy 


class TestSpider(scrapy.Spider): 

    name = 'testspider' 

    # start_urls is special and internally it builds Request objects for each of the URLs listed 
    start_urls = ['http://doc.scrapy.org/en/latest/_static/selectors-sample1.html'] 

    def parse(self, response): 
     yield { 
      'title': response.xpath('//h1/text()').extract_first() 
     } 

然後你需要運行蜘蛛。Scrapy具有運行單個文件的蜘蛛命令：

$ scrapy runspider test.py 

，你會得到這個在控制檯：

2016-06-14 10:48:05 [scrapy] INFO: Scrapy 1.1.0 started (bot: scrapybot) 
2016-06-14 10:48:05 [scrapy] INFO: Overridden settings: {} 
2016-06-14 10:48:06 [scrapy] INFO: Enabled extensions: 
['scrapy.extensions.logstats.LogStats', 'scrapy.extensions.corestats.CoreStats'] 
2016-06-14 10:48:06 [scrapy] INFO: Enabled downloader middlewares: 
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 
'scrapy.downloadermiddlewares.retry.RetryMiddleware', 
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 
'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware', 
'scrapy.downloadermiddlewares.stats.DownloaderStats'] 
2016-06-14 10:48:06 [scrapy] INFO: Enabled spider middlewares: 
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 
'scrapy.spidermiddlewares.referer.RefererMiddleware', 
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 
'scrapy.spidermiddlewares.depth.DepthMiddleware'] 
2016-06-14 10:48:06 [scrapy] INFO: Enabled item pipelines: 
[] 
2016-06-14 10:48:06 [scrapy] INFO: Spider opened 
2016-06-14 10:48:06 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
2016-06-14 10:48:06 [scrapy] DEBUG: Crawled (200) <GET http://doc.scrapy.org/en/latest/_static/selectors-sample1.html> (referer: None) 
2016-06-14 10:48:06 [scrapy] DEBUG: Scraped from <200 http://doc.scrapy.org/en/latest/_static/selectors-sample1.html> 
{'title': 'Example website'} 
2016-06-14 10:48:06 [scrapy] INFO: Closing spider (finished) 
2016-06-14 10:48:06 [scrapy] INFO: Dumping Scrapy stats: 
{'downloader/request_bytes': 252, 
'downloader/request_count': 1, 
'downloader/request_method_count/GET': 1, 
'downloader/response_bytes': 501, 
'downloader/response_count': 1, 
'downloader/response_status_count/200': 1, 
'finish_reason': 'finished', 
'finish_time': datetime.datetime(2016, 6, 14, 8, 48, 6, 564591), 
'item_scraped_count': 1, 
'log_count/DEBUG': 2, 
'log_count/INFO': 7, 
'response_received_count': 1, 
'scheduler/dequeued': 1, 
'scheduler/dequeued/memory': 1, 
'scheduler/enqueued': 1, 
'scheduler/enqueued/memory': 1, 
'start_time': datetime.datetime(2016, 6, 14, 8, 48, 6, 85693)} 
2016-06-14 10:48:06 [scrapy] INFO: Spider closed (finished) 

如果你真的想選擇玩，實際上並沒有下載任何網絡數據，假設你擁有的數據已經在本地（從view-source:在瀏覽器中複製例子），你可以這樣做，但你需要提供身體：

>>> response = HtmlResponse(url=URL, body=''' 
... <!DOCTYPE html> 
... <html> 
... <head> 
... </head> 
... <body> 
...  <h1>Herman Melville - Moby-Dick</h1> 
... 
...  <div> 
...   <p> 
...   Availing himself of the mild, summer-cool weather that now reigned in these latitudes, ... them a care-killing competency. 
...   </p> 
...  </div> 
... </body> 
... </html>''', encoding='utf8') 
>>> response.xpath('//h1') 
[<Selector xpath='//h1' data='<h1>Herman Melville - Moby-Dick</h1>'>] 
>>> response.xpath('//h1').extract() 
['<h1>Herman Melville - Moby-Dick</h1>'] 
>>>