Scrapy單元測試

Question

我想在Scrapy（屏幕抓取/網絡爬蟲）中實現一些單元測試。由於一個項目是通過「scrapy crawl」命令運行的，我可以通過鼻子之類的東西運行它。由於scrapy建立在扭曲的頂端，我可以使用它的單元測試框架Trial？如果是這樣，怎麼樣？否則，我想獲得鼻子工作。

更新：

我一直在談論關於Scrapy-Users，我想我應該「建立測試碼的響應，然後調用與響應的方法和斷言[I]得到輸出中的預期項目/請求「。我似乎無法得到這個工作，雖然。

我可以建立一個單元測試測試類和測試：

• 創建一個響應對象
• 嘗試調用我的蜘蛛的解析方法與響應對象

但它最終會產生this回溯。任何洞察力爲什麼？

Answer 1

我這樣做的方式是創建假響應，這種方式可以離線測試解析函數。但是你通過使用真實的HTML來獲得真實的情況。

此方法的一個問題是您的本地HTML文件可能不會反映最新的狀態在線。所以如果HTML在線改變，你可能會有一個大錯誤，但你的測試用例仍然會通過。所以這可能不是測試這種方式的最佳方式。

我目前的工作流程是，無論何時出現錯誤，我都會發送一封電子郵件給管理員，並附有網址。然後，對於那個特定的錯誤，我創建了一個html文件，其中包含導致錯誤的內容。然後我爲它創建一個單元測試。

這是我使用從一個本地HTML文件創建用於測試樣品Scrapy HTTP響應代碼：

# scrapyproject/tests/responses/__init__.py 

import os 

from scrapy.http import Response, Request 

def fake_response_from_file(file_name, url=None): 
    """ 
    Create a Scrapy fake HTTP response from a HTML file 
    @param file_name: The relative filename from the responses directory, 
         but absolute paths are also accepted. 
    @param url: The URL of the response. 
    returns: A scrapy HTTP response which can be used for unittesting. 
    """ 
    if not url: 
     url = 'http://www.example.com' 

    request = Request(url=url) 
    if not file_name[0] == '/': 
     responses_dir = os.path.dirname(os.path.realpath(__file__)) 
     file_path = os.path.join(responses_dir, file_name) 
    else: 
     file_path = file_name 

    file_content = open(file_path, 'r').read() 

    response = Response(url=url, 
     request=request, 
     body=file_content) 
    response.encoding = 'utf-8' 
    return response 

樣本HTML文件位於scrapyproject /測試/響應/ osdir/sample.html

然後測試用例可以看看如下： 測試用例位置是scrapyproject /測試/ test_osdir.py

import unittest 
from scrapyproject.spiders import osdir_spider 
from responses import fake_response_from_file 

class OsdirSpiderTest(unittest.TestCase): 

    def setUp(self): 
     self.spider = osdir_spider.DirectorySpider() 

    def _test_item_results(self, results, expected_length): 
     count = 0 
     permalinks = set() 
     for item in results: 
      self.assertIsNotNone(item['content']) 
      self.assertIsNotNone(item['title']) 
     self.assertEqual(count, expected_length) 

    def test_parse(self): 
     results = self.spider.parse(fake_response_from_file('osdir/sample.html')) 
     self._test_item_results(results, 10) 

這是BASICA lly我如何測試我的解析方法，但它不僅用於解析方法。如果它變得更復雜，我建議看看Mox

Answer 2

您可以按照scrapy網站上的this代碼片段從腳本中運行它。然後，您可以對返回的項目進行任何形式的聲明。

Answer 3

新增加的Spider Contracts值得一試。它給你一個簡單的方法來添加測試，而不需要太多的代碼。

Answer 4

我用Betamax在真正的現場運行測試的第一時間，保持本地的HTTP響應以便下次測試運行超快後：

Betamax intercepts every request you make and attempts to find a matching request that has already been intercepted and recorded.

當你需要獲得最新版本的網站，只是刪除了什麼betamax已記錄並重新運行測試。

例子：

from scrapy import Spider, Request 
from scrapy.http import HtmlResponse 


class Example(Spider): 
    name = 'example' 

    url = 'http://doc.scrapy.org/en/latest/_static/selectors-sample1.html' 

    def start_requests(self): 
     yield Request(self.url, self.parse) 

    def parse(self, response): 
     for href in response.xpath('//a/@href').extract(): 
      yield {'image_href': href} 


# Test part 
from betamax import Betamax 
from betamax.fixtures.unittest import BetamaxTestCase 


with Betamax.configure() as config: 
    # where betamax will store cassettes (http responses): 
    config.cassette_library_dir = 'cassettes' 
    config.preserve_exact_body_bytes = True 


class TestExample(BetamaxTestCase): # superclass provides self.session 

    def test_parse(self): 
     example = Example() 

     # http response is recorded in a betamax cassette: 
     response = self.session.get(example.url) 

     # forge a scrapy response to test 
     scrapy_response = HtmlResponse(body=response.content, url=example.url) 

     result = example.parse(scrapy_response) 

     self.assertEqual({'image_href': u'image1.html'}, result.next()) 
     self.assertEqual({'image_href': u'image2.html'}, result.next()) 
     self.assertEqual({'image_href': u'image3.html'}, result.next()) 
     self.assertEqual({'image_href': u'image4.html'}, result.next()) 
     self.assertEqual({'image_href': u'image5.html'}, result.next()) 

     with self.assertRaises(StopIteration): 
      result.next() 

僅供參考，我發現在Betamax的2015年PYCON由於Ian Cordasco's talk。

Answer 5

我使用Twisted的trial來運行測試，類似於Scrapy自己的測試。它已經啓動了一個反應堆，所以我利用了CrawlerRunner，而不用擔心在測試中啓動和停止一個反應堆。

從check和parse Scrapy竊取一些想法命令我結束了以下基礎TestCase類抗活站點上運行的斷言：

from twisted.trial import unittest 

from scrapy.crawler import CrawlerRunner 
from scrapy.http import Request 
from scrapy.item import BaseItem 
from scrapy.utils.spider import iterate_spider_output 

class SpiderTestCase(unittest.TestCase): 
    def setUp(self): 
     self.runner = CrawlerRunner() 

    def make_test_class(self, cls, url): 
     """ 
     Make a class that proxies to the original class, 
     sets up a URL to be called, and gathers the items 
     and requests returned by the parse function. 
     """ 
     class TestSpider(cls): 
      # This is a once used class, so writing into 
      # the class variables is fine. The framework 
      # will instantiate it, not us. 
      items = [] 
      requests = [] 

      def start_requests(self): 
       req = super(TestSpider, self).make_requests_from_url(url) 
       req.meta["_callback"] = req.callback or self.parse 
       req.callback = self.collect_output 
       yield req 

      def collect_output(self, response): 
       try: 
        cb = response.request.meta["_callback"] 
        for x in iterate_spider_output(cb(response)): 
         if isinstance(x, (BaseItem, dict)): 
          self.items.append(x) 
         elif isinstance(x, Request): 
          self.requests.append(x) 
       except Exception as ex: 
        print("ERROR", "Could not execute callback: ",  ex) 
        raise ex 

       # Returning any requests here would make the  crawler follow them. 
       return None 

     return TestSpider 

實施例：

@defer.inlineCallbacks 
def test_foo(self): 
    tester = self.make_test_class(FooSpider, 'https://foo.com') 
    yield self.runner.crawl(tester) 
    self.assertEqual(len(tester.items), 1) 
    self.assertEqual(len(tester.requests), 2) 

或執行一個請求在設置和運行結果多重測試：

@defer.inlineCallbacks 
def setUp(self): 
    super(FooTestCase, self).setUp() 
    if FooTestCase.tester is None: 
     FooTestCase.tester = self.make_test_class(FooSpider, 'https://foo.com') 
     yield self.runner.crawl(self.tester) 

def test_foo(self): 
    self.assertEqual(len(self.tester.items), 1) 

Answer 6

我使用scrapy 1.3.0和功能：fake_response_from_file，引發錯誤：

response = Response(url=url, request=request, body=file_content) 

我得到：

raise AttributeError("Response content isn't text") 

的解決方案是使用TextResponse，而是和它的作品確定，例如：

response = TextResponse(url=url, request=request, body=file_content)  

非常感謝。

Answer 7

稍微簡單，由所選擇的答案取出def fake_response_from_file：

import unittest 
from spiders.my_spider import MySpider 
from scrapy.selector import Selector 


class TestParsers(unittest.TestCase): 


    def setUp(self): 
     self.spider = MySpider(limit=1) 
     self.html = Selector(text=open("some.htm", 'r').read()) 


    def test_some_parse(self): 
     expected = "some-text" 
     result = self.spider.some_parse(self.html) 
     self.assertEqual(result, expected) 


if __name__ == '__main__': 
    unittest.main()