如何將2個蜘蛛的所有結果與scrapy放在一個XML中？

Question

我用Scrapy製作了2個蜘蛛，我需要將它們放在一個腳本中，並將所有結果放在一個XML中。

以下是一些管理2個蜘蛛的方法，但我無法將結果放在一個XML中。

http://doc.scrapy.org/en/latest/topics/practices.html

有什麼辦法，推出1個腳本2個蜘蛛和收集在一個文件中所有的結果？

Answer 1

在你scrapy項目中創建一個名爲script.py添加蜘蛛文件code.Assuming名 以下行Python腳本是spider_one.py和spider_two.py和你的蜘蛛分別SpiderOne和SpiderTwo。 所以在你script.py中添加。

from spiders.spider_one import SpiderOne 
from spiders.spider_two import SpiderTwo 

# scrapy api 
from scrapy import signals, log 
from twisted.internet import reactor 
from scrapy.crawler import Crawler 
from scrapy.settings import Settings 


file = "your_file.json" #your results 
TO_CRAWL = [SpiderOne,SpiderTwo] 


# list of crawlers that are running 
RUNNING_CRAWLERS = [] 

def spider_closing(spider): 
    """Activates on spider closed signal""" 
    log.msg("Spider closed: %s" % spider, level=log.INFO) 
    RUNNING_CRAWLERS.remove(spider) 
    if not RUNNING_CRAWLERS: 
     reactor.stop() 

log.start(loglevel=log.DEBUG) 
for spider in TO_CRAWL: 
    settings = Settings() 
    settings.set("USER_AGENT", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36") 
    settings.set("FEED_FORMAT",'json') 
    settings.set("FEED_URI",file) 
    # settings.set("ITEM_PIPELINES",{ 'pipelines.CustomPipeline': 300}) 
    settings.set("DOWNLOAD_DELAY",1) 
    crawler = Crawler(settings) 
    crawler_obj = spider() 
    RUNNING_CRAWLERS.append(crawler_obj) 

    # stop reactor when spider closes 
    crawler.signals.connect(spider_closing, signal=signals.spider_closed) 
    crawler.configure() 
    crawler.crawl(crawler_obj) 
    crawler.start() 

# blocks process so always keep as the last statement 
reactor.run() 

該示例適用於json，但也可以適用於xml。