Split scrapy的大型CSV文件

Question

是否可以將scrapy寫入每個文件不超過5000行的CSV文件？我怎樣才能給它一個自定義的命名方案？我應該修改CsvItemExporter？

Answer 1

您使用的是Linux嗎？

split命令對於這種情況非常有用。

split -l 5000 -d --additional-suffix .csv items.csv items- 

查看split --help的選項。

Answer 2

嘗試這條管道：

# -*- coding: utf-8 -*- 

# Define your item pipelines here 
# 
# Don't forget to add your pipeline to the ITEM_PIPELINES setting 
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html 

from scrapy.exporters import CsvItemExporter 

import datetime 

class MyPipeline(object): 

    def __init__(self, stats): 
     self.stats = stats 
     self.base_filename = "result/amazon_{}.csv" 
     self.next_split = self.split_limit = 50000 # assuming you want to split 50000 items/csv 
     self.create_exporter() 

    @classmethod 
    def from_crawler(cls, crawler): 
     return cls(crawler.stats) 

    def create_exporter(self): 
     now = datetime.datetime.now() 
     datetime_stamp = now.strftime("%Y%m%d%H%M") 
     self.file = open(self.base_filename.format(datetime_stamp),'w+b') 
     self.exporter = CsvItemExporter(self.file) 
     self.exporter.start_exporting()  

    def process_item(self, item, spider): 
     if (self.stats.get_stats()['item_scraped_count'] >= self.next_split): 
      self.next_split += self.split_limit 
      self.exporter.finish_exporting() 
      self.file.close() 
      self.create_exporter 
     self.exporter.export_item(item) 
     return item 

不要忘了管道添加到您的設置：

ITEM_PIPELINES = { 
    'myproject.pipelines.MyPipeline': 300, 
}