Scrapy按照鏈接並收集電子郵件

Question

我需要使用Scrapy保存電子郵件的幫助。電子郵件應該收集的.csv文件中的行是空白的。任何幫助非常感謝。下面是代碼：

# -*- coding: utf-8 -*- 
import scrapy 


# item class included here 
class DmozItem(scrapy.Item): 
    # define the fields for your item here like: 
    link = scrapy.Field() 
    attr = scrapy.Field() 


class DmozSpider(scrapy.Spider): 
    name = "dmoz" 
    allowed_domains = ["craigslist.org"] 
    start_urls = [ 
    "http://chicago.craigslist.org/search/vgm?" 
    ] 

    BASE_URL = 'http://chicago.craigslist.org/' 

    def parse(self, response): 
     links = response.xpath('//a[@class="hdrlnk"]/@href').extract() 
     for link in links: 
      absolute_url = self.BASE_URL + link 
      yield scrapy.Request(absolute_url, callback=self.parse_attr) 

    def parse_attr(self, response): 
     item = DmozItem() 
     item["link"] = response.url 
     item["attr"] = "".join(response.xpath("//div[@class='anonemail']//text()").extract()) 
     return item 

Answer 1

爲了讓看到Craiglist上的商品頁面上的電子郵件，該人會點擊「回覆」按鈕，從而啓動一個新的請求「回覆/卡/ VGM /」 URL。這是你需要通過發行新Request和回調解析結果Scrapy模擬的東西：

# -*- coding: utf-8 -*- 
import re 
import scrapy 


# item class included here 
class DmozItem(scrapy.Item): 
    # define the fields for your item here like: 
    link = scrapy.Field() 
    attr = scrapy.Field() 


class DmozSpider(scrapy.Spider): 
    name = "dmoz" 
    allowed_domains = ["craigslist.org"] 
    start_urls = [ 
    "http://chicago.craigslist.org/search/vgm?" 
    ] 

    BASE_URL = 'http://chicago.craigslist.org/' 

    def parse(self, response): 
     links = response.xpath('//a[@class="hdrlnk"]/@href').extract() 
     for link in links: 
      absolute_url = self.BASE_URL + link 
      yield scrapy.Request(absolute_url, callback=self.parse_attr) 

    def parse_attr(self, response): 
     match = re.search(r"(\w+)\.html", response.url) 
     if match: 
      item_id = match.group(1) 
      url = self.BASE_URL + "reply/chi/vgm/" + item_id 

      item = DmozItem() 
      item["link"] = response.url 

      return scrapy.Request(url, meta={'item': item}, callback=self.parse_contact) 

    def parse_contact(self, response): 
     item = response.meta['item'] 
     item["attr"] = "".join(response.xpath("//div[@class='anonemail']//text()").extract()) 
     return item