如何抓取分類的網站

Question

我正在嘗試使用Scrapy編寫抓取工具來抓取分類類型（目標）網站並從目標網站上的鏈接獲取信息。關於Scrapy的教程只幫助我從目標URL獲取鏈接，但不能獲得我尋求的第二層數據收集。任何線索？

因此，例如，目標站點將是：

start_url = "http://newyork.craigslist.org/search/cta" 

和所有目標站點上我想去給每個上市，並獲得價格和賣家，地點，電話或電子郵件

鏈接

Answer 1

import scrapy 
from scrapy.contrib.spiders import CrawlSpider, Rule 
from scrapy.selector import Selector 
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor 
from urlparse import urljoin 


class CompItem(scrapy.Item): 
    name = scrapy.Field() 
    price = scrapy.Field() 
    location = scrapy.Field() 




class criticspider(CrawlSpider): 
    name = "craig" 
    allowed_domains = ["newyork.craigslist.org"] 
    start_urls = ["http://newyork.craigslist.org/search/cta"] 


    def parse(self, response): 
     sites = response.xpath('//div[@class="content"]') 
     items = [] 

     for site in sites: 
      item = CompItem() 
      item['name'] = site.xpath('.//p[@class="row"]/span[@class="txt"]/span[@class="pl"]/a/text()').extract(). 
      item['price'] = site.xpath('.//p[@class="row"]/span[@class="txt"]/span[@class="l2"]/span[@class="price"]/text()').extract() 
      item['location'] = site.xpath('.//p[@class="row"]/span[@class="txt"]/span[@class="l2"]/span[@class="pnr"]/small/text()').extract() 
      items.append(item) 
      return items