Scrapy與Scrapy蜘蛛

Question

我想刮website。我想要做的提取是文檔列表，作者姓名和日期。我觀看了一些scrapy蜘蛛視頻，並能夠找出3個shell腳本命令，從網站上提供所需的數據。該命令是

scrapy shell https://www.cato.org/research/34/commentary 

日期：

response.css('span.date-display-single::text').extract() 

作者：

response.css('p.text-sans::text').extract() 

在頁面的文檔鏈接

response.css('p.text-large.experts-more-h > a::text').extract() 

我試圖得到它通過Python，但都是徒勞的。由於有多個數據。

這裏是Python代碼：

import scrapy 
class CatoSpider(scrapy.Spider): 

    name = 'cato' 

    allowed_domains = ['cato.org'] 

    start_urls = ['https://www.cato.org/research/34/commentary'] 


def parse(self, response): 

    pass 

Answer 1

這應該工作。所有你需要的就是運行這個命令： scrapy runspider cato.py -o out.json 但我所看到的，有錯誤的鏈接，你將只從文字鏈接，而不是HREF

import scrapy 

class CatoItem(scrapy.Item): 
    date = scrapy.Field() 
    author = scrapy.Field() 
    links = scrapy.Field() 


class CatoSpider(scrapy.Spider): 

    name = 'cato' 

    allowed_domains = ['cato.org'] 

    start_urls = ['https://www.cato.org/research/34/commentary'] 


    def parse(self, response): 
     date = response.css('span.date-display-single::text').extract() 
     author = response.css('p.text-sans::text').extract() 
     links = response.css('p.text-large.experts-more-h > a::text').extract() 
     for d, a, l in zip(date, author, links): 
      item = CatoItem() 
      item['date'] = d 
      item['author'] = a 
      item['links'] = l 
      yield item