Python的網頁抓取腳本，下面的教程和有問題

Question

繼在YouTube上的教程： Scraping Web Pages with Scrapy

這是老，爲Python 2.x和我學習版本3.x到目前爲止，我遇到了幾個我通過Google可以找到的問題。不過目前，我得到一個錯誤：

File "/usr/lib64/python3.5/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/home/skeer/PycharmProjects/scrape_craigslists/scrape_cl/scrape_cl/spiders/scrape.py", line 11, in parse xpath = scrapy.selector(response) TypeError: 'module' object is not callable

早些時候谷歌搜索，我發現引用到其他有這是由於非大寫字符，就好像「S」的選擇應該是資本。試了一下，並與scrapy.Selector模塊無法找到一個錯誤迎接。

這裏是我的代碼：

from scrapy.spider import Spider 
import scrapy.selector 


class MySpider(Spider): 
name = "craigslist" 
allowed_domains = ["craigslist.org"] 
start_urls = ["https://helena.craigslist.org/search/sad"] 

def parse(self, response): 
    xpath = scrapy.selector(response) 
    titles = xpath.select("//p") 
    for titles in titles: 
     title = xpath("/body/section/form/div/li/p[@class]()").extract()  
     link = 
xpath("/body/section/form/div/ul/li/a[@href]").extract() 
     print (title, link) 

Answer 1

scrapy.selector是包含選擇的模塊。嘗試

from scrapy.selector import Selector 

然而，這是因爲響應對象已經有selector interface and an xpath method是沒有必要的，所以你應該做的：

def parse(self, response): 
    xpath = response.xpath 
    titles = xpath("//p") 
    for titles in titles: 
     title = xpath("/body/section/form/div/li/p[@class]()").extract()  
     link = xpath("/body/section/form/div/ul/li/a[@href]").extract() 
     print (title, link) 

此外，您將需要一個非常好的代理的列表，如果你正計劃刮craigslist。他們迅速禁止ip，特別是爲了防止刮傷。

Answer 2

我會推薦學習與official docs，還有curated resources。

對於您的問題，檢查official docs for Scrapy Selectors：

from scrapy.selector import Selector 

class MySpider(Spider): 
... 
    def parse(self, response): 
     xpath = Selector(response) 
     ... 

Answer 3

更改函數的定義：

def parse(self, response): 
    xpath = scrapy.selector.Selector(response) 
    titles = xpath.select("//p") 
    for titles in titles: 
     title = xpath.xpath("/body/section/form/div/li/p[@class]()").extract() 
     link = xpath.xpath("/body/section/form/div/ul/li/a[@href]").extract() 
     print(title, link) 

注xpath("/body/section/form/div/li/p[@class]()") - >xpath.xpath("/body/section/form/div/li/p[@class]()")