僅在scrapy中返回特定網址

Question

我正在使用scrapy從網站刮取網址。目前它返回所有網址，但我希望它只返回包含單詞「下載」的網址。我怎樣才能做到這一點？

from scrapy.selector import HtmlXPathSelector 
from scrapy.spider import BaseSpider 
from scrapy.http import Request 
import scrapy 

DOMAIN = 'somedomain.com' 
URL = 'http://' +str(DOMAIN) 

class MySpider(scrapy.Spider): 
    name = DOMAIN 
    allowed_domains = [DOMAIN] 
    start_urls = [ 
     URL 
    ] 

    def parse(self, response): 
     hxs = HtmlXPathSelector(response) 
     for url in hxs.select('//a/@href').extract(): 
      if not (url.startswith('http://') or url.startswith('https://')): 
       url= URL + url 
      print url 
      yield Request(url, callback=self.parse) 

編輯：

我實現了以下建議。它仍然會拋出一些錯誤，但至少只返回包含下載的鏈接。

from scrapy.selector import HtmlXPathSelector 
from scrapy.spider import BaseSpider 
from scrapy.http import Request 
import scrapy 
from scrapy.linkextractors import LinkExtractor 


DOMAIN = 'somedomain.com' 
URL = 'http://' +str(DOMAIN) 

class MySpider(scrapy.Spider): 
    name = DOMAIN 
    allowed_domains = [DOMAIN] 
    start_urls = [ 
     URL 
    ] 

# First parse returns all the links of the website and feeds them to parse2 

    def parse(self, response): 
     hxs = HtmlXPathSelector(response) 
     for url in hxs.select('//a/@href').extract(): 
      if not (url.startswith('http://') or url.startswith('https://')): 
       url= URL + url 
      yield Request(url, callback=self.parse2) 

# Second parse selects only the links that contains download 

    def parse2(self, response): 
     le = LinkExtractor(allow=("download")) 
     for link in le.extract_links(response): 
       yield Request(url=link.url, callback=self.parse2) 
       print link.url 

Answer 1

一個更Python和乾淨的解決方案，是使用LinkExtractor：

from scrapy.linkextractors import LinkExtractor 

... 

le = LinkExtractor(deny="download") 
for link in le.extract_links(response): 
    yield Request(url=link.url, callback=self.parse) 

Answer 2

您試圖檢查一個字符串中是否存在子字符串。

例子：

string = 'this is a simple string' 

'simple' in string 
True 

'zimple' in string 
False 

---

所以，你只需要添加一個if語句，如：

：

if 'download' in url:

行後210

即：

for url in hxs.select('//a/@href').extract(): 
    if 'download' in url: 
     if not (url.startswith('http://') or url.startswith('https://')): 
      url = URL + url 
     print url 
     yield Request(url, callback=self.parse) 

因此，如果條件'download' in url返回True如果鏈接開始與http://代碼將只檢查。