Scrapy SgmlLinkExtractor添加任意網址

Question

如何向SgmlLinkExtractor添加網址？也就是說，我如何添加一個任意的URL來運行回調？

爲了詳細描述，使用dirbot作爲一個例子：https://github.com/scrapy/dirbot/blob/master/dirbot/spiders/googledir.py

parse_category只訪問相匹配的SgmlLinkExtractor SgmlLinkExtractor（一切允許='directory.google.com/[AZ] [A-ZA-Z_ /] + $'）

Answer 1

使用BaseSpider代替CrawlSpider，然後設置添加到start_requests或start_urls []

class MySpider(BaseSpider): 
    name = "myspider" 

    def start_requests(self): 
     return [Request("https://www.example.com", 
      callback=self.parse)] 

    def parse(self, response): 
     hxs = HtmlXPathSelector(response) 
     ... 

Answer 2

類ThemenHubSpider（CrawlSpider）：

name = 'themenHub' 
allowed_domains = ['themen.t-online.de'] 
start_urls = ["http://themen.t-online.de/themen-a-z/a"] 
rules = [Rule(SgmlLinkExtractor(allow=['id_\d+']), 'parse_news')]