Scrapy SGML鏈接提取與自定義標記和屬性

Question

我想知道爲什麼我的鏈接提取沒有得到任何鏈接。我正在擴展CrawlSpider並在鏈接提取器中設置tags和attrs屬性。我試圖從網頁上的選擇框中取消鏈接（http://www.realclearpolitics.com/epolls/latest_polls/president/），所以我正在尋找帶有值屬性的選項標籤。

結帳代碼在這裏：https://github.com/jtfairbank/RCP-Poll-Scraper/blob/master/polldata/spiders/pres.py

而這裏的根目錄：https://github.com/jtfairbank/RCP-Poll-Scraper

任何想法？謝謝！

Answer 1

你的鏈接提取沒有得到任何的聯繫，因爲你與tags=['option']尋找<option>標籤和有上this page沒有選擇或選項標籤。

嘗試restrict_xpaths='//td[@class="lp-race"]'：

>>> SgmlLinkExtractor(
...  allow=(r"epolls/2012/president/[a-z]{2}/[a-z]+_romney_vs_obama-[0-9]{4}\.html"), 
...  allow_domains=('realclearpolitics.com',), 
...  restrict_xpaths='//td[@class="lp-race"]', 
...).extract_links(response) 

[Link(url='http://www.realclearpolitics.com/epolls/2012/president/fl/florida_romney_vs_ob...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/oh/ohio_romney_vs_obama...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/va/virginia_romney_vs_o...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/ia/iowa_romney_vs_obama...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/wi/wisconsin_romney_vs_...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/or/oregon_romney_vs_oba...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/mo/missouri_romney_vs_o...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/ca/california_romney_vs...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/md/maryland_romney_vs_o...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/mi/michigan_romney_vs_o...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/co/colorado_romney_vs_o...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/mn/minnesota_romney_vs_...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/wa/washington_romney_vs...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/ct/connecticut_romney_v...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/nv/nevada_romney_vs_oba...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/mt/montana_romney_vs_ob...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/pa/pennsylvania_romney_...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/in/indiana_romney_vs_ob...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/ma/massachusetts_romney...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/ga/georgia_romney_vs_ob...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/id/idaho_romney_vs_obam...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/az/arizona_romney_vs_ob...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/me/maine_romney_vs_obam...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/la/louisiana_romney_vs_...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/hi/hawaii_romney_vs_oba...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/tx/texas_romney_vs_obam...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/ar/arkansas_romney_vs_o...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/ne/nebraska_romney_vs_o...',