Question

我在Python中使用scrapy，並想檢索位於另一個「擴展」元素後面的元素的內容。在檢查DOM樹時，直到第一次單擊父元素時纔會加載div標籤和文本本身。一旦父母被點擊，文本可以被重新隱藏，但至少會在DOM中。

示例網站是here。我正在尋找抽象文本（直到單擊「摘要」鏈接才加載）。

鬥志旺盛的命令是： response.xpath("//div[@class='previewBox abstract hidden']").extract()而是返回了一堆空的div像這樣： u'<div id="abs_S0740002015000179" class="previewBox abstract hidden"></div>'

如果我用這個：response.xpath("//div[@class='previewBox abstract']").extract()那麼它不返回任何東西。

Answer 1

您需要模擬額外的HTTP GET請求正在發送的abtract鏈接點擊。

這個想法是提取並請求「Abstract」鏈接的data-url屬性值。從"Scrapy Shell"

演示：

$ scrapy shell "http://www.sciencedirect.com/science?_ob=ArticleListURL&_method=list&_ArticleListID=-764831607&_sort=r&_st=13&view=c&md5=a41e9f25739feae932862575251c1e0d&searchtype=a" 
In [1]: url = response.xpath("//a[@data-type='abstract']/@data-url").extract()[0] 
In [2]: fetch(url) 
In [3]: print "".join(response.xpath("//div[@class='articleText']//text()").extract()) 
AbstractThe aim of the present study was to investigate the effect of lactic acid against Shiga toxin producing Escherichia coli (O157:H7 and non-O157 serogroups including O103, O111, O145 and O26) at different conditions. Soybean sprouts and spinach leaves inoculated with each serogroup of E. coli (∼7.00 + 1.00 log10 cfu/g) were treated with the lactic acid solutions at different concentrations (0% (control), 1.5%, 2.0%, or 2.5%) and at different temperatures (20, 40, or 50 °C) for 3 min. Results indicated that regardless of the treatment temperature, no significant reduction in the numbers of any serogroup occurred in the control group (0%) (p > 0.05). However, lactic acid at concentration of 1.5%, 2% and 2.5% was found to be effective against all organisms tested. There was no significant difference (p > 0.05) between E. coli O157:H7 and non-O157 STEC serogroups at any treatment group. The highest reductions (ca. 4.00 log10 cfu/g) of all serotypes in both produces were observed after immersing into 2.5% lactic acid at 50 °C. The results of this study showed that decontamination of fresh produces such as spinach and soybean sprout with lactic acid solutions prepared at mild temperatures (40 °C and 50 °C) might be an effective safety measure in preventing public health risks associated with these products contaminated with STEC. 

注意，這fetch()調用是使外殼其他請求一種特殊的方式。在您的Scrapy蜘蛛中，您需要yield或returnscrapy.http.Request()實例並解析callback中的結果。