BS未能得到部分ID硒檢索

Question

頁

我有一個問題，下面的代碼

import re 
from lxml import html 
from bs4 import BeautifulSoup as BS 
from selenium import webdriver 
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary 
import requests 
import sys 
import datetime 

print ('start!') 
print(datetime.datetime.now()) 

list_file = 'list2.csv' 
#This should be the regular input list 

url_list=["http://www.genecards.org/cgi-bin/carddisp.pl?gene=ENO3&keywords=ENO3"] 
#This is an example input instead 

binary = FirefoxBinary('C:/Program Files (x86)/Mozilla Firefox/firefox.exe') 
#Read somewhere it could be a variable useful to supply but anyway, the program fails randomly at time with [WinError 6] Invalid Descriptor while having nothing different from when it is able to at least get the webpage; even when not able to perform further operation. 

for page in url_list: 
    print(page) 
    browser = webdriver.Firefox(firefox_binary=binary) 
    #I tried this too to solve the [WinError 6] but it is not working 
    browser.get(page) 
    print ("TEST BEGINS") 
    soup=BS(browser.page_source,"lxml") 
    soup=soup.find("summaries") 
    # This fails here. It finds nothing, while there is a section id termed summaries. soup.find_all("p") works but i don't want all the p's outside of summaries 
    print(soup) #It prints "None" indeed. 
    print ("TEST ENDS") 

我正源代碼包含「摘要」。首先出現的是

<li> <a href="#summaries" ng-click="scrollTo('summaries')">Summaries</a></li> 

再有就是

<section id="summaries" data-ga-label="Summaries" data-section="Summaries"> 

如這裏（Webscraping in python: BS, selenium, and None error）由@alexce建議，我試圖

summary = soup.find('section', attrs={'id':'summaries'}) 

（編輯：建議是_summaries但我沒有測試摘要也是）

但它也不起作用。 所以我的問題是： 爲什麼BS找不到摘要，並且爲什麼硒不斷打破，當我使用的腳本連續過多（重新啓動控制檯的作品，而另一方面，但這是乏味），或一個包含四個以上實例的列表？ 感謝

Answer 1

此：

summary = soup.find('section', attrs={'id':'_summaries'})

搜索元素section有屬性id設置爲_summaries：

<section id="_summary" />

有這些屬性沒有元素一世在頁面中。
你想要的那個可能是<section id="summaries" data-ga-label="Summaries" data-section="Summaries">。並且可以匹配：

results = soup.find('section', id_='summaries') 

另外，關於爲什麼使用硒的附註。如果您不轉發cookie，該頁面將返回錯誤。所以爲了使用請求，你需要發送cookies。

我全碼：

1 from __future__ import unicode_literals 
    2 
    3 import re 
    4 import requests 
    5 from bs4 import BeautifulSoup as BS 
    6 
    7 
    8 data = requests.get(
    9  'http://www.genecards.org/cgi-bin/carddisp.pl?gene=ENO3&keywords=ENO3', 
10  cookies={ 
11   'nlbi_146342': '+fhjaf6NSntlOWmvFHlFeAAAAAAwHqv5tJUsy3kqgNQOt77C', 
12   'visid_incap_146342': 'tEumui9aQoue4yMuu9tuUcly6VYAAAAAQUIPAAAAAABcQsCGxBC1gj0OdNFoMEx+', 
13   'incap_ses_189_146342': 'bNY8PNPZJzroIFLs6nefAspy6VYAAAAAYlWrxz2UrYFlrqgcQY9AuQ==' 
14  }).content 
15 
16 soup=BS(data) 
17 results=soup.find_all(string=re.compile('summary', re.I)) 
18 print(results) 
19 summary_re = re.compile('summary', re.I) 
20 results = soup.find('section', id_='summaries') 
21 print(results) 

Answer 2

元素，也許是尚未在頁面上。我會等元素與BS解析頁面源之前：

from selenium import webdriver 
from selenium.webdriver.common.keys import Keys 
from selenium.webdriver.common.by import By 
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC 

driver = webdriver.Firefox() 
driver.get("http://www.genecards.org/cgi-bin/carddisp.pl?gene=ENO3&keywords=ENO3") 
wait = WebDriverWait(driver, 10) 
wait.until(EC.visibility_of_element_located((By.ID, "summaries"))) 
soup = BS(driver.page_source,"lxml") 

我注意到，你永遠不會調用driver.quit（），這可能是你的最新問題的原因。 所以一定要調用它或嘗試重用相同的會話。

，並使其更加穩定和高性能的話，我會因爲拉扯和解析頁面的源代碼是昂貴的，因爲玉米粥與硒API工作成爲可能。