網上刮刮香港電子立法

Question

我想摘一下香港立法的內容。但是，我無法訪問不可見的內容，除非我向下滾動頁面。

網站我訪問：https://www.elegislation.gov.hk/hk/cap211

from selenium import webdriver 
from selenium.webdriver.common.by import By 
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC 
from selenium.common.exceptions import TimeoutException 
from selenium.common.exceptions import ElementNotVisibleException 
from selenium.webdriver.common.action_chains import ActionChains 

def init_driver(profile): 
    driver = webdriver.Firefox(profile) 
    driver.wait = WebDriverWait(driver, 5) 
    return driver 

def convert2text2(webElement): 
    if webElement != []: 
     webElements = [] 
     for element in webElement: 
      e = element.text.encode('utf8') 
      webElements.append(e) 
    else: 
     webElements = ['NA'] 
    return webElements 

profile = webdriver.FirefoxProfile() 
driver = init_driver(profile) 
url = 'https://www.elegislation.gov.hk/hk/cap211' 
driver.get(url) 
driver.wait = WebDriverWait(driver, 5) 

content = driver.find_elements_by_xpath("//div[@class='hklm_content' or @class='hklm_leadIn' or @class='hklm_continued']") 
content = convert2text2(content) 

瞭解，從How can I scroll a web page using selenium webdriver in python?採取下面的代碼被用於滾動瀏覽器：

SCROLL_PAUSE_TIME = 0.5 

# Get scroll height 
last_height = driver.execute_script("return document.body.scrollHeight") 

while True: 
    # Scroll down to bottom 
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") 

    # Wait to load page 
    time.sleep(SCROLL_PAUSE_TIME) 

    # Calculate new scroll height and compare with last scroll height 
    new_height = driver.execute_script("return document.body.scrollHeight") 
    if new_height == last_height: 
     break 
    last_height = new_height 

但我不能想出如何指定內容窗口的滾動條並滾動到底部。

Answer 1

你只要把last_height在JavaScript代碼如下所示：

while True: 
    # Scroll down to 'last_height' 
    driver.execute_script("window.scrollTo(0, {});".format(last_height)) 

    # Wait to load page 
    time.sleep(SCROLL_PAUSE_TIME) 

    # Calculate new scroll height and compare with last scroll height 
    new_height = driver.execute_script("return document.body.scrollHeight;") 
    if new_height == last_height: 
     break 
    last_height = new_height 

的要對此是根本談不上把數據無硒的另一種方式。如果您查看網頁所做的調用（Chrome檢查器，網絡選項卡），您會看到每個新元素都使用小塊xml加載到網站中。

爲出發點，網址爲「https://www.elegislation.gov.hk/xml?skipHSC=true&LANGUAGE=E&BILINGUAL=&LEG_PROV_MASTER_ID=181740&QUERY=.&INDEX_CS=N&PUBLISHED=true」

的PROV_MASTER_ID參數將由1對於每個大塊，該網站負荷增加。

你可以抓住它，像這樣全部採用請求：

import requests 
url = 'https://www.elegislation.gov.hk/xml?skipHSC=true&LANGUAGE=E&BILINGUAL=&LEG_PROV_MASTER_ID={}&QUERY=.&INDEX_CS=N&PUBLISHED=true' 
starting_count = 181740 
stop_count = "" # integer - you need to figure out, when you got all you need 
count = starting_count 
while count <= stop_count: 
    response = requests.get(url.format(count)) 
    # parse the xml and grab the parts you need... 
    count +=1