Python的phantomjs加載網頁不正確

Question

我有一個問題，從這個鏈接

http://www.bursamalaysia.com/market/listed-companies/company-announcements/#/?category=FA&sub_category=FA1&alphabetical=All&company=5250

提取帶給我從這個鏈接，而不是它本身的主要頁面的數據。 http://www.bursamalaysia.com/market/listed-companies/company-announcements/#/?category=all

任何想法爲什麼發生這種情況？ 我使用PhantomJS硒和美麗的湯來幫助我。

# The standard library modules 
import os 
import sys 
import re 
import sqlite3 
import locale 
# The wget module 
import wget 
import time 
import calendar 
from datetime import datetime 
# The BeautifulSoup module 
from bs4 import BeautifulSoup 

# The selenium module 
from selenium import webdriver 
from selenium.webdriver.common.keys import Keys 
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC 
from selenium.webdriver.common.by import By 


def getURLS(url): 
    driver = webdriver.PhantomJS(service_args=['--ignore-ssl-errors=true']) 
    driver.get(url) # load the web page 
    src = driver.page_source 
    #Get text and split it 
    soup = BeautifulSoup(src, 'html5lib') 

    print soup 

link ='http://www.bursamalaysia.com/market/listed-companies/company-announcements/#/?category=FA&sub_category=FA1&alphabetical=All&company=5250' 
getURLS(link) 

亞歷克斯Lucaci解決方案

def getURLS(url): 
    driver = webdriver.PhantomJS(service_args=['--ignore-ssl-errors=true']) 
    driver.get(url) # load the web page 
    src = driver.page_source 
    category_select = Select(driver.find_element_by_xpath('//*[@id="bm_announcement_types"]')) 
    category_select.select_by_visible_text("Financial Results") 
    category_select2 = Select(driver.find_element_by_xpath('//*[@id="bm_sub_announcement_types"]')) 
    category_select2.select_by_visible_text("Financial Results") 
    category_select3 = Select(driver.find_element_by_xpath('//*[@id="bm_company_list"]')) 
    category_select3.select_by_visible_text("7-ELEVEN MALAYSIA HOLDINGS BERHAD (5250)") 
    driver.find_element_by_xpath('//*[@id="bm_company_announcements_search_form"]/input[1]').click() 
    src = driver.page_source 
    soup = BeautifulSoup(src, 'html5lib') 
    link="http://www.bursamalaysia.com/market/listed-companies/company-announcements/#/?category=all" 
    getURLS(link) 

Answer 1

當你保存頁面沒有完全加載您提交後的源，以便嘗試抓取的網頁源之前等待一對夫婦秒：

def getURLS(url): 
driver = webdriver.PhantomJS(service_args=['--ignore-ssl-errors=true']) 
driver.get(url) # load the web page 
time.sleep(5)# waiting for 5 seconds before fetching the source 
src = driver.page_source 
#Get text and split it 
soup = BeautifulSoup(src, 'html5lib') 

print soup 

要執行下拉菜單中選擇你導入的Select類如下：from selenium.webdriver.support.ui import Select，然後你必須選擇這樣的下拉元素：

category_select = Select(driver.find_element_by_xpath('//*[@id="bm_announcement_types"]')) 
category_select.select_by_visible_text('Financial Results') 

在我的例子我已經做到了爲 - 分類 - 下拉列表，按照每個類別的具體步驟。 請注意，通過xpath選擇下拉列表是最好的方法，您可以通過使用谷歌瀏覽器 - >點擊元素 - >檢查 - >右鍵點擊<select>出現的右側菜單 - >複製 - >複製Xpath

當您選擇所有元素時，您必須單擊提交併等待幾秒鐘才能加載，之後您將獲取源代碼。

讓我知道如果我的答案幫助你。