用BeautifulSoup和Python刮取Javascript網頁

我想從網站http://www.jobs.ch刮取內容。結果應該是一個腳本，我可以指定一個工作條件，例如業務分析師，並獲得標題中所有的職位。我假設我應該使用多步驟方法，首先收集每個頁面的所有匹配鏈接，存儲它們，然後提取作業說明。用BeautifulSoup和Python刮取Javascript網頁

這是實現這一目標的可能方法嗎？或者我還需要使用硒，因爲該網站在react.js中？

這裏我的劇本的開頭：

from bs4 import BeautifulSoup 
import urllib2 

jobsFile = urllib2.urlopen("http://www.jobs.ch/en/vacancies/?term=business+analyst") 
jobsHtml = jobsFile.read() 
jobsFile.close() 

soup = BeautifulSoup(jobsHtml) 
jobsAll = soup.find_all("a") 
for links in soup.find_all('a'): 
    print (links.get('href'))

從控制檯輸出：

python jobplatform.py 
/Library/Python/2.7/site-packages/bs4/__init__.py:181: UserWarning: 
No parser was explicitly specified, so I'm using the best available 
HTML parser for this system ("lxml"). This usually isn't a problem, 
but if you run this code on another system, or in a different virtual 
environment, it may use a different parser and behave differently. 

The code that caused this warning is on line 8 of the file 
jobplatform.py. To get rid of this warning, change code that looks 
like this: 

BeautifulSoup([your markup]) 

to this: 

BeautifulSoup([your markup], "lxml") 

markup_type=markup_type)) 
None 
/en/ 
/en/login/ 
/en/register/ 
/en/vacancies/ 
/en/companies/ 
http://www.jobs.ch/en/sucheBerater.php 
http://www.jobs.ch/en/tipps 
http://www.jobs.ch/en/ecom/ 
/de/stellenangebote/?term=business 
/fr/offres-emplois/?term=business 
/en/vacancies/?term=business 
/en/vacancies/ 
None 
None 
None 
None 
/en/vacancies/?page=1&term=business&web-results=1 
None 
None 
/en/companies/79912-bayer-business-services-gmbh/ 
/en/vacancies/detail/7376115/?source=vacancy_search 
/en/companies/79912-bayer-business-services-gmbh/ 
/en/companies/48196-kotra-korea-business-center/ 
/en/vacancies/detail/7397077/?source=vacancy_search 
/en/companies/48196-kotra-korea-business-center/ 
/en/companies/66172-diwisa-distillerie-willisau-sa/ 
/en/vacancies/detail/7363589/?source=vacancy_search 
/en/companies/66172-diwisa-distillerie-willisau-sa/ 
/en/companies/2859-paul-scherrer-institut/ 
/en/vacancies/detail/7359642/?source=vacancy_search 
/en/companies/2859-paul-scherrer-institut/ 
/en/companies/49314-pit-offices-gmbh/ 
/en/vacancies/detail/7344672/?source=vacancy_search 
/en/companies/49314-pit-offices-gmbh/ 
/en/companies/27786-zuehlke-engineering-ag/ 
/en/vacancies/detail/7176356/?source=vacancy_search 
/en/companies/27786-zuehlke-engineering-ag/ 
/en/companies/1802-six-payment-services-ag/ 
/en/vacancies/detail/7396870/?source=vacancy_search 
/en/companies/1802-six-payment-services-ag/ 
/en/companies/49420-mettler-toledo-gruppe/ 
/en/vacancies/detail/7384998/?source=vacancy_search 
/en/companies/49420-mettler-toledo-gruppe/ 
/en/companies/16414-partners-group/ 
/en/vacancies/detail/7279253/?source=vacancy_search 
/en/companies/16414-partners-group/ 
/en/companies/4005-johnson-johnson/ 
/en/vacancies/detail/7397184/?source=vacancy_search 
/en/companies/4005-johnson-johnson/ 
/en/companies/44340-amgen/ 
/en/vacancies/detail/7359993/?source=vacancy_search 
/en/companies/44340-amgen/ 
/en/companies/1802-six-payment-services-ag/ 
/en/vacancies/detail/7357631/?source=vacancy_search 
/en/companies/1802-six-payment-services-ag/ 
/en/companies/16649-fritschi-unternehmensberatung-gmbh/ 
/en/vacancies/detail/7369054/?source=vacancy_search 
/en/companies/16649-fritschi-unternehmensberatung-gmbh/ 
/en/companies/19002-hays-schweiz-ag/ 
/en/vacancies/detail/7389632/?source=vacancy_search 
/en/companies/19002-hays-schweiz-ag/ 
/en/companies/5977-canon-schweiz-ag/ 
/en/vacancies/detail/7236919/?source=vacancy_search 
/en/companies/5977-canon-schweiz-ag/ 
/en/companies/40039-vorwerk-international-strecker-co/ 
/en/vacancies/detail/7374142/?source=vacancy_search 
/en/companies/40039-vorwerk-international-strecker-co/ 
/en/companies/2263-zuercher-kantonalbank/ 
/en/vacancies/detail/7299359/?source=vacancy_search 
/en/companies/2263-zuercher-kantonalbank/ 
/en/companies/10673-accenture/ 
/en/vacancies/detail/6664788/?source=vacancy_search 
/en/companies/10673-accenture/ 
/en/companies/38308-addexpert-gmbh/ 
/en/vacancies/detail/7386047/?source=vacancy_search 
/en/companies/38308-addexpert-gmbh/ 
/en/companies/1802-six-swiss-exchange-ag/ 
/en/vacancies/detail/7357633/?source=vacancy_search 
/en/companies/1802-six-swiss-exchange-ag/ 
/en/vacancies/?page=1&term=business 
/en/vacancies/?page=2&term=business 
/en/vacancies/?page=3&term=business 
/en/vacancies/?page=4&term=business 
/en/vacancies/?page=5&term=business 
/en/vacancies/?page=6&term=business 
/en/vacancies/?page=124&term=business 
/en/vacancies/?page=2&term=business 
None 
http://jobcloud.ch/c/en/about/ 
http://jobcloud.ch/c/en/about/ 
http://jobcloud.ch/c/en/about/team/ 
http://jobcloud.ch/c/en/we-are-jobcloud/ 
None 
http://www.jobs.ch/en/newest.php 
http://www.jobs.ch/en/info.php?info=agb 
http://www.jobs.ch/en/info.php?info=pp 
None 
http://jobcloud.ch/c/en/products/international-recruiting/ 
/en/ 
http://www.jobs.ch/en/sitemap.php 
http://jobcloud.ch/c/en/about/contact/ 
http://jobcloud.ch/ 
http://www.facebook.com/jobs.ch 
http://twitter.com/jobs_ch 
http://www.xing.com/company/jobcloudag 
http://www.youtube.com/jobspunktch 
http://plus.google.com/113239437813300663024/ 
http://www.flickr.com/photos/jobsag

來源

2017-02-22 Stefan Badertscher

[網絡刮JavaScript頁面的可能的複製與Python]（http://stackoverflow.com/questions/8049520/web-scraping-javascript-page-with-python） –

正如@Teemu Risikko評論說，你可以使用dryscrape或硒。這裏是一個解決方案使用dryscrape：

from bs4 import BeautifulSoup 
import dryscrape 

my_url = "http://www.jobs.ch/en/vacancies/?term=business+analyst" 
session = dryscrape.Session() 
session.visit(my_url) 
response = session.body() 
soup = BeautifulSoup(response) 
jobsAll = soup.find_all("a") 
for links in soup.find_all('a'): 
    print (links.get('href'))

溶液與dryscrape非常簡單，但安裝包可能會非常棘手（使用QT < = 55）...

來源

2017-02-22 17:39:28

用BeautifulSoup和Python刮取Javascript網頁

回答

相關問題