無法使用python和beautifulsoup在網頁中抓取某些href

我目前使用Python 3.4和bs4抓取網頁，以收集塞爾維亞在Rio2016中發揮的匹配結果。所以url here包含指向她所有比賽結果的鏈接，例如this。無法使用python和beautifulsoup在網頁中抓取某些href

後來我發現，鏈接位於這樣的HTML源代碼：

<a href="/en/volleyball/women/7168-serbia-italy/post" ng-href="/en/volleyball/women/7168-serbia-italy/post"> 
    <span class="score ng-binding">3 - 0</span> 
</a>

但經過幾次試驗，這href="/en/volleyball/women/7168-serbia-italy/post"從未露面。然後我試圖運行下面的代碼從URL得到所有的HREF：

from bs4 import BeautifulSoup 
import requests 

Countryr = requests.get('http://rio2016.fivb.com/en/volleyball/women/teams/srb-serbia#wcbody_0_wcgridpadgridpad1_1_wcmenucontent_3_Schedule') 
countrySoup = BeautifulSoup(Countryr.text) 

for link in countrySoup.find_all('a'): 
    print(link.get('href'))

然後奇怪的事情發生。根本沒有輸出href="/en/volleyball/women/7168-serbia-italy/post"。

我發現，這href是位於一側這個網址的標籤頁href="#scheduldedOver"之一，它是由下面的HTML代碼來控制：

<nav class="tabnav"> 
    <a href="#schedulded" ng-class="{selected: chosenStatus == 'Pre' }" ng-click="setStatus('Pre')" ng-href="#schedulded">Scheduled</a> 
    <a href="#scheduldedLive" ng-class="{selected: chosenStatus == 'Live' }" ng-click="setStatus('Live')" ng-href="#scheduldedLive">Live</a> 
    <a href="#scheduldedOver" class="selected" ng-class="{selected: chosenStatus == 'Over' }" ng-click="setStatus('Over')" ng-href="#scheduldedOver">Complete</a> 
</nav>

那我應該如何開始使用BeautifulSoup裏面的HREF一個標籤頁？

來源

2016-08-22 Benson

由於數據是從不同的URL來你不會找到源代碼，網址 - http://rio2016.fivb.com/en/ api/volley/matches/WOG2016/en/user/team/3017。爲了構建該網址，請參閱源代碼 –

中的'data-serviceteammatches ='是的。這是因爲你的html沒有這個信息你可以通過'print Countryr.text'來看到它。上面的評論鏈接是獲取網址的方式。 – giaosudau

非常感謝！所以現在我得到了正確的URL，看起來它只是一個純文本文件。這是否意味着美麗的事情在這裏完成，我需要使用其他搜索字符串函數來獲取信息？我已經嘗試了以下，似乎'matchSoup'只是一個沒有任何類別分隔的長字符串 'Matchr = requests.get（'http://rio2016.fivb.com'+ linkUrl） matchSoup = BeautifulSoup（ Matchr.text） print（matchSoup。文字）' – Benson

數據是動態創建的，如果你看看實際的源代碼，你可以看到Angularjs模板。

您仍然可以通過模仿一個Ajax調用，在源獲取JSON格式的所有信息yuuuuou也可以看到像一個div：

<div id="AngularPanel" class="main-wrapper" ng-app="fivb" 
data-servicematchcenterbar="/en/api/volley/matches/341/en/user/lives" 
data-serviceteammatches="/en/api/volley/matches/WOG2016/en/user/team/3017" 
data-servicelabels="/en/api/labels/Volley/en" 
data-servicelive="/en/api/volley/matches/WOG2016/en/user/live/">

使用data-servicematchcenterbar的href會給你所有的信息：

from bs4 import BeautifulSoup 
import requests 
from urlparse import urljoin 

r = requests.get('http://rio2016.fivb.com/en/volleyball/women/teams/srb-serbia#wcbody_0_wcgridpadgridpad1_1_wcmenucontent_3_Schedule') 
soup = BeautifulSoup(r.content) 

base = "http://rio2016.fivb.com/" 

json = requests.get(urljoin(base, soup.select_one("#AngularPanel")["data-serviceteammatches"])).json()

在JSON，你會看到這樣的輸出：

{"Id": 7168, "MatchNumber": "006", "TournamentCode": "WOG2016", "TournamentName": "Women's Olympic Games 2016", 
     "TournamentGroupName": "", "Gender": "", "LocalDateTime": "2016-08-06T22:35:00", 
     "UtcDateTime": "2016-08-07T01:35:00+00:00", "CalculatedMatchDate": "2016-08-07T03:35:00+02:00", 
     "CalculatedMatchDateType": "user", "LocalDateTimeText": "August 06 2016", 
     "Pool": {"Code": "B", "Name": "Pool B", "Url": "/en/volleyball/women/results and ranking/round1#anchorB"}, 
     "Round": 68, 
     "Location": {"Arena": "Maracanãzinho", "City": "Maracanãzinho", "CityUrl": "", "Country": "Brazil"}, 
     "TeamA": {"Code": "SRB", "Name": "Serbia", "Url": "/en/volleyball/women/teams/srb-serbia", 
        "FlagUrl": "/~/media/flags/flag_SRB.png?h=60&w=60"}, 
     "TeamB": {"Code": "ITA", "Name": "Italy", "Url": "/en/volleyball/women/teams/ita-italy", 
        "FlagUrl": "/~/media/flags/flag_ITA.png?h=60&w=60"}, 
     "Url": "/en/volleyball/women/7168-serbia-italy/post", "TicketUrl": "", "Status": "Over", "MatchPointsA": 3, 
     "MatchPointsB": 0, "Sets": [{"Number": 1, "PointsA": 27, "PointsB": 25, "Hours": 0, "Minutes": "28"}, 
            {"Number": 2, "PointsA": 25, "PointsB": 20, "Hours": 0, "Minutes": "25"}, 
            {"Number": 3, "PointsA": 25, "PointsB": 23, "Hours": 0, "Minutes": "27"}], 
     "PoolRoundName": "Preliminary Round", "DayInfo": "Weekend Day", 
     "WeekInfo": {"Number": 31, "Start": 7, "End": 13}, "LiveStreamUri": ""},

你可以從這些解析任何你需要的東西。

來源

2016-08-22 12:11:26

謝謝@ padraic！這很清楚，現在我可以獲得鏈接。 – Benson

@Benson，沒有問題，不客氣。 –

感謝您的所有幫助，我現在可以得到正確的網址。對我來說這是一個很好的學習經歷。非常感謝:)

from bs4 import BeautifulSoup 
import requests 

Countryr = requests.get('http://rio2016.fivb.com/en/volleyball/women/teams/srb-serbia#wcbody_0_wcgridpadgridpad1_1_wcmenucontent_3_Schedule') 
countrySoup = BeautifulSoup(Countryr.text) 

for link in countrySoup.find_all('div', {'id': 'AngularPanel'}): 
    linkUrl = link.get('data-serviceteammatches') 

json = requests.get('http://rio2016.fivb.com' + linkUrl).json() 

for item in json: 
    print(item.get('Url'))

輸出：

/en/volleyball/women/7168-serbia-italy/post 
/en/volleyball/women/7172-serbia-puerto rico/post 
/en/volleyball/women/7177-usa-serbia/post 
/en/volleyball/women/7181-china-serbia/post 
/en/volleyball/women/7187-serbia-netherlands/post 
/en/volleyball/women/7195-russia-serbia/post 
/en/volleyball/women/7198-serbia-usa/post 
/en/volleyball/women/7200-china-serbia/post

來源

2016-08-22 13:44:35 Benson

無法使用python和beautifulsoup在網頁中抓取某些href

回答

相關問題