抓取某個網頁與Python

Question

我要抓取所有這些結果在頁面： http://www.carnival.com.au/Find-A-Cruise/search-results.aspx?ShipCode=LE&

的問題是，沒有任何選項來顯示他們。到目前爲止，我成功地抓取了初始頁面，但是我無法進入其他頁面。如何完成這項工作？

Answer 1

後續頁面用JavaScript分頁加載。您看到該請求正在發送到"http://www.carnival.com.au/DomainData/SailingSearch/Get/"，並且POST請求中有一些參數。如果您嘲笑相同的請求，則會返回包含巡航信息的JSON數據。

import requests 
sesh = requests.Session() 
first_page = sesh.get("http://www.carnival.com.au/Find-A-Cruise/search-results.aspx?ShipCode=LE&#UBSELBWf2tB4Rs1H.97") 
data = {"ShipCode": "LE", "CurrencyCode": "AUD", "PageSize": 5, "PageNumber": 2, "SortExpression": "FirstSailDate"} 
page_2 = sesh.post("http://www.carnival.com.au/DomainData/SailingSearch/Get/", data=data) 
cruise_data = page_2.json() 

JSON響應甚至還跟顯示有多少總成績也有，你可以用更有效地請求後續頁面。

該JSON的一些示例輸出。

{'CurrentPage': '2', 
'CurrentResultsCount': '6 - 10', 
'LastPage': '9', 
'SortExpression': 'FirstSailDate', 
'TotalResultsCount': 44, 
'Voyages': [{'BookNowUrl': 'http://booking.carnival.com.au/index.asp?AIID=44&overridePageID=651&currentPageID=650&processingObjectIDList=21604&search 
Mode=searchByNumber&searchByNumberCriteria=G639&searchByCriteriaStatus=go&voyageCode=G639&voyageName=G639&shipCode=LE&shipName=Legend&brandCode=CL&bra 
ndName=Carnival%20Cruise%20Lines&homeCityCode=SYD&airCityCode=SYD&homeCityName=Sydney&airCityNameSydney&tDef=&tourName=&duration=10&embarkDate=2016121 
7&tType=O&tDirection=R&destinationCode=I&destinationName=Pacific+Islands&cruiseSelected=yes&unbundling=-&switchPolarRegion=prd&currencyCode=AUD', 
       'CruiseCode': 'G639 ', 
       'CruiseNights': 10, 
       'DateRangeText': '17 Dec 2016 (Sat - Tue)', 
       'DeparturePortCode': 'SYD', 
       'DeparturePortName': 'Sydney', 
       'FromBPrice': '1,699.00 AUD', 
       'FromIPrice': '1,549.00 AUD', 
       'FromOPrice': '1,649.00 AUD', 
       'FromQuadPrice': '1,689.00 AUD', 
       'FromSPrice': '2,649.00 AUD', 
       'FromTwinPrice': '1,549.00 AUD', 
       'MetaCategory': 'P', 
       'MetaCategoryDescription': 'Pacific Islands', 
       'PortsVisited': [{'CruiseDay': 0, 
           'PortCode': 'SYD', 
           'PortName': 'Sydney'}, 
           {'CruiseDay': 1, 
           'PortCode': 'NOU', 
           'PortName': 'Noumea'}, 
           {'CruiseDay': 2, 
           'PortCode': 'MY2', 
           'PortName': 'Mystery Island'}, 
           {'CruiseDay': 3, 
           'PortCode': 'LIF', 
           'PortName': 'Lifou Isle'}, 
           {'CruiseDay': 4, 
           'PortCode': 'MEE', 
           'PortName': 'Mare'}, 
           {'CruiseDay': 5, 
           'PortCode': 'SYD', 
           'PortName': 'Sydney'}], 
       'RegionCode': 'I', 
       'RegionName': 'Pacific Islands', 
       'SailDate': '/Date(1481950800000)/', 
       'ShipCode': 'LE', 
       'ShipName': 'Legend',