Python3請求庫提交表單，不允許發佈請求

Question

我試圖從Philly Police webpage的給定位置獲得警區。我有太多的地方需要手工完成，所以我正在嘗試使用Python的請求庫自動化這個過程。保存位置值網頁的形式如下：

<form id="search-form" method="post" action="districts/searchAddress"> 
<fieldset> 
    <div class="clearfix"> 
     <label for="search-address-box"><span>Enter Your Street Address</span></label> 
     <div class="input"> 
      <input tabindex="1" class="district-street-address-input" id="search-address-box" name="name" type="text" value=""> 
     </div> 
    </div> 
    <div class="actions" style="float: left;"> 
     <button tabindex="3" type="submit" class="btn btn-success">Search</button> 
    </div> 
    <a id="use-location" href="https://www.phillypolice.com/districts/index.html?_ID=7&_ClassName=DistrictsHomePage#" style="float: left; margin: 7px 0 0 12px;"><i class="icon-location-arrow"></i>Use Current Location</a> 
    <div id="current-location-display" style="display: none;"><p>Where I am right now.</p></div> 
</fieldset> 
</form> 

然而，當我嘗試發佈或使用以下付諸網頁：

r = requests.post('http://www.phillypolice.com/districts',data={'search-address-box':'425 E. Roosevelt Blvd'}) 

我收到錯誤405，POST是不允許的。然後我關掉Javascript並試圖在網頁上找到該區域，當我點擊提交時，我收到了相同的405錯誤消息。因此，該表格絕對不會被提交，並且該區域是使用JavaScript來查找的。

有沒有一種方法來模擬'點擊'提交按鈕來使用請求庫觸發JavaScript？

Answer 1

的數據後，首先查詢谷歌地圖在這裏的最後請求是得到像下面這樣的座標檢索你需要做的GET請求：

import requests 

key = "my_key" 
coord_params = {"output": "json", 
       "key": key} 

# This provides the coordinates. 
coords_url = "https://dev.virtualearth.net/REST/v1/Locations" 

# Template to pass each address to in your actual loop. 
template = "{add},US" 
url = "https://api.phillypolice.com/jsonservice/Map/searchAddress.json" 
with requests.Session() as s: 
    # Add the query param passing in each zipcode 
    coord_params["query"] = template.format(add="425 E. Roosevelt Blvd") 
    js = s.get(coords_url, params=coord_params).json() 
    # Parse latitude and longitude from the returned json. 
    # Call str to make make it into `(lat, lon)` 
    latitude_longitude = str((js[u'resourceSets'][0][u'resources'][0]["point"][u'coordinates'])) 
    data = s.get(url, params={"latlng": latitude_longitude}) 

    print(data.json()) 

如果我們運行它減去我的鑰匙：

In [2]: import requests 
    ...: 
    ...: key = "my_key..." 
    ...: 
    ...: coord_params = {"output": "json", 
    ...:     "key": key} 
    ...: coords_url = "https://dev.virtualearth.net/REST/v1/Locations" 
    ...: template = "{add},US" 
    ...: url = "https://api.phillypolice.com/jsonservice/Map/searchAddress.json" 
    ...: with requests.Session() as s: 
    ...:  coord_params["query"] = template.format(add="425 E. Roosevelt Blvd") 
    ...: 
    ...:  js = s.get(coords_url, params=coord_params).json() 
    ...:  latitude_longitude = str(js[u'resourceSets'][0][u'resources'][0]["po 
    ...: int"][u'coordinates']) 
    ...:  print(latitude_longitude) 
    ...:  data = s.get(url, params={"latlng": latitude_longitude}) 
    ...:  print(data.json()) 
    ...:  
[40.02735900878906, -75.1153564453125] 
{'response': ['35', '2', 'Marques Newsome', 'PPD.35_PSA2@phila.gov ', '267-357-1436']} 

如果您在瀏覽器中查看請求，可以看到它與您看到的響應相匹配。

Answer 2

當您點擊「提交」時，會發生兩件大事： - 谷歌地址解析服務和對使用地理編碼服務返回的座標的「searchAddress.json」端點的XHR請求。

您可以嘗試模擬上述請求，小心處理所有API密鑰和所需參數，或者您可以保持更高級別並通過selenium使用瀏覽器自動化。

使用PhantomJS headless browser工作例如：

In [2]: from selenium import webdriver 

In [3]: driver = webdriver.PhantomJS() 

In [4]: driver.get("https://www.phillypolice.com/districts/") 

In [5]: address = "425 E. Roosevelt Blvd" 

In [6]: search_box = driver.find_element_by_id("search-address-box") 

In [7]: search_box.send_keys(address) 

In [8]: search_box.submit() 

In [9]: driver.find_element_by_css_selector("#district-menu h2").text 
Out[9]: u'35th District' 

In [10]: driver.find_element_by_css_selector("#district-menu h4").text 
Out[10]: u'PSA 2' 

而且，你可能需要Explicit Waits處理 「時機」 問題。

你可以設置一個免費帳戶與bing maps api，並獲得座標：