HTTP錯誤403：禁止的urlib2 Python 2.7

Question

我已經成功地使用了urllib2，但是對於這個網站，我突然測試了它並沒有工作。我在論壇上看過，並嘗試了一些修復程序，它似乎並沒有工作。下面是一個解決方案的例子，但不適合我。有人可以幫助我連接到它。

，讓錯誤的代碼：

from bs4 import BeautifulSoup 
import urllib2 

proxy_support = urllib2.ProxyHandler({"http":"http://username:password@ip:port"}) 
hdr = {'Accept': 'text/html,application/xhtml+xml,*/*'} 
url = 'http://www.carnextdoor.com.au/' 
opener = urllib2.build_opener(proxy_support) 
urllib2.install_opener(opener) 
req=urllib2.Request(url,headers=hdr) 
#Here I get the error with and without using the header or going html = urllib2.urlopen(url).read() 
html = urllib2.urlopen(req).read() 
soup=BeautifulSoup(html,"html5lib") 
print soup 

Answer 1

我得到了403，直到我添加了一個用戶代理，下面就足以爲我工作：

hdr = {'Accept': 'text/html,application/xhtml+xml,*/*',"user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"} 
url = 'http://www.carnextdoor.com.au/' 


req=urllib2.Request(url,headers=hdr) 
#Here I get the error with and without using the header or going html = urllib2.urlopen(url).read() 
html = urllib2.urlopen(req).read() 
soup=BeautifulSoup(html,"html5lib") 
print soup 

不包含用戶代理：

In [10]: hdr = {'Accept': 'text/html,application/xhtml+xml,*/*'} 

In [11]: url = 'http://www.carnextdoor.com.au/' 

In [12]: req=urllib2.Request(url,headers=hdr) 

In [13]: html = urllib2.urlopen(req).read() 
--------------------------------------------------------------------------- 
HTTPError         Traceback (most recent call last) 
<ipython-input-13-dbeb64d95cd3> in <module>() 
----> 1 html = urllib2.urlopen(req).read() 

在User-Agent：

In [20]: hdr = {'Accept': 'text/html,application/xhtml+xml,*/*',"user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"} 

In [21]: req=urllib2.Request(url,headers=hdr) 
In [22]: html = urllib2.urlopen(req).read() 
In [23]: 

使用requests沒有任何用戶代理也工作正常。

In [28]: import requests 

In [29]: r = requests.get(url) 

In [30]: r.status_code 
Out[30]: 200