Python的SSL網頁抓取

我想使用Python 2.7和BeautifulSoup刮網頁，但我不能通過一個協議錯誤，這對我來說沒有多大意義。這只是發生在特定的網站，我需要做的這對：https://edd.telstra.com/telstra Python的SSL網頁抓取

的代碼，我只使用了基本測試：

#! /usr/bin/python 

from urllib import urlopen 
from BeautifulSoup import BeautifulSoup 
import re 

# Copy all of the content from the provided web page 
webpage = urlopen("https://edd.telstra.com/telstra/").read()

而且我得到以下錯誤（在Ubuntu 12.10上運行）：

Traceback (most recent call last): 
File "e.py", line 8, in <module> 
webpage = urlopen("https://edd.telstra.com/telstra/").read() 
File "/usr/lib/python2.7/urllib.py", line 86, in urlopen 
return opener.open(url) 
File "/usr/lib/python2.7/urllib.py", line 207, in open 
return getattr(self, name)(url) 
File "/usr/lib/python2.7/urllib.py", line 436, in open_https 
h.endheaders(data) 
File "/usr/lib/python2.7/httplib.py", line 958, in endheaders 
self._send_output(message_body) 
File "/usr/lib/python2.7/httplib.py", line 818, in _send_output 
self.send(msg) 
File "/usr/lib/python2.7/httplib.py", line 780, in send 
self.connect() 
File "/usr/lib/python2.7/httplib.py", line 1165, in connect 
self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file) 
File "/usr/lib/python2.7/ssl.py", line 381, in wrap_socket 
ciphers=ciphers) 
File "/usr/lib/python2.7/ssl.py", line 143, in __init__ 
self.do_handshake() 
File "/usr/lib/python2.7/ssl.py", line 305, in do_handshake 
self._sslobj.do_handshake() 
IOError: [Errno socket error] [Errno 1] _ssl.c:504: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac

有人能告訴我是否有一些參數需要指定，以便讓此頁面在Python中下載？這似乎是這個網頁上的問題，因爲上面的代碼（以及我嘗試過的其他許多代碼）在我嘗試的其他HTTPS/SSL頁面上工作正常。

感謝您的幫助！

來源

2013-04-10 Tom

我有一個類似的問題，和整個這次來到http://bugs.debian.org/cgi-bin /bugreport.cgi?bug=678353這似乎表明openssl版本問題，並且該問題將在1.0.1e-2中解決。 – 2014-01-15 22:54:04

我可以推薦使用請求lib目錄下：

def get_page(login, password): 
    '''Docstring 
    ''' 
    url = 'https://qwe.qwe' 

    payload = { 
     'user': login, 
     'pass': password 
    } 

    with requests.Session() as my_session: 
     my_session.post(url, data=payload) 
     data = my_session.get(url) 
    return data.text

來源

2015-01-20 11:47:57 dezz

Python的SSL網頁抓取

回答

相關問題