如何在做多個請求時加速Python的urllib2

Question

我正在使用python的urllib2庫向特定主機發出多個http請求。每次發出請求時，都會創建一個新的tcp和http連接，這會花費大量時間。有沒有辦法讓使用urllib2的tcp/http連接保持活動狀態？

Answer 1

如果切換到httplib，您將更好地控制底層連接。

例如：

import httplib 

conn = httplib.HTTPConnection(url) 

conn.request('GET', '/foo') 
r1 = conn.getresponse() 
r1.read() 

conn.request('GET', '/bar') 
r2 = conn.getresponse() 
r2.read() 

conn.close() 

這將發送相同的基礎TCP連接上2 HTTP的GET。

Answer 2

我以前使用第三方庫urllib3效果不錯。它旨在通過彙集連接以便重用來補充urllib2。從the wiki

修改例如：

>>> from urllib3 import HTTPConnectionPool 
>>> # Create a connection pool for a specific host 
... http_pool = HTTPConnectionPool('www.google.com') 
>>> # simple GET request, for example 
... r = http_pool.urlopen('GET', '/') 
>>> print r.status, len(r.data) 
200 28050 
>>> r = http_pool.urlopen('GET', '/search?q=hello+world') 
>>> print r.status, len(r.data) 
200 79124 

Answer 3

如果你需要的東西比普通httplib的更加自動化，這可能幫助，但它不是線程安全的。

try: 
    from http.client import HTTPConnection, HTTPSConnection 
except ImportError: 
    from httplib import HTTPConnection, HTTPSConnection 
import select 
connections = {} 


def request(method, url, body=None, headers={}, **kwargs): 
    scheme, _, host, path = url.split('/', 3) 
    h = connections.get((scheme, host)) 
    if h and select.select([h.sock], [], [], 0)[0]: 
     h.close() 
     h = None 
    if not h: 
     Connection = HTTPConnection if scheme == 'http:' else HTTPSConnection 
     h = connections[(scheme, host)] = Connection(host, **kwargs) 
    h.request(method, '/' + path, body, headers) 
    return h.getresponse() 


def urlopen(url, data=None, *args, **kwargs): 
    resp = request('POST' if data else 'GET', url, data, *args, **kwargs) 
    assert resp.status < 400, (resp.status, resp.reason, resp.read()) 
    return resp