多線程與urlib2

Question

我試圖加快我的程序時使用urilib2查詢api。爲了做到這一點，我試着用多線程寫作。

但是，當我運行代碼它與以下錯誤，我沒有得到任何響應通過urlib2調用。

Traceback (most recent call last): 
    File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner 
    File "/usr/lib64/python2.6/threading.py", line 484, in run 
    File "./myscript", line 333, in query_api 
    File "./myscript", line 350, in issue_request 
    File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen 
    File "/usr/lib64/python2.6/urllib2.py", line 391, in open 
    File "/usr/lib64/python2.6/urllib2.py", line 409, in _open 
    File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain 
    File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open 
    File "/usr/lib64/python2.6/urllib2.py", line 1164, in do_open 
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error' 
Exception in thread Thread-2 (most likely raised during interpreter shutdown): 
Traceback (most recent call last): 
    File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner 
    File "/usr/lib64/python2.6/threading.py", line 484, in run 
    File "./myscript", line 333, in query_api 
    File "./myscript", line 350, in issue_request 
    File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen 
    File "/usr/lib64/python2.6/urllib2.py", line 391, in open 
    File "/usr/lib64/python2.6/urllib2.py", line 409, in _open 
    File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain 
    File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open 
    File "/usr/lib64/python2.6/urllib2.py", line 1164, in do_open 
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error' 

我的代碼是：

class ApiMultiThreadHelper: 

    def __init__(self,api_calls): 
     self.q = Queue.Queue() 
     self.api_datastore = {} 
     self.api_calls = api_calls 
     self.userpass = '#####' 

    def query_api(self,q,api_query): 
     self.q.put(self.issue_request(api_query)) 

    def issue_request(self,api_query): 

     self.api_datastore.update({api_query:{}}) 

     for lookup in ["call1","call2"]: 
      query = api_query+lookup 

      request = urllib2.Request(query) 
      request.add_header("Authorization", "Basic %s" % self.userpass) 
      f = urllib2.urlopen(request) 
      response = f.read() 
      f.close() 

      self.api_datastore[api_query].update({lookup:response}) 

     return True 

    def go(self): 
     for i in self.api_calls: 
      t = threading.Thread(target=self.query_api, args = (self.q,i)) 
      t.daemon = True 
      t.start() 

     return self.api_datastore 

感謝，

Answer 1

urllib2應該是線程安全的。這是一個猜測，但...螺紋加工-2

異常（解釋關閉過程中最有可能提出的）

這表明，你是不是在等待你的線程來完成這項工作。該問題的行是這一個：

t.daemon = True 

的結果是，當主線程退出一切都被殺死（主線程只能等待非守護進程）。嘗試刪除該行。你也應該做一個適當的加入：

threads = [] 
for i in self.api_calls: 
    t = threading.Thread(target=self.query_api, args = (self.q,i)) 
    t.start() 
    threads.append(t) 

for t in threads: 
    t.join() # <--- wait for all threads to finish their jobs