Python的多線程和HTTP請求

Question

我從付費的第三方API抓取實時數據的Python的後端Web服務器上的工作。 我需要非常快速地查詢這個API（每10秒約150個查詢）。因此，我創建了一個產生200個線程並將URL寫入隊列的小概念證明。線程然後從隊列中的url中讀取併發送HTTP請求。第三方API返回一個名爲delay的值，這是他們的服務器處理請求所花費的時間。 這裏是POC代碼，只需下載所有的URL（不重複）。

_http_pool = urllib3.PoolManager() 

def getPooledResponse(url): 
    return _http_pool.request("GET", url, timeout=30) 

class POC: 
    _worker_threads = [] 
    WORKER_THREAD_COUNT = 200 
    q = Queue.Queue() 

    @staticmethod 
    def worker(): 
     while True: 
      url = POC.q.get() 
      t0 = datetime.datetime.now() 
      r = getPooledResponse(item) 
      print "thread %s took %d seconds to process the url (service delay %d)" % (threading.currentThread().ident, (datetime.datetime.now() - t0).seconds, getDelayFromResponse(r)) 
      POC.q.task_done() 

    @staticmethod 
    def run(): 
      # start the threads if we have less than the desired amount 
      if len(POC._worker_threads) < POC.WORKER_THREAD_COUNT: 
       for i in range(POC.WORKER_THREAD_COUNT - len(POC._worker_threads)): 
        t = threading.Thread(target=POC.worker) 
        t.daemon = True 
        t.start() 
        POC._worker_threads.append(t) 

      # put the urls in the queue 
      for url in urls: 
       POC.q.put(url) 
       # sleep for just a bit so that the requests don't get sent out together (this is a limitation of the API I am using) 
       time.sleep(0.3) 
POC.run() 

當我運行此，前幾個結果返回一個合理的延遲：

thread 140544300453053 took 2 seconds to process the url (service delay 1.782) 

然而，約10-20秒後，我得到這樣的東西：

thread 140548049958656 took 23 seconds to process the url (service delay 1.754) 

換句話說，即使服務器返回一個小延遲，我的線程需要更長的時間才能完成...

如何測試其他21秒運行秒數？

謝謝！

Answer 1

您應該在代碼上使用profiler。