Python使用工作隊列處理隊列中的多個項目

Question

我試圖使用隊列構建一個簡單的多處理應用程序。

我開始4個進程來處理來自多個網站的數據。我希望每個進程都能處理不同的網站，但出於某種原因，這些進程會多次運行並且永遠不會退出。

from multiprocessing import Process 
import Queue 
import requests 

def readdata(item): 
    print item 
    r = requests.get(item) 
    print 'read data' 
    print r.status_code 


def worker(queue): 
    while True: 
     try: 
      print 'start process' 
      item = queue.get() 
      readdata(item) 
      q.task_done() 
     except: 
      print "the end" 
      break 

if __name__ == "__main__": 
    nthreads = 4 
    queue = Queue.Queue() 
    # put stuff in the queue here 
    moreStuff = ['http://www.google.com','http://www.yahoo.com','http://www.cnn.com'] 
    for stuff in moreStuff: 
     queue.put(stuff) 
    procs = [Process(target = worker, args = (queue,)) for i in xrange(nthreads)] 
    for p in procs: 
     p.start() 
    for p in procs: 
     p.join() 

輸出：

start process 
http://www.google.com 
start process 
http://www.google.com 
start process 
http://www.google.com 
start process 
http://www.google.com 
read data 
200 
start process 
http://www.yahoo.com 
read data 
200 
start process 
http://www.yahoo.com 
read data 
200 
start process 
http://www.yahoo.com 
read data 
200 
start process 
http://www.yahoo.com 
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. 
    InsecurePlatformWarning 
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. 
    InsecurePlatformWarning 
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. 
    InsecurePlatformWarning 
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. 
    InsecurePlatformWarning 
read data 
200 
start process 
http://www.cnn.com 
read data 
200 
start process 
http://www.cnn.com 
read data 
200 
start process 
http://www.cnn.com 
read data 
200 
start process 
read data 
200 
start process 
http://www.cnn.com 
read data 
200 
start process 
read data 
200 
start process 
read data 
200 
start process 

如何檢查隊列爲空，並退出？

Answer 1

使用.empty()爲queue

此外，作爲一個建議，因爲您的隊列並沒有改變，我應該這樣做，而不是：

while not queue.empty(): # Wait for the queue to finish 
    pass 

print('Queue finished') 

代替：

for p in procs: 
    p.join() 

或甚至更好地使用JoinableQueue代替：

for p in procs: 
    p.start() 
queue.join()