我使用multiprocessing.Queue
來處理廣告創建。但是,在使用Queue.get()時,似乎有些元素會丟失(例如,我在隊列中放置了10個廣告,但在多處理完成後只處理了8個)。多處理.Queue可能會丟失它內部的元素
的代碼是:
from multiProcessing import Process, Queue, JoinableQueue
def create_ad(origin_queue, ad_queue):
''' ad creation'''
for ad in iter(origin_queue.get, None):
try:
# do sth else
ad_queue.put(ad)
except:
pass
# actually there exists logging here, no error outside
finally:
origin_queue.task_done()
origin_queue.task_done()
if __name__ == '__main__':
origin_queue = JoinableQueue()
# mock to put ads inside
ads = [1,2,3,4,5]
[ origin_queue.put(ad) for ad in ads ]
ad_queue = Queue()
process_list = []
for p in range(4): # PROCESS_NUM
process = Process(target=create_ad, args=(origin_queue, ad_queue))
process_list.append(process)
for process in process_list:
process.daemon = True
process.start()
origin_queue.join()
for process in process_list:
origin_queue.put(None)
origin_queue.join()
while not ad_queue.empty():
ad = ad_queue.get() # number of ad got here is different from ads put inside it
是我的隊列的用法錯了嗎?
感謝您的回覆,但爲什麼它與異常部分有關?正如我的理解,因爲最終代碼將始終執行,它應該是與異常無關〜實際上,我在實際代碼中使用logger.exception(e),但沒有例外記錄 – linpingta