刪除列表中的對象以傳遞給多處理

Question

我需要將大列表中的每個對象傳遞給一個函數。函數完成後，我不再需要傳遞給該函數的對象，並希望刪除該對象以節省內存。如果我用一個單一的過程中工作，我將做到以下幾點：

result = [] 
while len(mylist) > 0: 
    result.append(myfunc(mylist.pop()) 

正如我遍歷MYLIST它傳遞給我的功能後，我突然掉在這樣的對象不再存儲在MYLIST列表中的每個對象。如何使用multiprocessing並行實現同樣的效果？

Answer 1

一個簡單的消費者例如（credits go here）：

import multiprocessing 
import time 
import random 

class Consumer(multiprocessing.Process): 

    def __init__(self, task_queue, result_queue): 
     multiprocessing.Process.__init__(self) 
     self.task_queue = task_queue 
     self.result_queue = result_queue 

    def run(self): 
     while True: 
      task = self.task_queue.get() 
      if task is None: 
       # Poison pill means shutdown 
       self.task_queue.task_done() 
       break 
      answer = task.process() 
      self.task_queue.task_done() 
      self.result_queue.put(answer) 
     return 


class Task(object): 

    def process(self): 
     time.sleep(0.1) # pretend to take some time to do the work 
     return random.randint(0, 100) 


if __name__ == '__main__': 
    # Establish communication queues 
    tasks = multiprocessing.JoinableQueue() 
    results = multiprocessing.Queue() 

    # Start consumers 
    num_consumers = multiprocessing.cpu_count() * 2 
    consumers = [Consumer(tasks, results) for i in xrange(num_consumers)] 
    for consumer in consumers: 
     consumer.start() 

    # Enqueue jobs 
    num_jobs = 10 
    for _ in xrange(num_jobs): 
     tasks.put(Task()) 

    # Add a poison pill for each consumer 
    for _ in xrange(num_consumers): 
     tasks.put(None) 

    # Wait for all tasks to finish 
    tasks.join() 

    # Start printing results 
    while num_jobs: 
     result = results.get() 
     print 'Result:', result 
     num_jobs -= 1