多進程命令列表

Question

我試圖運行一個命令列表。問題是列表可能很長，所以同時運行多個命令會很好。

如何在多處理模塊中執行此操作？

list_of_commands = [cmd foo, cmd bar, ...] 

main_log_file = open(os.getcwd() + '/Error.log', 'w+') 

Count = 0 
for Job in list_of_commands: 
    Count += 1 
    child = subprocess.Popen(Job, stdout=subprocess.PIPE, stderr=subprocess.PIPE) 
    streamdata = child.communicate()[0] 
    errcode = child.returncode 
    if errcode == 0: 
     print ('Job', Count, 'Success') 
    elif errcode == 1: 
     print ('Job', Count, 'Completed With Errors') 
    elif errcode == 2: 
     print ('Job', Count, 'Error') 
     main_log_file.write (streamdata.decode('ascii') + str(errcode) + '\n') 

main_log_file.close() 

執行的順序並不重要

Answer 1

可以使用concurrent.futures.ThreadPoolExecutormap功能以運行並行執行的costant數。

WORKERS = 5 # amount of concurrent executions you want 

def launcher(job): 
    child = subprocess.Popen(job, ...) 
    streamdata = child.communicate()[0] 
    ... 

with concurrent.futures.ThreadPoolExecutor(max_workers=WORKERS) as pool: 
    pool.map(launcher, jobs)