用於計算特徵值的多處理

Question

我生成了大小爲1000x1000的100個隨機int矩陣。我正在使用多處理模塊來計算100個矩陣的特徵值。

的代碼如下：

import timeit 
import numpy as np 
import multiprocessing as mp 

def calEigen(): 

S, U = np.linalg.eigh(a) 

def multiprocess(processes): 
pool = mp.Pool(processes=processes) 
#Start timing here as I don't want to include time taken to initialize the processes 
start = timeit.default_timer() 
results = [pool.apply_async(calEigen, args=())] 
stop = timeit.default_timer() 
print (processes":", stop - start) 

results = [p.get() for p in results] 
results.sort() # to sort the results 


if __name__ == "__main__": 

global a 
a=[] 

for i in range(0,100): 
    a.append(np.random.randint(1,100,size=(1000,1000))) 

#Print execution time without multiprocessing 
start = timeit.default_timer() 
calEigen() 
stop = timeit.default_timer() 
print stop - start 

#With 1 process 
multiprocess(1) 

#With 2 processes 
multiprocess(2) 

#With 3 processes 
multiprocess(3) 

#With 4 processes 
multiprocess(4) 

輸出是

0.510247945786 
('Process:', 1, 5.1021575927734375e-05) 
('Process:', 2, 5.698204040527344e-05) 
('Process:', 3, 8.320808410644531e-05) 
('Process:', 4, 7.200241088867188e-05) 

另一次迭代顯示輸出：

69.7296020985 
('Process:', 1, 0.0009050369262695312) 
('Process:', 2, 0.023727893829345703) 
('Process:', 3, 0.0003509521484375) 
('Process:', 4, 0.057518959045410156) 

我的問題是：

1. 爲什麼時間執行時間不會隨着 進程數量的增加而減少？我是否正確使用多處理模塊？
2. 我是否正確計算執行時間？

我編輯了下面評論中給出的代碼。我想要串行和多處理函數來查找100個矩陣的相同列表的特徵值。編輯後的代碼是 -

import numpy as np 
import time 
from multiprocessing import Pool 

a=[] 

for i in range(0,100): 
a.append(np.random.randint(1,100,size=(1000,1000))) 

def serial(z): 
result = [] 
start_time = time.time() 
for i in range(0,100):  
    result.append(np.linalg.eigh(z[i])) #calculate eigen values and append to result list 
end_time = time.time() 
print("Single process took :", end_time - start_time, "seconds") 


def caleigen(c): 
result = []   
result.append(np.linalg.eigh(c)) #calculate eigenvalues and append to result list 
return result 

def mp(x): 
start_time = time.time() 
with Pool(processes=x) as pool: # start a pool of 4 workers 
    result = pool.map_async(caleigen,a) # distribute work to workers 
    result = result.get() # collect result from MapResult object 
end_time = time.time() 
print("Mutltiprocessing took:", end_time - start_time, "seconds") 

if __name__ == "__main__": 

serial(a) 
mp(1,a) 
mp(2,a) 
mp(3,a) 
mp(4,a) 

隨着進程數量的增加，時間不會減少。我哪裏錯了？多處理將列表劃分爲進程的塊還是必須進行劃分？

Answer 1

您沒有正確使用多處理模塊。正如@dopstar指出的那樣，你不會分裂你的任務。進程池只有一個任務，所以不管你分配了多少工人，只有一個人能夠得到這份工作。至於你的第二個問題，我沒有使用timeit來準確地測量處理時間。我只是使用time模塊來粗略地瞭解事物的速度。儘管如此，它在大多數時間都是有用的。如果我理解你想要正確地做什麼，這應該是你的代碼

import numpy as np 
import time 

result = [] 
start_time = time.time() 
for i in range(100): 
    a = np.random.randint(1, 100, size=(1000,1000)) #generate random matrix 
    result.append(np.linalg.eigh(a))     #calculate eigen values and append to result list 
end_time = time.time() 
print("Single process took :", end_time - start_time, "seconds") 

單流程版本了15.27秒我的電腦上的單進程版本。下面是多進程版本，我的電腦只花了0.46秒。我還包括單一的流程版本進行比較。 （單進程版本也必須包含在if塊中，並放在多進程版本之後。）因爲您想重複計算100次，創建一個工作池並讓它更容易他們會自動執行未完成的任務，而不是手動啓動每個進程並指定每個進程應該執行的操作。在我的代碼中，調用caleigen的參數只是爲了跟蹤任務執行的次數。最後，map_async通常比apply_async更快，它的缺點是消耗稍多的內存，並且只有一個參數用於函數調用。使用map_async而不是map的原因是，在這種情況下，返回結果的順序無關緊要，map_async比map快得多。

from multiprocessing import Pool 
import numpy as np 
import time 

def caleigen(x):  # define work for each worker 
    a = np.random.randint(1,100,size=(1000,1000)) 
    S, U = np.linalg.eigh(a)       
    return S, U 


if __name__ == "main": 
    start_time = time.time() 
    with Pool(processes=4) as pool:  # start a pool of 4 workers 
     result = pool.map_async(caleigen, range(100)) # distribute work to workers 
     result = result.get()  # collect result from MapResult object 
    end_time = time.time() 
    print("Mutltiprocessing took:", end_time - start_time, "seconds") 

    # Run the single process version for comparison. This has to be within the if block as well. 
    result = [] 
    start_time = time.time() 
    for i in range(100): 
     a = np.random.randint(1, 100, size=(1000,1000)) #generate random matrix 
     result.append(np.linalg.eigh(a))     #calculate eigen values and append to result list 
    end_time = time.time() 
    print("Single process took :", end_time - start_time, "seconds")