用於數值計算的多處理python函數

Question

希望能夠在這裏獲得一些幫助，並行化我的python代碼，我一直在苦苦掙扎了一段時間，並以我嘗試的方式提出了幾個錯誤，目前正在運行的代碼將會涉及2-3小時完成，代碼如下：

import numpy as np 
from scipy.constants import Boltzmann, elementary_charge as kb, e 
import multiprocessing 
from functools import partial 
Tc = 9.2 
x = [] 
g= [] 
def Delta(T): 
''' 
Delta(T) takes a temperature as an input and calculates a 
temperature dependent variable based on Tc which is defined as a 
global parameter 
''' 
    d0 = (pi/1.78)*kb*Tc 
    D0 = d0*(np.sqrt(1-(T**2/Tc**2))) 
    return D0 

def element_in_sum(T, n, phi): 
    D = Delta(T) 
    matsubara_frequency = (np.pi * kb * T) * (2*n + 1) 
    factor_d = np.sqrt((D**2 * cos(phi/2)**2) + matsubara_frequency**2) 
    element = ((2 * D * np.cos(phi/2))/ factor_d) * np.arctan((D * np.sin(phi/2))/factor_d) 
    return element 

def sum_elements(T, M, phi): 
''' 
    sum_elements(T,M,phi) is the most computationally heavy part 
    of the calculations, the larger the M value the more accurate the 
    results are. 
    T: temperature 
    M: number of steps for matrix calculation the larger the more accurate the calculation 
phi: The phase of the system can be between 0- pi 
''' 
    X = list(np.arange(0,M,1)) 
    Y = [element_in_sum(T, n, phi) for n in X] 
    return sum(Y) 

def KO_1(M, T, phi): 
    Iko1Rn = (2 * np.pi * kb * T /e) * sum_elements(T, M, phi) 
    return Iko1Rn 

def main(): 
    for j in range(1, 92): 
     T = 0.1*j 
     for i in range(1, 314): 
      phi = 0.01*i 
      pool = multiprocessing.Pool() 
      result = pool.apply_async(KO_1,args=(26000, T, phi,)) 
      g.append(result) 
      pool.close() 
      pool.join()  
    A = max(g); 
    x.append(A) 
    del g[:]  

我的做法是試圖發送KO1功能成多池，但我要麼得到一個Pickling錯誤或too many files open，任何幫助是極大的讚賞，如果多是錯誤的做法我很樂意的任何指南。

Answer 1

我還沒有測試過你的代碼，但你可以做幾件事情來改善它。首先，不要不必要地創建數組。當它只能使用一個生成器時，sum_elements會創建三個類似數組的對象。首先，np.arange創建一個numpy數組，然後list函數創建一個列表對象，然後列表理解創建另一個列表。該功能是它應該做的4倍。

正確的方法來實現它（在python3）將是：

def sum_elements(T, M, phi): 
    return sum(element_in_sum(T, n, phi) for n in range(0, M, 1)) 

如果使用python2，與xrange更換range。 這篇技巧可能會幫助你編寫任何python腳本。

此外，嘗試更好地利用多處理。看來你需要做的是創建一個multiprocessing.Pool對象一次，並使用pool.map函數。

主要功能應該是這樣的：我爲了用一個元組

def job(args): 
    i, j = args 
    T = 0.1*j 
    phi = 0.01*i 
    return K0_1(26000, T, phi) 

def main():   
    pool = multiprocessing.Pool(processes=4) # You can change this number 
    x = [max(pool.imap(job, ((i, j) for i in range(1, 314)) for j in range(1, 92)] 

通知到多個參數傳遞給工作。

Answer 2

這不是問題的答案，但如果我可以，我會建議如何使用簡單的numpy數組操作來加速代碼。看看下面的代碼：

import numpy as np 
from scipy.constants import Boltzmann, elementary_charge as kb, e 
import time 
Tc = 9.2 
RAM = 4*1024**2 # 4GB 

def Delta(T): 
    ''' 
    Delta(T) takes a temperature as an input and calculates a 
    temperature dependent variable based on Tc which is defined as a 
    global parameter 
    ''' 
    d0 = (np.pi/1.78)*kb*Tc 
    D0 = d0*(np.sqrt(1-(T**2/Tc**2))) 
    return D0 

def element_in_sum(T, n, phi): 
    D = Delta(T) 
    matsubara_frequency = (np.pi * kb * T) * (2*n + 1) 
    factor_d = np.sqrt((D**2 * np.cos(phi/2)**2) + matsubara_frequency**2) 
    element = ((2 * D * np.cos(phi/2))/ factor_d) * np.arctan((D * np.sin(phi/2))/factor_d) 
    return element 


def KO_1(M, T, phi): 
    X = np.arange(M)[:,np.newaxis,np.newaxis] 
    sizeX = int((float(RAM)/sum(T.shape))/sum(phi.shape)/8) #8byte 
    i0 = 0 
    Iko1Rn = 0. * T * phi 
    while (i0+sizeX) <= M: 
     print "X = %i"%i0 
     indices = slice(i0, i0+sizeX) 
     Iko1Rn += (2 * np.pi * kb * T /e) * element_in_sum(T, X[indices], phi).sum(0) 
     i0 += sizeX 
    return Iko1Rn 

def main(): 
    T = np.arange(0.1,9.2,0.1)[:,np.newaxis] 
    phi = np.linspace(0,np.pi, 361) 
    M = 26000 
    result = KO_1(M, T, phi) 
    return result, result.max() 

T0 = time.time() 
r, rmax = main() 
print time.time() - T0 

它在我的電腦上運行了超過20秒。必須小心不要使用太多的內存，這就是爲什麼仍然存在一個循環，其構造有點複雜，只能使用X片段。如果存在足夠的內存，則不是必需的。

還應該注意，這只是加速的第一步。仍然可以使用例如只是及時編譯或cython。