最快的方式

Question

我有一個列表的列表，像這樣：

import numpy as np 
import random 
import time 
import itertools 

N = 1000 
x =np.random.random((N,N)) 
y = np.zeros((N,N)) 
z = np.random.random((N,N)) 

list_of_lists = [[x, y], [y,z], [z,x]] 

併爲每個子表我想計算非零的個數，均值和標準差。

我已經做到了，像這樣：

distribution = [] 
alb_mean = [] 
alb_std = [] 


start = time.time() 

for i in range(len(list_of_lists)): 

    one_mean = [] 
    non_zero_l = [] 
    one_list = list_of_lists[i] 

    for n in one_list: 


     #count non_zeros 
     non_zero_count = np.count_nonzero(n) 
     non_zero_l.append(non_zero_count) 

     #assign nans 
     n = n.astype(float) 
     n[n == 0.0] = np.nan 

     #flatten the matrix 
     n = np.array(n.flatten()) 
     one_mean.append(n) 

    #append means and stds 
    distribution.append(sum(non_zero_l)) 
    alb_mean.append(np.nanmean(one_mean)) 
    alb_std.append(np.nanstd(one_mean)) 


end = time.time() 
print "Loop took {} seconds".format((end - start)) 

這需要0.23秒。

我試圖使這個更快了第二個選項：

distribution = [] 
alb_mean = [] 
alb_std = [] 


start = time.time() 

for i in range(len(list_of_lists)): 

    for_mean = [] 

    #get one list 
    one_list = list_of_lists[i] 

    #flatten the list 
    chain = itertools.chain(*one_list) 
    flat = list(chain) 

    #count non_zeros 
    non_zero_count = np.count_nonzero(flat) 
    distribution.append(non_zero_count) 

    #remove zeros 
    remove_zero = np.setdiff1d(flat ,[0.0]) 
    alb_mean.append(np.nanmean(remove_zero)) 
    alb_std.append(np.nanstd(remove_zero)) 

end = time.time() 
print "Loop took {} seconds".format((end - start)) 

這實際上是慢，需要0.88秒。

絕對數量的循環讓我覺得有一個更好的方法來做到這一點。我已經嘗試過numba，但它並沒有像在函數中追加一樣。一個與3迭代，另一個2迭代 -

Answer 1

版本＃1

很好用的多圈解決您的樣品中，你有兩個迴路循環。所以，它已經接近於矢量化了。唯一的瓶頸是append步驟。

外出時完全矢量化，這裏有一個方法 -

a = np.array(list_of_lists, dtype=float) 
zm = a!=0 
avgs = np.einsum('ijkl,ijkl->i',zm,a)/zm.sum(axis=(1,2,3)).astype(float) 

a[~zm] = np.nan 
stds = np.nanstd(a, axis=(1,2,3)) 

使用相同的設置中的問題，這是我得到的時序 -

Loop took 0.150925159454 seconds 
Proposed solution took 0.121352910995 seconds 

版本＃2

我們可以使用average來計算std，因此重新使用avgs作進一步提升：

因此，修改後的版本將是 -

a = np.asarray(list_of_lists) 
zm = a!=0 
N = zm.sum(axis=(1,2,3)).astype(float) 
avgs = np.einsum('ijkl,ijkl->i',zm,a)/N 

diffs = ((a-avgs[:,None,None,None])**2) 
stds = np.sqrt(np.einsum('ijkl,ijkl->i',zm,diffs)/N) 

更新時機 -

Loop took 0.155035018921 seconds 
Proposed solution took 0.0648851394653 seconds