快python numpy哪裏有功能？

Question

我在幾個for循環內使用numpy的函數很多次，但它變得太慢了。有什麼方法可以更快地執行此功能？我讀過你應該嘗試做in-line for循環，並且在for循環之前爲函數創建局部變量，但似乎沒有什麼能夠提高速度（< 1％）。 len(UNIQ_IDS)〜800. emiss_data和obj_data是形狀=（2600,5200）的numpy ndarrays。我已經使用import profile來處理瓶頸的位置，並且for環路中的where是一個很大的問題。

import numpy as np 
max = np.max 
where = np.where 
MAX_EMISS = [max(emiss_data[where(obj_data == i)]) for i in UNIQ_IDS)] 

Answer 1

你就不能這樣做

emiss_data[obj_data == i] 

？我不知道你爲什麼使用where。

Answer 2

根據Are tuples more efficient than lists in Python?，分配元組比分配列表要快得多，所以也許只需構建一個元組而不是一個列表，這樣可以提高效率。

Answer 3

事實證明，在這種情況下，純Python循環比NumPy索引（或調用np.where）要快得多。

考慮以下選擇：

import numpy as np 
import collections 
import itertools as IT 

shape = (2600,5200) 
# shape = (26,52) 
emiss_data = np.random.random(shape) 
obj_data = np.random.random_integers(1, 800, size=shape) 
UNIQ_IDS = np.unique(obj_data) 

def using_where(): 
    max = np.max 
    where = np.where 
    MAX_EMISS = [max(emiss_data[where(obj_data == i)]) for i in UNIQ_IDS] 
    return MAX_EMISS 

def using_index(): 
    max = np.max 
    MAX_EMISS = [max(emiss_data[obj_data == i]) for i in UNIQ_IDS] 
    return MAX_EMISS 

def using_max(): 
    MAX_EMISS = [(emiss_data[obj_data == i]).max() for i in UNIQ_IDS] 
    return MAX_EMISS 

def using_loop(): 
    result = collections.defaultdict(list) 
    for val, idx in IT.izip(emiss_data.ravel(), obj_data.ravel()): 
     result[idx].append(val) 
    return [max(result[idx]) for idx in UNIQ_IDS] 

def using_sort(): 
    uind = np.digitize(obj_data.ravel(), UNIQ_IDS) - 1 
    vals = uind.argsort() 
    count = np.bincount(uind) 
    start = 0 
    end = 0 
    out = np.empty(count.shape[0]) 
    for ind, x in np.ndenumerate(count): 
     end += x 
     out[ind] = np.max(np.take(emiss_data, vals[start:end])) 
     start += x 
    return out 

def using_split(): 
    uind = np.digitize(obj_data.ravel(), UNIQ_IDS) - 1 
    vals = uind.argsort() 
    count = np.bincount(uind) 
    return [np.take(emiss_data, item).max() 
      for item in np.split(vals, count.cumsum())[:-1]] 

for func in (using_index, using_max, using_loop, using_sort, using_split): 
    assert using_where() == func() 

下面是基準，與shape = (2600,5200)：

In [57]: %timeit using_loop() 
1 loops, best of 3: 9.15 s per loop 

In [90]: %timeit using_sort() 
1 loops, best of 3: 9.33 s per loop 

In [91]: %timeit using_split() 
1 loops, best of 3: 9.33 s per loop 

In [61]: %timeit using_index() 
1 loops, best of 3: 63.2 s per loop 

In [62]: %timeit using_max() 
1 loops, best of 3: 64.4 s per loop 

In [58]: %timeit using_where() 
1 loops, best of 3: 112 s per loop 

因此using_loop（純Python）原來是超過11倍比using_where更快。

我不完全確定爲什麼純Python在這裏比NumPy快。我的猜測是，純Python版本通過兩個數組拉一次（是的，雙關）。它利用了這樣一個事實，即儘管所有的花式索引，我們真的只想訪問每個值。因此，它必須確定emiss_data中的每個值究竟屬於哪個組，但是這只是模糊的猜測。我不知道在我進行基準測試之前它會更快。

Answer 4

可以使用np.unique與return_index：

def using_sort(): 
    #UNIQ_IDS,uind=np.unique(obj_data, return_inverse=True) 
    uind= np.digitize(obj_data.ravel(), UNIQ_IDS) - 1 
    vals=uind.argsort() 
    count=np.bincount(uind) 

    start=0 
    end=0 

    out=np.empty(count.shape[0]) 
    for ind,x in np.ndenumerate(count): 
     end+=x 
     out[ind]=np.max(np.take(emiss_data,vals[start:end])) 
     start+=x 
    return out 

使用@ unutbu的答案爲基準shape = (2600,5200)：

np.allclose(using_loop(),using_sort()) 
True 

%timeit using_loop() 
1 loops, best of 3: 12.3 s per loop 

#With np.unique inside the definition 
%timeit using_sort() 
1 loops, best of 3: 9.06 s per loop 

#With np.unique outside the definition 
%timeit using_sort() 
1 loops, best of 3: 2.75 s per loop 

#Using @Jamie's suggestion for uind 
%timeit using_sort() 
1 loops, best of 3: 6.74 s per loop 

Answer 5

我相信做到這一點的最快方法是使用在該groupby()操作pandas包。到@比較俄菲翁的using_sort()功能，大熊貓是關於10快的一個因素：

import numpy as np 
import pandas as pd 

shape = (2600,5200) 
emiss_data = np.random.random(shape) 
obj_data = np.random.random_integers(1, 800, size=shape) 
UNIQ_IDS = np.unique(obj_data) 

def using_sort(): 
    #UNIQ_IDS,uind=np.unique(obj_data, return_inverse=True) 
    uind= np.digitize(obj_data.ravel(), UNIQ_IDS) - 1 
    vals=uind.argsort() 
    count=np.bincount(uind) 

    start=0 
    end=0 

    out=np.empty(count.shape[0]) 
    for ind,x in np.ndenumerate(count): 
     end+=x 
     out[ind]=np.max(np.take(emiss_data,vals[start:end])) 
     start+=x 
    return out 

def using_pandas(): 
    return pd.Series(emiss_data.ravel()).groupby(obj_data.ravel()).max() 

print('same results:', np.allclose(using_pandas(), using_sort())) 
# same results: True 

%timeit using_sort() 
# 1 loops, best of 3: 3.39 s per loop 

%timeit using_pandas() 
# 1 loops, best of 3: 397 ms per loop 

Answer 6

如果obj_data由相對較小的整數，則可以使用numpy.maximum.at（自v1.8.0）：

def using_maximumat(): 
    n = np.max(UNIQ_IDS) + 1 
    temp = np.full(n, -np.inf) 
    np.maximum.at(temp, obj_data, emiss_data) 
    return temp[UNIQ_IDS]