使用numpy爲RNN準備數據的最快方法是什麼？

我目前有一個(1631160,78) NP陣列作爲我的神經網絡輸入。我想嘗試使用需要3D結構作爲輸入數據的LSTM。我目前使用下面的代碼來生成所需的3D結構，但它超級慢（ETA> 1day）。有沒有更好的方式與numpy做到這一點？使用numpy爲RNN準備數據的最快方法是什麼？

我當前的代碼生成數據：

def transform_for_rnn(input_x, input_y, window_size): 
    output_x = None 
    start_t = time.time() 
    for i in range(len(input_x)): 
     if i > 100 and i % 100 == 0: 
      sys.stdout.write('\rTransform Data: %d/%d\tETA:%s'%(i, len(input_x), str(datetime.timedelta(seconds=(time.time()-start_t)/i * (len(input_x) - i))))) 
      sys.stdout.flush() 
     if output_x is None: 
      output_x = np.array([input_x[i:i+window_size, :]]) 
     else: 
      tmp = np.array([input_x[i:i+window_size, :]]) 
      output_x = np.concatenate((output_x, tmp)) 

    print 
    output_y = input_y[window_size:] 
    assert len(output_x) == len(output_y) 
    return output_x, output_y

來源

2016-11-01 Tengyu Liu

下面是使用NumPy strides向量化的output_x建立一個辦法 -

nrows = input_x.shape[0] - window_size + 1 
p,q = input_x.shape 
m,n = input_x.strides 
strided = np.lib.stride_tricks.as_strided 
out = strided(input_x,shape=(nrows,window_size,q),strides=(m,m,n))

採樣運行 -

In [83]: input_x 
Out[83]: 
array([[ 0.73089384, 0.98555845, 0.59818726], 
     [ 0.08763718, 0.30853945, 0.77390923], 
     [ 0.88835985, 0.90506367, 0.06204614], 
     [ 0.21791334, 0.77523643, 0.47313278], 
     [ 0.93324799, 0.61507976, 0.40587073], 
     [ 0.49462016, 0.00400835, 0.66401908]]) 

In [84]: window_size = 4 

In [85]: out 
Out[85]: 
array([[[ 0.73089384, 0.98555845, 0.59818726], 
     [ 0.08763718, 0.30853945, 0.77390923], 
     [ 0.88835985, 0.90506367, 0.06204614], 
     [ 0.21791334, 0.77523643, 0.47313278]], 

     [[ 0.08763718, 0.30853945, 0.77390923], 
     [ 0.88835985, 0.90506367, 0.06204614], 
     [ 0.21791334, 0.77523643, 0.47313278], 
     [ 0.93324799, 0.61507976, 0.40587073]], 

     [[ 0.88835985, 0.90506367, 0.06204614], 
     [ 0.21791334, 0.77523643, 0.47313278], 
     [ 0.93324799, 0.61507976, 0.40587073], 
     [ 0.49462016, 0.00400835, 0.66401908]]])

這將創建一個查看輸入數組，正如我們所記憶的那樣e高效。在大多數情況下，這應該轉化爲對性能的好處，而且涉及到進一步的操作。讓我們來驗證它的一個觀點確實 -

In [86]: np.may_share_memory(out,input_x) 
Out[86]: True # Doesn't guarantee, but is sufficient in most cases

另一個肯定拍的方式來驗證是設置一些值output和檢查輸入 -

In [87]: out[0] = 0 

In [88]: input_x 
Out[88]: 
array([[ 0.  , 0.  , 0.  ], 
     [ 0.  , 0.  , 0.  ], 
     [ 0.  , 0.  , 0.  ], 
     [ 0.  , 0.  , 0.  ], 
     [ 0.93324799, 0.61507976, 0.40587073], 
     [ 0.49462016, 0.00400835, 0.66401908]])

來源

2016-11-01 19:47:27 Divakar

使用numpy爲RNN準備數據的最快方法是什麼？

回答

相關問題