在python tensorflow中劃分批處理

Question

我試圖設置批處理大小並運行Autoencoder程序，因爲沒有足夠的內存來使用完整批處理。所以我試圖使用tf.train.batch。但由於函數的參數是一個張量，我試圖用tf.convert_to_tensor將np數組轉換爲張量。但是內存超過2GB，無法變成張量。我怎樣才能用小批量培訓？下面是 是我的代碼。

N_img=47000000 
batch_size=100 
X_train = np.zeros(shape=(N_img, Freq_LEN, LOOK_LEN, 1), dtype='float32') 
x = tf.placeholder(tf.float32, [None, FRM_LEN/2,FRM_LEN/2,1]) #FRM_LEN=256 
y = tf.placeholder(tf.float32, [None, FRM_LEN/2,FRM_LEN/2,1]) 
X_train=tf.convert_to_tensor(X_train) 
X_train_batch= tf.train.batch(X_train,batch_size=batch_size) 

print("Start training..") 

for step in range(n_iters): 
    sess.run(optm, feed_dict={x: X_train_batch, y: X_train_batch, keepprob: 0.7}) 
    if step % 100 == 0: 
     print(step,sess.run(cost, feed_dict={x: X_train_batch, y: X_train_batch, keepprob: 1})) 

print("finish training") 

Answer 1

嘗試創建不需要張量參數（避免tf.convert_to_tensor操作）的自定義功能generate_batch，例如：

import numpy as np 
batch_size = 100 

X_train = np.zeros(shape=(N_img, Freq_LEN, LOOK_LEN, 1), dtype='float32') 
y_train = np.zeros(shape=(N_img, Freq_LEN, LOOK_LEN, 1), dtype='float32') 

data_index = 0 

def generate_batch(batch_size): 
    global data_index 
    batch = np.ndarray(shape=(batch_size, Freq_LEN, LOOK_LEN, 1), dtype=np.float32) #the same shapes as train data 
    labels = np.ndarray(shape=(batch_size, Freq_LEN, LOOK_LEN, 1), dtype=np.float32) 
    for i in range(batch_size): 
     batch[i] = X_train[data_index] 
     labels[i] = y_train[data_index] 
     data_index = (data_index + 1) % len(X_train) 
    return batch, labels 

for step in range(n_iters): 
    X_train_batch, X_train_batch = generate_batch(batch_size) 
    sess.run(optm, feed_dict={x: X_train_batch, y: X_train_batch, keepprob: 0.7}) 
    if step % 100 == 0: 
     print(step,sess.run(cost, feed_dict={x: X_train_batch, y: X_train_batch, keepprob: 1})) 

Answer 2

力批次處理的CPU上發生：

.... 
with tf.device('/cpu:0'): 
    x = tf.placeholder(tf.float32, [None, FRM_LEN/2,FRM_LEN/2,1]) #FRM_LEN=256 
    y = tf.placeholder(tf.float32, [None, FRM_LEN/2,FRM_LEN/2,1]) 
    X_train=tf.convert_to_tensor(X_train) 
    X_train_batch= tf.train.batch(X_train,batch_size=batch_size) 
print("Start training..") 
....