Tensorflow耗盡資源

Question

我寫了一個Tensorflow程序，該程序讀取128x128圖像。該程序在我的筆記本電腦上運行良好，我用它來檢查代碼是否正常。 第一個程序是基於MNIST教程的，第二個是使用MNIST示例的convNN。從我在網上閱讀

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[16384,20000] 
[[Node: inputLayer_1/weights/Variable/Adam_1/Assign = Assign[T=DT_FLOAT, _class=["loc:@inputLayer_1/weights/Variable"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](inputLayer_1/weights/Variable/Adam_1, inputLayer_1/weights/Variable/Adam_1/Initializer/Const)]] 

：當我嘗試在GPU上運行它們，我收到以下錯誤消息。我必須在我的測試中使用批次，以下是餵養的工作方式：

........................................... 
    batchSize = 40 
img_height = 128 
img_width = 128 


# 1st function to read images form TF_Record 
def getImage(filename): 
    # convert filenames to a queue for an input pipeline. 
    filenameQ = tf.train.string_input_producer([filename],num_epochs=None) 

    # object to read records 
    recordReader = tf.TFRecordReader() 

    # read the full set of features for a single example 
    key, fullExample = recordReader.read(filenameQ) 

    # parse the full example into its' component features. 
    features = tf.parse_single_example(
     fullExample, 
     features={ 
      'image/height': tf.FixedLenFeature([], tf.int64), 
      'image/width': tf.FixedLenFeature([], tf.int64), 
      'image/colorspace': tf.FixedLenFeature([], dtype=tf.string,default_value=''), 
      'image/channels': tf.FixedLenFeature([], tf.int64), 
      'image/class/label': tf.FixedLenFeature([],tf.int64), 
      'image/class/text': tf.FixedLenFeature([], dtype=tf.string,default_value=''), 
      'image/format': tf.FixedLenFeature([], dtype=tf.string,default_value=''), 
      'image/filename': tf.FixedLenFeature([], dtype=tf.string,default_value=''), 
      'image/encoded': tf.FixedLenFeature([], dtype=tf.string, default_value='') 
     }) 

    # now we are going to manipulate the label and image features 
    label = features['image/class/label'] 
    image_buffer = features['image/encoded'] 
    # Decode the jpeg 
    with tf.name_scope('decode_jpeg',[image_buffer], None): 
     # decode 
     image = tf.image.decode_jpeg(image_buffer, channels=3) 

     # and convert to single precision data type 
     image = tf.image.convert_image_dtype(image, dtype=tf.float32) 
    # cast image into a single array, where each element corresponds to the greyscale 
    # value of a single pixel. 
    # the "1-.." part inverts the image, so that the background is black. 
    image=tf.reshape(1-tf.image.rgb_to_grayscale(image),[img_height*img_width]) 
    # re-define label as a "one-hot" vector 
    # it will be [0,1] or [1,0] here. 
    # This approach can easily be extended to more classes. 
    label=tf.stack(tf.one_hot(label-1, numberOFclasses)) 
    return label, image 

train_img,train_label = getImage(TF_Records+"/train-00000-of-00001") 
validation_img,validation_label=getImage(TF_Records+"/validation-00000-of-00001") 
# associate the "label_batch" and "image_batch" objects with a randomly selected batch--- 
# of labels and images respectively 
train_imageBatch, train_labelBatch = tf.train.shuffle_batch([train_img, train_label], batch_size=batchSize,capacity=50,min_after_dequeue=10) 

# and similarly for the validation data 
validation_imageBatch, validation_labelBatch = tf.train.shuffle_batch([validation_img, validation_label], 
               batch_size=batchSize,capacity=50,min_after_dequeue=10) 

......................... ...............................

sess.run(tf.global_variables_initializer()) 

# start the threads used for reading files 
coord = tf.train.Coordinator() 
threads = tf.train.start_queue_runners(sess=sess,coord=coord) 

# feeding function 
def feed_dict(train): 
    if True : 
     #img_batch, labels_batch= tf.train.shuffle_batch([train_label,train_img],batch_size=batchSize,capacity=500,min_after_dequeue=200) 
     img_batch , labels_batch = sess.run([ train_labelBatch ,train_imageBatch]) 
     dropoutValue = 0.7 
    else: 
     # img_batch,labels_batch = tf.train.shuffle_batch([validation_label,validation_img],batch_size=batchSize,capacity=500,min_after_dequeue=200) 
     img_batch,labels_batch = sess.run([ validation_labelBatch,validation_imageBatch]) 
     dropoutValue = 1 
    return {x:img_batch,y_:labels_batch,keep_prob:dropoutValue} 

for i in range(max_numberofiteretion): 
    if i%10 == 0:#Run a Test 
     summary, acc = sess.run([merged,accuracy],feed_dict=feed_dict(False)) 
     test_writer.add_summary(summary,i)# Save to TensorBoard 
    else: # Training 
     if i % 100 == 99: # Record execution stats 
     run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE) 
     run_metadata = tf.RunMetadata() 
     summary, _ = sess.run([merged, train_step], 
           feed_dict=feed_dict(True), 
           options=run_options, 
           run_metadata=run_metadata) 
     train_writer.add_run_metadata(run_metadata, 'step%03d' % i) 
     train_writer.add_summary(summary, i) 
     print('Adding run metadata for', i) 
     else: # Record a summary 
     summary, _ = sess.run([merged, train_step], feed_dict=feed_dict(True)) 
     train_writer.add_summary(summary, i) 

# finalise 
coord.request_stop() 
coord.join(threads) 
train_writer.close() 
test_writer.close() 

............ ......................................

驗證文件夾包含2100個文件，所以是的，我明白這是太多了，

我發現這個suggestion

config = tf.ConfigProto() 
config.gpu_options.allocator_type = 'BFC' 
with tf.Session(config = config) as s:...... 

但是這並沒有解決問題！ 任何想法如何解決這個問題？

Answer 1

問題似乎是圖中的所有內容都是在GPU上完成的。您應該將CPU資源用於預處理功能以及GPU上的其他圖形。因此，使輸入處理函數（如getImage（）和隊列）可以在CPU而不是GPU上運行。基本上，當GPU在張量上工作時，CPU應該填充輸入流水線隊列，所以CPU和GPU都被有效地使用。 這在tensorflow性能指南中說明：

在CPU上放置預處理可導致 樣本/秒的6X +增加處理，這可能導致的 時間的1 /第六訓練。 https://www.tensorflow.org/performance/performance_guide

例如，您可以創建一個函數get_batch對CPU這樣運行：

def get_batch(dataset): 
     with tf.device('/cpu:0'): 
      'File Name Queue' 
      'Get image function implementation' 
      'Shuffle batch to make batches' 
    return image, labels 
train_imageBatch, train_labelBatch = get_batch('train_dataset') 
validation_imageBatch, validation_labelBatch = get_batch('valid_dataset') 

使用隊列時，還要檢查如何測試和驗證之間切換以下鏈接：Tensorflow Queues - Switching between train and validation data。您的代碼應該是這樣的：

# A bool tensor to figure out whether in training loop or tesing loop 
_is_train = tf.placeholder(dtype=tf.bool, name='is_train') 

# Select train or test database based on the _is_train tensor 
images = tf.cond(_is_train, lambda:train_imageBatch, lambda:validation_imageBatch) 
labels = tf.cond(_is_train, lambda:train_labelBatch, lambda:validation_labelBatch) 

train_op = ... 
... 
for step in num_steps: 

    # each step 
    summary, _ = sess.run([merged, train_step], fead_dict={_is_train:True} 
    ... 
    if (validate_step) 
     summary, acc = sess.run([merged,accuracy],feed_dict={_is_train:False) 
     ... 

爲了實現get_batch的，你可以看到從tensorflow這個例子：https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/how_tos/reading_data/fully_connected_reader.py。