Tensorflow：「GraphDef不能大於2GB」。在分配變量後保存模型時發生錯誤

Question

我想使用預訓練模型熱烈啓動另一個模型，但有一點區別。簡單地說，我創建一個新模型，並使用預訓練模型權重爲變量賦予相同的名稱。但是，保存模型時發生錯誤。

Traceback (most recent call last): File "tf_test.py", line 23, in <module> save_path = saver.save(sess, "./model.ckpt") File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1308, in save self.export_meta_graph(meta_graph_filename) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1331, in export_meta_graph graph_def=ops.get_default_graph().as_graph_def(add_shapes=True), File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2268, in as_graph_def result, _ = self._as_graph_def(from_version, add_shapes) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2231, in _as_graph_def raise ValueError("GraphDef cannot be larger than 2GB.") ValueError: GraphDef cannot be larger than 2GB.

的示例代碼如下：

import tensorflow as tf 
import numpy as np 

v1 = tf.get_variable("L_enc", [400000, 1024]) 
v2 = tf.get_variable("L_dec", [400000, 1024]) 

init_op = tf.initialize_all_variables() 

saver = tf.train.Saver(tf.all_variables()) 

with tf.Session() as sess: 
    sess.run(init_op) 
    for v in tf.trainable_variables(): 
    embedding = np.random.uniform(-1, 1, (400000, 1024)) 
    sess.run(v.assign(embedding)) 
    # Save the variables to disk. 
    save_path = saver.save(sess, "./model.ckpt") 
    print("Model saved in file: %s" % save_path) 

Answer 1

有2GB的，因爲在protobuf的32位帶符號的大小的序列化個體張量 的硬性限制。

https://github.com/tensorflow/tensorflow/issues/4291

Answer 2

法布里奇奧correctly points out有關於協議緩衝區的大小硬2GB的限制，但爲什麼程序打這個限制，你可能想知道。該問題由這些線莖：

for v in tf.trainable_variables(): 
    embedding = np.random.uniform(-1, 1, (400000, 1024)) 
    sess.run(v.assign(embedding)) 

當執行命中v.assign(embedding)，新節點被添加到TensorFlow曲線圖。特別是，每個embedding陣列都被轉換爲tf.constant()張量，這個張量將非常大（根據我的估計大約爲328MB）。

避免這種情況的最佳方法是使用tf.train.Saver將以前模型中的變量直接加載到新模型中。由於模型可能具有不同的結構，因此可能需要指定從舊模型中的變量名稱到新模型中的tf.Variable對象的映射。

---

解決您的問題將是值分配給每個變量預先創建一個tf.placeholder()運的另一種方式。這可能需要你的代碼還多的重組，但對我下面的工作：

v1 = tf.get_variable("L_enc", [400000, 1024]) 
v2 = tf.get_variable("L_dec", [400000, 1024]) 

# Define a separate placeholder and assign op for each variable, so 
# that we can feed the initial value without adding it to the graph. 
vars = [v1, v2] 
placeholders = [tf.placeholder(tf.float32, shape=[400000, 1024]) for v in vars] 
assign_ops = [v.assign(p) for (v, p) in zip(vars, placeholders)] 

init_op = tf.global_variables_initializer() 

saver = tf.train.Saver(tf.all_variables()) 

with tf.Session() as sess: 
    sess.run(init_op) 
    for p, assign_op in zip(placeholders, assign_ops): 
    embedding = np.random.uniform(-1, 1, (400000, 1024)) 
    sess.run(assign_op, {p: embedding}) 

    # Save the variables to disk. 
    save_path = saver.save(sess, "./model.ckpt") 
    print("Model saved in file: %s" % save_path)