優化在tensorflow非線性不起作用工作

Question

我是新來tensorflow並試圖實現類似一個簡單的單層線性網絡https://www.tensorflow.org/get_started/mnist/beginners

x = tf.placeholder(tf.float32, [None, IN_SIZE], name="input") 

    W1 = tf.Variable(tf.zeros([IN_SIZE, OUT_SIZE]), name="Weight1") 
    b1 = tf.Variable(tf.zeros([OUT_SIZE]), name="bias1") 
    y = tf.matmul(x, W1) + b1 

    y_ = tf.placeholder(tf.float32, [None, OUT_SIZE], name="target") 

    cross_entropy = tf.reduce_mean( 
     tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y)) 
    train_step = tf.train.AdamOptimizer(1e-3).minimize(cross_entropy) 

該項目按預期工作和我有沒有問題那。但是，我嘗試添加另一個圖層，但只發現W1,b1,W2瞭解的都是零矩陣，只有偏差b2包含非零值。下面是我修改的網絡

x = tf.placeholder(tf.float32, [None, IN_SIZE], name="input") 

    W1 = tf.Variable(tf.zeros([IN_SIZE, L1_SIZE]), name="Weight1") 
    b1 = tf.Variable(tf.zeros([L1_SIZE]), name="bias1") 
    y = tf.matmul(x, W1) + b1 

    W2 = tf.Variable(tf.zeros([L1_SIZE, OUT_SIZE]), name="Weight2") 
    b2 = tf.Variable(tf.zeros([OUT_SIZE]), name="bias2") 
    y = tf.nn.relu(y) 
    y = tf.matmul(y, W2) + b2 

    # Define loss and optimizer 
    y_ = tf.placeholder(tf.float32, [None, OUT_SIZE], name="target") 

    cross_entropy = tf.reduce_mean( 
     tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y)) 
    train_step = tf.train.AdamOptimizer(1e-3).minimize(cross_entropy) 

Answer 1

的問題是，如果你有一個零之前RELU初始化權重矩陣梯度將永遠是零，沒有學習會發生。你需要做隨機初始化。