相同的批次但不同的批量生成不同的結果

Question

例如，我們有64 * 100的輸入數據將被髮送到張量流圖中，並且它將在輸入softmax或任何損失之前生成64 *（n_hidden節點）輸出功能。我們把1 * 100放在同一個圖中，結果應該是前一個輸出的第一行，但結果不是。我使用Mnist上的張量流量實例來測試比較。

''' 
A Multilayer Perceptron implementation example using TensorFlow library. 
This example is using the MNIST database of handwritten digits 
(http://yann.lecun.com/exdb/mnist/) 
Author: Aymeric Damien 
Project: https://github.com/aymericdamien/TensorFlow-Examples/ 
''' 

from __future__ import print_function 
import numpy as np 
# Import MNIST data 
from tensorflow.examples.tutorials.mnist import input_data 

mnist = input_data.read_data_sets("/tmp/data/", one_hot=True) 

import tensorflow as tf 



# Parameters 
learning_rate = 0.001 
training_epochs = 15 
batch_size = 100 
display_step = 1 

# Network Parameters 
n_hidden_1 = 256 # 1st layer number of features 
n_hidden_2 = 256 # 2nd layer number of features 
n_input = 784 # MNIST data input (img shape: 28*28) 
n_classes = 10 # MNIST total classes (0-9 digits) 

# tf Graph input 
x = tf.placeholder("float", [None, n_input]) 
y = tf.placeholder("float", [None, n_classes]) 




# Create model 
def multilayer_perceptron(x, weights, biases): 
    # Hidden layer with RELU activation 
    layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['b1']) 
    layer_1 = tf.nn.relu(layer_1) 
    # Hidden layer with RELU activation 
    layer_2 = tf.add(tf.matmul(layer_1, weights['h2']), biases['b2']) 
    layer_2 = tf.nn.relu(layer_2) 
    # Output layer with linear activation 
    out_layer = tf.matmul(layer_2, weights['out']) + biases['out'] 
    return out_layer 

# Store layers weight & bias 
weights = { 
    'h1': tf.Variable(tf.random_normal([n_input, n_hidden_1]), name ='layer1'), 
    'h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2]), name = 'layer2'), 
    'out': tf.Variable(tf.random_normal([n_hidden_2, n_classes]), name = 'layer3') 
} 
biases = { 
    'b1': tf.Variable(tf.random_normal([n_hidden_1]), name = 'layer1_b'), 
    'b2': tf.Variable(tf.random_normal([n_hidden_2]), name = 'layer2_b'), 
    'out': tf.Variable(tf.random_normal([n_classes]), name = 'layer3_b') 
} 

# Construct model 
pred = multilayer_perceptron(x, weights, biases) 

# Define loss and optimizer 
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(pred, y)) 
#optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost) 
var = tf.all_variables() 
trainer = tf.train.AdamOptimizer(learning_rate=learning_rate) 
grads = trainer.compute_gradients(cost, var) 
update = trainer.apply_gradients(grads) 

# Initializing the variables 
init = tf.initialize_all_variables() 

# Launch the graph 
with tf.Session() as sess: 
    sess.run(init) 

    # Training cycle 
    for epoch in range(training_epochs): 
     avg_cost = 0. 
     total_batch = int(mnist.train.num_examples/batch_size) 
     # Loop over all batches 
     for i in range(total_batch): 
      batch_x, batch_y = mnist.train.next_batch(batch_size) 
      # Run optimization op (backprop) and cost op (to get loss value) 
      #_, c = sess.run([optimizer, cost], feed_dict={x: batch_x, y: batch_y}) 
      #c, v,grad, Pred, bi = sess.run([cost, var,grads, pred, biases], feed_dict={x: batch_x, y: batch_y}) 

      Pred_2 = sess.run(pred, feed_dict={x: batch_x, y: batch_y}) 

      Pred_1 = sess.run(pred , feed_dict={x: batch_x[0:1,:], y: batch_y[0:1]}) 
      print(Pred_2[0] == Pred_1) 
      # Compute average loss 
      avg_cost += c/total_batch 
     # Display logs per epoch step 
     if epoch % display_step == 0: 
      print("Epoch:", '%04d' % (epoch+1), "cost=", \ 
       "{:.9f}".format(avg_cost)) 
      # print(len(v)) 
      # g1 = np.array(grad[0]) 
      # g2 = np.array(grad[1]) 
      # g3 = np.array(grad[2]) 
      # g4 = np.array(grad[3]) 
      # g5 = np.array(grad[4]) 
      # g6 = np.array(grad[5]) 
      # print(g1.shape) 
      # print(g2.shape) 
      # print(g3.shape) 
      # print(g4.shape) 
      # print(g5.shape) 
      # print(g6.shape) 
      # print(g6[0,:]) 
      # print(g6[1,:]) 
      # print(bi['out']) 

      #print(type(updating)) 
    print("Optimization Finished!") 

    # Test model 
    correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1)) 
    # Calculate accuracy 
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float")) 
    print("Accuracy:", accuracy.eval({x: mnist.test.images, y: mnist.test.labels})) 

print（Pred_2 [0] == Pred_1）應該是相同的，但它們不是。它很奇怪。

Answer 1

如果您的權重和偏差初始化是隨機的，且每次的梯度不同，則梯度下降路徑應該不同，並且可能需要朝不同的最小值的路徑。