導致神經網絡停滯在學習中的問題是什麼？

Question

我想知道神經網絡如何準確地逼近簡單函數，如多個變量中的標量值多項式。所以我有這些想法：

1. 修復了幾個變量的多項式，比如f（x_1，..，x_n）。
2. 使用numpy.random作爲訓練數據生成長度爲n的50000個向量。
3. 評估這些點的f（x），該值將被用作標籤。
4. 以相同的方式製作測試數據和標籤
5. 編寫一個神經網絡，看看它如何準確地估計f（x）對測試集。

這裏是tensorflow

import tensorflow as tf 
import numpy as np 

input_vector_length = int(10) 
output_vector_length = int(1) 
train_data_size = int(50000) 
test_data_size = int(10000) 
train_input_domain = [-10, 10] #Each component in an input vector is between -10 and 10 
test_input_domain = [-10, 10] 
iterations = 20000 
batch_size = 200 
regularizer = 0.01 
sess = tf.Session() 

x = tf.placeholder(tf.float32, shape=[None, input_vector_length], name="x") 
y = tf.placeholder(tf.float32, shape =[None, output_vector_length], name="y") 

function = tf.reduce_sum(x, 1) + 0.25*tf.pow(tf.reduce_sum(x,1), 2) + 0.025*tf.pow(tf.reduce_sum(x,1), 3) 

#make train data input 
train_input = (train_input_domain[1]-train_input_domain[0])*np.random.rand(train_data_size, input_vector_length) + train_input_domain[0] 

#make train data label 
train_label = sess.run(function, feed_dict = {x : train_input}) 
train_label = train_label.reshape(train_data_size, output_vector_length) 

#make test data input 
test_input = (test_input_domain[1]-test_input_domain[0])*np.random.rand(test_data_size, input_vector_length) + test_input_domain[0] 

#make test data label 
test_label = sess.run(function, feed_dict = {x : test_input}) 
test_label = test_label.reshape(test_data_size, output_vector_length) 

def weight_variables(shape, name): 
    initial = 10*tf.truncated_normal(shape, stddev=0.1) 
    return tf.Variable(initial) 
def bias_variables(shape, name): 
    initial = 10*tf.truncated_normal(shape, stddev=0.1) 
    return tf.Variable(initial) 
def take_this_batch(data, batch_index=[]): 
    A = [] 
    for i in range(len(batch_index)): 
     A.append(data[i]) 
    return A 

W_0 = weight_variables(shape=[input_vector_length, 10], name="W_0") 
B_0 = bias_variables(shape=[10], name="W_0") 
y_1 = tf.sigmoid(tf.matmul(x, W_0) + B_0) 


W_1 = weight_variables(shape=[10, 20], name="W_1") 
B_1 = bias_variables(shape=[20], name="B_1") 
y_2 = tf.sigmoid(tf.matmul(y_1, W_1) + B_1) 

W_2 = weight_variables(shape=[20,40], name="W_2") 
B_2 = bias_variables(shape=[40], name="B_2") 
y_3 = tf.sigmoid(tf.matmul(y_2, W_2) + B_2) 

keep_prob = tf.placeholder(tf.float32, name="keep_prob") 
y_drop = tf.nn.dropout(y_3, keep_prob) 

W_output = weight_variables(shape=[40, output_vector_length], name="W_output") 
B_output = bias_variables(shape=[output_vector_length], name="B_output") 
y_output = tf.matmul(y_drop, W_output) + B_output 


weight_sum = tf.reduce_sum(tf.square(W_0)) + tf.reduce_sum(tf.square(W_1)) + tf.reduce_sum(tf.square(W_2)) + tf.reduce_sum(tf.square(W_3)) 
cost = tf.reduce_mean(tf.square(y - y_output)) + regularizer*(weight_sum) 
train_step = tf.train.GradientDescentOptimizer(0.01).minimize(cost) 
error = cost 

sess.run(tf.initialize_all_variables()) 
with sess.as_default(): 
    for step in range(iterations): 
     batch_index = np.random.randint(low=0, high=train_data_size, size=batch_size) 
     batch_input = take_this_batch(train_input, batch_index) 
     batch_label = take_this_batch(train_label, batch_index) 
     train_step.run(feed_dict = {x : batch_input, y:batch_label, keep_prob:0.5}) 
     if step % 1000 == 0: 
      current_error = error.eval(feed_dict = {x:batch_input, y:batch_label, keep_prob:1.0}) 
      print("step %d, Current error is %f" % (step,current_error)) 


    print(error.eval(feed_dict={x:test_input, y:test_label, keep_prob:1.0})) 

簡單地說實現我的樣本的神經網絡，該神經網絡的性能是令人震驚！我的神經網絡有三個大小爲10,20和40的隱藏層。輸入層的大小爲10，輸出層的大小爲1.我使用了一個簡單的L^2代價函數，並且我用正方形的權重和調節器0.01。

在訓練階段，我注意到錯誤似乎卡住，拒絕下降。我想知道會出現什麼問題？非常感謝您閱讀這個長長的問題。任何建議表示讚賞。

Answer 1

由於您使用sigmoid作爲隱藏層中的激活函數，因此這些神經元的值會減少到（0,1）的範圍。因此，對這個網絡的輸入數據進行規範化是個好主意。