連體模型LSTM網絡出現故障時使用tensorflow

Question

數據集描述

數據集包含了一組問題對，如果問題是相同告訴標籤訓練。例如

「我如何閱讀和查找我的YouTube評論？」 ，「我怎樣才能看到我所有的 Youtube評論？」 ，「1」

該模型的目標是確定給定的問題對是相同還是不同。

方法

我創建了一個Siamese network來識別，如果兩個問題都是一樣的。下面是該模型：

with graph.as_default(): 
    diff = tf.sqrt(tf.reduce_sum(tf.square(tf.subtract(question1_outputs[:, -1, :], question2_outputs[:, -1, :])), reduction_indices=1)) 

    margin = tf.constant(1.) 
    labels = tf.to_float(labels) 
    match_loss = tf.expand_dims(tf.square(diff, 'match_term'), 0) 
    mismatch_loss = tf.expand_dims(tf.maximum(0., tf.subtract(margin, tf.square(diff)), 'mismatch_term'), 0) 

    loss = tf.add(tf.matmul(labels, match_loss), tf.matmul((1 - labels), mismatch_loss), 'loss_add') 
    distance = tf.reduce_mean(loss) 

    optimizer = tf.train.AdamOptimizer(learning_rate).minimize(distance) 

以下是代碼訓練模型：

with graph.as_default(): 
    saver = tf.train.Saver() 

with tf.Session(graph=graph) as sess: 
    sess.run(tf.global_variables_initializer(), feed_dict={embedding_placeholder: embedding_matrix}) 

    iteration = 1 
    for e in range(epochs): 
     summary_writer = tf.summary.FileWriter('/Users/mithun/projects/kaggle/quora_question_pairs/logs', sess.graph) 
     summary_writer.add_graph(sess.graph) 

     for ii, (x1, x2, y) in enumerate(get_batches(question1_train, question2_train, label_train, batch_size), 1): 
      feed = {question1_inputs: x1, 
        question2_inputs: x2, 
        labels: y[:, None], 
        keep_prob: 0.9 
        } 
      loss1 = sess.run([distance], feed_dict=feed) 

      if iteration%5==0: 
       print("Epoch: {}/{}".format(e, epochs), 
         "Iteration: {}".format(iteration), 
         "Train loss: {:.3f}".format(loss1)) 

      if iteration%50==0: 
       val_acc = [] 
       for x1, x2, y in get_batches(question1_val, question2_val, label_val, batch_size): 
        feed = {question1_inputs: x1, 
          question2_inputs: x2, 
          labels: y[:, None], 
          keep_prob: 1 
          } 
        batch_acc = sess.run([accuracy], feed_dict=feed) 
        val_acc.append(batch_acc) 
       print("Val acc: {:.3f}".format(np.mean(val_acc))) 
      iteration +=1 

    saver.save(sess, "checkpoints/quora_pairs.ckpt") 

我已經訓練上述模型

graph = tf.Graph() 

with graph.as_default(): 
    embedding_placeholder = tf.placeholder(tf.float32, shape=embedding_matrix.shape, name='embedding_placeholder') 
    with tf.variable_scope('siamese_network') as scope: 
     labels = tf.placeholder(tf.int32, [batch_size, None], name='labels') 
     keep_prob = tf.placeholder(tf.float32, name='question1_keep_prob') 

     with tf.name_scope('question1') as question1_scope: 
      question1_inputs = tf.placeholder(tf.int32, [batch_size, seq_len], name='question1_inputs') 

      question1_embedding = tf.get_variable(name='embedding', initializer=embedding_placeholder, trainable=False) 
      question1_embed = tf.nn.embedding_lookup(question1_embedding, question1_inputs) 

      question1_lstm = tf.contrib.rnn.BasicLSTMCell(lstm_size) 
      question1_drop = tf.contrib.rnn.DropoutWrapper(question1_lstm, output_keep_prob=keep_prob) 
      question1_multi_lstm = tf.contrib.rnn.MultiRNNCell([question1_drop] * lstm_layers) 

      q1_initial_state = question1_multi_lstm.zero_state(batch_size, tf.float32) 

      question1_outputs, question1_final_state = tf.nn.dynamic_rnn(question1_multi_lstm, question1_embed, initial_state=q1_initial_state) 

     scope.reuse_variables() 

     with tf.name_scope('question2') as question2_scope: 
      question2_inputs = tf.placeholder(tf.int32, [batch_size, seq_len], name='question2_inputs') 

      question2_embedding = question1_embedding 
      question2_embed = tf.nn.embedding_lookup(question2_embedding, question2_inputs) 

      question2_lstm = tf.contrib.rnn.BasicLSTMCell(lstm_size) 
      question2_drop = tf.contrib.rnn.DropoutWrapper(question2_lstm, output_keep_prob=keep_prob) 
      question2_multi_lstm = tf.contrib.rnn.MultiRNNCell([question2_drop] * lstm_layers) 

      q2_initial_state = question2_multi_lstm.zero_state(batch_size, tf.float32) 

      question2_outputs, question2_final_state = tf.nn.dynamic_rnn(question2_multi_lstm, question2_embed, initial_state=q2_initial_state) 

使用RNN輸出計算餘弦距離與約10,000標記的數據。但是，準確度在0.630左右停滯不前，奇怪的是所有迭代中的驗證準確性都是相同的。

lstm_size = 64 
lstm_layers = 1 
batch_size = 128 
learning_rate = 0.001 

我創建模型的方式有什麼問題嗎？

Answer 1

這是一個非平衡數據集的常見問題，例如您正在使用的最近發佈的Quora數據集。由於Quora數據集不平衡（〜63％的負數和〜37％的正數示例），您需要對權重進行適當的初始化。如果不進行初始化，您的解決方案將會停留在局部最小值，並且它將訓練以僅預測負面類別。因此，準確率爲63％，因爲這是驗證數據中「不相似」問題的百分比。如果你檢查你的驗證集上得到的結果，你會注意到它預測了所有的零。 He等人提出的截斷的正態分佈http://arxiv.org/abs/1502.01852是初始化權重的良好替代方案。