NaN在張量流中的損失LSTM模型

Question

下面的網絡代碼應該是您的經典簡單LSTM語言模型，在一段時間後開始輸出nan損失...在我的訓練集上需要幾個小時，我無法複製它很容易在較小的數據集上。但它總是在嚴肅的訓練中發生。

Sparse_softmax_with_cross_entropy應該在數值上是穩定的，所以它不可能是原因...但除此之外，我沒有看到任何其他可能導致圖形問題的節點。可能是什麼問題呢？

class MyLM(): 
    def __init__(self, batch_size, embedding_size, hidden_size, vocab_size): 
     self.x = tf.placeholder(tf.int32, [batch_size, None]) # [batch_size, seq-len] 
     self.lengths = tf.placeholder(tf.int32, [batch_size]) # [batch_size] 

     # remove padding. [batch_size * seq_len] -> [batch_size * sum(lengths)] 
     mask = tf.sequence_mask(self.lengths) # [batch_size, seq_len] 
     mask = tf.cast(mask, tf.int32) # [batch_size, seq_len] 
     mask = tf.reshape(mask, [-1]) # [batch_size * seq_len] 

     # remove padding + last token. [batch_size * seq_len] -> [batch_size * sum(lengths-1)] 
     mask_m1 = tf.cast(tf.sequence_mask(self.lengths - 1, maxlen=tf.reduce_max(self.lengths)), tf.int32) # [batch_size, seq_len] 
     mask_m1 = tf.reshape(mask_m1, [-1]) # [batch_size * seq_len] 

     # remove padding + first token. [batch_size * seq_len] -> [batch_size * sum(lengths-1)] 
     m1_mask = tf.cast(tf.sequence_mask(self.lengths - 1), tf.int32) # [batch_size, seq_len-1] 
     m1_mask = tf.concat([tf.cast(tf.zeros([batch_size, 1]), tf.int32), m1_mask], axis=1) # [batch_size, seq_len] 
     m1_mask = tf.reshape(m1_mask, [-1]) # [batch_size * seq_len] 

     embedding = tf.get_variable("TokenEmbedding", shape=[vocab_size, embedding_size]) 
     x_embed = tf.nn.embedding_lookup(embedding, self.x) # [batch_size, seq_len, embedding_size] 

     lstm = tf.nn.rnn_cell.LSTMCell(hidden_size, use_peepholes=True) 

     # outputs shape: [batch_size, seq_len, hidden_size] 
     outputs, final_state = tf.nn.dynamic_rnn(lstm, x_embed, dtype=tf.float32, 
               sequence_length=self.lengths) 
     outputs = tf.reshape(outputs, [-1, hidden_size]) # [batch_size * seq_len, hidden_size] 

     w = tf.get_variable("w_out", shape=[hidden_size, vocab_size]) 
     b = tf.get_variable("b_out", shape=[vocab_size]) 
     logits_padded = tf.matmul(outputs, w) + b # [batch_size * seq_len, vocab_size] 
     self.logits = tf.dynamic_partition(logits_padded, mask_m1, 2)[1] # [batch_size * sum(lengths-1), vocab_size] 

     predict = tf.argmax(logits_padded, axis=1) # [batch_size * seq_len] 
     self.predict = tf.dynamic_partition(predict, mask, 2)[1] # [batch_size * sum(lengths)] 

     flat_y = tf.dynamic_partition(tf.reshape(self.x, [-1]), m1_mask, 2)[1] # [batch_size * sum(lengths-1)] 

     self.cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=self.logits, labels=flat_y) 
     self.cost = tf.reduce_mean(self.cross_entropy) 
     self.train_step = tf.train.AdamOptimizer(learning_rate=0.01).minimize(self.cost) 

Answer 1

這可能是exploding gradients，其中梯度可以在LSTMs反向傳播期間發生爆炸的情況下，得到的數字溢出。處理爆炸梯度的常用技術是執行Gradient Clipping。