Tensorflow SEQ2SEQ培訓突然超慢

Question

我正在使用https://google.github.io/seq2seq/的代碼訓練seq2seq NMT（神經機器翻譯）。在我中斷訓練過程之後，重新啓動的過程變得非常慢（從1.2步/秒到0.07步/秒）。有其他人有這種經歷嗎？我該如何調試？我已經運行此數週，真的不想放棄...謝謝！〜

正常訓練的最後一行，

INFO:tensorflow:loss = 0.585205, step = 830853 (79.477 sec) 
INFO:tensorflow:global_step/sec: 1.24179 
INFO:tensorflow:loss = 0.267574, step = 830953 (80.529 sec) 

的超慢訓練前幾行...

INFO:tensorflow:global_step/sec: 0.0746058 
INFO:tensorflow:loss = 0.554718, step = 830854 (1340.379 sec) 

Answer 1

啊，經過仔細審查日誌後，似乎Cuda上的最新更新造成了這個問題。後來自動更新程序解決了這個問題。這裏是日誌..

2017-06-02 22:47:22.684229: E tensorflow/stream_executor/cuda/cuda_driver.cc:405] failed call to cuInit: CUDA_ERROR_NO_DEVICE 
2017-06-02 22:47:22.684260: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:158] retrieving CUDA diagnostic information for host: POSingularity 
2017-06-02 22:47:22.684276: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:165] hostname: POSingularity 
2017-06-02 22:47:22.684305: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:189] libcuda reported version is: 375.66.0 
2017-06-02 22:47:22.684331: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:369] driver version file contents: """NVRM version: NVIDIA UNIX x86_64 Kernel Module 375.51 Wed Mar 22 10:26:12 PDT 
2017 
GCC version: gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) 
""" 
2017-06-02 22:47:22.684352: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:193] kernel reported version is: 375.51.0 
2017-06-02 22:47:22.684368: E tensorflow/stream_executor/cuda/cuda_diagnostics.cc:303] kernel version 375.51.0 does not match DSO version 375.66.0 -- cannot find working devices in this configuration