Theano和Celery：工人過早退出：信號11（SIGSEGV）

我正在構建一個網絡應用程序，我通過從客戶端發送ajax請求開始在服務器端使用theano實現的神經網絡的培訓。顯然，我不想等到服務器完全訓練網絡，以便將答案發回給我的客戶，因爲這將是長期的方式。Theano和Celery：工人過早退出：信號11（SIGSEGV）

所以我想出了芹菜，使我能夠在服務器端做異步代碼。我用命令celery -A CBIR worker -l info運行芹菜工人。

[2015-12-14 19:15:06,790: ERROR/MainProcess] Process 'Worker-3' pid:1610 exited with 'signal 11 (SIGSEGV)' 
[2015-12-14 19:15:07,001: ERROR/MainProcess] Task fit[ac40d4d4-5b56-4278-b270-647ef76f3a49] raised unexpected: WorkerLostError('Worker exited prematurely: signal 11 (SIGSEGV).',) 
Traceback (most recent call last): 
File "/Users/leo/anaconda/envs/ImgRet/lib/python3.5/site-packages/billiard/pool.py", line 1175, in mark_as_worker_losthuman_status(exitcode)), 
billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 11 (SIGSEGV).

我一直在尋找，爲什麼會出現這個錯誤，從我的理解，我的代碼：不幸的是我一個工人正在運行我的任務（這是訓練使用theano我的網絡）每次收到以下消息運行正在遭受內存泄漏。我不明白的是爲什麼我的代碼在不使用芹菜時運行沒有問題，但是使用芹菜時出現這個錯誤？

而最重要的是，我不知道如何解決這個問題。我用LLDB看生成的轉儲文件，這裏是回溯我：

thread #1: tid = 0x0000, 0x00007fff93b4a9b3 libdispatch.dylib`dispatch_group_async + 533, stop reason = signal SIGSTOP 
* frame #0: 0x00007fff93b4a9b3 libdispatch.dylib`dispatch_group_async + 533 
frame #1: 0x00007fff7c5b8d40 libdispatch.dylib`_dispatch_root_queues + 1280 
frame #2: 0x00007fff9519b228 libBLAS.dylib`APL_dgemm + 1100 
frame #3: 0x00007fff951d27aa libBLAS.dylib`cblas_dgemm + 1420 
frame #4: 0x0000000104beeb18 multiarray.cpython-35m-darwin.so`gemm + 200 
frame #5: 0x0000000104bee3b9 multiarray.cpython-35m-darwin.so`cblas_matrixproduct + 3097 
frame #6: 0x0000000104bc01af multiarray.cpython-35m-darwin.so`PyArray_MatrixProduct2 + 207 
frame #7: 0x0000000104bc4808 multiarray.cpython-35m-darwin.so`array_matrixproduct + 264 
frame #8: 0x00000001000671a9 libpython3.5m.dylib`PyCFunction_Call + 281 
frame #9: 0x00000001000f2fbd libpython3.5m.dylib`PyEval_EvalFrameEx + 32029 
frame #10: 0x00000001000f4053 libpython3.5m.dylib`PyEval_EvalFrameEx + 36275 
frame #11: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400 
frame #12: 0x00000001000f4ef7 libpython3.5m.dylib`PyEval_EvalCodeEx + 71 
frame #13: 0x0000000100041d2a libpython3.5m.dylib`function_call + 186 
frame #14: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99 
frame #15: 0x00000001000e95a7 libpython3.5m.dylib`PyEval_CallObjectWithKeywords + 87 
frame #16: 0x00000001042fae3a lazylinker_ext.so`pycall(self=0x0000000108fad3d8, node_idx=13, verbose=0) + 442 at mod.cpp:510 
frame #17: 0x00000001042fa869 lazylinker_ext.so`lazy_rec_eval(self=0x0000000108fad3d8, var_idx=24, one=0x000000010026cf60, zero=0x000000010026cf40) + 2089 at mod.cpp:704 
frame #18: 0x00000001042fa789 lazylinker_ext.so`lazy_rec_eval(self=0x0000000108fad3d8, var_idx=28, one=0x000000010026cf60, zero=0x000000010026cf40) + 1865 at mod.cpp:690 
frame #19: 0x00000001042fa16d lazylinker_ext.so`lazy_rec_eval(self=0x0000000108fad3d8, var_idx=30, one=0x000000010026cf60, zero=0x000000010026cf40) + 301 at mod.cpp:576 
frame #20: 0x00000001042fa789 lazylinker_ext.so`lazy_rec_eval(self=0x0000000108fad3d8, var_idx=33, one=0x000000010026cf60, zero=0x000000010026cf40) + 1865 at mod.cpp:690 
frame #21: 0x00000001042fa789 lazylinker_ext.so`lazy_rec_eval(self=0x0000000108fad3d8, var_idx=36, one=0x000000010026cf60, zero=0x000000010026cf40) + 1865 at mod.cpp:690 
frame #22: 0x00000001042fa789 lazylinker_ext.so`lazy_rec_eval(self=0x0000000108fad3d8, var_idx=41, one=0x000000010026cf60, zero=0x000000010026cf40) + 1865 at mod.cpp:690 
frame #23: 0x00000001042fa789 lazylinker_ext.so`lazy_rec_eval(self=0x0000000108fad3d8, var_idx=42, one=0x000000010026cf60, zero=0x000000010026cf40) + 1865 at mod.cpp:690 
frame #24: 0x00000001042f83db lazylinker_ext.so`CLazyLinker_call(_self=0x0000000108fad3d8, args=0x0000000100382048, kwds=0x0000000000000000) + 811 at mod.cpp:838 
frame #25: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99 
frame #26: 0x00000001000ed08c libpython3.5m.dylib`PyEval_EvalFrameEx + 7660 
frame #27: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400 
frame #28: 0x00000001000f4ef7 libpython3.5m.dylib`PyEval_EvalCodeEx + 71 
frame #29: 0x0000000100041d2a libpython3.5m.dylib`function_call + 186 
frame #30: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99 
frame #31: 0x000000010002a79c libpython3.5m.dylib`method_call + 140 
frame #32: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99 
frame #33: 0x0000000100080743 libpython3.5m.dylib`slot_tp_call + 67 
frame #34: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99 
frame #35: 0x00000001000ed08c libpython3.5m.dylib`PyEval_EvalFrameEx + 7660 
frame #36: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400 
frame #37: 0x00000001000f3d26 libpython3.5m.dylib`PyEval_EvalFrameEx + 35462 
frame #38: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400 
frame #39: 0x00000001000f4ef7 libpython3.5m.dylib`PyEval_EvalCodeEx + 71 
frame #40: 0x0000000100041d2a libpython3.5m.dylib`function_call + 186 
frame #41: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99 
frame #42: 0x00000001000eff0b libpython3.5m.dylib`PyEval_EvalFrameEx + 19563 
frame #43: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400 
frame #44: 0x00000001000f4ef7 libpython3.5m.dylib`PyEval_EvalCodeEx + 71 
frame #45: 0x0000000100041d2a libpython3.5m.dylib`function_call + 186 
frame #46: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99 
frame #47: 0x000000010002a79c libpython3.5m.dylib`method_call + 140 
frame #48: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99 
frame #49: 0x0000000100080743 libpython3.5m.dylib`slot_tp_call + 67 
frame #50: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99 
frame #51: 0x00000001000eff0b libpython3.5m.dylib`PyEval_EvalFrameEx + 19563 
frame #52: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400 
frame #53: 0x00000001000f3d26 libpython3.5m.dylib`PyEval_EvalFrameEx + 35462 
frame #54: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400 
frame #55: 0x00000001000f4ef7 libpython3.5m.dylib`PyEval_EvalCodeEx + 71 
frame #56: 0x0000000100041d2a libpython3.5m.dylib`function_call + 186 
frame #57: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99 
frame #58: 0x00000001000eff0b libpython3.5m.dylib`PyEval_EvalFrameEx + 19563 
frame #59: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400 
frame #60: 0x00000001000f3d26 libpython3.5m.dylib`PyEval_EvalFrameEx + 35462 
frame #61: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400 
frame #62: 0x00000001000f3d26 libpython3.5m.dylib`PyEval_EvalFrameEx + 35462 
frame #63: 0x00000001000f4053 libpython3.5m.dylib`PyEval_EvalFrameEx + 36275 
frame #64: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400 
frame #65: 0x00000001000f4ef7 libpython3.5m.dylib`PyEval_EvalCodeEx + 71 
frame #66: 0x0000000100041d2a libpython3.5m.dylib`function_call + 186 
frame #67: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99 
frame #68: 0x000000010002a79c libpython3.5m.dylib`method_call + 140 
frame #69: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99 
frame #70: 0x0000000100080471 libpython3.5m.dylib`slot_tp_init + 81 
frame #71: 0x000000010007b114 libpython3.5m.dylib`type_call + 212 
frame #72: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99 
frame #73: 0x00000001000ed08c libpython3.5m.dylib`PyEval_EvalFrameEx + 7660 
frame #74: 0x00000001000f4053 libpython3.5m.dylib`PyEval_EvalFrameEx + 36275 
frame #75: 0x00000001000f4053 libpython3.5m.dylib`PyEval_EvalFrameEx + 36275 
frame #76: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400 
frame #77: 0x00000001000f4ef7 libpython3.5m.dylib`PyEval_EvalCodeEx + 71 
frame #78: 0x0000000100041d2a libpython3.5m.dylib`function_call + 186 
frame #79: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99 
frame #80: 0x00000001000eff0b libpython3.5m.dylib`PyEval_EvalFrameEx + 19563 
frame #81: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400 
frame #82: 0x00000001000f4ef7 libpython3.5m.dylib`PyEval_EvalCodeEx + 71 
frame #83: 0x0000000100041d2a libpython3.5m.dylib`function_call + 186 
frame #84: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99 
frame #85: 0x000000010002a79c libpython3.5m.dylib`method_call + 140 
frame #86: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99 
frame #87: 0x0000000100080471 libpython3.5m.dylib`slot_tp_init + 81 
frame #88: 0x000000010007b114 libpython3.5m.dylib`type_call + 212 
frame #89: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99 
frame #90: 0x00000001000eff0b libpython3.5m.dylib`PyEval_EvalFrameEx + 19563 
frame #91: 0x00000001000f4053 libpython3.5m.dylib`PyEval_EvalFrameEx + 36275 
frame #92: 0x00000001000f4053 libpython3.5m.dylib`PyEval_EvalFrameEx + 36275 
frame #93: 0x00000001000f4053 libpython3.5m.dylib`PyEval_EvalFrameEx + 36275 
frame #94: 0x00000001000f4053 libpython3.5m.dylib`PyEval_EvalFrameEx + 36275 
frame #95: 0x00000001000f4053 libpython3.5m.dylib`PyEval_EvalFrameEx + 36275 
frame #96: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400 
frame #97: 0x00000001000f4ef7 libpython3.5m.dylib`PyEval_EvalCodeEx + 71 
frame #98: 0x0000000100041d2a libpython3.5m.dylib`function_call + 186 
frame #99: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99 
frame #100: 0x00000001000eff0b libpython3.5m.dylib`PyEval_EvalFrameEx + 19563 
frame #101: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400 
frame #102: 0x00000001000f4ef7 libpython3.5m.dylib`PyEval_EvalCodeEx + 71 
frame #103: 0x0000000100041d2a libpython3.5m.dylib`function_call + 186 
frame #104: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99 
frame #105: 0x000000010002a79c libpython3.5m.dylib`method_call + 140 
frame #106: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99 
frame #107: 0x0000000100080743 libpython3.5m.dylib`slot_tp_call + 67 
frame #108: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99 
frame #109: 0x00000001000eff0b libpython3.5m.dylib`PyEval_EvalFrameEx + 19563 
frame #110: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400 
frame #111: 0x00000001000f3d26 libpython3.5m.dylib`PyEval_EvalFrameEx + 35462 
frame #112: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400 
frame #113: 0x00000001000f3d26 libpython3.5m.dylib`PyEval_EvalFrameEx + 35462 
frame #114: 0x00000001000f4053 libpython3.5m.dylib`PyEval_EvalFrameEx + 36275 
frame #115: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400 
frame #116: 0x00000001000f3d26 libpython3.5m.dylib`PyEval_EvalFrameEx + 35462 
frame #117: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400 
frame #118: 0x00000001000f3d26 libpython3.5m.dylib`PyEval_EvalFrameEx + 35462 
frame #119: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400 
frame #120: 0x00000001000f3d26 libpython3.5m.dylib`PyEval_EvalFrameEx + 35462 
frame #121: 0x00000001000f4053 libpython3.5m.dylib`PyEval_EvalFrameEx + 36275 
frame #122: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400 
frame #123: 0x00000001000f4f51 libpython3.5m.dylib`PyEval_EvalCode + 81 
frame #124: 0x0000000100123d4e libpython3.5m.dylib`PyRun_FileExFlags + 206 
frame #125: 0x0000000100123fef libpython3.5m.dylib`PyRun_SimpleFileExFlags + 447 
frame #126: 0x000000010013c7d7 libpython3.5m.dylib`Py_Main + 3479 
frame #127: 0x0000000100000e92 python3`main + 418 
frame #128: 0x0000000100000cc4 python3`start + 52

我真的不知道如何解釋這個回溯。預先感謝任何幫助！

來源

2015-12-14 Vetouz

是否提供核心轉儲？你有運行gdb嗎？ – scytale

我想使用gdb，但我想我錯過了一些東西。我所做的是運行worker'python manage.py芹菜worker'，然後執行'gdb python '並在gdb shell中輸入c。但在此之後，我的gdb外殼顯示Continuing，我無法再引用任何命令。當我的工作人員退出分段錯誤時，它不顯示任何內容。 – Vetouz

否 - 在segfault之後系統上應該有一個核心轉儲 - 分析它會更容易。檢查'ulimit -c'和'/ proc/sys/kernel/core_pattern'的值 – scytale

如果有人遇到同樣的問題，結果是一個解決方法是在任務中內聯導入theano庫，而不是在模塊級別。

這樣：

import baz 
import bar 

@app.task 
def foo(): 
    import theano 

    # do something with theano

檢查here更多解釋

來源

2015-12-18 17:17:58 Vetouz

FWIW，這也與sklearn.cluster.KMeans發生。如果我使用threading.Thread自己創建線程，則工作正常。如果我嘗試呼叫芹菜工人的fit，則獲得sig11。

我還沒有與sklearn.linear_model.LogisticRegression,Ridge或LinearRegression相同的問題。

來源

2017-01-20 00:08:21

這不是問題的答案，而是評論/新問題。一旦你有足夠的聲望（50），你將能夠評論任何帖子。與此同時，您可以通過提出自己的問題或提供有關您認識的主題的答案來獲得聲譽。 –

Theano和Celery：工人過早退出：信號11（SIGSEGV）

回答

相關問題