芹菜：所有工人卡住了，如何診斷

我所有的芹菜工人都會被卡住。我無法弄清楚是什麼原因造成的，因爲inspect不工作，因爲所有的工人都很忙。芹菜：所有工人卡住了，如何診斷

celery inspect active 

    Error: No nodes replied within time constraint

是否有可能得到芹菜狀態，如活動任務，即使節點正在做的事情（這似乎是造成問題）？我能以某種方式啓動一個臨時工來獲得inspect的輸出嗎？

有什麼其他策略可以診斷這個問題？

芹菜4.x. Redis後端。

來源

2017-02-13 Mikko Ohtamaa

的問題是不完整的，請至少提供了芹菜，以及如何運行你的員工配置設置。 CPU負載高嗎？他們在執行任務時是卡住還是閒置？只是一個好建議 - 您可以使用Redis作爲後端，但不要將其用作代理（這可能會導致您遇到的問題）。你是否在同一臺機器或遠程（檢查網絡）上有經紀人？檢查代理日誌以查找任何連接損失。 – baldr

事實證明，這是一個僵局問題與芹菜+ GEVENT（惡猴補丁）+哨兵的烏鴉記錄。

https://github.com/getsentry/raven-python/issues/305

要診斷問題

你可以開始芹菜工人不同的隊列（-q，-n）參數，看看當工人掛起。即使某些工作組掛起，其他工作組仍然可能會對inspect查詢做出響應。

芹菜文件記錄可能顯示錯誤

2017-02-27 08:36:34,371 CRITI [celery.worker][DummyThread-6] Unrecoverable error: AttributeError("'NoneType' object has no attribute 'readline'",) 
Traceback (most recent call last): 
    File "/srv/pyramid/xxx/venv/lib/python3.5/site-packages/celery/worker/worker.py", line 203, in start 
    self.blueprint.start(self) 
    File "/srv/pyramid/xxx/venv/lib/python3.5/site-packages/celery/bootsteps.py", line 119, in start 
    step.start(parent) 
    File "/srv/pyramid/xxx/venv/lib/python3.5/site-packages/celery/bootsteps.py", line 370, in start 
    return self.obj.start() 
    File "/srv/pyramid/xxx/venv/lib/python3.5/site-packages/celery/worker/consumer/consumer.py", line 318, in start 
    blueprint.start(self) 
    File "/srv/pyramid/xxx/venv/lib/python3.5/site-packages/celery/bootsteps.py", line 119, in start 
    step.start(parent) 
    File "/srv/pyramid/xxx/venv/lib/python3.5/site-packages/celery/worker/consumer/consumer.py", line 594, in start 
    c.loop(*c.loop_args()) 
    File "/srv/pyramid/xxx/venv/lib/python3.5/site-packages/celery/worker/loops.py", line 118, in synloop 
    connection.drain_events(timeout=2.0) 
    File "/srv/pyramid/xxx/venv/lib/python3.5/site-packages/kombu/connection.py", line 301, in drain_events 
    return self.transport.drain_events(self.connection, **kwargs) 
    File "/srv/pyramid/xxx/venv/lib/python3.5/site-packages/kombu/transport/virtual/base.py", line 961, in drain_events 
    get(self._deliver, timeout=timeout) 
    File "/srv/pyramid/xxx/venv/lib/python3.5/site-packages/kombu/transport/redis.py", line 359, in get 
    ret = self.handle_event(fileno, event) 
    File "/srv/pyramid/xxx/venv/lib/python3.5/site-packages/kombu/transport/redis.py", line 341, in handle_event 
    return self.on_readable(fileno), self 
    File "/srv/pyramid/xxx/venv/lib/python3.5/site-packages/kombu/transport/redis.py", line 337, in on_readable 
    chan.handlers[type]() 
    File "/srv/pyramid/xxx/venv/lib/python3.5/site-packages/kombu/transport/redis.py", line 714, in _brpop_read 
    **options) 
    File "/srv/pyramid/xxx/venv/lib/python3.5/site-packages/redis/client.py", line 585, in parse_response 
    response = connection.read_response() 
    File "/srv/pyramid/xxx/venv/lib/python3.5/site-packages/redis/connection.py", line 577, in read_response 
    response = self._parser.read_response() 
    File "/srv/pyramid/xxx/venv/lib/python3.5/site-packages/redis/connection.py", line 238, in read_response 
    response = self._buffer.readline() 
AttributeError: 'NoneType' object has no attribute 'readline'

來源

2017-02-27 13:19:22

芹菜：所有工人卡住了，如何診斷

回答

相關問題