Celery連接代理丟失使CPU使用率達到100％

當MainProcess失去與代理的連接時，我們一直在使用Celery獲得奇怪的問題。 Celery產生下面的錯誤，然後開始使用100％的CPU。工人們仍然會繼續良好的工作。我可以看到，RabbitMQ認爲連接超時。在更新到芹菜3後，我們經常得到這些錯誤。Celery連接代理丟失使CPU使用率達到100％

我有一種感覺，它與非阻塞消息有關，但我並沒有真正在理解代碼方面取得進展。

有沒有辦法檢測到這些更早或阻止芹菜使用100％的CPU？

芹菜：3.0.4
AMQP：1.0.10
RabbitMQ的：2.8.4

時間戳相距2小時因爲RabbitMQ的報告GMT和芹菜本地時間。

芹菜錯誤

[2013-05-09 18:20:20,204: ERROR/MainProcess] Consumer: Connection to broker lost. Trying to re-establish the connection... 
Traceback (most recent call last): 
    File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 369, in start 
    self.consume_messages() 
    File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 450, in consume_messages 
    readers[fileno](fileno, event) 
    File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 290, in drain_nowait 
    self.drain_events(timeout=0) 
    File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 279, in drain_events 
    return self.transport.drain_events(self.connection, **kwargs) 
    File "/usr/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 91, in drain_events 
    return connection.drain_events(**kwargs) 
    File "/usr/local/lib/python2.7/site-packages/amqp/connection.py", line 266, in drain_events 
    chanmap, None, timeout=timeout, 
    File "/usr/local/lib/python2.7/site-packages/amqp/connection.py", line 328, in _wait_multiple 
    channel, method_sig, args, content = read_timeout(timeout) 
    File "/usr/local/lib/python2.7/site-packages/amqp/connection.py", line 299, in read_timeout 
    return self.method_reader.read_method() 
    File "/usr/local/lib/python2.7/site-packages/amqp/method_framing.py", line 187, in read_method 
    raise m 
IOError: Socket closed

RabbitMQ的錯誤

=ERROR REPORT==== 9-May-2013::16:20:20 === 
closing AMQP connection <0.1813.0> (192.168.201.104:12809 -> 192.168.201.104:5672): 
{timeout,running}

來源

2013-05-10 Exelian

我轉身經紀人hearbeats關閉這似乎已經固定的問題。但是，我不太確定，因爲我沒有辦法重現錯誤。

來源

2013-05-16 06:55:42 Exelian

使用SQS經紀人爲我工作。謝謝！ – samuelgrigolato 2017-10-05 13:24:54

Celery連接代理丟失使CPU使用率達到100％

回答

相關問題