當通過'eta'選項安排時不考慮Celery rate_limit

我想安排一項任務在將來的不同時間運行。我在調用任務時嘗試使用eta選項來執行此操作。這部分工作正常，但同時，我想對任務進行限制。我無法做到這一點：安排任務在未來運行，同時也讓芹菜尊重費率限制。當通過'eta'選項安排時不考慮Celery rate_limit

下面是一個例子（rate_limit_eta.py）

from celery import Celery 
from datetime import datetime, timedelta 

app = Celery('rate_limit_eta', broker='amqp://[email protected]//') 

@app.task(rate_limit='4/m') 
def my_task(): 
    print 'execution timestamp: {}'.format(datetime.utcnow()) 


if __name__ == '__main__': 

    now = datetime.utcnow() 
    d = now + timedelta(seconds=5) 
    print now 

    for i in range(10): 
     my_task.apply_async(eta=d)

在該循環中，當我稱之爲 'my_task' 與ETA = d，任務得到儘可能快地執行。 rate_limit不受尊重。

my_task.apply_async(eta=d) 


[2015-06-12 13:45:15,750: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.748498 
[2015-06-12 13:45:15,757: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.757757 
[2015-06-12 13:45:15,760: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.759992 
[2015-06-12 13:45:15,763: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.763614 
[2015-06-12 13:45:15,766: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.765996 
[2015-06-12 13:45:15,768: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.768424 
[2015-06-12 13:45:15,771: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.771079 
[2015-06-12 13:45:15,774: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.774561 
[2015-06-12 13:45:15,777: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.777425 
[2015-06-12 13:45:15,780: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.780207

但是，如果我沒有安排任務（即沒有eta），那麼速率限制是強制執行的。

my_task.apply_async() 

[2015-06-12 13:47:05,844: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:05.843872 
[2015-06-12 13:47:05,850: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:05.850502 
[2015-06-12 13:47:21,919: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:21.919120 
[2015-06-12 13:47:35,937: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:35.937060 
[2015-06-12 13:47:51,959: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:51.959174 
[2015-06-12 13:48:05,976: WARNING/Worker-1] execution timestamp: 2015-06-12 13:48:05.976197 
[2015-06-12 13:48:21,998: WARNING/Worker-1] execution timestamp: 2015-06-12 13:48:21.998254 
[2015-06-12 13:48:36,014: WARNING/Worker-1] execution timestamp: 2015-06-12 13:48:36.014837 
[2015-06-12 13:48:52,041: WARNING/Worker-1] execution timestamp: 2015-06-12 13:48:52.041326 
[2015-06-12 13:49:06,057: WARNING/Worker-1] execution timestamp: 2015-06-12 13:49:06.057286

我做錯了什麼？

來源

2015-06-12 apatel

任何人有任何想法呢？ – apatel

是誰在這方面有想法？我可以使用幫助。謝謝。 – apatel

我有同樣的問題，我找到了原因。我進去一看芹菜碼而這種行爲是從the strategy代碼，做了沿着線的未來：

if req.eta: 
    apply_eta() 
else: 
    check_if_rate_limited()

恐怕這件事情做的ETA和速率限制爲2種不兼容的方式來告訴芹菜什麼時候應該執行任務。芹菜必須選擇一種，而恰恰是ETA。

我的鏈接是對芹菜v3（因爲我懷疑這個問題被要求爲v3），但在寫作時，這仍然是在芹菜的主分支上有效。我想這個限制可能會在某個地方記錄得更好一點（here?），這只是一個拉開請求！

更新：我已經在Celery項目上打開an issue。

來源

2017-03-03 11:59:08

當通過'eta'選項安排時不考慮Celery rate_limit

回答

相關問題