Python日誌記錄獲取錯誤：打開的文件太多

Question

我的情況是，我有一個配置文件，顯然每次系統運行時都會導入/調用。

在我的配置文件中，我有一個帶日誌記錄配置的字典，在程序的開頭，我得到了這個配置，並用dict做了logging.config.dictConfig（）。

我的系統在至少100名使用RQ的工人的子流程中運行。有時，系統運行時，我不斷收到錯誤：

Traceback (most recent call last): 
File "/home/manutencao/Heimdall/heimdall/worker.py", line 146, in heimdall_tradutor 
File "/usr/local/lib/python3.5/logging/__init__.py", line 1279, in info 
File "/usr/local/lib/python3.5/logging/__init__.py", line 1415, in _log 
File "/usr/local/lib/python3.5/logging/__init__.py", line 1425, in handle 
File "/usr/local/lib/python3.5/logging/__init__.py", line 1487, in callHandlers 
File "/usr/local/lib/python3.5/logging/__init__.py", line 855, in handle 
File "/home/manutencao/Heimdall/heimdall/logger.py", line 28, in emit 
File "/home/manutencao/Ratatosk/ratatosk/__init__.py", line 69, in __init__ 
File "/home/manutencao/Ratatosk/ratatosk/__init__.py", line 133, in get_logger 
File "/usr/local/lib/python3.5/logging/handlers.py", line 150, in __init__ 
File "/usr/local/lib/python3.5/logging/handlers.py", line 57, in __init__ 
File "/usr/local/lib/python3.5/logging/__init__.py", line 1008, in __init__ 
File "/usr/local/lib/python3.5/logging/__init__.py", line 1037, in _open 
OSError: [Errno 24] Too many open files: '/home/manutencao/Ratatosk/ratatosk/JobRQ.log' 

During handling of the above exception, another exception occurred: 

Traceback (most recent call last): 
File "/home/manutencao/.virtualenvs/heimdall/lib/python3.5/site-packages/rq/worker.py", line 588, in perform_job 
File "/home/manutencao/.virtualenvs/heimdall/lib/python3.5/site-packages/rq/job.py", line 498, in perform 
File "/home/manutencao/Heimdall/heimdall/worker.py", line 195, in heimdall_tradutor 
File "/usr/local/lib/python3.5/logging/__init__.py", line 1314, in exception 
File "/usr/local/lib/python3.5/logging/__init__.py", line 1308, in error 
File "/usr/local/lib/python3.5/logging/__init__.py", line 1415, in _log 
File "/usr/local/lib/python3.5/logging/__init__.py", line 1425, in handle 
File "/usr/local/lib/python3.5/logging/__init__.py", line 1487, in callHandlers 
File "/usr/local/lib/python3.5/logging/__init__.py", line 855, in handle 
File "/home/manutencao/Heimdall/heimdall/logger.py", line 28, in emit 
File "/home/manutencao/Ratatosk/ratatosk/__init__.py", line 69, in __init__ 
File "/home/manutencao/Ratatosk/ratatosk/__init__.py", line 133, in get_logger 
File "/usr/local/lib/python3.5/logging/handlers.py", line 150, in __init__ 
File "/usr/local/lib/python3.5/logging/handlers.py", line 57, in __init__ 
File "/usr/local/lib/python3.5/logging/__init__.py", line 1008, in __init__ 
File "/usr/local/lib/python3.5/logging/__init__.py", line 1037, in _open 
OSError: [Errno 24] Too many open files: '/home/manutencao/Ratatosk/ratatosk/JobRQ.log' 

我有一個自定義記錄器處理程序：

class RQHandler(logging.Handler): # Inherit from logging.Handler 
    def __init__(
     self, formatter=JSONFormatter(), level=logging.NOTSET, 
     connection_pool=None 
    ): 
     # run the regular Handler __init__ 
     logging.Handler.__init__(self, level) 
     self.formatter = formatter 
     self.connection_pool = connection_pool 

    def emit(self, record): 
     # record.message is the log message 
     ratatosk = Ratatosk(
      all_queues={'huginn_log': [1, 1]}, 
      debug=configuracoes['level_log'], 
      REDIS_HOST=conf_redis['host'], 
      REDIS_PORT=conf_redis['port'], 
      REDIS_DB=conf_redis['db'], 
      connection_pool=(
       self.connection_pool or conf_redis.get('connection_pool') 
      ) 
     ) 
     ratatosk.enqueue(
      'huginn_log', 
      'huginn.service.register_log', 
      (self.format(record)), 
     ) 

基本上，我只是得到記錄器，並將其添加到RQ隊列。

我讀過這個錯誤是由於在日誌記錄中有太多的處理程序造成的。發生這種情況是因爲我在代碼的開頭添加了logging.config.dictConfig（）？

在此先感謝

Answer 1

這很可能意味着有一個與在RataTosk實施一個未關閉連接的Redis或文件（插槽數爲在Linux打開的文件）的錯誤。如果你對執行有信心，你可以在linux中增加文件限制：

ulimit -Hn 
ulimit -Sn 
# increase the fs.file-max in /etc/sysctl.conf to *2 of greater of the two 
# restart the process hosting your python app