Python Logging getting error: Too many open files

Question

我的情况是，我有一个配置文件，显然，每次系统运行s.imported/called。

在我的配置文件中，我有一个带有日志记录配置的字典，在程序的开头，我得到了这个配置，并用字典执行 logging.config.dictConfig()。

我的系统运行在子进程中，至少有 100 个工人使用 RQ。有时，当系统运行ning 时，我不断收到错误消息：

Traceback (most recent call last):
File "/home/manutencao/Heimdall/heimdall/worker.py", line 146, in heimdall_tradutor
File "/usr/local/lib/python3.5/logging/__init__.py", line 1279, in info
File "/usr/local/lib/python3.5/logging/__init__.py", line 1415, in _log
File "/usr/local/lib/python3.5/logging/__init__.py", line 1425, in handle
File "/usr/local/lib/python3.5/logging/__init__.py", line 1487, in callHandlers
File "/usr/local/lib/python3.5/logging/__init__.py", line 855, in handle
File "/home/manutencao/Heimdall/heimdall/logger.py", line 28, in emit
File "/home/manutencao/Ratatosk/ratatosk/__init__.py", line 69, in __init__
File "/home/manutencao/Ratatosk/ratatosk/__init__.py", line 133, in get_logger
File "/usr/local/lib/python3.5/logging/handlers.py", line 150, in __init__
File "/usr/local/lib/python3.5/logging/handlers.py", line 57, in __init__
File "/usr/local/lib/python3.5/logging/__init__.py", line 1008, in __init__
File "/usr/local/lib/python3.5/logging/__init__.py", line 1037, in _open
OSError: [Errno 24] Too many open files: '/home/manutencao/Ratatosk/ratatosk/JobRQ.log'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/manutencao/.virtualenvs/heimdall/lib/python3.5/site-packages/rq/worker.py", line 588, in perform_job
File "/home/manutencao/.virtualenvs/heimdall/lib/python3.5/site-packages/rq/job.py", line 498, in perform
File "/home/manutencao/Heimdall/heimdall/worker.py", line 195, in heimdall_tradutor
File "/usr/local/lib/python3.5/logging/__init__.py", line 1314, in exception
File "/usr/local/lib/python3.5/logging/__init__.py", line 1308, in error
File "/usr/local/lib/python3.5/logging/__init__.py", line 1415, in _log
File "/usr/local/lib/python3.5/logging/__init__.py", line 1425, in handle
File "/usr/local/lib/python3.5/logging/__init__.py", line 1487, in callHandlers
File "/usr/local/lib/python3.5/logging/__init__.py", line 855, in handle
File "/home/manutencao/Heimdall/heimdall/logger.py", line 28, in emit
File "/home/manutencao/Ratatosk/ratatosk/__init__.py", line 69, in __init__
File "/home/manutencao/Ratatosk/ratatosk/__init__.py", line 133, in get_logger
File "/usr/local/lib/python3.5/logging/handlers.py", line 150, in __init__
File "/usr/local/lib/python3.5/logging/handlers.py", line 57, in __init__
File "/usr/local/lib/python3.5/logging/__init__.py", line 1008, in __init__
File "/usr/local/lib/python3.5/logging/__init__.py", line 1037, in _open
OSError: [Errno 24] Too many open files: '/home/manutencao/Ratatosk/ratatosk/JobRQ.log'

我有一个自定义记录器处理程序：

class RQHandler(logging.Handler):  # Inherit from logging.Handler
    def __init__(
        self, formatter=JSONFormatter(), level=logging.NOTSET,
        connection_pool=None
    ):
        # run the regular Handler __init__
        logging.Handler.__init__(self, level)
        self.formatter = formatter
        self.connection_pool = connection_pool

    def emit(self, record):
        # record.message is the log message
        ratatosk = Ratatosk(
            all_queues={'huginn_log': [1, 1]},
            debug=configuracoes['level_log'],
            REDIS_HOST=conf_redis['host'],
            REDIS_PORT=conf_redis['port'],
            REDIS_DB=conf_redis['db'],
            connection_pool=(
                self.connection_pool or conf_redis.get('connection_pool')
            )
        )
        ratatosk.enqueue(
            'huginn_log',
            'huginn.service.register_log',
            (self.format(record)),
        )

基本上，我只是获取记录器并将其添加到 RQ 队列中。

我了解到此错误是由于日志记录中的处理程序过多而导致的。发生这种情况是因为我在代码的开头添加了 logging.config.dictConfig() 吗？

提前致谢

Answer 1

这很可能意味着在 RataTosk 的实现中与 Redis 或文件的未关闭连接存在错误（在 linux 中，socket 算作一个打开的文件）。如果您对实施有信心，可以在 linux 中增加文件限制：

ulimit -Hn
ulimit -Sn
# increase the fs.file-max in /etc/sysctl.conf to *2 of greater of the two
# restart the process hosting your python app

Python Logging getting error: Too many open files

Python Logging getting error: Too many open files

python

logging

subprocess

python-3.5