CeleryD 似乎忽略了并发参数

Question

我最近将我的 Django 项目升级到 Celery 4.4.6，但进展并不顺利。我目前的头号问题是任务的并发性。因为任务锁定数据库表，有些任务非常占用内存，所以没有机会同时运行八个任务。我也只有一台 2 处理器机器可用。然而，这就是 celery 要做的事情。

以前我只能运行同时完成两项任务。

worker 是守护进程，只有一个 worker 处于活动状态（一个节点）。我将并发设置为两个。这是我的 /etc/default/celeryd:

#   most people will only start one node:
CELERYD_NODES="worker1"
#   but you can also start multiple and configure settings
#   for each in CELERYD_OPTS
#CELERYD_NODES="worker1 worker2 worker3"
#   alternatively, you can specify the number of nodes to start:
#CELERYD_NODES=3

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/home/ubuntu/dev/bin/python -m celery"
#CELERY_BIN="/virtualenvs/def/bin/celery"

# App instance to use
# comment out this line if you don't use an app
CELERY_APP="match2"
# or fully qualified:
#CELERY_APP="proj.tasks:app"

# Where to chdir at start.
export DJANGO_SETTINGS_MODULE="match2.settings"
CELERYD_CHDIR="/home/ubuntu/dev/match2/match2"

# Extra command-line arguments to the worker
CELERYD_OPTS="--concurrency=2"
# Configure node-specific settings by appending node name to arguments:
#CELERYD_OPTS="--time-limit=300 -c 8 -c:worker2 4 -c:worker3 2 -Ofair:worker1"

# Set logging level to DEBUG
CELERYD_LOG_LEVEL="INFO"

# %n will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"

# Workers should run as an unprivileged user.
#   You need to create this user manually (or you can choose
#   a user/group combination that already exists (e.g., nobody).
CELERYD_USER="ubuntu"
CELERYD_GROUP="users"

# If enabled pid and log directories will be created if missing,
# and owned by the userid/group configured.
CELERY_CREATE_DIRS=1

我非常假设这条线会处理可以同时执行的任务数量：CELERYD_OPTS="--concurrency=2" 但它似乎仍然从 RabbitMQ 消息队列中获取最多八个项目。

感谢任何帮助。

Answer 1

所以来来回回加入了Google群，终于得到了答案：

如果你希望 Celery 表现得像一个好小 worker 并且在完成旧任务之前不承担另一项任务，你需要在你的设置文件中同时拥有这两者：

task_acks_late = True
worker_prefetch_multiplier = 1

如果你随后在 Django 项目中使用它，使用 old-style 大写设置（参见：https://docs.celeryproject.org/en/stable/userguide/configuration.html#new-lowercase-settings），你将它变成：

CELERY_WORKER_PREFETCH_MULTIPLIER = 1
CELERY_TASK_ACKS_LATE = True

CeleryD 似乎忽略了并发参数

CeleryD seems to ignore concurrency argument

python

django

celery

django-celery