带 Redis 的 Celery：实例状态已更改（主 -> 副本？）

Question

我使用 celery 执行计划任务，使用 redis 服务器在 docker 容器内进行数据备份。我的工作有时运行正确。但是我随机出现以下错误，celery beat 任务无法再进行。

[2020-09-16 21:01:07,863: CRITICAL/MainProcess] Unrecoverable error: ResponseError('UNBLOCKED force unblock from blocking operation, instance sta
te changed (master -> replica?)',)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 599, in start
    c.loop(*c.loop_args())
  File "/usr/local/lib/python3.6/site-packages/celery/worker/loops.py", line 83, in asynloop
    next(loop)
  File "/usr/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 364, in create_loop
    cb(*cbargs)
  File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", line 1088, in on_readable
    self.cycle.on_readable(fileno)
  File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", line 359, in on_readable
    chan.handlers[type]()
  File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", line 739, in _brpop_read
    **options)
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 892, in parse_response
    response = connection.read_response()
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 752, in read_response
    raise response
redis.exceptions.ResponseError: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)

如有任何帮助，我们将不胜感激。如果您需要更多详细信息，请告诉我

Answer 1

正如我上面所说的，这个问题是随机发生的，并扰乱了我们生产中的应用程序。所以我决定花时间研究解决方案。我遇到了许多命题，例如硬件问题（内存或 CPU）。但是this one definitively solve the issue. I was not using authentication on redis server Those interesting on setting redis password easily in docker can refer to this Docker Tip。为 redis 设置密码后 url 看起来像 REDIS_URL=redis://user:myPass@localhost:6379

带 Redis 的 Celery：实例状态已更改（主 -> 副本？）

Celery with redis: instance state changed (master -> replica?)

redis

celery

docker