使用 django cache_page 时 Celery 容器崩溃（InvalidCacheBackendError）

Question

我的 Django webapp 项目遇到问题。我正在运行使用 Django、Postgres、Redis 和 Celery 构建容器化环境。主要是，我想使用 Redis 进行缓存，使用 Celery 来设置实时更新。到目前为止，我已经能够连接到 redis 和 celery，并将 celery 任务结果存储在 redis 缓存中。当我尝试使用 Redis 在 Django 中缓存页面时，事情变得一团糟。出于某种原因，使用 django 的缓存系统（cache_page 装饰器）破坏了我的芹菜容器。

错误

我的 Celery 容器遇到这个错误：

django.core.cache.backends.base.InvalidCacheBackendError: Could not find backend 'django_redis.cache.RedisCache': No module named 'django_redis'

这是完整的追溯：

Traceback (most recent call last):

File "/usr/local/lib/python3.9/site-packages/django/core/cache/__init__.py", line 50, in _create_cache

backend_cls = import_string(backend)

File "/usr/local/lib/python3.9/site-packages/django/utils/module_loading.py", line 17, in import_string

module = import_module(module_path)

File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module

return _bootstrap._gcd_import(name[level:], package, level)

File "<frozen importlib._bootstrap>", line 1030, in _gcd_import

File "<frozen importlib._bootstrap>", line 1007, in _find_and_load

File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked

File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed

File "<frozen importlib._bootstrap>", line 1030, in _gcd_import

File "<frozen importlib._bootstrap>", line 1007, in _find_and_load

File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked

ModuleNotFoundError: No module named 'django_redis'


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

File "/usr/local/bin/celery", line 8, in <module>

sys.exit(main())

File "/usr/local/lib/python3.9/site-packages/celery/__main__.py", line 15, in main

sys.exit(_main())

File "/usr/local/lib/python3.9/site-packages/celery/bin/celery.py", line 213, in main

return celery(auto_envvar_prefix="CELERY")

File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__

return self.main(*args, **kwargs)

File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main

rv = self.invoke(ctx)

File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke

return _process_result(sub_ctx.command.invoke(sub_ctx))

File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke

return ctx.invoke(self.callback, **ctx.params)

File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke

return callback(*args, **kwargs)

File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 21, in new_func

return f(get_current_context(), *args, **kwargs)

File "/usr/local/lib/python3.9/site-packages/celery/bin/base.py", line 132, in caller

return f(ctx, *args, **kwargs)

File "/usr/local/lib/python3.9/site-packages/celery/bin/worker.py", line 320, in worker

worker = app.Worker(

File "/usr/local/lib/python3.9/site-packages/celery/worker/worker.py", line 94, in __init__

self.app.loader.init_worker()

File "/usr/local/lib/python3.9/site-packages/celery/loaders/base.py", line 111, in init_worker

self.import_default_modules()

File "/usr/local/lib/python3.9/site-packages/celery/loaders/base.py", line 105, in import_default_modules

raise response

File "/usr/local/lib/python3.9/site-packages/celery/utils/dispatch/signal.py", line 276, in send

response = receiver(signal=self, sender=sender, **named)

File "/usr/local/lib/python3.9/site-packages/celery/fixups/django.py", line 82, in on_import_modules

self.worker_fixup.validate_models()

File "/usr/local/lib/python3.9/site-packages/celery/fixups/django.py", line 121, in validate_models

run_checks()

File "/usr/local/lib/python3.9/site-packages/django/core/checks/registry.py", line 70, in run_checks

new_errors = check(app_configs=app_configs, databases=databases)

File "/usr/local/lib/python3.9/site-packages/django/core/checks/urls.py", line 13, in check_url_config

return check_resolver(resolver)

File "/usr/local/lib/python3.9/site-packages/django/core/checks/urls.py", line 23, in check_resolver

return check_method()

File "/usr/local/lib/python3.9/site-packages/django/urls/resolvers.py", line 408, in check

for pattern in self.url_patterns:

File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 48, in __get__

res = instance.__dict__[self.name] = self.func(instance)

File "/usr/local/lib/python3.9/site-packages/django/urls/resolvers.py", line 589, in url_patterns

patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)

File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 48, in __get__

res = instance.__dict__[self.name] = self.func(instance)

File "/usr/local/lib/python3.9/site-packages/django/urls/resolvers.py", line 582, in urlconf_module

return import_module(self.urlconf_name)

File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module

return _bootstrap._gcd_import(name[level:], package, level)

File "<frozen importlib._bootstrap>", line 1030, in _gcd_import

File "<frozen importlib._bootstrap>", line 1007, in _find_and_load

File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked

File "<frozen importlib._bootstrap>", line 680, in _load_unlocked

File "<frozen importlib._bootstrap_external>", line 790, in exec_module

File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed

File "/code/footballdata/urls.py", line 23, in <module>

from api import views

File "/code/api/views.py", line 21, in <module>

from .mixins import CacheMixin

File "/code/api/mixins.py", line 10, in <module>

class CacheMixin(object):

File "/code/api/mixins.py", line 16, in CacheMixin

def list(self, request):

File "/usr/local/lib/python3.9/site-packages/django/utils/decorators.py", line 63, in _dec

return _multi_decorate(decorator, obj)

File "/usr/local/lib/python3.9/site-packages/django/utils/decorators.py", line 47, in _multi_decorate

_update_method_wrapper(_wrapper, dec)

File "/usr/local/lib/python3.9/site-packages/django/utils/decorators.py", line 17, in _update_method_wrapper

def dummy(*args, **kwargs):

File "/usr/local/lib/python3.9/site-packages/django/utils/decorators.py", line 117, in _decorator

middleware = middleware_class(view_func, *m_args, **m_kwargs)

File "/usr/local/lib/python3.9/site-packages/django/middleware/cache.py", line 174, in __init__

super().__init__(get_response)

File "/usr/local/lib/python3.9/site-packages/django/middleware/cache.py", line 67, in __init__

super().__init__(get_response)

File "/usr/local/lib/python3.9/site-packages/django/middleware/cache.py", line 133, in __init__

self.cache = caches[self.cache_alias]

File "/usr/local/lib/python3.9/site-packages/django/core/cache/__init__.py", line 79, in __getitem__

cache = _create_cache(alias)

File "/usr/local/lib/python3.9/site-packages/django/core/cache/__init__.py", line 52, in _create_cache

raise InvalidCacheBackendError(

django.core.cache.backends.base.InvalidCacheBackendError: Could not find backend 'django_redis.cache.RedisCache': No module named 'django_redis'

设置

将 Django 与 django-redis==4.12.1 和 redis==3.5.3 一起使用。使用 docker-compose.

进行 Docker 化

我的docker-compose.yml:

version: "3.9"
   
services:
  postgresdb:
    image: postgres
    container_name: postgresdb
    environment:
      - POSTGRES_DB=postgres
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
    volumes: 
      - pgdata:/code/pgdata/

  redisdb:
    image: redis
    container_name: redisdb
    volumes: 
      - redisdata:/code/redisdata/

  web:
    build: .
    container_name: django
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/code
    environment: 
      - CELERY_BROKER=redis://redisdb:6379/0
      - CELERY_BACKEND=redis://redisdb:6379/0
    ports:
      - "8000:8000"
    depends_on:
      - postgresdb
      - redisdb

  celery:
    build: .
    container_name: celery
    command: celery -A footballdata worker -l INFO
    volumes: 
      - .:/code
    depends_on: 
      - web
      - redisdb

volumes: 
  pgdata:
  redisdata:

在我的 settings.py:

        CACHES = {
        "default": {
            "BACKEND": "django_redis.cache.RedisCache",
            "LOCATION": "redis://redisdb:6379/",
            "OPTIONS": {
                "CLIENT_CLASS": "django_redis.client.DefaultClient",
            }
        }
    }

CACHE_TTL = 30

# Celery stuff 
CELERY_BROKER_URL = os.environ.get("CELERY_BROKER", "redis://redisdb:6379/0")
CELERY_RESULT_BACKEND = os.environ.get("CELERY_BROKER", "redis://redisdb:6379/0")
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Europe/Warsaw'

# Redis 
REDIS_HOST = 'redisdb'
REDIS_PORT = 6379

容器崩溃的原因

使用 django 的 cache_page 装饰器。我有一个 REST API，我想连接到缓存。我使用简单的 mixin 缓存页面：

CACHE_TTL = getattr(settings, 'CACHE_TTL', DEFAULT_TIMEOUT)

class CacheMixin(object):
    """
    Caching for REST API views.
    Caches list and retrieve results of the view for the specified TTL.
    """
    @method_decorator(cache_page(CACHE_TTL))
    def list(self, request):
        return super().list(request)

    @method_decorator(cache_page(CACHE_TTL))
    def retrieve(self, request, pk):
        return super().retrieve(request, pk)

使用这个 mixin 破坏了我的 celery 容器。当我注释掉这段代码时，容器运行就好了。

据我所知，服务器尝试创建或连接到我在 settings.py 中指定的缓存后端，但未找到 django_redis 模块。这很奇怪，因为我已经安装了它并在我的 requirements.txt 文件中。

更奇怪的是（至少对我而言）其他容器工作得很好——而且缓存 mixin 也工作得很好！当我连接到 redis 容器并使用此 mixin 缓存页面时，它已正确存储在缓存中。只是芹菜因为某些原因坏了。

我试过的

四处寻找类似的问题。我找到的最接近的是 this question。但是，此问题与 six 包有关。我的回溯是不同的。除此之外，我还没有找到任何对我有帮助的东西。
恢复到旧版本 django-redis (3.6.6) and/or redis (2.10) - 我看到有些人对 django 和 redis 有兼容性问题过去。这对我也不起作用。

有没有人遇到过类似的问题？如果有任何想法，我将不胜感激。

Answer 1

好的，我解决了。事实证明它比我想象的要简单（通常如此）。我应该只是听了错误消息并知道它来自哪里。

发生的事情是我在需求文件中做了一些更改，但我没有使用 docker-compose up --build 重建容器，而是使用 docker build . 重建了容器。所以我猜芹菜容器没有正确更新所需的包。

所以这个错误是由于我缺乏对 docker 和 docker-compose 的理解，本质上。

使用 django cache_page 时 Celery 容器崩溃（InvalidCacheBackendError）

Celery container crashes when using django cache_page (InvalidCacheBackendError)

django

caching

redis

celery

docker