如何使用 docker-compose 设置一个容器以允许其对另一个容器的整个卷访问
how to set one container to allow it's entire volume access to another container using docker-compose
我有 4 个服务 运行ning 使用 docker-compose
1) python-api
2) python-model
3)python-celery
4)redis-server。
流量：
1) python-api 通过邮递员以图像和一些文本作为参数在端口 8000 上被命中。
2) python-api 将图像和数据传递到端口 8001 上的 python-model 以进行一些 ML 预测。
3) JSON 格式的修改后的图像和响应数据然后传递给 python-celery 以触发邮件。[=42=]
错误： python-celery 能够在第 3 步中获取 python-model 发送的图像和响应。但目前无法读取图像
错误日志：
========================
python-celery_1 | Received task: classify_crack.tasks.queue_task_v3[d71f976f-b2e7-4b29-9147-35996668de17]
python-celery_1 | == unique_file_index
python-celery_1 | AANJkaNIJSDHURHQEYRQ(*R
python-celery_1 | /python-model/server/classify_crack/inference/images/202003251237371/202003251237371_0.jpg 64
python-celery_1 | Task classify_crack.tasks.queue_task_v3[d71f976f-b2e7-4b29-9147-35996668de17] raised unexpected: AttributeError("'NoneType' object has no attribute 'shape'",)
python-celery_1 | Traceback (most recent call last):
python-celery_1 | File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 385, in trace_task
python-celery_1 | R = retval = fun(*args, **kwargs)
python-celery_1 | File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 648, in __protected_call__
python-celery_1 | return self.run(*args, **kwargs)
python-celery_1 | File "/python-api/classify_crack/tasks.py", line 690, in queue_task_v3
python-celery_1 | save_heat_map_v2(predictions, img_path, _dir, unique_file_index, original_image_index, i, grid_size=grid_size, metadata=metadata, StoredFileLinks=StoredFileLinks, row_stride=row_stride, col_stride=col_stride)
python-celery_1 | File "/python-api/classify_crack/tasks.py", line 128, in save_heat_map_v2
python-celery_1 | num_row_splits = int(np.ceil(img.shape[0]/row_stride))
python-celery_1 | AttributeError: 'NoneType' object has no attribute 'shape'
python-celery 代码中出现错误的行：
img = cv2.imread(img_path)
print(img_path)
print(img)
# print(img_path,grid_size)
splits = int(np.ceil(img.shape[0]/row_stride))
这里，img_path是正在打印的容器内的有效路径。但是我无法将图像读取为 img returns None。 splits 行给我上述错误。
出现错误的原因：
我收到此错误，因为它正在尝试访问文件夹路径：
/python-model/server/classify_crack/inference/images/202003251237371/202003251237371.jpg，但 python-celery 无法访问名称为 202003251237371 的文件夹。
证明：
我尝试使用命令：
command: >
sh -c "ls '/python-model/server/classify_crack/inference/images' &&
在 docker-compose 的 python-model 和 python-celery 服务中，我再次 运行 所有容器时得到以下结果：
python-model_1 | 201801151543500
python-model_1 | 201801151543500.jpg
python-model_1 | IMG_20190307_184100
python-model_1 | IMG_20190307_184100.jpg
python-model_1 | extracted_input_0_0 (15)
python-model_1 | extracted_input_0_0 (15).jpg
python-model_1 | extracted_input_0_0 (16)
python-model_1 | extracted_input_0_0 (16).jpg
python-model_1 | extracted_input_0_0 (18)
python-model_1 | extracted_input_0_0 (18).jpg
python-model_1 | extracted_input_0_0 (19)
python-model_1 | extracted_input_0_0 (19).jpg
python-model_1 | extracted_input_0_0 (9)
python-model_1 | extracted_input_0_0 (9).jpg
python-model_1 | file
python-model_1 | image (2)
python-model_1 | image (2).png
python-model_1 | 202003251237371
python-model_1 | 202003251237371.jpg
python-model_1 | image_X
python-model_1 | image_X.png
python-model_1 | original
python-model_1 | original.jpg
python-model_1 | original_image_0
python-model_1 | original_image_0.jpg
python-celery_1 | 20013V_Y.JPG
python-celery_1 | extracted_input_0_0 (15).jpg
python-celery_1 | extracted_input_0_0 (16).jpg
python-celery_1 | extracted_input_0_0 (18).jpg
python-celery_1 | extracted_input_0_0 (19).jpg
python-celery_1 | extracted_input_0_0 (4).jpg
python-celery_1 | 202003251237371.jpg
现在很清楚，python-celery 无法显示图像名称为 202003251237371.jpg 的文件夹 202003251237371，我可以在 python-model.
中看到它
如何解决这种情况并允许 python-celery 访问此类图像文件夹？
docker-撰写
version: "3"
networks:
app-tier:
driver: bridge
volumes:
app-volume: {}
services:
python-api-celery: &python-api-celery
build:
context: /Users/AjayB/Desktop/python-api/
networks:
- app-tier
volumes:
- app-volume:/python-model/server/classify_crack/:rw
environment:
- PYTHON_API_ENV=development
command: >
sh -c "python manage.py makemigrations &&
python manage.py migrate"
python-api: &python-api
<<: *python-api-celery
ports:
- "8000:8000"
command: >
sh -c "python manage.py runserver 0.0.0.0:8000"
python-celery: &python-celery
<<: *python-api-celery
depends_on:
- redis
links:
- python-model
command: >
sh -c "ls '/python-model/server/classify_crack/inference/images' &&
celery -A server worker -l info"
redis:
image: redis:5.0.8-alpine
hostname: redis
networks:
- app-tier
expose:
- "6379"
volumes:
- app-volume:/python-model/server/classify_crack/:rw
ports:
- "6379:6379"
command: ["redis-server"]
python-model: &python-model
build:
context: /Users/AjayB/Desktop/Python/python/
ports:
- "8001:8001"
networks:
- app-tier
environment:
- PYTHON_API_ENV=development
volumes:
- app-volume
depends_on:
- python-api
command: >
sh -c "ls '/python-model/server/classify_crack/inference/images' &&
cd /python-model/server/ &&
python manage.py migrate &&
python manage.py runserver 0.0.0.0:8001"
容器实例：
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7374a7b0b051 integrated_python-celery "sh -c 'celery -A se…" 13 minutes ago Up 5 seconds 8000/tcp integrated_python-celery_1
8eb9a754996a integrated_python-model "sh -c 'cd /python-m…" 20 minutes ago Up 5 seconds 0.0.0.0:8001->8001/tcp integrated_python-model_1
b268b7bd1ac4 integrated_python-api-celery "sh -c 'python manag…" 20 minutes ago Up 6 seconds 8000/tcp integrated_python-api-celery_1
869bb5fc21b2 integrated_python-api "sh -c 'python manag…" 20 minutes ago Up 6 seconds 0.0.0.0:8000->8000/tcp integrated_python-api_1
c85a1becea34 redis:5.0.8-alpine "docker-entrypoint.s…" About an hour ago Up 6 seconds 0.0.0.0:6379->6379/tcp integrated_redis_1
在我看来，您应该遵循有关如何将 celery 与 python API 与 Docker 和 Kubernetes 一起使用的深入教程:)
来自 OpenCV-Python 文档：`
Even if the image path is wrong, it won’t throw any error, but print img will give you None
这似乎正是这里发生的事情。可能你的图片路径看似正确但实际上是错误的，可能原因：
- 您使用的相对或绝对路径不正确
- 图像文件存在但无效
- 文件夹或文件的权限
/python-model/server/classify_crack/inference/images/202003251237371/202003251237371.jpg 不允许用户 运行 python-celery 读取它
除此之外，您的 Compose 卷定义看起来还不错。
错误终于解决了：
它在 python-api 的 views.py 开头包括以下几行：
self.cur_file_dir_path = '/data/'
self.cur_file_folder_path = '/'.join(file_path.split('/')[:-1])
if not os.path.exists('{}/inference'.format(self.cur_file_folder_path)):
os.makedirs('{}/inference'.format(self.cur_file_folder_path))
另外，修改了 docker-compose 中的卷名称，以避免一些混淆。不敢相信花了将近 1 周的时间才解决了这个问题。
docker-撰写：
version: "3"
networks:
app-tier:
driver: bridge
volumes:
app-volume: {}
services:
python-api-celery: &python-api-celery
build:
context: /Users/AjayB/Desktop/python-api/
networks:
- app-tier
volumes:
- app-volume:/data/:rw
environment:
- PYTHON_API_ENV=development
command: >
sh -c "python manage.py makemigrations &&
python manage.py migrate"
python-api: &python-api
<<: *python-api-celery
ports:
- "8000:8000"
command: >
sh -c "python manage.py runserver 0.0.0.0:8000"
python-celery: &python-celery
<<: *python-api-celery
depends_on:
- redis
command: >
sh -c "ls '/data/' &&
celery -A server worker -l info"
redis:
image: redis:5.0.8-alpine
hostname: redis
networks:
- app-tier
expose:
- "6379"
volumes:
- app-volume:/data/:rw
ports:
- "6379:6379"
command: ["redis-server"]
python-model: &python-model
build:
context: /Users/AjayB/Desktop/Python/python/
ports:
- "8001:8001"
networks:
- app-tier
environment:
- PYTHON_API_ENV=development
volumes:
- app-volume:/data/:rw
depends_on:
- python-api
command: >
sh -c "ls '/data/' &&
cd /python-model/server/ &&
python manage.py migrate &&
python manage.py runserver 0.0.0.0:8001"
我有 4 个服务 运行ning 使用 docker-compose
1) python-api
2) python-model
3)python-celery
4)redis-server。
流量：
1) python-api 通过邮递员以图像和一些文本作为参数在端口 8000 上被命中。
2) python-api 将图像和数据传递到端口 8001 上的 python-model 以进行一些 ML 预测。
3) JSON 格式的修改后的图像和响应数据然后传递给 python-celery 以触发邮件。[=42=]
错误： python-celery 能够在第 3 步中获取 python-model 发送的图像和响应。但目前无法读取图像
错误日志：
========================
python-celery_1 | Received task: classify_crack.tasks.queue_task_v3[d71f976f-b2e7-4b29-9147-35996668de17]
python-celery_1 | == unique_file_index
python-celery_1 | AANJkaNIJSDHURHQEYRQ(*R
python-celery_1 | /python-model/server/classify_crack/inference/images/202003251237371/202003251237371_0.jpg 64
python-celery_1 | Task classify_crack.tasks.queue_task_v3[d71f976f-b2e7-4b29-9147-35996668de17] raised unexpected: AttributeError("'NoneType' object has no attribute 'shape'",)
python-celery_1 | Traceback (most recent call last):
python-celery_1 | File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 385, in trace_task
python-celery_1 | R = retval = fun(*args, **kwargs)
python-celery_1 | File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 648, in __protected_call__
python-celery_1 | return self.run(*args, **kwargs)
python-celery_1 | File "/python-api/classify_crack/tasks.py", line 690, in queue_task_v3
python-celery_1 | save_heat_map_v2(predictions, img_path, _dir, unique_file_index, original_image_index, i, grid_size=grid_size, metadata=metadata, StoredFileLinks=StoredFileLinks, row_stride=row_stride, col_stride=col_stride)
python-celery_1 | File "/python-api/classify_crack/tasks.py", line 128, in save_heat_map_v2
python-celery_1 | num_row_splits = int(np.ceil(img.shape[0]/row_stride))
python-celery_1 | AttributeError: 'NoneType' object has no attribute 'shape'
python-celery 代码中出现错误的行：
img = cv2.imread(img_path)
print(img_path)
print(img)
# print(img_path,grid_size)
splits = int(np.ceil(img.shape[0]/row_stride))
这里，img_path是正在打印的容器内的有效路径。但是我无法将图像读取为 img returns None。 splits 行给我上述错误。
出现错误的原因：
我收到此错误，因为它正在尝试访问文件夹路径：
/python-model/server/classify_crack/inference/images/202003251237371/202003251237371.jpg，但 python-celery 无法访问名称为 202003251237371 的文件夹。
证明：
我尝试使用命令：
command: >
sh -c "ls '/python-model/server/classify_crack/inference/images' &&
在 docker-compose 的 python-model 和 python-celery 服务中，我再次 运行 所有容器时得到以下结果：
python-model_1 | 201801151543500
python-model_1 | 201801151543500.jpg
python-model_1 | IMG_20190307_184100
python-model_1 | IMG_20190307_184100.jpg
python-model_1 | extracted_input_0_0 (15)
python-model_1 | extracted_input_0_0 (15).jpg
python-model_1 | extracted_input_0_0 (16)
python-model_1 | extracted_input_0_0 (16).jpg
python-model_1 | extracted_input_0_0 (18)
python-model_1 | extracted_input_0_0 (18).jpg
python-model_1 | extracted_input_0_0 (19)
python-model_1 | extracted_input_0_0 (19).jpg
python-model_1 | extracted_input_0_0 (9)
python-model_1 | extracted_input_0_0 (9).jpg
python-model_1 | file
python-model_1 | image (2)
python-model_1 | image (2).png
python-model_1 | 202003251237371
python-model_1 | 202003251237371.jpg
python-model_1 | image_X
python-model_1 | image_X.png
python-model_1 | original
python-model_1 | original.jpg
python-model_1 | original_image_0
python-model_1 | original_image_0.jpg
python-celery_1 | 20013V_Y.JPG
python-celery_1 | extracted_input_0_0 (15).jpg
python-celery_1 | extracted_input_0_0 (16).jpg
python-celery_1 | extracted_input_0_0 (18).jpg
python-celery_1 | extracted_input_0_0 (19).jpg
python-celery_1 | extracted_input_0_0 (4).jpg
python-celery_1 | 202003251237371.jpg
现在很清楚，python-celery 无法显示图像名称为 202003251237371.jpg 的文件夹 202003251237371，我可以在 python-model.
如何解决这种情况并允许 python-celery 访问此类图像文件夹？
docker-撰写
version: "3"
networks:
app-tier:
driver: bridge
volumes:
app-volume: {}
services:
python-api-celery: &python-api-celery
build:
context: /Users/AjayB/Desktop/python-api/
networks:
- app-tier
volumes:
- app-volume:/python-model/server/classify_crack/:rw
environment:
- PYTHON_API_ENV=development
command: >
sh -c "python manage.py makemigrations &&
python manage.py migrate"
python-api: &python-api
<<: *python-api-celery
ports:
- "8000:8000"
command: >
sh -c "python manage.py runserver 0.0.0.0:8000"
python-celery: &python-celery
<<: *python-api-celery
depends_on:
- redis
links:
- python-model
command: >
sh -c "ls '/python-model/server/classify_crack/inference/images' &&
celery -A server worker -l info"
redis:
image: redis:5.0.8-alpine
hostname: redis
networks:
- app-tier
expose:
- "6379"
volumes:
- app-volume:/python-model/server/classify_crack/:rw
ports:
- "6379:6379"
command: ["redis-server"]
python-model: &python-model
build:
context: /Users/AjayB/Desktop/Python/python/
ports:
- "8001:8001"
networks:
- app-tier
environment:
- PYTHON_API_ENV=development
volumes:
- app-volume
depends_on:
- python-api
command: >
sh -c "ls '/python-model/server/classify_crack/inference/images' &&
cd /python-model/server/ &&
python manage.py migrate &&
python manage.py runserver 0.0.0.0:8001"
容器实例：
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7374a7b0b051 integrated_python-celery "sh -c 'celery -A se…" 13 minutes ago Up 5 seconds 8000/tcp integrated_python-celery_1
8eb9a754996a integrated_python-model "sh -c 'cd /python-m…" 20 minutes ago Up 5 seconds 0.0.0.0:8001->8001/tcp integrated_python-model_1
b268b7bd1ac4 integrated_python-api-celery "sh -c 'python manag…" 20 minutes ago Up 6 seconds 8000/tcp integrated_python-api-celery_1
869bb5fc21b2 integrated_python-api "sh -c 'python manag…" 20 minutes ago Up 6 seconds 0.0.0.0:8000->8000/tcp integrated_python-api_1
c85a1becea34 redis:5.0.8-alpine "docker-entrypoint.s…" About an hour ago Up 6 seconds 0.0.0.0:6379->6379/tcp integrated_redis_1
在我看来，您应该遵循有关如何将 celery 与 python API 与 Docker 和 Kubernetes 一起使用的深入教程:)
来自 OpenCV-Python 文档：`
Even if the image path is wrong, it won’t throw any error, but print img will give you None
这似乎正是这里发生的事情。可能你的图片路径看似正确但实际上是错误的，可能原因：
- 您使用的相对或绝对路径不正确
- 图像文件存在但无效
- 文件夹或文件的权限
/python-model/server/classify_crack/inference/images/202003251237371/202003251237371.jpg不允许用户 运行python-celery读取它
除此之外，您的 Compose 卷定义看起来还不错。
错误终于解决了：
它在 python-api 的 views.py 开头包括以下几行：
self.cur_file_dir_path = '/data/'
self.cur_file_folder_path = '/'.join(file_path.split('/')[:-1])
if not os.path.exists('{}/inference'.format(self.cur_file_folder_path)):
os.makedirs('{}/inference'.format(self.cur_file_folder_path))
另外，修改了 docker-compose 中的卷名称，以避免一些混淆。不敢相信花了将近 1 周的时间才解决了这个问题。
docker-撰写：
version: "3"
networks:
app-tier:
driver: bridge
volumes:
app-volume: {}
services:
python-api-celery: &python-api-celery
build:
context: /Users/AjayB/Desktop/python-api/
networks:
- app-tier
volumes:
- app-volume:/data/:rw
environment:
- PYTHON_API_ENV=development
command: >
sh -c "python manage.py makemigrations &&
python manage.py migrate"
python-api: &python-api
<<: *python-api-celery
ports:
- "8000:8000"
command: >
sh -c "python manage.py runserver 0.0.0.0:8000"
python-celery: &python-celery
<<: *python-api-celery
depends_on:
- redis
command: >
sh -c "ls '/data/' &&
celery -A server worker -l info"
redis:
image: redis:5.0.8-alpine
hostname: redis
networks:
- app-tier
expose:
- "6379"
volumes:
- app-volume:/data/:rw
ports:
- "6379:6379"
command: ["redis-server"]
python-model: &python-model
build:
context: /Users/AjayB/Desktop/Python/python/
ports:
- "8001:8001"
networks:
- app-tier
environment:
- PYTHON_API_ENV=development
volumes:
- app-volume:/data/:rw
depends_on:
- python-api
command: >
sh -c "ls '/data/' &&
cd /python-model/server/ &&
python manage.py migrate &&
python manage.py runserver 0.0.0.0:8001"