不能将来自另一个 python3 容器的 postgres 容器与 psycopg2 一起使用
can't use postgres container from another python3 container with psycopg2
我正在尝试使用 docker-compose 制作一个多容器 docker 应用程序。
这是我想要完成的： 我有一个 python3 应用程序，它从 API 调用中获取数字列表作为输入（fastAPI 与 gunicorn 服务器）并将数字传递给一个 returns 数字的函数（实际上是一个 ML 模型），然后将其作为结果发送回（当然在 json 中）那个 API 电话。那部分工作得很好。当我引入一个 postgres 容器来存储我收到的输入到 postgres table 时，问题就开始了，我还没有添加我也应该从我的本地 pgadmin4 应用程序访问这个 postgres 数据库的数据的部分。
这是我到目前为止所做的：我正在使用“docker-compose.yml”文件来设置这两个容器，这里是是：
version: '3.8'
services:
postgres:
image: postgres:12.4
restart: always
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres_password
- POSTGRES_DATABASE=postgres
docker_fastapi:
# use the Dockerfile in the current directory.
build: .
ports:
# 3000 is what I send API calls to
- "3000:3000"
# this is postgres's port
- "5432:5432"
environment:
# these are the environment variables that I am using inside psycop2 to make connection.
- POSTGRES_HOST=postgres
- POSTGRES_PORT=5432
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres_password
- POSTGRES_DATABASE=postgres
他
以下是我在 psycopg2:
中使用这些环境变量的方式
import os
from psycopg2 import connect
# making database connection using environement variables.
connection = connect(host=os.environ['POSTGRES_HOST'], port=os.environ['POSTGRES_PORT'],
user=os.environ['POSTGRES_USER'], password=os.environ['POSTGRES_PASSWORD'],
database=os.environ['POSTGRES_DATABASE']
)
这是 Dockerfile：
FROM tiangolo/uvicorn-gunicorn:python3.8-slim
# slim = debian-based. Not using alpine because it has poor python3 support.
LABEL maintainer="Sebastian Ramirez <tiangolo@gmail.com>"
RUN apt-get update
RUN apt-get install -y libpq-dev gcc
# copy and install from requirements.txt file
COPY requirements.txt /app/requirements.txt
RUN pip install --no-cache-dir -r /app/requirements.txt
# remove all the dependency files to reduce the final image size
RUN apt-get autoremove -y gcc
# copying all the code files to the container's file system
COPY ./api /app/api
WORKDIR /app/api
EXPOSE 3000
ENTRYPOINT ["uvicorn"]
CMD ["api.main:app", "--host", "0.0.0.0", "--port", "3000"]
这是它为我发送的 API 呼叫生成的错误：
root@naveen-hp:/home/naveen/Videos/ML-Model-serving-with-fastapi-and-Docker# # docker-compose up
Starting ml-model-serving-with-fastapi-and-docker_docker_fastapi_1 ... done
Starting ml-model-serving-with-fastapi-and-docker_postgres_1 ... done
Attaching to ml-model-serving-with-fastapi-and-docker_postgres_1, ml-model-serving-with-fastapi-and-docker_docker_fastapi_1
postgres_1 |
postgres_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres_1 |
postgres_1 | 2020-10-22 13:17:14.080 UTC [1] LOG: starting PostgreSQL 12.4 (Debian 12.4-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
postgres_1 | 2020-10-22 13:17:14.080 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
postgres_1 | 2020-10-22 13:17:14.080 UTC [1] LOG: listening on IPv6 address "::", port 5432
postgres_1 | 2020-10-22 13:17:14.092 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1 | 2020-10-22 13:17:14.120 UTC [24] LOG: database system was shut down at 2020-10-22 12:48:50 UTC
postgres_1 | 2020-10-22 13:17:14.130 UTC [1] LOG: database system is ready to accept connections
docker_fastapi_1 | INFO: Started server process [1]
docker_fastapi_1 | INFO: Waiting for application startup.
docker_fastapi_1 | INFO: Application startup complete.
docker_fastapi_1 | INFO: Uvicorn running on http://0.0.0.0:3000 (Press CTRL+C to quit)
docker_fastapi_1 | INFO: 172.18.0.1:56094 - "POST /predict HTTP/1.1" 500 Internal Server Error
docker_fastapi_1 | ERROR: Exception in ASGI application
docker_fastapi_1 | Traceback (most recent call last):
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 391, in run_asgi
docker_fastapi_1 | result = await app(self.scope, self.receive, self.send)
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
docker_fastapi_1 | return await self.app(scope, receive, send)
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/fastapi/applications.py", line 179, in __call__
docker_fastapi_1 | await super().__call__(scope, receive, send)
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/starlette/applications.py", line 111, in __call__
docker_fastapi_1 | await self.middleware_stack(scope, receive, send)
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
docker_fastapi_1 | raise exc from None
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
docker_fastapi_1 | await self.app(scope, receive, _send)
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
docker_fastapi_1 | raise exc from None
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
docker_fastapi_1 | await self.app(scope, receive, sender)
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 566, in __call__
docker_fastapi_1 | await route.handle(scope, receive, send)
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 227, in handle
docker_fastapi_1 | await self.app(scope, receive, send)
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 41, in app
docker_fastapi_1 | response = await func(request)
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 182, in app
docker_fastapi_1 | raw_response = await run_endpoint_function(
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 135, in run_endpoint_function
docker_fastapi_1 | return await run_in_threadpool(dependant.call, **values)
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/starlette/concurrency.py", line 34, in run_in_threadpool
docker_fastapi_1 | return await loop.run_in_executor(None, func, *args)
docker_fastapi_1 | File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
docker_fastapi_1 | result = self.fn(*self.args, **self.kwargs)
docker_fastapi_1 | File "/app/api/main.py", line 83, in predict
docker_fastapi_1 | insert_into_db(X)
docker_fastapi_1 | File "/app/api/main.py", line 38, in insert_into_db
docker_fastapi_1 | cursor.execute(f"INSERT INTO public.\"API_Test\""
docker_fastapi_1 | IndexError: index 1 is out of bounds for axis 0 with size 1
以下是我发送 API 电话的方式：
curl -X POST "http://0.0.0.0:3000/predict" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"input_data\":[[
1.354e+01, 1.436e+01, 8.746e+01, 5.663e+02, 9.779e-02, 8.129e-02,
6.664e-02, 4.781e-02, 1.885e-01, 5.766e-02, 2.699e-01, 7.886e-01,
2.058e+00, 2.356e+01, 8.462e-03, 1.460e-02, 2.387e-02, 1.315e-02,
1.980e-02, 2.300e-03, 1.511e+01, 1.926e+01, 9.970e+01, 7.112e+02,
1.440e-01, 1.773e-01, 2.390e-01, 1.288e-01, 2.977e-01, 7.259e-02]]}"
当我使用 AWS RDS 的 postgres 实例的凭证构建它而没有第二个 postgres 容器并直接在 psycopg2.connect() 内指定凭证而不使用环境变量和 docker-compose 和直接使用上面显示的 Dockerfile 构建；因此，我将接收到的数据插入 postgres 的代码大概没问题。当我引入第二个容器时，问题就开始了。是什么导致了这些错误，我该如何解决？
您必须添加网络 网络 和 depend_on 标志。试试这个：
version: '3.8'
services:
postgres:
image: postgres:12.4
restart: always
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres_password
- POSTGRES_DB=postgres
networks:
- default
docker_fastapi:
# use the Dockerfile in the current directory.
build: .
ports:
# 3000 is what I send API calls to
- "3000:3000"
# this is postgres's port
# no need for this
# - "5432:5432"
networks:
- default
depends_on:
- postgres
# no need for this
# environment:
# these are the environment variables that I am using inside psycop2 to make connection.
# - POSTGRES_HOST=postgres
# - POSTGRES_PORT=5432
# - POSTGRES_USER=postgres
# - POSTGRES_PASSWORD=postgres_password
# - POSTGRES_DATABASE=postgres
问题是因为“docker-compose.yml”文件中有很多 postgres。
在 的帮助下，以及我的实现；这是有效的方法，如果有人可能需要的话。
version: '3.8'
services:
postgres_instance:
image: postgres:12.4
# to expose postgres to local machine and monitor it in something like pgadmin
ports:
- "5432:5432"
restart: unless-stopped
# to persist data, if containers are stoopped and resumed.
volumes:
- ./postgres-data:/var/lib/postgresql/data
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres_password
- POSTGRES_DB=postgres
networks:
- default
docker_fastapi:
# using Dockerfile in current directory
build: .
# port I send API calls to
ports:
- "3000:3000"
restart: always
depends_on:
- postgres_instance
networks:
- default
# these environment variables must be specified here, to be able to use from .py file inside this container.
environment:
- POSTGRES_HOST=postgres_instance
- POSTGRES_PORT=5432
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres_password
- POSTGRES_DB=postgres
可以看出，将服务名称从 postgres 更改为 postgres_instance 可以解决所有问题。大概是因为，postgres 实际上指的是数据库的托管位置，但标签与用户名和数据库名称相同。
我正在尝试使用 docker-compose 制作一个多容器 docker 应用程序。
这是我想要完成的： 我有一个 python3 应用程序，它从 API 调用中获取数字列表作为输入（fastAPI 与 gunicorn 服务器）并将数字传递给一个 returns 数字的函数（实际上是一个 ML 模型），然后将其作为结果发送回（当然在 json 中）那个 API 电话。那部分工作得很好。当我引入一个 postgres 容器来存储我收到的输入到 postgres table 时，问题就开始了，我还没有添加我也应该从我的本地 pgadmin4 应用程序访问这个 postgres 数据库的数据的部分。
这是我到目前为止所做的：我正在使用“docker-compose.yml”文件来设置这两个容器，这里是是：
version: '3.8'
services:
postgres:
image: postgres:12.4
restart: always
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres_password
- POSTGRES_DATABASE=postgres
docker_fastapi:
# use the Dockerfile in the current directory.
build: .
ports:
# 3000 is what I send API calls to
- "3000:3000"
# this is postgres's port
- "5432:5432"
environment:
# these are the environment variables that I am using inside psycop2 to make connection.
- POSTGRES_HOST=postgres
- POSTGRES_PORT=5432
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres_password
- POSTGRES_DATABASE=postgres
他
以下是我在 psycopg2:
import os
from psycopg2 import connect
# making database connection using environement variables.
connection = connect(host=os.environ['POSTGRES_HOST'], port=os.environ['POSTGRES_PORT'],
user=os.environ['POSTGRES_USER'], password=os.environ['POSTGRES_PASSWORD'],
database=os.environ['POSTGRES_DATABASE']
)
这是 Dockerfile：
FROM tiangolo/uvicorn-gunicorn:python3.8-slim
# slim = debian-based. Not using alpine because it has poor python3 support.
LABEL maintainer="Sebastian Ramirez <tiangolo@gmail.com>"
RUN apt-get update
RUN apt-get install -y libpq-dev gcc
# copy and install from requirements.txt file
COPY requirements.txt /app/requirements.txt
RUN pip install --no-cache-dir -r /app/requirements.txt
# remove all the dependency files to reduce the final image size
RUN apt-get autoremove -y gcc
# copying all the code files to the container's file system
COPY ./api /app/api
WORKDIR /app/api
EXPOSE 3000
ENTRYPOINT ["uvicorn"]
CMD ["api.main:app", "--host", "0.0.0.0", "--port", "3000"]
这是它为我发送的 API 呼叫生成的错误：
root@naveen-hp:/home/naveen/Videos/ML-Model-serving-with-fastapi-and-Docker# # docker-compose up
Starting ml-model-serving-with-fastapi-and-docker_docker_fastapi_1 ... done
Starting ml-model-serving-with-fastapi-and-docker_postgres_1 ... done
Attaching to ml-model-serving-with-fastapi-and-docker_postgres_1, ml-model-serving-with-fastapi-and-docker_docker_fastapi_1
postgres_1 |
postgres_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres_1 |
postgres_1 | 2020-10-22 13:17:14.080 UTC [1] LOG: starting PostgreSQL 12.4 (Debian 12.4-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
postgres_1 | 2020-10-22 13:17:14.080 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
postgres_1 | 2020-10-22 13:17:14.080 UTC [1] LOG: listening on IPv6 address "::", port 5432
postgres_1 | 2020-10-22 13:17:14.092 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1 | 2020-10-22 13:17:14.120 UTC [24] LOG: database system was shut down at 2020-10-22 12:48:50 UTC
postgres_1 | 2020-10-22 13:17:14.130 UTC [1] LOG: database system is ready to accept connections
docker_fastapi_1 | INFO: Started server process [1]
docker_fastapi_1 | INFO: Waiting for application startup.
docker_fastapi_1 | INFO: Application startup complete.
docker_fastapi_1 | INFO: Uvicorn running on http://0.0.0.0:3000 (Press CTRL+C to quit)
docker_fastapi_1 | INFO: 172.18.0.1:56094 - "POST /predict HTTP/1.1" 500 Internal Server Error
docker_fastapi_1 | ERROR: Exception in ASGI application
docker_fastapi_1 | Traceback (most recent call last):
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 391, in run_asgi
docker_fastapi_1 | result = await app(self.scope, self.receive, self.send)
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
docker_fastapi_1 | return await self.app(scope, receive, send)
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/fastapi/applications.py", line 179, in __call__
docker_fastapi_1 | await super().__call__(scope, receive, send)
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/starlette/applications.py", line 111, in __call__
docker_fastapi_1 | await self.middleware_stack(scope, receive, send)
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
docker_fastapi_1 | raise exc from None
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
docker_fastapi_1 | await self.app(scope, receive, _send)
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
docker_fastapi_1 | raise exc from None
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
docker_fastapi_1 | await self.app(scope, receive, sender)
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 566, in __call__
docker_fastapi_1 | await route.handle(scope, receive, send)
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 227, in handle
docker_fastapi_1 | await self.app(scope, receive, send)
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 41, in app
docker_fastapi_1 | response = await func(request)
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 182, in app
docker_fastapi_1 | raw_response = await run_endpoint_function(
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 135, in run_endpoint_function
docker_fastapi_1 | return await run_in_threadpool(dependant.call, **values)
docker_fastapi_1 | File "/usr/local/lib/python3.8/site-packages/starlette/concurrency.py", line 34, in run_in_threadpool
docker_fastapi_1 | return await loop.run_in_executor(None, func, *args)
docker_fastapi_1 | File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
docker_fastapi_1 | result = self.fn(*self.args, **self.kwargs)
docker_fastapi_1 | File "/app/api/main.py", line 83, in predict
docker_fastapi_1 | insert_into_db(X)
docker_fastapi_1 | File "/app/api/main.py", line 38, in insert_into_db
docker_fastapi_1 | cursor.execute(f"INSERT INTO public.\"API_Test\""
docker_fastapi_1 | IndexError: index 1 is out of bounds for axis 0 with size 1
以下是我发送 API 电话的方式：
curl -X POST "http://0.0.0.0:3000/predict" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"input_data\":[[
1.354e+01, 1.436e+01, 8.746e+01, 5.663e+02, 9.779e-02, 8.129e-02,
6.664e-02, 4.781e-02, 1.885e-01, 5.766e-02, 2.699e-01, 7.886e-01,
2.058e+00, 2.356e+01, 8.462e-03, 1.460e-02, 2.387e-02, 1.315e-02,
1.980e-02, 2.300e-03, 1.511e+01, 1.926e+01, 9.970e+01, 7.112e+02,
1.440e-01, 1.773e-01, 2.390e-01, 1.288e-01, 2.977e-01, 7.259e-02]]}"
当我使用 AWS RDS 的 postgres 实例的凭证构建它而没有第二个 postgres 容器并直接在 psycopg2.connect() 内指定凭证而不使用环境变量和 docker-compose 和直接使用上面显示的 Dockerfile 构建；因此，我将接收到的数据插入 postgres 的代码大概没问题。当我引入第二个容器时，问题就开始了。是什么导致了这些错误，我该如何解决？
您必须添加网络 网络 和 depend_on 标志。试试这个：
version: '3.8'
services:
postgres:
image: postgres:12.4
restart: always
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres_password
- POSTGRES_DB=postgres
networks:
- default
docker_fastapi:
# use the Dockerfile in the current directory.
build: .
ports:
# 3000 is what I send API calls to
- "3000:3000"
# this is postgres's port
# no need for this
# - "5432:5432"
networks:
- default
depends_on:
- postgres
# no need for this
# environment:
# these are the environment variables that I am using inside psycop2 to make connection.
# - POSTGRES_HOST=postgres
# - POSTGRES_PORT=5432
# - POSTGRES_USER=postgres
# - POSTGRES_PASSWORD=postgres_password
# - POSTGRES_DATABASE=postgres
问题是因为“docker-compose.yml”文件中有很多 postgres。
在
version: '3.8'
services:
postgres_instance:
image: postgres:12.4
# to expose postgres to local machine and monitor it in something like pgadmin
ports:
- "5432:5432"
restart: unless-stopped
# to persist data, if containers are stoopped and resumed.
volumes:
- ./postgres-data:/var/lib/postgresql/data
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres_password
- POSTGRES_DB=postgres
networks:
- default
docker_fastapi:
# using Dockerfile in current directory
build: .
# port I send API calls to
ports:
- "3000:3000"
restart: always
depends_on:
- postgres_instance
networks:
- default
# these environment variables must be specified here, to be able to use from .py file inside this container.
environment:
- POSTGRES_HOST=postgres_instance
- POSTGRES_PORT=5432
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres_password
- POSTGRES_DB=postgres
可以看出，将服务名称从 postgres 更改为 postgres_instance 可以解决所有问题。大概是因为，postgres 实际上指的是数据库的托管位置，但标签与用户名和数据库名称相同。