服务之间的 Kubernetes GRPC 内部通信连接被拒绝
Kubernetes GRPC internal communication between services Connection refused
我正在尝试通过 grpc 在 kubernetes 内部的两个微服务之间进行通信，但我收到连接被拒绝的错误。
这些是尝试通信的服务的 yaml 文件。
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/name: source-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
name: source-api
spec:
ports:
- name: grpc-server
port: 8081
protocol: TCP
targetPort: 8081
- name: http
port: 8080
protocol: TCP
targetPort: 8080
selector:
app.kubernetes.io/name: source-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/name: source-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
name: source-api
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: source-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
template:
metadata:
labels:
app.kubernetes.io/name: source-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
spec:
containers:
- env:
- name: QUARKUS_GRPC_CLIENTS_STORE_PORT
value: "8081"
- name: QUARKUS_DATASOURCE_PASSWORD
valueFrom:
secretKeyRef:
key: datasourcePassword
name: liop
- name: KAFKA_BOOTSTRAP_SERVERS
value: kafka-service:9092
- name: QUARKUS_DATASOURCE_USERNAME
valueFrom:
secretKeyRef:
key: datasourceUsername
name: liop
- name: QUARKUS_HTTP_PORT
value: "8080"
- name: QUARKUS_GRPC_SERVER_PORT
value: "8081"
- name: QUARKUS_GRPC_SERVER_HOST
value: localhost
- name: QUARKUS_DATASOURCE_JDBC_URL
value: jdbc:mysql://mysql:3306/product
- name: QUARKUS_GRPC_CLIENTS_STORE_HOST
value: store-api
image: tools_source-api:latest
imagePullPolicy: Never
name: source-api
ports:
- containerPort: 8081
name: grpc-server
protocol: TCP
- containerPort: 8080
name: http
protocol: TCP
imagePullSecrets:
- name: gitlab-registry
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/name: product-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
name: product-api
spec:
ports:
- name: http
port: 8080
protocol: TCP
targetPort: 8080
selector:
app.kubernetes.io/name: product-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/name: product-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
name: product-api
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: product-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
template:
metadata:
labels:
app.kubernetes.io/name: product-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
spec:
containers:
- env:
- name: KAFKA_BOOTSTRAP_SERVERS
value: kafka-service:9092
- name: QUARKUS_DATASOURCE_JDBC_URL
value: jdbc:mysql://mysql:3306/product
- name: QUARKUS_GRPC_CLIENTS_IMAGE_PORT
value: "8081"
- name: QUARKUS_GRPC_CLIENTS_SOURCE_HOST
value: source-api
- name: QUARKUS_DATASOURCE_PASSWORD
valueFrom:
secretKeyRef:
key: datasourcePassword
name: liop
- name: QUARKUS_DATASOURCE_USERNAME
valueFrom:
secretKeyRef:
key: datasourceUsername
name: liop
- name: QUARKUS_GRPC_CLIENTS_SOURCE_PORT
value: "8081"
- name: QUARKUS_GRPC_CLIENTS_IMAGE_HOST
value: media-api
image: tools_product-api:latest
imagePullPolicy: Always
name: product-api
ports:
- containerPort: 8080
name: http
protocol: TCP
imagePullSecrets:
- name: gitlab-registry
这是我的 API-gateway 的 yaml，它确实通过 HTTP 与微服务正确通信：
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/version: latest
app.kubernetes.io/part-of: liop
app.kubernetes.io/name: api-gateway
name: api-gateway
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/version: latest
app.kubernetes.io/part-of: liop
app.kubernetes.io/name: api-gateway
template:
metadata:
labels:
app.kubernetes.io/version: latest
app.kubernetes.io/part-of: liop
app.kubernetes.io/name: api-gateway
spec:
containers:
- env:
- name: product_api
value: http://product-api:8080/api/products/v1/
- name: source_api
value: http://source-api:8080/api/sources/v1/
- name: store_api
value: http://store-api:8080/api/stores/v1/
- name: report_api
value: http://report-api:8080/api/reports/v1/
- name: category_api
value: http://category-api:8080/api/categories/v1/
- name: AUTH0_ISSUER_URL
value: xxxx
- name: AUTH0_AUDIENCE
value: xxxxxxx
- name: PORT
value: "7000"
image: tools_webgateway:latest
imagePullPolicy: Never
name: api-gateway
ports:
- containerPort: 7000
hostPort: 7000
name: http
protocol: TCP
imagePullSecrets:
- name: gitlab-registry
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/name: api-gateway
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
name: api-gateway
spec:
ports:
- name: http
port: 7000
protocol: TCP
targetPort: 7000
selector:
app.kubernetes.io/name: api-gateway
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
错误产品-api抛出：
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at io.grpc.Status.asRuntimeException(Status.java:533)
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at io.quarkus.grpc.runtime.supports.IOThreadClientInterceptor.onClose(IOThreadClientInterceptor.java:68)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:617)
at io.grpc.internal.ClientCallImpl.access0(ClientCallImpl.java:70)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImplStreamClosed.runInternal(ClientCallImpl.java:803)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImplStreamClosed.runInContext(ClientCallImpl.java:782)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: source-api/10.101.237.82:8081
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
来源-api:
source-api ClusterIP 10.101.237.82 <none> 8081/TCP,8080/TCP
2021-04-01 06:38:08,973 INFO [io.qua.grp.run.GrpcServerRecorder] (vert.x-eventloop-thread-1) gRPC Server started on localhost:8081 [SSL enabled: false]
在内部使用 grpcurl 也会给我一个连接被拒绝的错误。但是转发 source-api:8081 的端口确实允许我做请求。
QUARKUS_GRPC_SERVER_HOST 应该是 0.0.0.0 而不是 localhost
我正在尝试通过 grpc 在 kubernetes 内部的两个微服务之间进行通信，但我收到连接被拒绝的错误。
这些是尝试通信的服务的 yaml 文件。
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/name: source-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
name: source-api
spec:
ports:
- name: grpc-server
port: 8081
protocol: TCP
targetPort: 8081
- name: http
port: 8080
protocol: TCP
targetPort: 8080
selector:
app.kubernetes.io/name: source-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/name: source-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
name: source-api
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: source-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
template:
metadata:
labels:
app.kubernetes.io/name: source-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
spec:
containers:
- env:
- name: QUARKUS_GRPC_CLIENTS_STORE_PORT
value: "8081"
- name: QUARKUS_DATASOURCE_PASSWORD
valueFrom:
secretKeyRef:
key: datasourcePassword
name: liop
- name: KAFKA_BOOTSTRAP_SERVERS
value: kafka-service:9092
- name: QUARKUS_DATASOURCE_USERNAME
valueFrom:
secretKeyRef:
key: datasourceUsername
name: liop
- name: QUARKUS_HTTP_PORT
value: "8080"
- name: QUARKUS_GRPC_SERVER_PORT
value: "8081"
- name: QUARKUS_GRPC_SERVER_HOST
value: localhost
- name: QUARKUS_DATASOURCE_JDBC_URL
value: jdbc:mysql://mysql:3306/product
- name: QUARKUS_GRPC_CLIENTS_STORE_HOST
value: store-api
image: tools_source-api:latest
imagePullPolicy: Never
name: source-api
ports:
- containerPort: 8081
name: grpc-server
protocol: TCP
- containerPort: 8080
name: http
protocol: TCP
imagePullSecrets:
- name: gitlab-registry
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/name: product-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
name: product-api
spec:
ports:
- name: http
port: 8080
protocol: TCP
targetPort: 8080
selector:
app.kubernetes.io/name: product-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/name: product-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
name: product-api
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: product-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
template:
metadata:
labels:
app.kubernetes.io/name: product-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
spec:
containers:
- env:
- name: KAFKA_BOOTSTRAP_SERVERS
value: kafka-service:9092
- name: QUARKUS_DATASOURCE_JDBC_URL
value: jdbc:mysql://mysql:3306/product
- name: QUARKUS_GRPC_CLIENTS_IMAGE_PORT
value: "8081"
- name: QUARKUS_GRPC_CLIENTS_SOURCE_HOST
value: source-api
- name: QUARKUS_DATASOURCE_PASSWORD
valueFrom:
secretKeyRef:
key: datasourcePassword
name: liop
- name: QUARKUS_DATASOURCE_USERNAME
valueFrom:
secretKeyRef:
key: datasourceUsername
name: liop
- name: QUARKUS_GRPC_CLIENTS_SOURCE_PORT
value: "8081"
- name: QUARKUS_GRPC_CLIENTS_IMAGE_HOST
value: media-api
image: tools_product-api:latest
imagePullPolicy: Always
name: product-api
ports:
- containerPort: 8080
name: http
protocol: TCP
imagePullSecrets:
- name: gitlab-registry
这是我的 API-gateway 的 yaml，它确实通过 HTTP 与微服务正确通信：
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/version: latest
app.kubernetes.io/part-of: liop
app.kubernetes.io/name: api-gateway
name: api-gateway
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/version: latest
app.kubernetes.io/part-of: liop
app.kubernetes.io/name: api-gateway
template:
metadata:
labels:
app.kubernetes.io/version: latest
app.kubernetes.io/part-of: liop
app.kubernetes.io/name: api-gateway
spec:
containers:
- env:
- name: product_api
value: http://product-api:8080/api/products/v1/
- name: source_api
value: http://source-api:8080/api/sources/v1/
- name: store_api
value: http://store-api:8080/api/stores/v1/
- name: report_api
value: http://report-api:8080/api/reports/v1/
- name: category_api
value: http://category-api:8080/api/categories/v1/
- name: AUTH0_ISSUER_URL
value: xxxx
- name: AUTH0_AUDIENCE
value: xxxxxxx
- name: PORT
value: "7000"
image: tools_webgateway:latest
imagePullPolicy: Never
name: api-gateway
ports:
- containerPort: 7000
hostPort: 7000
name: http
protocol: TCP
imagePullSecrets:
- name: gitlab-registry
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/name: api-gateway
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
name: api-gateway
spec:
ports:
- name: http
port: 7000
protocol: TCP
targetPort: 7000
selector:
app.kubernetes.io/name: api-gateway
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
错误产品-api抛出：
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at io.grpc.Status.asRuntimeException(Status.java:533)
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at io.quarkus.grpc.runtime.supports.IOThreadClientInterceptor.onClose(IOThreadClientInterceptor.java:68)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:617)
at io.grpc.internal.ClientCallImpl.access0(ClientCallImpl.java:70)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImplStreamClosed.runInternal(ClientCallImpl.java:803)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImplStreamClosed.runInContext(ClientCallImpl.java:782)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: source-api/10.101.237.82:8081
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
来源-api:
source-api ClusterIP 10.101.237.82 <none> 8081/TCP,8080/TCP
2021-04-01 06:38:08,973 INFO [io.qua.grp.run.GrpcServerRecorder] (vert.x-eventloop-thread-1) gRPC Server started on localhost:8081 [SSL enabled: false]
在内部使用 grpcurl 也会给我一个连接被拒绝的错误。但是转发 source-api:8081 的端口确实允许我做请求。
QUARKUS_GRPC_SERVER_HOST 应该是 0.0.0.0 而不是 localhost