如何从 Kubernetes 集群中的应用程序连接 Redis 集群?

How to connect Redis cluster from application in Kubernetes cluster?

这是我的申请代码:

from flask import Flask
from redis import Redis, RedisError
import os
import socket

# Connect to Redis
redis = Redis(host=os.getenv("REDIS", "redis"), db=0, socket_connect_timeout=2, socket_timeout=2)

app = Flask(__name__)

@app.route("/")
def hello():
    try:
        visits = redis.incr("counter")
    except RedisError:
        visits = "<i>cannot connect to Redis, counter disabled</i>"

    html = "<h3>Hello {name}!</h3>" \
           "<b>Hostname:</b> {hostname}<br/>" \
           "<b>Visits:</b> {visits}"
    return html.format(name=os.getenv("NAME", "world"), hostname=socket.gethostname(), visits=visits)

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=80)

这里想在Kubernetes集群中连接一台Redis主机。所以在 Kubernetes 的清单文件中放一个环境变量 REDIS 来设置值。

这是 Kubernetes 部署清单文件:

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: {{ template "fullname" . }}
  labels:
    app: {{ template "name" . }}
    chart: {{ template "chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ template "name" . }}
      release: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app: {{ template "name" . }}
        release: {{ .Release.Name }}
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          env:
            - name: REDIS
              value: {{ template "fullname" . }}-master-svc
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
          resources:
{{ toYaml .Values.resources | indent 12 }}
    {{- with .Values.nodeSelector }}
      nodeSelector:
{{ toYaml . | indent 8 }}
    {{- end }}
    {{- with .Values.affinity }}
      affinity:
{{ toYaml . | indent 8 }}
    {{- end }}
    {{- with .Values.tolerations }}
      tolerations:
{{ toYaml . | indent 8 }}
    {{- end }}

为了连接同一集群中的Redis主机,将环境值设置为:

      env:
        - name: REDIS
          value: {{ template "fullname" . }}-master-svc

关于Redis集群,官方redis-ha图安装。它的主服务清单文件是:

apiVersion: v1
kind: Service
metadata:
  name: {{ template "fullname" . }}-master-svc
  annotations:
{{ toYaml .Values.servers.annotations | indent 4 }}
spec:
  ports:
  - port: 6379
    protocol: TCP
    targetPort: 6379
  selector:
    app: "redis-ha"
    redis-node: "true"
    redis-role: "master"
    release: "{{ .Release.Name }}"
  type: "{{ .Values.servers.serviceType }}"

但是application pod好像没有成功连接到Redis master服务名。当我访问 URL:

export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services wishful-rabbit-mychart)
export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT

从浏览器得到这个结果:

Hello World!

Hostname: wishful-rabbit-mychart-85dc7658c6-9blg5
Visits: cannot connect to Redis, counter disabled

有关 podsservices 的更多信息:

kubectl get po
NAME                                               READY     STATUS    RESTARTS   AGE
wishful-rabbit-mychart-bdfdf6558-8fjmb             1/1       Running   0          8m
wishful-rabbit-mychart-bdfdf6558-9khfs             1/1       Running   0          7m
wishful-rabbit-mychart-bdfdf6558-hgqxv             1/1       Running   0          8m
wishful-rabbit-mychart-sentinel-8667dd57d4-9njwq   1/1       Running   0          37m
wishful-rabbit-mychart-sentinel-8667dd57d4-jsq6d   1/1       Running   0          37m
wishful-rabbit-mychart-sentinel-8667dd57d4-ndqss   1/1       Running   0          37m
wishful-rabbit-mychart-server-746f47dfdd-2fn4s     1/1       Running   0          37m
wishful-rabbit-mychart-server-746f47dfdd-bwgrq     1/1       Running   0          37m
wishful-rabbit-mychart-server-746f47dfdd-spkkm     1/1       Running   0          37m

kubectl get svc
NAME                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes                          ClusterIP   10.96.0.1        <none>        443/TCP        3h
wishful-rabbit-mychart              NodePort    10.101.103.224   <none>        80:30033/TCP   37m
wishful-rabbit-mychart-master-svc   ClusterIP   10.108.128.167   <none>        6379/TCP       37m
wishful-rabbit-mychart-sentinel     ClusterIP   10.107.63.208    <none>        26379/TCP      37m
wishful-rabbit-mychart-slave-svc    ClusterIP   10.99.211.111    <none>        6379/TCP       37m

正确的用法是什么?


Redis 服务器 Pod 环境

kubectl exec -it wishful-rabbit-mychart-server-746f47dfdd-2fn4s -- sh
/ # printenv
KUBERNETES_PORT=tcp://10.96.0.1:443
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP_PORT=6379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP_ADDR=10.108.128.167
KUBERNETES_SERVICE_PORT=443
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP_PROTO=tcp
WISHFUL_RABBIT_MYCHART_SERVICE_HOST=10.101.103.224
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP_PORT=6379
HOSTNAME=wishful-rabbit-mychart-server-746f47dfdd-2fn4s
SHLVL=1
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP_PROTO=tcp
HOME=/root
WISHFUL_RABBIT_MYCHART_SENTINEL_SERVICE_HOST=10.107.63.208
WISHFUL_RABBIT_MYCHART_PORT=tcp://10.101.103.224:80
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP=tcp://10.99.211.111:6379
WISHFUL_RABBIT_MYCHART_SERVICE_PORT=80
WISHFUL_RABBIT_MYCHART_SENTINEL_SERVICE_PORT=26379
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT=tcp://10.107.63.208:26379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP=tcp://10.108.128.167:6379
WISHFUL_RABBIT_MYCHART_PORT_80_TCP_ADDR=10.101.103.224
REDIS_CHART_PREFIX=wishful-rabbit-mychart-
TERM=xterm
WISHFUL_RABBIT_MYCHART_PORT_80_TCP_PORT=80
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
WISHFUL_RABBIT_MYCHART_PORT_80_TCP_PROTO=tcp
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP_ADDR=10.107.63.208
WISHFUL_RABBIT_MYCHART_PORT_80_TCP=tcp://10.101.103.224:80
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_SERVICE_HOST=10.99.211.111
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP_PORT=26379
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP_PROTO=tcp
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
WISHFUL_RABBIT_MYCHART_MASTER_SVC_SERVICE_HOST=10.108.128.167
PWD=/
KUBERNETES_SERVICE_HOST=10.96.0.1
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_SERVICE_PORT=6379
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT=tcp://10.99.211.111:6379
WISHFUL_RABBIT_MYCHART_SERVICE_PORT_HTTP=80
REDIS_SENTINEL_SERVICE_HOST=redis-sentinel
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP_ADDR=10.99.211.111
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP=tcp://10.107.63.208:26379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT=tcp://10.108.128.167:6379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_SERVICE_PORT=6379

应用程序 Pod 环境

kubectl exec -it wishful-rabbit-mychart-85dc7658c6-8wlq6 -- sh
# printenv
KUBERNETES_SERVICE_PORT=443
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP_PORT=6379
KUBERNETES_PORT=tcp://10.96.0.1:443
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP_ADDR=10.108.128.167
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP_PROTO=tcp
WISHFUL_RABBIT_MYCHART_SERVICE_HOST=10.101.103.224
HOSTNAME=wishful-rabbit-mychart-85dc7658c6-8wlq6
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP_PORT=6379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP_PROTO=tcp
PYTHON_PIP_VERSION=9.0.1
WISHFUL_RABBIT_MYCHART_SENTINEL_SERVICE_HOST=10.107.63.208
HOME=/root
GPG_KEY=C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF
REDIS=wishful-rabbit-mychart-master-svc
WISHFUL_RABBIT_MYCHART_SERVICE_PORT=80
WISHFUL_RABBIT_MYCHART_PORT=tcp://10.101.103.224:80
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP=tcp://10.99.211.111:6379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP=tcp://10.108.128.167:6379
WISHFUL_RABBIT_MYCHART_SENTINEL_SERVICE_PORT=26379
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT=tcp://10.107.63.208:26379
WISHFUL_RABBIT_MYCHART_PORT_80_TCP_ADDR=10.101.103.224
NAME=World
TERM=xterm
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
WISHFUL_RABBIT_MYCHART_PORT_80_TCP_PORT=80
WISHFUL_RABBIT_MYCHART_PORT_80_TCP_PROTO=tcp
PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
LANG=C.UTF-8
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP_ADDR=10.107.63.208
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_SERVICE_HOST=10.99.211.111
WISHFUL_RABBIT_MYCHART_PORT_80_TCP=tcp://10.101.103.224:80
PYTHON_VERSION=2.7.14
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP_PORT=26379
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_SERVICE_PORT_HTTPS=443
WISHFUL_RABBIT_MYCHART_MASTER_SVC_SERVICE_HOST=10.108.128.167
KUBERNETES_SERVICE_HOST=10.96.0.1
PWD=/app
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT=tcp://10.99.211.111:6379
WISHFUL_RABBIT_MYCHART_SERVICE_PORT_HTTP=80
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_SERVICE_PORT=6379
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP_ADDR=10.99.211.111
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP=tcp://10.107.63.208:26379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_SERVICE_PORT=6379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT=tcp://10.108.128.167:6379

所有端点

kubectl get ep
NAME                                ENDPOINTS                                              AGE
kubernetes                          10.0.2.15:8443                                         4h
wishful-rabbit-mychart              172.17.0.5:80,172.17.0.6:80,172.17.0.7:80              1h
wishful-rabbit-mychart-master-svc   <none>                                                 1h
wishful-rabbit-mychart-sentinel     172.17.0.11:26379,172.17.0.12:26379,172.17.0.8:26379   1h
wishful-rabbit-mychart-slave-svc    <none>

描述 Redis Master 服务

kubectl describe svc wishful-rabbit-mychart-master-svc
Name:              wishful-rabbit-mychart-master-svc
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          app=redis-ha,redis-node=true,redis-role=master,release=wishful-rabbit
Type:              ClusterIP
IP:                10.108.128.167
Port:              <unset>  6379/TCP
TargetPort:        6379/TCP
Endpoints:         <none>
Session Affinity:  None
Events:            <none>

描述 Redis 服务器 Pod

kubectl describe po wishful-rabbit-mychart-server-746f47dfdd-2fn4s
Name:           wishful-rabbit-mychart-server-746f47dfdd-2fn4s
Namespace:      default
Node:           minikube/192.168.99.100
Start Time:     Fri, 02 Feb 2018 15:28:42 +0900
Labels:         app=mychart
                chart=mychart-0.1.0
                heritage=Tiller
                name=redis-server
                pod-template-hash=3029038988
                podIP=172.17.0.10
                redis-node=true
                redis-role=master
                release=wishful-rabbit
                runID=cbf8e0
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"wishful-rabbit-mychart-server-746f47dfdd","uid":"4fcb0dfc-07e2-1...
Status:         Running
IP:             172.17.0.10
Controlled By:  ReplicaSet/wishful-rabbit-mychart-server-746f47dfdd
Containers:
  redis:
    Container ID:   docker://2734d60bd44a1446ec6556369359ed15b866a4589abe1e6d394f9252114c6d4d
    Image:          quay.io/smile/redis:4.0.6r2
    Image ID:       docker-pullable://quay.io/smile/redis@sha256:8948a952920d4495859c984546838d4c9b4c71e0036eef86570922d91cacb3df
    Port:           6379/TCP
    State:          Running
      Started:      Fri, 02 Feb 2018 15:28:44 +0900
    Ready:          True
    Restart Count:  0
    Environment:
      REDIS_SENTINEL_SERVICE_HOST:  redis-sentinel
      REDIS_CHART_PREFIX:           wishful-rabbit-mychart-
    Mounts:
      /redis-master-data from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wfv2q (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          True
  PodScheduled   True
Volumes:
  data:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  default-token-wfv2q:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-wfv2q
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     <none>
Events:          <none>

您的 wishful-rabbit-mychart-master-svc 服务 Selector 和您的主 Redis pods' labels.

之间存在差异

您的服务正在搜索具有以下标签的 pods:

app=redis-ha
redis-node=true
redis-role=master
release=wishful-rabbit

虽然您的 pods 具有以下标签:

app=mychart
chart=mychart-0.1.0
heritage=Tiller
name=redis-server
pod-template-hash=3029038988
podIP=172.17.0.10
redis-node=true
redis-role=master
release=wishful-rabbit
runID=cbf8e0

如您所见,app 标签不同(redis-ha 在您的服务中,mychart 在您的 pods 中)。

这会导致服务不受限制(它不知道将传入流量转发到哪里)。

虽然我不知道此配置错误的实际原因,但我可以建议一个解决方案以使其正常工作。

您必须编辑您的 redis 服务并更改其选择器属性以匹配 pod 的标签。只是 运行:

kubectl edit svc wishful-rabbit-mychart-master-svc

并将 app: "redis-ha" 更改为 app: "mychart"

您的应用程序应该突然能够访问您的 redis 实例。

A Kubernetes Service is an abstraction which defines a logical set of Pods. The set of Pods targeted by a Service is (usually) determined by a Label Selector

因此,Pods 可以在选择器的帮助下被服务定位。

The Service’s selector will be evaluated continuously and the results will be POSTed to an Endpoints object.

如果您提供的选择器未找到 Pod,则不会创建有效的端点。

在这种情况下,如果您使用此服务访问 Pod,这将不起作用,因为 DNS 不会从 Endpoint 获取任何 IP 地址

要了解更多信息,read how Service works