mongo 裸机 kubernetese 集群上的数据库 pod 始终处于待处理状态

mongo db pod on bare metal kubernetese cluster is always pending

在我的裸机 kubernetese 集群上,我使用 helm from bitnami on kubernetese 安装了 mongo 数据库,如下所示。

helm install mongodb bitnami/mongodb

我立即得到以下输出结果。

vagrant@kmasterNew:~$ helm install mongodb bitnami/mongodb
NAME: mongodb
LAST DEPLOYED: Tue May  4 12:26:58 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **

MongoDB(R) can be accessed on the following DNS name(s) and ports from within your cluster:

    mongodb.default.svc.cluster.local

To get the root password run:

    export MONGODB_ROOT_PASSWORD=$(kubectl get secret --namespace default mongodb -o jsonpath="{.data.mongodb-root-password}" | base64 --decode)

To connect to your database, create a MongoDB(R) client container:

    kubectl run --namespace default mongodb-client --rm --tty -i --restart='Never' --env="MONGODB_ROOT_PASSWORD=$MONGODB_ROOT_PASSWORD" --image docker.io/bitnami/mongodb:4.4.5-debian-10-r21 --command -- bash

Then, run the following command:
    mongo admin --host "mongodb" --authenticationDatabase admin -u root -p $MONGODB_ROOT_PASSWORD

To connect to your database from outside the cluster execute the following commands:

    kubectl port-forward --namespace default svc/mongodb 27017:27017 &
    mongo --host 127.0.0.1 --authenticationDatabase admin -p $MONGODB_ROOT_PASSWORD
    

然后在检查 pods 后,我发现无论我等待多长时间,pod 始终处于待处理状态。

vagrant@kmasterNew:~$ kubectl get pod -o wide
NAME                      READY   STATUS    RESTARTS   AGE     IP       NODE     NOMINATED NODE   READINESS GATES
mongodb-d9b6d589c-zpmb6   0/1     Pending   0          9m21s   <none>   <none>   <none>           <none>

我错过了什么?

如 helm install 输出所示,我 运行 以下命令获取秘密。

export MONGODB_ROOT_PASSWORD=$(kubectl get secret --namespace default mongodb -o jsonpath="{.data.mongodb-root-password}" | base64 --decode)

执行成功。当我这样做时

echo $MONGODB_ROOT_PASSWORD

我得到root密码为

rMjjciN8An

正如 helm install 输出中的说明所示,我尝试通过 运行ning

连接到数据库
kubectl run --namespace default mongodb-client --rm --tty -i --restart='Never' --env="MONGODB_ROOT_PASSWORD=rMjjciN8An" --image docker.io/bitnami/mongodb:4.4.5-debian-10-r21 --command -- bash

mongo admin --host "mongodb" --authenticationDatabase admin -u root -p rMjjciN8An

我得到以下输出。

MongoDB shell version v4.4.5
connecting to: mongodb://mongodb:27017/admin?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
Error: couldn't connect to server mongodb:27017, connection attempt failed: SocketException: Error connecting to mongodb:27017 (10.111.99.8:27017) :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:374:17
@(connect):2:6
exception: connect failed
exiting with code 1

如您所见,连接尝试失败。我猜这是因为 pod 本身首先处于 pending 状态。

因此,为了获得有关 pod 的更多信息,我退出 mongodb-client pod(我在上面的步骤中创建),然后 运行 以下命令。

kubectl get pod -o yaml

我得到了冗长的输出。

apiVersion: v1
items:
- apiVersion: v1
  kind: Pod
  metadata:
    creationTimestamp: "2021-05-04T12:26:59Z"
    generateName: mongodb-d9b6d589c-
    labels:
      app.kubernetes.io/component: mongodb
      app.kubernetes.io/instance: mongodb
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: mongodb
      helm.sh/chart: mongodb-10.15.0
      pod-template-hash: d9b6d589c
    name: mongodb-d9b6d589c-zpmb6
    namespace: default
    ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: ReplicaSet
      name: mongodb-d9b6d589c
      uid: c99bfa3e-9a8d-425f-acdc-74d8acaba71b
    resourceVersion: "52012"
    uid: 97f77766-f400-424c-9651-9839a7506721
  spec:
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchLabels:
                app.kubernetes.io/component: mongodb
                app.kubernetes.io/instance: mongodb
                app.kubernetes.io/name: mongodb
            namespaces:
            - default
            topologyKey: kubernetes.io/hostname
          weight: 1
    containers:
    - env:
      - name: BITNAMI_DEBUG
        value: "false"
      - name: MONGODB_ROOT_PASSWORD
        valueFrom:
          secretKeyRef:
            key: mongodb-root-password
            name: mongodb
      - name: ALLOW_EMPTY_PASSWORD
        value: "no"
      - name: MONGODB_SYSTEM_LOG_VERBOSITY
        value: "0"
      - name: MONGODB_DISABLE_SYSTEM_LOG
        value: "no"
      - name: MONGODB_DISABLE_JAVASCRIPT
        value: "no"
      - name: MONGODB_ENABLE_JOURNAL
        value: "yes"
      - name: MONGODB_ENABLE_IPV6
        value: "no"
      - name: MONGODB_ENABLE_DIRECTORY_PER_DB
        value: "no"
      image: docker.io/bitnami/mongodb:4.4.5-debian-10-r21
      imagePullPolicy: IfNotPresent
      livenessProbe:
        exec:
          command:
          - mongo
          - --disableImplicitSessions
          - --eval
          - db.adminCommand('ping')
        failureThreshold: 6
        initialDelaySeconds: 30
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
      name: mongodb
      ports:
      - containerPort: 27017
        name: mongodb
        protocol: TCP
      readinessProbe:
        exec:
          command:
          - bash
          - -ec
          - |
            mongo --disableImplicitSessions $TLS_OPTIONS --eval 'db.hello().isWritablePrimary || db.hello().secondary' | grep -q 'true'
        failureThreshold: 6
        initialDelaySeconds: 5
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
      resources: {}
      securityContext:
        runAsNonRoot: true
        runAsUser: 1001
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /bitnami/mongodb
        name: datadir
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-g5kx8
        readOnly: true
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    preemptionPolicy: PreemptLowerPriority
    priority: 0
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext:
      fsGroup: 1001
    serviceAccount: mongodb
    serviceAccountName: mongodb
    terminationGracePeriodSeconds: 30
    tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    volumes:
    - name: datadir
      persistentVolumeClaim:
        claimName: mongodb
    - name: kube-api-access-g5kx8
      projected:
        defaultMode: 420
        sources:
        - serviceAccountToken:
            expirationSeconds: 3607
            path: token
        - configMap:
            items:
            - key: ca.crt
              path: ca.crt
            name: kube-root-ca.crt
        - downwardAPI:
            items:
            - fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
              path: namespace
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2021-05-04T12:26:59Z"
      message: '0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.'
      reason: Unschedulable
      status: "False"
      type: PodScheduled
    phase: Pending
    qosClass: BestEffort
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

但我觉得在输出的最后有一个重要的线索。

message: '0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.'

PVC 似乎出了点问题。现在,当我查看 运行ning

生成的清单时
helm get menifest mongodb

我得到清单如下。

---
# Source: mongodb/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: mongodb
  namespace: default
  labels:
    app.kubernetes.io/name: mongodb
    helm.sh/chart: mongodb-10.15.0
    app.kubernetes.io/instance: mongodb
    app.kubernetes.io/managed-by: Helm
secrets:
  - name: mongodb
---
# Source: mongodb/templates/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: mongodb
  namespace: default
  labels:
    app.kubernetes.io/name: mongodb
    helm.sh/chart: mongodb-10.15.0
    app.kubernetes.io/instance: mongodb
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: mongodb
type: Opaque
data:
  mongodb-root-password: "ck1qamNpTjhBbg=="
---
# Source: mongodb/templates/standalone/pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mongodb
  namespace: default
  labels:
    app.kubernetes.io/name: mongodb
    helm.sh/chart: mongodb-10.15.0
    app.kubernetes.io/instance: mongodb
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: mongodb
spec:
  accessModes:
    - "ReadWriteOnce"
  resources:
    requests:
      storage: "8Gi"
---
# Source: mongodb/templates/standalone/svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: mongodb
  namespace: default
  labels:
    app.kubernetes.io/name: mongodb
    helm.sh/chart: mongodb-10.15.0
    app.kubernetes.io/instance: mongodb
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: mongodb
spec:
  type: ClusterIP
  ports:
    - name: mongodb
      port: 27017
      targetPort: mongodb
      nodePort: null
  selector:
    app.kubernetes.io/name: mongodb
    app.kubernetes.io/instance: mongodb
    app.kubernetes.io/component: mongodb
---
# Source: mongodb/templates/standalone/dep-sts.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb
  namespace: default
  labels:
    app.kubernetes.io/name: mongodb
    helm.sh/chart: mongodb-10.15.0
    app.kubernetes.io/instance: mongodb
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: mongodb
spec:
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app.kubernetes.io/name: mongodb
      app.kubernetes.io/instance: mongodb
      app.kubernetes.io/component: mongodb
  template:
    metadata:
      labels:
        app.kubernetes.io/name: mongodb
        helm.sh/chart: mongodb-10.15.0
        app.kubernetes.io/instance: mongodb
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/component: mongodb
    spec:

      serviceAccountName: mongodb
      affinity:
        podAffinity:

        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: mongodb
                    app.kubernetes.io/instance: mongodb
                    app.kubernetes.io/component: mongodb
                namespaces:
                  - "default"
                topologyKey: kubernetes.io/hostname
              weight: 1
        nodeAffinity:

      securityContext:
        fsGroup: 1001
        sysctls: []
      containers:
        - name: mongodb
          image: docker.io/bitnami/mongodb:4.4.5-debian-10-r21
          imagePullPolicy: "IfNotPresent"
          securityContext:
            runAsNonRoot: true
            runAsUser: 1001
          env:
            - name: BITNAMI_DEBUG
              value: "false"
            - name: MONGODB_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mongodb
                  key: mongodb-root-password
            - name: ALLOW_EMPTY_PASSWORD
              value: "no"
            - name: MONGODB_SYSTEM_LOG_VERBOSITY
              value: "0"
            - name: MONGODB_DISABLE_SYSTEM_LOG
              value: "no"
            - name: MONGODB_DISABLE_JAVASCRIPT
              value: "no"
            - name: MONGODB_ENABLE_JOURNAL
              value: "yes"
            - name: MONGODB_ENABLE_IPV6
              value: "no"
            - name: MONGODB_ENABLE_DIRECTORY_PER_DB
              value: "no"
          ports:
            - name: mongodb
              containerPort: 27017
          livenessProbe:
            exec:
              command:
                - mongo
                - --disableImplicitSessions
                - --eval
                - "db.adminCommand('ping')"
            initialDelaySeconds: 30
            periodSeconds: 10
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 6
          readinessProbe:
            exec:
              command:
                - bash
                - -ec
                - |
                  mongo --disableImplicitSessions $TLS_OPTIONS --eval 'db.hello().isWritablePrimary || db.hello().secondary' | grep -q 'true'
            initialDelaySeconds: 5
            periodSeconds: 10
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 6
          resources:
            limits: {}
            requests: {}
          volumeMounts:
            - name: datadir
              mountPath: /bitnami/mongodb
              subPath:
      volumes:
        - name: datadir
          persistentVolumeClaim:
            claimName: mongodb

总而言之,以下是上述清单所代表的 5 种不同类型的对象。

kind: ServiceAccount
kind: Secret
kind: PersistentVolumeClaim
kind: Service
kind: Deployment

我们可以看到有 PersistentVolumeClaim,但没有 PersistentVolume。

我想,我按照 here 给出的说明在 kubernetese 上安装了 mongo 数据库图表。

那里没有提到 PersistentVolume。我在这里错过了什么吗?我必须以某种方式自己创建持久卷吗?

所以问题是

  1. 为什么 pod 会无限期地处于 pending 状态?

  2. 为什么没有创建 Persistant Volume 对象(我用命令 kubectl get pv --all-namespaces 检查过)

  3. 最后令我困惑的是,当尝试获取日志时,我什么也没看到!!

    vagrant@kmasterNew:~$ kubectl get pods
    NAME                      READY   STATUS    RESTARTS   AGE
    mongodb-d9b6d589c-zpmb6   0/1     Pending   0          60m
    vagrant@kmasterNew:~$ kubectl logs mongodb-d9b6d589c-zpmb6
    vagrant@kmasterNew:~$ 
    

将其从评论中移除,因为我能够使用 kubeadm.

在 kubernetes 集群设置上重现它

1 - 它处于待定状态,因为它没有要继续的持久卷。 可以通过以下方式检查:

kubectl get pvc 输出是:

kubectl get pvc
NAME      STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mongodb   Pending                                                     8s

然后kubectl describe pvc mongodb

Name:          mongodb
Namespace:     default
StorageClass:  
Status:        Pending
Volume:        
Labels:        app.kubernetes.io/component=mongodb
               app.kubernetes.io/instance=mongodb
               app.kubernetes.io/managed-by=Helm
               app.kubernetes.io/name=mongodb
               helm.sh/chart=mongodb-10.15.0
Annotations:   meta.helm.sh/release-name: mongodb
               meta.helm.sh/release-namespace: default
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       mongodb-d9b6d589c-7mbf8
Events:
  Type    Reason         Age               From                         Message
  ----    ------         ----              ----                         -------
  Normal  FailedBinding  2s (x8 over 97s)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

2 - 有 three main prerequisites 从 bitnami/mongodb 图表开始:

  • Kubernetes 1.12+
  • Helm 3.1.0
  • PV provisioner support in the underlying infrastructure

在您的情况下,pod 无法启动,因为它没有创建 PersistentVolume。发生这种情况是因为没有使用供应商。例如。在云或 minikube 中,它会自动为您处理,而对于裸机集群,您应该自己处理。以下是您如何操作的两个示例:

您可以检查是否有任何存储 类 和配置器用于:

kubectl get storageclasses

3 - 您看不到日志,因为容器甚至没有启动。你可以随时参考troubleshooting pending or crashing pods