GKE v1.14 上的 nginx 入口控制器 0.26.1 returns 504（连接到上游时超时）

Question

我最近将我的 gke 集群升级到 1.14.x 并将 nginx ingress 升级到最新版本 0.26.1。在某些时候，我的入口停止工作。

例如，当尝试使用 curl INGRESS_IP -H "host:nexus.myorg.com" 访问 Nexus 时，这些是入口控制器日志：

2019/11/07 08:35:49 [error] 350#350: *2664 upstream timed out (110: Connection timed out) while connecting to upstream, client: 82.81.2.76, server: nexus.myorg.com, request: "GET / HTTP/1.1", upstream: "http://10.8.25.3:8081/", host: "nexus.myorg.com"
2019/11/07 08:35:54 [error] 350#350: *2664 upstream timed out (110: Connection timed out) while connecting to upstream, client: 82.81.2.76, server: nexus.myorg.com, request: "GET / HTTP/1.1", upstream: "http://10.8.25.3:8081/", host: "nexus.myorg.com"
2019/11/07 08:35:59 [error] 350#350: *2664 upstream timed out (110: Connection timed out) while connecting to upstream, client: 82.81.2.76, server: nexus.myorg.com, request: "GET / HTTP/1.1", upstream: "http://10.8.25.3:8081/", host: "nexus.myorg.com"
82.81.2.76 - - [07/Nov/2019:08:35:59 +0000] "GET / HTTP/1.1" 504 173 "-" "curl/7.64.1" 79 15.003 [some-namespace-nexus-service-8081] [] 10.8.25.3:8081, 10.8.25.3:8081, 10.8.25.3:8081 0, 0, 0 5.001, 5.001, 5.001 504, 504, 504 a03f13a3bfc943e44f2df3d82a6ecaa4

如您所见，它尝试连接到 pod IP 10.8.25.3:8081 三次，但都超时。

我进入了一个 pod 并使用相同的 IP 访问了该 pod，没有问题：curl 10.8.25.3:8081。所以服务设置正确。

这是我的 Ingress 配置：

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  namespace: some-namespace
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/add-base-url: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: 30M
spec:
  rules:
  - host: nexus.myorg.com
    http:
      paths:
      - backend:
          serviceName: nexus-service
          servicePort: 8081

知道如何解决这个问题吗？

Answer 1

问题与网络策略有关。我们有一些政策禁止从其他命名空间访问 pods，只允许从入口命名空间访问

  apiVersion: extensions/v1beta1
  kind: NetworkPolicy
  metadata:
    name: allow-from-ingress-namespace
    namespace: some-namespace
  spec:
    ingress:
    - from:
      - namespaceSelector:
          matchLabels:
            type: ingress
    podSelector: {}
    policyTypes:
    - Ingress

  apiVersion: extensions/v1beta1
  kind: NetworkPolicy
  metadata:
    name: deny-from-other-namespaces
    namespace: some-namespace
  spec:
    ingress:
    - from:
      - podSelector: {}
    podSelector: {}
    policyTypes:
    - Ingress

随着升级，我们丢失了策略中匹配的标签 (type=ingress)。只需添加它即可解决问题：kubectl label namespaces ingress-nginx type=ingress

GKE v1.14 上的 nginx 入口控制器 0.26.1 returns 504（连接到上游时超时）

nginx ingress controller 0.26.1 returns 504 (timeout while connecting to upstream) on GKE v1.14

kubernetes

google-kubernetes-engine

kubernetes-ingress

nginx-ingress