如何在 GKE 中使用 HTTPS 部署 Echo 应用程序？

Question

使用 Echo framework developed a web app. Set https://<DOMAIN> feature with its Auto TLS.

package main

import (
    "net/http"

    "github.com/labstack/echo/v4"
    "github.com/labstack/echo/v4/middleware"
    "golang.org/x/crypto/acme/autocert"
)

func main() {
    e := echo.New()
    env := os.Getenv("ENV")
    if env == "prod" {
        e.AutoTLSManager.HostPolicy = autocert.HostWhitelist("arealdomain.com")
        e.AutoTLSManager.Cache = autocert.DirCache("/var/www/cert")
        e.Pre(middleware.HTTPSWWWRedirect())
    }

    e.GET("/healthcheck", func(c echo.Context) error {
        return c.JSON(http.StatusOK, {"ok"})
    })

    switch env {
    case "prod":
        e.Logger.Fatal(e.StartAutoTLS(":8443"))
    case "dev":
        e.Logger.Fatal(e.Start(":9000"))
    default:
        e.Logger.Fatal(e.Start(":9000"))
    }
}

部署在 Kubernetes 中。

development.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: testapp
spec:
  selector:
    matchLabels:
      app: testapp
  replicas: 3
  template:
    metadata:
      labels:
        app: testapp
    spec:
      containers:
        - name: testapp
          image: gcr.io/<PROJECT_ID>/testapp
      ports:
      - containerPort: 9000
      - containerPort: 8443
      livenessProbe:
        initialDelaySeconds: 10
        periodSeconds: 10
        exec:
          command:
            - "true"
      readinessProbe:
        initialDelaySeconds: 5
        periodSeconds: 20
        httpGet:
          path: /healthcheck
          port: 9000

service.yml

apiVersion: v1
kind: Service
metadata:
  name: testapp
spec:
  type: NodePort
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 9000
  selector:
    app: testapp

ingress.yml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: testingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: testip // a real IP
    networking.gke.io/managed-certificates: testcertificate
    kubernetes.io/ingress.class: "gce"
spec:
  backend:
    serviceName: testapp
    servicePort: 80

managedcertificate.yml

apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
  name: testcertificate
spec:
  domains:
    - arealdomain.com  // a real domain

部署这些资源后，访问域arealdomain.com，出现502错误：

Error: Server Error
The server encountered a temporary error and could not complete your request.
Please try again in 30 seconds.

从 GCP 的 network loadbalancer，得到一项服务不健康。

可能是防火墙问题：https://cloud.google.com/load-balancing/docs/https/ext-http-lb-simple#firewall

检查kubernetes的deployment pod，发现这个错误：

Readiness probe failed: HTTP probe failed with statuscode: 400

由于要部署这个web app可以通过https使用，所以只部署了8443端口。使用像这样的网站：https://arealdomain.com 但是如果做healthcheck，GCP需要其他端口吗？将此部署到 GCP 的最佳做法是什么？应用内9000和8443端口都需要用Nginx服务吗？

更新

当 Ingress 改为使用 80 端口，同时部署应用到 9000 端口时，pod 可以启动，但 neg 检查似乎无法通过。

  Events:
  Type    Reason                             Age    From                                                  Message
  ----    ------                             ----   ----                                                  -------
  Normal  LoadBalancerNegNotReady            115s   neg-readiness-reflector                               Waiting for pod to become healthy in at least one of the NEG(s): [k8s1-173zho00-default-testapp-80-l4cadn29]
  Normal  Scheduled                          115s   default-scheduler                                     Successfully assigned default/testapp-2c6f02f021-afjls to gke-<project_name>-default-pool-asfjo2c5-afjl
  Normal  Pulling                            114s   kubelet, gke-<project_name>-default-pool-asfdl0-asl3  Pulling image "gcr.io/<project_name>/testapp"
  Normal  Pulled                             109s   kubelet, gke-<project_name>-default-pool-asfdl0-asl3  Successfully pulled image "gcr.io/<project_name>/testapp"
  Normal  Created                            109s   kubelet, gke-<project_name>-default-pool-asfdl0-asl3  Created container testapp
  Normal  Started                            109s   kubelet, gke-<project_name>-default-pool-asfdl0-asl3  Started container testapp
  Normal  LoadBalancerNegWithoutHealthCheck  94s    neg-readiness-reflector                               Pod is in NEG "Key{\"k8s1-173zho00-default-testapp-80-l4cadn29\", zone: \"southamerica-east1-c\"}". NEG is not attached to any BackendService with health checking. Marking condition "cloud.google.com/load-balancer-neg-ready" to True.

我找到了这个article，如何将cloud.google.com/load-balancer-neg-ready设置为True？

GCP 网络的 Load balancer details 仍然显示不健康。

根据 Health Check 的详细信息，路径未设置为 /healthcheck，而是 /。哪里错了？

Answer 1

如果您刚开始使用 GKE，我建议您只创建服务和部署并使用 UI 创建入口和托管证书

我创建并部署了一个示例应用程序：

main.go

中的代码

package main

import (
    "log"
    "net/http"
)

func main() {
    // change this handlers for echo handlers
    http.HandleFunc("/", http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
        rw.WriteHeader(http.StatusOK)
        rw.Write([]byte("Hello World..."))
    }))
    http.HandleFunc("/health", http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
        rw.WriteHeader(http.StatusOK)
    }))
    log.Panic(http.ListenAndServe(":8080", nil))
}

Docker 文件

FROM golang:alpine AS builder
RUN apk add --no-cache git
WORKDIR /go/src/app
COPY . .
RUN go build -o bin main.go

#final stage
FROM alpine:latest
RUN apk --no-cache add ca-certificates
COPY --from=builder /go/src/app/bin /app
ENTRYPOINT ./app
EXPOSE 8080

k8s-artifacts.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: testapp
spec:
  selector:
    matchLabels:
      app: testapp
  replicas: 3
  template:
    metadata:
      labels:
        app: testapp
    spec:
      containers:
        - name: testapp
          image: gcr.io/<ACCOUNT>/test  
          ports:
            - containerPort: 8080
          livenessProbe:
            initialDelaySeconds: 10
            periodSeconds: 10
            exec:
              command:
                - "true"
          readinessProbe:
            initialDelaySeconds: 5
            periodSeconds: 20
            httpGet:
              path: /health
              port: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: testapp
spec:
  type: NodePort
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 8080
  selector:
    app: testapp
---
apiVersion: "extensions/v1beta1"
kind: "Ingress"
metadata:
  name: "lb-2"
  namespace: "default"
spec:
  backend:
    serviceName: "testapp"
    servicePort: 80

有了它，您将至少有一个可以通过互联网访问的 http 入口。之后，当您确认您的服务已启动并且运行时，您可以编辑负载均衡器的前端以添加 https 规则和您的托管证书

更新

验证 lb 已启动并且运行

要编辑负载均衡器，请转到入口详细信息

页面底部有指向由 GCE 管理的负载均衡器的链接

Select 负载均衡器并编辑它

转到前端配置并为 https 配置新的前端和端口

更新 2

您也可以使用托管证书直接创建新入口。例如：首先，创建托管证书

apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
  name: test-cert
spec:
  domains:
    - test-domain.com

之后，只需在 GKE 的服务和入口部分创建一个新入口

如何在 GKE 中使用 HTTPS 部署 Echo 应用程序？

How to deploy an Echo app with HTTPS in GKE?

https

go

google-cloud-platform

kubernetes

google-kubernetes-engine

更新