在 https 通道上接收到明文 http 流量，关闭连接

Question

我已经在我的 k8s 集群上部署了 ECK（使用 helm），我正在尝试按照文档安装 elasticsearch。 https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-elasticsearch.html

我已经在外部公开了 service/elasticsearch-prod-es-http 以便我可以从我的 k8s 集群外部连接到它。但是，正如您所看到的，当我尝试从 curl 或浏览器连接到它时，我收到错误“502 Bad Gateway”错误。

curl elasticsearch.dev.acme.com
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
</body>
</html>

检查 pod (elasticsearch-prod-es-default-0) 后，我可以看到重复出现以下消息。

{"type": "server", "timestamp": "2021-04-27T13:12:20,048Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch-prod", "node.name": "elasticsearch-prod-es-default-0", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.0.5.81:9200, remoteAddress=/10.0.3.50:46380}", "cluster.uuid": "t0mRfv7kREGQhXW9DVM3Vw", "node.id": "nCyAItDmSqGZRa3lApsC6g" }

你能帮我理解为什么会发生这种情况以及如何解决它吗？

我怀疑这与我的 TLS 配置有关，因为当我禁用 TLS 时，我可以毫无问题地从外部连接到它。但是在生产环境中，我认为保持启用 TLS 很重要？

仅供参考，我能够端口转发服务并使用 -k 标志通过 curl 连接到它。

我试过的

我已尝试将我的域添加到此处所述的部分 https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-http-settings-tls-sans.html#k8s-elasticsearch-http-service-san
我曾尝试使用 openssl 生成自签名证书，但没有成功。尝试在本地连接 returns 出现以下错误消息。

curl -u "elastic:$PASSWORD" "https://localhost:9200" curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.haxx.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.

我尝试使用工具生成证书 https://www.elastic.co/guide/en/elasticsearch/reference/7.9/configuring-tls.html#tls-transport

bin/elasticsearch-certutil ca bin/elasticsearch-certutil cert --ca elastic-stack-ca.12 --pem

然后使用生成的 .crt 和 .key 创建了一个 kubectl 密钥 elastic-tls-cert。但是再次在没有 -k 的情况下卷曲本地主机给出了以下错误：

curl --cacert cacert.pem -u "elastic:$PASSWORD" -XGET "https://localhost:9200" curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.haxx.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.

elasticsearch.yml

# This sample sets up an Elasticsearch cluster with 3 nodes.
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-prod
  namespace: elastic-system
spec:
  version: 7.12.0
  nodeSets:
  - name: default
    config:
      # most Elasticsearch configuration parameters are possible to set, e.g: node.attr.attr_name: attr_value
      node.roles: ["master", "data", "ingest", "ml"]
      # this allows ES to run on nodes even if their vm.max_map_count has not been increased, at a performance cost
      node.store.allow_mmap: false
      xpack.security.enabled: true
    podTemplate:
      metadata:
        labels:
          # additional labels for pods
          foo: bar
      spec:
        nodeSelector: 
          acme/node-type: ops

        # this changes the kernel setting on the node to allow ES to use mmap
        # if you uncomment this init container you will likely also want to remove the
        # "node.store.allow_mmap: false" setting above
        # initContainers:
        # - name: sysctl
        #   securityContext:
        #     privileged: true
        #   command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        ###
        # uncomment the line below if you are using a service mesh such as linkerd2 that uses service account tokens for pod identification.
        # automountServiceAccountToken: true
        containers:
        - name: elasticsearch
          # specify resource limits and requests
          resources:
            limits:
              memory: 4Gi
              cpu: 1
          env:
          - name: ES_JAVA_OPTS
            value: "-Xms2g -Xmx2g"
    count: 3
  #   # request 2Gi of persistent data storage for pods in this topology element
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 250Gi
        storageClassName: elasticsearch
  # # inject secure settings into Elasticsearch nodes from k8s secrets references
  # secureSettings:
  # - secretName: ref-to-secret
  # - secretName: another-ref-to-secret
  #   # expose only a subset of the secret keys (optional)
  #   entries:
  #   - key: value1
  #     path: newkey # project a key to a specific path (optional)
  http:
    service:
      spec:
        # expose this cluster Service with a LoadBalancer
        type: NodePort
    # tls:
      # selfSignedCertificate:
        # add a list of SANs into the self-signed HTTP certificate
        subjectAltNames:
        # - ip: 192.168.1.2
        # - ip: 192.168.1.3
        # - dns: elasticsearch.dev.acme.com
        # - dns: localhost
      # certificate:
      #   # provide your own certificate
      #   secretName: elastic-tls-cert

kubectl 版本

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.6-eks-49a6c0", GitCommit:"49a6c0bf091506e7bafcdb1b142351b69363355a", GitTreeState:"clean", BuildDate:"2020-12-23T22:10:21Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

头盔列表

    NAME                NAMESPACE       REVISION    UPDATED                                 STATUS      CHART               APP VERSION
elastic-operator    elastic-system  1           2021-04-26 11:18:02.286692269 +0100 BST deployed    eck-operator-1.5.0  1.5.0

资源

pod/elastic-operator-0                1/1     Running   0          4h58m   10.0.5.142   ip-10-0-5-71.us-east-2.compute.internal    <none>           <none>
pod/elasticsearch-prod-es-default-0   1/1     Running   0          9m5s    10.0.5.81    ip-10-0-5-71.us-east-2.compute.internal    <none>           <none>
pod/elasticsearch-prod-es-default-1   1/1     Running   0          9m5s    10.0.1.128   ip-10-0-1-207.us-east-2.compute.internal   <none>           <none>
pod/elasticsearch-prod-es-default-2   1/1     Running   0          9m5s    10.0.5.60    ip-10-0-5-71.us-east-2.compute.internal    <none>           <none>

NAME                                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE    SELECTOR
service/elastic-operator-webhook          ClusterIP   172.20.218.208   <none>        443/TCP          26h    app.kubernetes.io/instance=elastic-operator,app.kubernetes.io/name=elastic-operator
service/elasticsearch-prod-es-default     ClusterIP   None             <none>        9200/TCP         9m5s   common.k8s.elastic.co/type=elasticsearch,elasticsearch.k8s.elastic.co/cluster-name=elasticsearch-prod,elasticsearch.k8s.elastic.co/statefulset-name=elasticsearch-prod-es-default
service/elasticsearch-prod-es-http        NodePort    172.20.229.173   <none>        9200:30604/TCP   9m6s   common.k8s.elastic.co/type=elasticsearch,elasticsearch.k8s.elastic.co/cluster-name=elasticsearch-prod
service/elasticsearch-prod-es-transport   ClusterIP   None             <none>        9300/TCP         9m6s   common.k8s.elastic.co/type=elasticsearch,elasticsearch.k8s.elastic.co/cluster-name=elasticsearch-prod

aws alb 入口控制器

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: elastic-ingress
  namespace: elastic-system
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group.name: "<redacted>"
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80,"HTTPS": 443}]'
    alb.ingress.kubernetes.io/certificate-arn: <redacted>
    alb.ingress.kubernetes.io/tags: Environment=prod,Team=dev
    alb.ingress.kubernetes.io/healthcheck-path: /health
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: '300'
    alb.ingress.kubernetes.io/load-balancer-attributes: access_logs.s3.enabled=true,access_logs.s3.bucket=acme-aws-ingress-logs,access_logs.s3.prefix=dev-ingress
spec:
  rules:
    - host: elasticsearch.dev.acme.com
      http:
        paths:
          - path: /*
            pathType: Prefix
            backend:
              service:
                name: elasticsearch-prod-es-http
                port:
                  number: 9200
    # - host: kibana.dev.acme.com
    #   http:
    #     paths:
    #       - path: /*
    #         pathType: Prefix
    #         backend:
    #           service:
    #             name: kibana-prod-kb-http
    #             port:
    #               number: 5601

Answer 1

如果以后有人遇到这个问题，请确保您的入口配置正确。错误消息表明它的入口配置错误。

received plaintext http traffic on an https channel, closing connection

在我的例子中，我使用的是 aws-load-balancer-controller。我必须在我的入口附加一个注释，强制连接为 HTTPS 而不是 HTTP。

alb.ingress.kubernetes.io/backend-protocol: "HTTPS"

对于我的情况，通过将上述注释设置到我的入口文件解决了这个问题，它与设置 custom/private TLS 证书无关。

Answer 2

我的解决方案：http => https

Answer 3

您必须禁用 http ssl，为此您必须修改 config/elasticsearch.yml 文件并将相关变量更改为 false：

xpack.security.http.ssl:
   enabled: false
   keystore.path: certs/http.p12

在 https 通道上接收到明文 http 流量，关闭连接

Received plaintext http traffic on an https channel, closing connection

elasticsearch

kubernetes

elastic-stack