AKS 外部负载均衡器未与 PODS 通信

Question

我创建了一个存储在 Azure 容器注册表中的 ASP.NET CORE 2.x 应用程序 Kestrel 图像，其中证书存储在 Azure 存储中，证书密码是秘密的。我已经确认图像是运行在本地使用自签名证书。我已将图像推送到 ACR，它正在成功拉入 AKS。当我远程进入 pod 时，我可以卷曲并看到 kestrel 响应。

我认为我在 AKS 群集中创建的外部负载平衡器没有转发到我的 POD。当我调用提供的外部 IP 端点时，出现超时，我不明白为什么。

我按照这里的 k8s 调试服务指南进行操作，但是，我仍然不明白我做错了什么： https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/

这是我的 DockerFile：

FROM microsoft/dotnet:2.2-sdk AS build
WORKDIR /app

# set up node
ENV NODE_VERSION 10.15.1
ENV NODE_DOWNLOAD_SHA ca1dfa9790876409c8d9ecab7b4cdb93e3276cedfc64d56ef1a4ff1778a40214
RUN curl -SL "https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-x64.tar.gz" --output nodejs.tar.gz \
    && echo "$NODE_DOWNLOAD_SHA nodejs.tar.gz" | sha256sum -c - \
    && tar -xzf "nodejs.tar.gz" -C /usr/local --strip-components=1 \
    && rm nodejs.tar.gz \
    && ln -s /usr/local/bin/node /usr/local/bin/nodejs

# copy and build projects
COPY . .
RUN dotnet restore
RUN dotnet publish -c Release -o out
WORKDIR /app/MyApp


FROM microsoft/dotnet:2.2-aspnetcore-runtime AS runtime
WORKDIR /app
COPY --from=build /app/MyApp/out ./
ENV ASPNETCORE_URLS https://+:443
ENV ASPNETCORE_Kestrel__Certificates__Default__Path /mnt/certs/MyApp.pfx
ENTRYPOINT ["dotnet", "MyApp.dll"]

这是我的部署

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-website
  labels:
    app: myapp-website
spec:
  replicas: 1
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      app: myapp-website
  template:
    metadata:
      labels:
        app: myapp-website
    spec:
      nodeSelector:
        "beta.kubernetes.io/os": linux    
      containers:
      - name: myapp-website
        image: myappdev.azurecr.io/myapp:v1
        ports:
        - containerPort: 443
        env:
        - name: ASPNETCORE_ENVIRONMENT
          value: dev
        - name: ASPNETCORE_Kestrel__Certificates__Default__Password
          valueFrom:
            secretKeyRef:
              name: myapp-secrets
              key: cert-pass   
        volumeMounts:
        - name: certs
          mountPath: /mnt/certs
          readOnly: true
      imagePullSecrets:
      - name: docker-reg-credential
      volumes:
      - name: certs
        azureFile:
          secretName: myapp-secrets
          shareName: myapp/certs
          readOnly: true   
---
kind: Service
apiVersion: v1
metadata:
  name: myapp-website
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
spec:
  type: LoadBalancer
  ports:
  - name:
    port: 443
    targetPort: 443
  selector:
    app: myapp-website

Kubectl 描述服务：

# kubectl describe services                                                                               
Name:              kubernetes                                                                             
Namespace:         default                                                                                
Labels:            component=apiserver                                                                    
                   provider=kubernetes                                                                    
Annotations:       <none>                                                                                 
Selector:          <none>                                                                                 
Type:              ClusterIP                                                                              
IP:                10.0.0.1                                                                               
Port:              https  443/TCP                                                                         
TargetPort:        443/TCP                                                                                
Endpoints:         172.31.3.237:443                                                                       
Session Affinity:  None                                                                                   
Events:            <none>                                                                                 


Name:                     myapp-website                                                                    
Namespace:                default                                                                         
Labels:                   <none>                                                                          
Annotations:              kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Serv
ice","metadata":{"annotations":{"service.beta.kubernetes.io/azure-load-balancer-internal":"true"},"name":"
myapp-website",...                                                                                         
                          service.beta.kubernetes.io/azure-load-balancer-internal=true                    
Selector:                 app=myapp-website                                                                
Type:                     LoadBalancer                                                                    
IP:                       10.0.194.20                                                                     
LoadBalancer Ingress:     10.240.0.7                                                                      
Port:                     <unset>  443/TCP                                                                
TargetPort:               443/TCP                                                                         
NodePort:                 <unset>  30872/TCP                                                              
Endpoints:                10.244.1.53:443                                                                 
Session Affinity:         None                                                                            
External Traffic Policy:  Cluster                                                                         
Events:                   <none>

kubectl 描述 pods:

# kubectl describe pods
Name:               myapp-website-9d89dd8b-plvs8
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               aks-agentpool-20448790-1/10.240.0.5
Start Time:         Thu, 04 Jul 2019 23:36:11 -0700
Labels:             app=myapp-website
                    pod-template-hash=9d89dd8b
Annotations:        <none>
Status:             Running
IP:                 10.244.1.43
Controlled By:      ReplicaSet/myapp-website-9d89dd8b
Containers:
  myapp-website:
    Container ID:   docker://fb3c91fe13f74b8e970f786e3cb70e1875c3e2d1d2198ac2e761120dc4655134
    Image:          myappdev.azurecr.io/transimyappapp:v1
    Image ID:       docker-pullable://myappdev.azurecr.io/transimyappapp@sha256:1bb5cb2cae935866e538f231c08ff33188ac4bed94a662a86a3c859da3bb28db
    Port:           443/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu, 04 Jul 2019 23:36:12 -0700
    Ready:          True
    Restart Count:  0
    Environment:
      ASPNETCORE_ENVIRONMENT:                               dev
      ASPNETCORE_Kestrel__Certificates__Default__Password:  <set to the key 'cert-pass' in secret 'myapp-secrets'>  Optional: false
      KUBERNETES_PORT_443_TCP_ADDR:                         aks-tso-dev-dns-1b650af7.hcp.canadacentral.azmk8s.io
      KUBERNETES_PORT:                                      tcp://aks-tso-dev-dns-1b650af7.hcp.canadacentral.azmk8s.io:443
      KUBERNETES_PORT_443_TCP:                              tcp://aks-tso-dev-dns-1b650af7.hcp.canadacentral.azmk8s.io:443
      KUBERNETES_SERVICE_HOST:                              aks-tso-dev-dns-1b650af7.hcp.canadacentral.azmk8s.io
    Mounts:
      /mnt/certs from certs (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-fs8sn (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  certs:
    Type:        AzureFile (an Azure File Service mount on the host and bind mount to the pod)
    SecretName:  myapp-secrets
    ShareName:   myapp/certs
    ReadOnly:    true
  default-token-fs8sn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-fs8sn
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

获取端点：

# kubectl get endpoints myapp-website
NAME           ENDPOINTS         AGE
myapp-website   10.244.1.43:443   12h

远程进入 pod 以确保服务响应。

root@myapp-website-9d89dd8b-plvs8:/app# nslookup myapp-website
Server:         10.0.0.10
Address:        10.0.0.10#53

Name:   myapp-website.default.svc.cluster.local
Address: 10.0.194.20

curl -k https://10.0.194.20
# kubectl exec -it myapp-website-9d89dd8b-plvs8 /bin/bash                                                  
root@myapp-website-9d89dd8b-plvs8:/app# curl                                                               
curl: try 'curl --help' or 'curl --manual' for more information                                           
root@myapp-website-9d89dd8b-plvs8:/app# curl -k https://10.0.194.20                                        


<!DOCTYPE html>                                                                                           
<html>... continues

这可能很简单，但是，我似乎无法理解。即将进行 1.5 周的尝试来完成这项工作。请提供帮助，非常感谢您提供的任何帮助。

Answer 1

你有这个注释：

annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"

这基本上是说在 kubernetes 内部网络中公开此服务，而不是在外部。删除此注释，它应该可以工作（其他一切似乎都已正确配置）。

AKS 外部负载均衡器未与 PODS 通信

AKS External Load Balancer is not Communicating with PODS

load-balancing

docker

kubernetes

asp.net-core

azure-aks