.Net 5 Kestrel 背后的 Kubernetes 负载均衡器无法识别的 HTTP 版本

Question

我有一个 .NET 5 API 应用运行在 Docker 容器中安装了一个 Kestrel 服务器。我可以在 Docker 容器内运行本地应用程序，没有任何问题。当我部署到 Kubernetes 时会出现问题，而 Kubernetes 的 LoadBalancer 似乎会导致问题。

我反复收到以下错误：

←[40m←[37mdbug←[39m←[22m←[49m: Microsoft.AspNetCore.Server.Kestrel[17]
      Connection id "0HMAMHC4BPDGN" bad request data: "Unrecognized HTTP version: '10.98.6.196 10.123.90.100 62168 443'"
      Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Unrecognized HTTP version: '10.98.6.196 10.123.90.100 62168 443'
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpParser`1.RejectUnknownVersion(Int32 offset, ReadOnlySpan`1 requestLine)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpParser`1.ParseRequestLine(TRequestHandler handler, ReadOnlySpan`1 requestLine)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpParser`1.ParseRequestLine(TRequestHandler handler, SequenceReader`1& reader)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.ParseRequest(SequenceReader`1& reader)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.TryParseRequest(ReadResult result, Boolean& endConnection)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
←[40m←[37mdbug←[39m←[22m←[49m: Microsoft.AspNetCore.Server.Kestrel[10]
      Connection id "0HMAMHC4BPDGN" disconnecting.
←[40m←[37mdbug←[39m←[22m←[49m: Microsoft.AspNetCore.Server.Kestrel[2]
      Connection id "0HMAMHC4BPDGN" stopped.
←[40m←[37mdbug←[39m←[22m←[49m: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[7]
      Connection id "0HMAMHC4BPDGN" sending FIN because: "The Socket transport's send loop completed gracefully."

我的 kubernetes 基础设施部署。它正在将所有 443 个请求映射到运行ning Docker 容器的端口 80。

kind: ConfigMap
apiVersion: v1
metadata:
  name: api-base
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-base
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api-base
  template:
    metadata:
      labels:
        app: api-base
    spec:
      containers:
      - name: api-base
        image: /path/to/image      
        ports:
        - containerPort: 80
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
   name: api-base
   labels:
      app: api-base
   annotations:
      service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: mycert/path
      service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
spec:
   selector:
      app: api-base
   ports:
    - port: 443
      targetPort: 80
      protocol: TCP
      name: https
   type: LoadBalancer

Docker文件（缩写）暴露80端口，配置.net core到80端口运行。

FROM mcr.microsoft.com/dotnet/aspnet:5.0 
WORKDIR /app
EXPOSE 80
ENV ASPNETCORE_URLS http://*:80
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "API.dll"]

Startup.cs

public class Startup
{
    public Startup(IConfiguration configuration)
    {
        Configuration = configuration;
    }

    public IConfiguration Configuration { get; }

    private const string CorsOriginName = "AllowedOrigins";

    public void ConfigureServices(IServiceCollection services)
    {

        services.AddCors(options =>
        {
            options.AddPolicy(name: CorsOriginName,
                builder =>
                {
                    builder.AllowAnyOrigin()
                        .AllowAnyMethod()
                        .AllowAnyHeader();
                });
        });

        services.Configure<ForwardedHeadersOptions>(options =>
        {
            options.ForwardedHeaders =
                ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
        });

        services.AddControllers();
        services.AddSwaggerGen(c =>
        {
            c.SwaggerDoc("v1", new OpenApiInfo { Title = "API", Version = "v1" });
        });
    }

    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
        }

        app.UseSwagger();
        app.UseSwaggerUI(c => c.SwaggerEndpoint("/swagger/v1/swagger.json", "API v1"));

        app.UseCors(CorsOriginName);
        app.UseForwardedHeaders();
        app.UseStaticFiles();

        app.UseRouting();

        app.UseAuthentication();
        app.UseAuthorization();

        app.UseEndpoints(endpoints =>
        {
            endpoints.MapControllers();
        });
    }
}

Program.cs

public class Program
{
    public static void Main(string[] args)
    {
        CreateHostBuilder(args).Build().Run();
    }

    public static IHostBuilder CreateHostBuilder(string[] args) =>
        Host.CreateDefaultBuilder(args)
            .UseServiceProviderFactory(new AutofacServiceProviderFactory())
            .ConfigureAppConfiguration((hostContext, builder) =>
            {
                // Add other providers for JSON, etc.
                if (hostContext.HostingEnvironment.IsDevelopment())
                {
                    builder.AddUserSecrets<Program>();
                }
            })
            .ConfigureWebHostDefaults(webBuilder =>
            {
                webBuilder.UseKestrel().UseStartup<Startup>();
            });
 }

如何配置 Kestrel 以接受来自 LoadBalancer 的请求？

更新：

为此，我终止了 https 流量并使用 http

与容器通信

apiVersion: v1
kind: Service
metadata:
  name: api-base
  annotations:
    # Note that the backend talks over HTTP.
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    # TODO: Fill in with the ARN of your certificate.
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:{region}:{user id}:certificate/{id}
    # Only run SSL on the port named "https" below.
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
spec:
  selector:
    app: api-base
  ports:
  - name: http
    port: 80
    targetPort: 80
  - name: https
    port: 443
    targetPort: 80
  type: LoadBalancer

https://aws.amazon.com/premiumsupport/knowledge-center/terminate-https-traffic-eks-acm/

Answer 1

您需要决定是否要在您的应用程序中或在此之前终止 https。我建议您使用 Ingress 而不是 LB 类型的服务来将您的服务暴露给外部世界。这样你就可以在它自己的入口上配置 https 终止，并且应用程序将保持基于 http，无需在它自己的应用程序上配置证书、加密等，尽管你仍然需要正确配置入口。

.Net 5 Kestrel 背后的 Kubernetes 负载均衡器无法识别的 HTTP 版本

.Net 5 Kestrel behind Kubernetes Load Balancer Unrecognized HTTP Version

docker

kubernetes

asp.net-core

.net-5