为什么 nginx 不接受来自 apache bench 的超过 1024 个连接

Question

我有一个 clojure/jetty 服务器（在端口 8081 上）并使用 nginx（端口 8080）代理它。我一直在尝试单独对 clojure 应用程序以及 nginx 的代理角色进行基准测试。

当我运行直接针对 clojure 进行测试时，我可能会 Connection reset by peer，十分之一运行。通常，测试已完成，性能可以接受。

$ ulimit -n 4096
$ ab -n 20000 -c 2048 -k localhost:8081
...
Concurrency Level:      2048
Time taken for tests:   8.713 seconds
Complete requests:      20000
Failed requests:        0
Keep-Alive requests:    20000
Total transferred:      15160000 bytes
HTML transferred:       11720000 bytes
Requests per second:    2295.43 [#/sec] (mean)
Time per request:       892.208 [ms] (mean)
Time per request:       0.436 [ms] (mean, across all concurrent requests)
Transfer rate:          1699.16 [Kbytes/sec] received
...

我开始测试完整的本地配置，端口 8080 上的 nginx 和端口 8081 上的 clojure。一切顺利，直到超过 1024 个并发连接。

我注意到，使用 ss -tl，接收队列不会激增，或者至少如果它们是在一瞬间。但我确实发现，使用 netstat -s，正在发送大量 TCP RST。有时，dmesg 告诉我似乎发生了 SYN 泛洪。此外，nginx 正在响应 HTTP 状态 499，这应该表示客户端关闭连接...

所以，我的诊断是交叉的，apache bench 和 nginx 都声称对方关闭了连接！？

$ ulimit -n 4096
$ ab -n 20000 -c 2048 -k localhost:8080
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
apr_socket_recv: Connection reset by peer (104)
Total of 26 requests completed

配置

我将 nginx 和 clojure 的 ulimit 最大打开文件数设置为 4096。

无效的网络更改

net.core.netdev_max_backlog=30000
# yes, we are using jumbo frames
net.ipv4.tcp_mtu_probing=1

net.core.somaxconn=4096
net.ipv4.ip_local_port_range=4096     61000
net.ipv4.tcp_fin_timeout=30
net.ipv4.tcp_max_syn_backlog=2048

/etc/nginx/nginx.conf

user www-data;
worker_processes  2;
worker_rlimit_nofile 100000;

error_log  /var/log/nginx/error.log;
pid        /run/nginx.pid;

events {
  worker_connections  2048;
  use epoll;
}

http {

  include       /etc/nginx/mime.types;
  default_type  application/octet-stream;


  access_log    /var/log/nginx/access.log;

  sendfile on;
  tcp_nopush on;
  tcp_nodelay on;

  keepalive_requests 1000;
  keepalive_timeout  65;


  gzip  on;
  gzip_http_version 1.0;
  gzip_comp_level 2;
  gzip_proxied any;
  gzip_vary off;
  gzip_types text/plain text/css application/x-javascript text/xml application/xml application/rss+xml application/atom+xml text/javascript application/javascript application/json text/mathml;
  gzip_min_length  1000;
  gzip_disable     "MSIE [1-6]\.";


  variables_hash_max_size 1024;
  variables_hash_bucket_size 64;
  server_names_hash_bucket_size 128;
  types_hash_max_size 2048;
  types_hash_bucket_size 64;



  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/sites-enabled/*;
}

/etc/nginx/sites-enabled/ups

upstream ups {
  server localhost:8081 fail_timeout=0;
}


server {
  listen 8080 backlog=1024;

  server_name example.com;

  proxy_buffer_size   128k;
  proxy_buffers   4 256k;
  proxy_busy_buffers_size   256k;

  client_max_body_size 3M;

  large_client_header_buffers 4 128k;

  proxy_read_timeout 300;
  proxy_send_timeout 300;
  send_timeout 300;
  keepalive_timeout 300;

  server_tokens off;

  access_log /var/log/nginx/ups_access.log enhanced-combined;
  error_log /var/log/nginx/ups_error.log;
  root /apps/ups/current/public/;

  error_page 403 /errors/403_maintenance.html;
  error_page 500 /errors/500.html;
  location ^~ /errors/ {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-Request-Id $request_uuid;
    satisfy any;
    allow all;
  }

  location / {

    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-Request-Id $request_uuid;

    if ($http_x_forwarded_proto = 'http') {
      rewrite ^ https://$host$request_uri? permanent;
    }


    if (-f $request_filename/index.html) {
      rewrite (.*) /index.html break;
    }

    if (-f $request_filename.html) {
      rewrite (.*) .html break;
    }

    if (!-f $request_filename) {
      proxy_pass http://ups;
      break;
    }
  }
}

Answer 1

@Terra 几乎找到了答案。

accept_mutex off;
worker_connections 4096;

我尝试单独添加每个更改并重新加载，但我仍然看到相同的错误。直到我改变了两者，我才可以超过 1024 个连接。

看来，由于代理，我需要有两倍于我打算接受的工作连接数。

为什么 nginx 不接受来自 apache bench 的超过 1024 个连接

Why won't nginx accept more than 1024 connections from apache bench

linux

benchmarking

tcp

clojure

nginx

配置

无效的网络更改

/etc/nginx/nginx.conf

/etc/nginx/sites-enabled/ups