php curl localhost 在发出并发请求时很慢
php curl localhost is slow when making concurrent requests
我有一个有趣的问题，我不确定根本原因是什么。我有一个服务器和两个虚拟主机 A 和 B，端口 运行 分别位于 80 和 81。我在 A 上写了一个简单的 PHP 代码，如下所示：
<?php
echo "from A server\n";
B 上的另一个简单 PHP 代码：
<?php
echo "B server:\n";
// create curl resource
$ch = curl_init();
// set url
curl_setopt($ch, CURLOPT_URL, "localhost:81/a.php");
//return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// $output contains the output string
$output = curl_exec($ch);
// close curl resource to free up system resources
curl_close($ch);
echo $output;
当使用 ab 发出并发请求时，我得到以下结果：
ab -n 10 -c 5 http://192.168.10.173/b.php
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.10.173 (be patient).....done
Server Software: nginx/1.10.0
Server Hostname: 192.168.10.173
Server Port: 80
Document Path: /b.php
Document Length: 26 bytes
Concurrency Level: 5
Time taken for tests: 2.680 seconds
Complete requests: 10
Failed requests: 0
Total transferred: 1720 bytes
HTML transferred: 260 bytes
Requests per second: 3.73 [#/sec] (mean)
Time per request: 1340.197 [ms] (mean)
Time per request: 268.039 [ms] (mean, across all concurrent requests)
Transfer rate: 0.63 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 1
Processing: 2 1339 1408.8 2676 2676
Waiting: 2 1339 1408.6 2676 2676
Total: 3 1340 1408.8 2676 2677
Percentage of the requests served within a certain time (ms)
50% 2676
66% 2676
75% 2676
80% 2676
90% 2677
95% 2677
98% 2677
99% 2677
100% 2677 (longest request)
但是发出 1000 个并发级别 1 的请求非常快：
$ ab -n 1000 -c 1 http://192.168.10.173/b.php
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.10.173 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
Server Software: nginx/1.10.0
Server Hostname: 192.168.10.173
Server Port: 80
Document Path: /b.php
Document Length: 26 bytes
Concurrency Level: 1
Time taken for tests: 1.659 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 172000 bytes
HTML transferred: 26000 bytes
Requests per second: 602.86 [#/sec] (mean)
Time per request: 1.659 [ms] (mean)
Time per request: 1.659 [ms] (mean, across all concurrent requests)
Transfer rate: 101.26 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 1
Processing: 1 1 10.3 1 201
Waiting: 1 1 10.3 1 201
Total: 1 2 10.3 1 201
Percentage of the requests served within a certain time (ms)
50% 1
66% 1
75% 1
80% 1
90% 1
95% 1
98% 1
99% 2
100% 201 (longest request)
谁能解释为什么会这样？我真的很想知道根本原因。这是卷曲的问题吗？它感觉不像是网络瓶颈或打开文件问题，因为并发性仅为 5。顺便说一下，我也用 guzzlehttp 尝试了同样的事情，但结果是一样的。我在我的笔记本电脑上使用 ab 并且服务器在同一个本地网络中。另外，它肯定与网络带宽无关，因为主机 A 和 B 之间的请求是在本地主机上完成的。
我修改了代码，让测试更灵活：
<?php
require 'vendor/autoload.php';
use GuzzleHttp\Client;
$opt = 1;
$url = 'http://localhost:81/a.php';
switch ($opt) {
case 1:
// create curl resource
$ch = curl_init();
// set url
curl_setopt($ch, CURLOPT_URL, $url);
//return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// $output contains the output string
$output = curl_exec($ch);
curl_close($ch);
echo $output;
break;
case 2:
$client = new Client();
$response = $client->request('GET', $url);
echo $response->getBody();
break;
case 3:
echo file_get_contents($url);
break;
default:
echo "no opt";
}
echo "app server:\n";
我尝试了file_get_contents，但切换到file_get_contents时并没有明显的区别。当并发为1时，所有方法都很好。但是当并发增加时，它们都开始降级。
我想我找到了与此问题相关的内容，所以我只是 post 另一个问题 concurrent curl could not resolve host。这可能是根本原因，但我还没有任何答案。
试了这么久，我想这肯定是和名字解析有关。这里是 php 可以并发执行的脚本 500
<?php
require 'vendor/autoload.php';
use GuzzleHttp\Client;
$opt = 1;
$url = 'http://localhost:81/a.php';
switch ($opt) {
case 1:
// create curl resource
$ch = curl_init();
// set url
curl_setopt($ch, CURLOPT_URL, $url);
//return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_PROXY, 'localhost');
// $output contains the output string
$output = curl_exec($ch);
curl_close($ch);
echo $output;
break;
case 2:
$client = new Client();
$response = $client->request('GET', $url, ['proxy' => 'localhost']);
echo $response->getBody();
break;
case 3:
echo file_get_contents($url);
break;
default:
echo "no opt";
}
echo "app server:\n";
真正重要的是 curl_setopt($ch, CURLOPT_PROXY, 'localhost'); 和 $response = $client->request('GET', $url, ['proxy' => 'localhost']);。它告诉 curl 使用 localhost 作为代理。
这里是 ab 测试的结果
ab -n 1000 -c 500 http://192.168.10.173/b.php
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.10.173 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
Server Software: nginx/1.10.0
Server Hostname: 192.168.10.173
Server Port: 80
Document Path: /b.php
Document Length: 182 bytes
Concurrency Level: 500
Time taken for tests: 0.251 seconds
Complete requests: 1000
Failed requests: 184
(Connect: 0, Receive: 0, Length: 184, Exceptions: 0)
Non-2xx responses: 816
Total transferred: 308960 bytes
HTML transferred: 150720 bytes
Requests per second: 3985.59 [#/sec] (mean)
Time per request: 125.452 [ms] (mean)
Time per request: 0.251 [ms] (mean, across all concurrent requests)
Transfer rate: 1202.53 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 6 4.9 5 14
Processing: 9 38 42.8 22 212
Waiting: 8 38 42.9 22 212
Total: 11 44 44.4 31 214
Percentage of the requests served within a certain time (ms)
50% 31
66% 37
75% 37
80% 38
90% 122
95% 135
98% 207
99% 211
100% 214 (longest request)
但为什么当不使用 localhost 作为代理时名称解析在并发级别 5 失败？
虚拟主机设置非常简单干净，几乎所有内容都是默认配置。我没有在这个服务器上使用 iptables，也没有配置任何特殊的东西。
server {
listen 81 default_server;
listen [::]:81 default_server;
root /var/www/html;
index index.html index.htm index.nginx-debian.html;
server_name _;
location / {
try_files $uri $uri/ =404;
}
location ~ \.php$ {
include snippets/fastcgi-php.conf;
fastcgi_pass unix:/run/php/php7.0-fpm.sock;
}
}
找到有趣的东西！如果您在第一个 ab 测试后大约 3 秒内立即进行另一个 ab 测试。第二次 ab 测试很快。
不使用本地主机作为代理
ab -n 10 -c 5 http://192.168.10.173/b.php <-- This takes 2.8 seconds to finish.
ab -n 10 -c 5 http://192.168.10.173/b.php <-- This takes 0.008 seconds only.
使用本地主机作为代理
ab -n 10 -c 5 http://192.168.10.173/b.php <-- This takes 0.006 seconds.
ab -n 10 -c 5 http://192.168.10.173/b.php <-- This takes 0.006 seconds.
我认为这仍然意味着问题是名称解析。但是为什么？
假设：nginx 没有监听 localhost:81
我试过在nginx中添加listen 127.0.0.1:81;，没有效果。
发现自己在使用 curl 代理时犯了一些错误，这不起作用！稍后更新其他详细信息。
已解决，与代理或其他任何内容无关。根本原因是 php-fpm 的 www.conf.
中的 pm.start_servers
好的，经过这么多天的努力解决这个问题，我终于找到了原因。而且这不是名称解析。我不敢相信花了这么多天的时间才找到根本原因，即 php-fpm 的 www.conf 中 pm.start_servers 的数量。最初，我将 pm.start_servers 的数量设置为 3，这就是为什么对本地主机的 ab 测试总是在并发级别 3 之后变得更糟 。虽然 php-cli 没有 php 进程数量有限的问题，因此，php-cli 总是表现出色。将pm.start_servers增加到5后，ab测试结果和php-cli一样快。如果这是您的 php-fpm 速度慢的原因，您还应该考虑更改 pm.min_spare_servers、pm.max_spare_servers、pm.max_children 和任何相关的数字。
我有一个有趣的问题，我不确定根本原因是什么。我有一个服务器和两个虚拟主机 A 和 B，端口 运行 分别位于 80 和 81。我在 A 上写了一个简单的 PHP 代码，如下所示：
<?php
echo "from A server\n";
B 上的另一个简单 PHP 代码：
<?php
echo "B server:\n";
// create curl resource
$ch = curl_init();
// set url
curl_setopt($ch, CURLOPT_URL, "localhost:81/a.php");
//return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// $output contains the output string
$output = curl_exec($ch);
// close curl resource to free up system resources
curl_close($ch);
echo $output;
当使用 ab 发出并发请求时，我得到以下结果：
ab -n 10 -c 5 http://192.168.10.173/b.php
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.10.173 (be patient).....done
Server Software: nginx/1.10.0
Server Hostname: 192.168.10.173
Server Port: 80
Document Path: /b.php
Document Length: 26 bytes
Concurrency Level: 5
Time taken for tests: 2.680 seconds
Complete requests: 10
Failed requests: 0
Total transferred: 1720 bytes
HTML transferred: 260 bytes
Requests per second: 3.73 [#/sec] (mean)
Time per request: 1340.197 [ms] (mean)
Time per request: 268.039 [ms] (mean, across all concurrent requests)
Transfer rate: 0.63 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 1
Processing: 2 1339 1408.8 2676 2676
Waiting: 2 1339 1408.6 2676 2676
Total: 3 1340 1408.8 2676 2677
Percentage of the requests served within a certain time (ms)
50% 2676
66% 2676
75% 2676
80% 2676
90% 2677
95% 2677
98% 2677
99% 2677
100% 2677 (longest request)
但是发出 1000 个并发级别 1 的请求非常快：
$ ab -n 1000 -c 1 http://192.168.10.173/b.php
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.10.173 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
Server Software: nginx/1.10.0
Server Hostname: 192.168.10.173
Server Port: 80
Document Path: /b.php
Document Length: 26 bytes
Concurrency Level: 1
Time taken for tests: 1.659 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 172000 bytes
HTML transferred: 26000 bytes
Requests per second: 602.86 [#/sec] (mean)
Time per request: 1.659 [ms] (mean)
Time per request: 1.659 [ms] (mean, across all concurrent requests)
Transfer rate: 101.26 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 1
Processing: 1 1 10.3 1 201
Waiting: 1 1 10.3 1 201
Total: 1 2 10.3 1 201
Percentage of the requests served within a certain time (ms)
50% 1
66% 1
75% 1
80% 1
90% 1
95% 1
98% 1
99% 2
100% 201 (longest request)
谁能解释为什么会这样？我真的很想知道根本原因。这是卷曲的问题吗？它感觉不像是网络瓶颈或打开文件问题，因为并发性仅为 5。顺便说一下，我也用 guzzlehttp 尝试了同样的事情，但结果是一样的。我在我的笔记本电脑上使用 ab 并且服务器在同一个本地网络中。另外，它肯定与网络带宽无关，因为主机 A 和 B 之间的请求是在本地主机上完成的。
我修改了代码，让测试更灵活：
<?php
require 'vendor/autoload.php';
use GuzzleHttp\Client;
$opt = 1;
$url = 'http://localhost:81/a.php';
switch ($opt) {
case 1:
// create curl resource
$ch = curl_init();
// set url
curl_setopt($ch, CURLOPT_URL, $url);
//return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// $output contains the output string
$output = curl_exec($ch);
curl_close($ch);
echo $output;
break;
case 2:
$client = new Client();
$response = $client->request('GET', $url);
echo $response->getBody();
break;
case 3:
echo file_get_contents($url);
break;
default:
echo "no opt";
}
echo "app server:\n";
我尝试了file_get_contents，但切换到file_get_contents时并没有明显的区别。当并发为1时，所有方法都很好。但是当并发增加时，它们都开始降级。
我想我找到了与此问题相关的内容，所以我只是 post 另一个问题 concurrent curl could not resolve host。这可能是根本原因，但我还没有任何答案。
试了这么久，我想这肯定是和名字解析有关。这里是 php 可以并发执行的脚本 500
<?php
require 'vendor/autoload.php';
use GuzzleHttp\Client;
$opt = 1;
$url = 'http://localhost:81/a.php';
switch ($opt) {
case 1:
// create curl resource
$ch = curl_init();
// set url
curl_setopt($ch, CURLOPT_URL, $url);
//return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_PROXY, 'localhost');
// $output contains the output string
$output = curl_exec($ch);
curl_close($ch);
echo $output;
break;
case 2:
$client = new Client();
$response = $client->request('GET', $url, ['proxy' => 'localhost']);
echo $response->getBody();
break;
case 3:
echo file_get_contents($url);
break;
default:
echo "no opt";
}
echo "app server:\n";
真正重要的是 curl_setopt($ch, CURLOPT_PROXY, 'localhost'); 和 $response = $client->request('GET', $url, ['proxy' => 'localhost']);。它告诉 curl 使用 localhost 作为代理。
这里是 ab 测试的结果
ab -n 1000 -c 500 http://192.168.10.173/b.php
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.10.173 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
Server Software: nginx/1.10.0
Server Hostname: 192.168.10.173
Server Port: 80
Document Path: /b.php
Document Length: 182 bytes
Concurrency Level: 500
Time taken for tests: 0.251 seconds
Complete requests: 1000
Failed requests: 184
(Connect: 0, Receive: 0, Length: 184, Exceptions: 0)
Non-2xx responses: 816
Total transferred: 308960 bytes
HTML transferred: 150720 bytes
Requests per second: 3985.59 [#/sec] (mean)
Time per request: 125.452 [ms] (mean)
Time per request: 0.251 [ms] (mean, across all concurrent requests)
Transfer rate: 1202.53 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 6 4.9 5 14
Processing: 9 38 42.8 22 212
Waiting: 8 38 42.9 22 212
Total: 11 44 44.4 31 214
Percentage of the requests served within a certain time (ms)
50% 31
66% 37
75% 37
80% 38
90% 122
95% 135
98% 207
99% 211
100% 214 (longest request)
但为什么当不使用 localhost 作为代理时名称解析在并发级别 5 失败？
虚拟主机设置非常简单干净，几乎所有内容都是默认配置。我没有在这个服务器上使用 iptables，也没有配置任何特殊的东西。
server {
listen 81 default_server;
listen [::]:81 default_server;
root /var/www/html;
index index.html index.htm index.nginx-debian.html;
server_name _;
location / {
try_files $uri $uri/ =404;
}
location ~ \.php$ {
include snippets/fastcgi-php.conf;
fastcgi_pass unix:/run/php/php7.0-fpm.sock;
}
}
找到有趣的东西！如果您在第一个 ab 测试后大约 3 秒内立即进行另一个 ab 测试。第二次 ab 测试很快。
不使用本地主机作为代理
ab -n 10 -c 5 http://192.168.10.173/b.php <-- This takes 2.8 seconds to finish.
ab -n 10 -c 5 http://192.168.10.173/b.php <-- This takes 0.008 seconds only.
使用本地主机作为代理
ab -n 10 -c 5 http://192.168.10.173/b.php <-- This takes 0.006 seconds.
ab -n 10 -c 5 http://192.168.10.173/b.php <-- This takes 0.006 seconds.
我认为这仍然意味着问题是名称解析。但是为什么？
假设：nginx 没有监听 localhost:81
我试过在nginx中添加listen 127.0.0.1:81;，没有效果。
发现自己在使用 curl 代理时犯了一些错误，这不起作用！稍后更新其他详细信息。
已解决，与代理或其他任何内容无关。根本原因是 php-fpm 的 www.conf.
pm.start_servers
好的，经过这么多天的努力解决这个问题，我终于找到了原因。而且这不是名称解析。我不敢相信花了这么多天的时间才找到根本原因，即 php-fpm 的 www.conf 中 pm.start_servers 的数量。最初，我将 pm.start_servers 的数量设置为 3，这就是为什么对本地主机的 ab 测试总是在并发级别 3 之后变得更糟 。虽然 php-cli 没有 php 进程数量有限的问题，因此，php-cli 总是表现出色。将pm.start_servers增加到5后，ab测试结果和php-cli一样快。如果这是您的 php-fpm 速度慢的原因，您还应该考虑更改 pm.min_spare_servers、pm.max_spare_servers、pm.max_children 和任何相关的数字。