Kubernetes HTTP 活性探测失败并显示 "connection refused"，即使 URL 在没有它的情况下也能正常工作
Kubernetes HTTP liveness probe fails with "connection refused" even though URL works without it
环境：
Kubernetes version: v1.16.3
OS: CentOS 7
Kernel: Linux k8s02-master01 3.10.0-1062.4.3.el7.x86_64 #1 SMP Wed Nov 13 23:58:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
发生了什么：
我有一个 Wordpress 部署 运行 一个从自定义 Apache/Wordpress 图像构建的容器。该图像公开端口 8080 而不是 80 （下面的 Dockerfile）。 Pod 通过 Traefik 反向代理暴露给世界。一切正常，无需任何活动或就绪检查。 Pod 准备就绪，可以从 https://www.example.com/.
访问 Wordpress
我尝试添加 liveness 和 readiness 探测器，但它们都反复失败 "connection refused"。当我删除两个探测器并重新应用 Deployment 时，它再次工作。它会一直工作，直到探测器达到故障阈值，此时容器会进入无休止的重启循环并变得无法访问。
POD 事件：
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned development/blog-wordpress-5dbcd9c7c7-kdgpc to gg-k8s02-worker02
Normal Killing 16m (x2 over 17m) kubelet, gg-k8s02-worker02 Container blog-wordpress failed liveness probe, will be restarted
Normal Created 16m (x3 over 18m) kubelet, gg-k8s02-worker02 Created container blog-wordpress
Normal Started 16m (x3 over 18m) kubelet, gg-k8s02-worker02 Started container blog-wordpress
Normal Pulled 13m (x5 over 18m) kubelet, gg-k8s02-worker02 Container image "wordpress-test:test12" already present on machine
Warning Unhealthy 8m17s (x35 over 18m) kubelet, gg-k8s02-worker02 Liveness probe failed: Get http://10.244.3.83/: dial tcp 10.244.3.83:80: connect: connection refused
Warning BackOff 3m27s (x27 over 11m) kubelet, gg-k8s02-worker02 Back-off restarting failed container
POD 日志：
WordPress not found in /var/www/html - copying now...
WARNING: /var/www/html is not empty! (copying anyhow)
Complete! WordPress has been successfully copied to /var/www/html
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.244.3.83. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.244.3.83. Set the 'ServerName' directive globally to suppress this message
[Wed Dec 11 06:39:07.502247 2019] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.38 (Debian) PHP/7.3.11 configured -- resuming normal operations
[Wed Dec 11 06:39:07.502323 2019] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'
10.244.3.1 - - [11/Dec/2019:06:39:18 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16"
10.244.3.1 - - [11/Dec/2019:06:39:33 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16"
10.244.3.1 - - [11/Dec/2019:06:39:48 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16"
10.244.3.1 - - [11/Dec/2019:06:40:03 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16"
10.244.3.1 - - [11/Dec/2019:06:40:18 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16"
DOCKERFILE ("wordpress-test:test12"):
FROM wordpress:5.2.4-apache
RUN sed -i 's/Listen 80/Listen 8080/g' /etc/apache2/ports.conf;
RUN sed -i 's/:80/:8080/g' /etc/apache2/sites-enabled/000-default.conf;
# RUN sed -i 's/#ServerName www.example.com/ServerName localhost/g' /etc/apache2/sites-enabled/000-default.conf;
EXPOSE 8080
CMD ["apache2-foreground"]
部署：
apiVersion: apps/v1
kind: Deployment
metadata:
name: blog-wordpress
namespace: development
labels:
app: blog
spec:
selector:
matchLabels:
app: blog
tier: wordpress
replicas: 4
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 2
maxUnavailable: 2
template:
metadata:
labels:
app: blog
tier: wordpress
spec:
volumes:
- name: blog-wordpress
persistentVolumeClaim:
claimName: blog-wordpress
containers:
- name: blog-wordpress
# image: wordpress:5.2.4-apache
image: wordpress-test:test12
securityContext:
runAsUser: 65534
allowPrivilegeEscalation: false
capabilities:
add:
- "NET_ADMIN"
- "NET_BIND_SERVICE"
- "SYS_TIME"
resources:
requests:
cpu: "250m"
memory: "64Mi"
limits:
cpu: "500m"
memory: "128Mi"
ports:
- name: liveness-port
containerPort: 8080
readinessProbe:
initialDelaySeconds: 15
httpGet:
path: /index.php
port: 8080
timeoutSeconds: 15
periodSeconds: 15
failureThreshold: 5
livenessProbe:
initialDelaySeconds: 10
httpGet:
path: /index.php
port: 8080
timeoutSeconds: 10
periodSeconds: 15
failureThreshold: 5
env:
# Database
- name: WORDPRESS_DB_HOST
value: blog-mysql
- name: WORDPRESS_DB_NAME
value: wordpress
- name: WORDPRESS_DB_USER
valueFrom:
secretKeyRef:
name: blog-mysql
key: username
- name: WORDPRESS_DB_PASSWORD
valueFrom:
secretKeyRef:
name: blog-mysql
key: password
- name: WORDPRESS_TABLE_PREFIX
value: wp_
- name: WORDPRESS_AUTH_KEY
valueFrom:
secretKeyRef:
name: blog-wordpress
key: auth-key
- name: WORDPRESS_SECURE_AUTH_KEY
valueFrom:
secretKeyRef:
name: blog-wordpress
key: secure-auth-key
- name: WORDPRESS_LOGGED_IN_KEY
valueFrom:
secretKeyRef:
name: blog-wordpress
key: logged-in-key
- name: WORDPRESS_NONCE_KEY
valueFrom:
secretKeyRef:
name: blog-wordpress
key: nonce-key
- name: WORDPRESS_AUTH_SALT
valueFrom:
secretKeyRef:
name: blog-wordpress
key: auth-salt
- name: WORDPRESS_SECURE_AUTH_SALT
valueFrom:
secretKeyRef:
name: blog-wordpress
key: secure-auth-salt
- name: WORDPRESS_LOGGED_IN_SALT
valueFrom:
secretKeyRef:
name: blog-wordpress
key: logged-in-salt
- name: WORDPRESS_NONCE_SALT
valueFrom:
secretKeyRef:
name: blog-wordpress
key: nonce-salt
- name: WORDPRESS_CONFIG_EXTRA
value: |
define('WPLANG', 'fr_FR');
define('WP_CACHE', false);
define('WP_MEMORY_LIMIT', '64M');
volumeMounts:
- name: blog-wordpress
mountPath: "/var/www/html/wp-content"
部署服务：
apiVersion: v1
kind: Service
metadata:
name: blog-wordpress
namespace: development
labels:
app: blog
spec:
ports:
- protocol: TCP
port: 80
targetPort: 8080
selector:
app: blog
tier: wordpress
type: ClusterIP
TRAEFIK 入口：
##
# HTTP
##
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: blog
namespace: development
spec:
entryPoints:
- http
routes:
- match: Host(`example.com`)
kind: Rule
services:
- name: blog-wordpress
port: 80
middlewares:
- name: redirect-to-https
namespace: kube-system
---
##
# HTTPS
##
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: blog-https
namespace: development
spec:
entryPoints:
- https
routes:
- match: Host(`example.com`) && PathPrefix(`/`)
kind: Rule
services:
- name: blog-wordpress
port: 80
tls:
certResolver: letsencrypt
谢谢！
10.244.3.1 - - [11/Dec/2019:06:39:18 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16"
您收到来自 Apache 的 301 重定向响应。您需要获得 2xx 才能被视为成功。
要检查重定向的路径，您可以尝试 curl --location --verbose http://url/index.php
如果您无法找到绕过 Apache 或 Wordpress 重定向的方法，您可以考虑使用 tcpSocket 探测而不是 httpGet
我认为 WP 会将您重定向到 / 的“干净”url。删除 index.php 部分
对于任何感兴趣的人，我已经设法解决了这个问题。
由于 Wordpress 强制我的域名 example.com，我从 Wordpress 收到了 301 重定向响应。通过为特定请求禁用 Wordpress 规范重定向功能解决了这个问题 http://POD_IP:8080/index.php.
方法如下：
将 Pod IP 地址添加为环境变量：
- name: K8S_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
创建了一个带有自定义 redirect_canonical 过滤器的 Wordpress 插件，可防止 Wordpress 重定向 http://POD_IP:8080/index.php:
<?php
/**
* Plugin Name: Kubernetes Liveness Probe Exception
*/
add_filter('redirect_canonical', function($redirect_url, $requested_url) {
$K8S_POD_IP = getenv('K8S_POD_IP');
$LIVENESS_URL = "http://" . $K8S_POD_IP . ":8080/index.php";
if ($requested_url == $LIVENESS_URL) {
return $requested_url;
}
return $redirect_url;
}, 10, 2);
只是提供另一种方式 - wordpress 将尝试重定向，因为您缺少 X-Forwarded http headers 如果您通过代理连接到 wordpress，则您应该拥有。
像这样的东西不需要自定义 php:
livenessProbe:
initialDelaySeconds: 10
httpGet:
path: /
port: 8080
httpHeaders:
- name: X-Forwarded-Proto
value: https
- name: X-Forwarded-Host
value: www.your-wordpress-domain-here.com
- name: Host
value: www.your-wordpress-domain-here.com
timeoutSeconds: 10
periodSeconds: 15
failureThreshold: 5
环境：
Kubernetes version: v1.16.3
OS: CentOS 7
Kernel: Linux k8s02-master01 3.10.0-1062.4.3.el7.x86_64 #1 SMP Wed Nov 13 23:58:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
发生了什么：
我有一个 Wordpress 部署 运行 一个从自定义 Apache/Wordpress 图像构建的容器。该图像公开端口 8080 而不是 80 （下面的 Dockerfile）。 Pod 通过 Traefik 反向代理暴露给世界。一切正常，无需任何活动或就绪检查。 Pod 准备就绪，可以从 https://www.example.com/.
访问 Wordpress我尝试添加 liveness 和 readiness 探测器，但它们都反复失败 "connection refused"。当我删除两个探测器并重新应用 Deployment 时，它再次工作。它会一直工作，直到探测器达到故障阈值，此时容器会进入无休止的重启循环并变得无法访问。
POD 事件：
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned development/blog-wordpress-5dbcd9c7c7-kdgpc to gg-k8s02-worker02
Normal Killing 16m (x2 over 17m) kubelet, gg-k8s02-worker02 Container blog-wordpress failed liveness probe, will be restarted
Normal Created 16m (x3 over 18m) kubelet, gg-k8s02-worker02 Created container blog-wordpress
Normal Started 16m (x3 over 18m) kubelet, gg-k8s02-worker02 Started container blog-wordpress
Normal Pulled 13m (x5 over 18m) kubelet, gg-k8s02-worker02 Container image "wordpress-test:test12" already present on machine
Warning Unhealthy 8m17s (x35 over 18m) kubelet, gg-k8s02-worker02 Liveness probe failed: Get http://10.244.3.83/: dial tcp 10.244.3.83:80: connect: connection refused
Warning BackOff 3m27s (x27 over 11m) kubelet, gg-k8s02-worker02 Back-off restarting failed container
POD 日志：
WordPress not found in /var/www/html - copying now...
WARNING: /var/www/html is not empty! (copying anyhow)
Complete! WordPress has been successfully copied to /var/www/html
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.244.3.83. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.244.3.83. Set the 'ServerName' directive globally to suppress this message
[Wed Dec 11 06:39:07.502247 2019] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.38 (Debian) PHP/7.3.11 configured -- resuming normal operations
[Wed Dec 11 06:39:07.502323 2019] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'
10.244.3.1 - - [11/Dec/2019:06:39:18 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16"
10.244.3.1 - - [11/Dec/2019:06:39:33 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16"
10.244.3.1 - - [11/Dec/2019:06:39:48 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16"
10.244.3.1 - - [11/Dec/2019:06:40:03 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16"
10.244.3.1 - - [11/Dec/2019:06:40:18 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16"
DOCKERFILE ("wordpress-test:test12"):
FROM wordpress:5.2.4-apache
RUN sed -i 's/Listen 80/Listen 8080/g' /etc/apache2/ports.conf;
RUN sed -i 's/:80/:8080/g' /etc/apache2/sites-enabled/000-default.conf;
# RUN sed -i 's/#ServerName www.example.com/ServerName localhost/g' /etc/apache2/sites-enabled/000-default.conf;
EXPOSE 8080
CMD ["apache2-foreground"]
部署：
apiVersion: apps/v1
kind: Deployment
metadata:
name: blog-wordpress
namespace: development
labels:
app: blog
spec:
selector:
matchLabels:
app: blog
tier: wordpress
replicas: 4
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 2
maxUnavailable: 2
template:
metadata:
labels:
app: blog
tier: wordpress
spec:
volumes:
- name: blog-wordpress
persistentVolumeClaim:
claimName: blog-wordpress
containers:
- name: blog-wordpress
# image: wordpress:5.2.4-apache
image: wordpress-test:test12
securityContext:
runAsUser: 65534
allowPrivilegeEscalation: false
capabilities:
add:
- "NET_ADMIN"
- "NET_BIND_SERVICE"
- "SYS_TIME"
resources:
requests:
cpu: "250m"
memory: "64Mi"
limits:
cpu: "500m"
memory: "128Mi"
ports:
- name: liveness-port
containerPort: 8080
readinessProbe:
initialDelaySeconds: 15
httpGet:
path: /index.php
port: 8080
timeoutSeconds: 15
periodSeconds: 15
failureThreshold: 5
livenessProbe:
initialDelaySeconds: 10
httpGet:
path: /index.php
port: 8080
timeoutSeconds: 10
periodSeconds: 15
failureThreshold: 5
env:
# Database
- name: WORDPRESS_DB_HOST
value: blog-mysql
- name: WORDPRESS_DB_NAME
value: wordpress
- name: WORDPRESS_DB_USER
valueFrom:
secretKeyRef:
name: blog-mysql
key: username
- name: WORDPRESS_DB_PASSWORD
valueFrom:
secretKeyRef:
name: blog-mysql
key: password
- name: WORDPRESS_TABLE_PREFIX
value: wp_
- name: WORDPRESS_AUTH_KEY
valueFrom:
secretKeyRef:
name: blog-wordpress
key: auth-key
- name: WORDPRESS_SECURE_AUTH_KEY
valueFrom:
secretKeyRef:
name: blog-wordpress
key: secure-auth-key
- name: WORDPRESS_LOGGED_IN_KEY
valueFrom:
secretKeyRef:
name: blog-wordpress
key: logged-in-key
- name: WORDPRESS_NONCE_KEY
valueFrom:
secretKeyRef:
name: blog-wordpress
key: nonce-key
- name: WORDPRESS_AUTH_SALT
valueFrom:
secretKeyRef:
name: blog-wordpress
key: auth-salt
- name: WORDPRESS_SECURE_AUTH_SALT
valueFrom:
secretKeyRef:
name: blog-wordpress
key: secure-auth-salt
- name: WORDPRESS_LOGGED_IN_SALT
valueFrom:
secretKeyRef:
name: blog-wordpress
key: logged-in-salt
- name: WORDPRESS_NONCE_SALT
valueFrom:
secretKeyRef:
name: blog-wordpress
key: nonce-salt
- name: WORDPRESS_CONFIG_EXTRA
value: |
define('WPLANG', 'fr_FR');
define('WP_CACHE', false);
define('WP_MEMORY_LIMIT', '64M');
volumeMounts:
- name: blog-wordpress
mountPath: "/var/www/html/wp-content"
部署服务：
apiVersion: v1
kind: Service
metadata:
name: blog-wordpress
namespace: development
labels:
app: blog
spec:
ports:
- protocol: TCP
port: 80
targetPort: 8080
selector:
app: blog
tier: wordpress
type: ClusterIP
TRAEFIK 入口：
##
# HTTP
##
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: blog
namespace: development
spec:
entryPoints:
- http
routes:
- match: Host(`example.com`)
kind: Rule
services:
- name: blog-wordpress
port: 80
middlewares:
- name: redirect-to-https
namespace: kube-system
---
##
# HTTPS
##
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: blog-https
namespace: development
spec:
entryPoints:
- https
routes:
- match: Host(`example.com`) && PathPrefix(`/`)
kind: Rule
services:
- name: blog-wordpress
port: 80
tls:
certResolver: letsencrypt
谢谢！
10.244.3.1 - - [11/Dec/2019:06:39:18 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16"
您收到来自 Apache 的 301 重定向响应。您需要获得 2xx 才能被视为成功。
要检查重定向的路径，您可以尝试 curl --location --verbose http://url/index.php
如果您无法找到绕过 Apache 或 Wordpress 重定向的方法，您可以考虑使用 tcpSocket 探测而不是 httpGet
我认为 WP 会将您重定向到 / 的“干净”url。删除 index.php 部分
对于任何感兴趣的人，我已经设法解决了这个问题。
由于 Wordpress 强制我的域名 example.com，我从 Wordpress 收到了 301 重定向响应。通过为特定请求禁用 Wordpress 规范重定向功能解决了这个问题 http://POD_IP:8080/index.php.
方法如下：
将 Pod IP 地址添加为环境变量：
- name: K8S_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
创建了一个带有自定义 redirect_canonical 过滤器的 Wordpress 插件，可防止 Wordpress 重定向 http://POD_IP:8080/index.php:
<?php
/**
* Plugin Name: Kubernetes Liveness Probe Exception
*/
add_filter('redirect_canonical', function($redirect_url, $requested_url) {
$K8S_POD_IP = getenv('K8S_POD_IP');
$LIVENESS_URL = "http://" . $K8S_POD_IP . ":8080/index.php";
if ($requested_url == $LIVENESS_URL) {
return $requested_url;
}
return $redirect_url;
}, 10, 2);
只是提供另一种方式 - wordpress 将尝试重定向，因为您缺少 X-Forwarded http headers 如果您通过代理连接到 wordpress，则您应该拥有。
像这样的东西不需要自定义 php:
livenessProbe:
initialDelaySeconds: 10
httpGet:
path: /
port: 8080
httpHeaders:
- name: X-Forwarded-Proto
value: https
- name: X-Forwarded-Host
value: www.your-wordpress-domain-here.com
- name: Host
value: www.your-wordpress-domain-here.com
timeoutSeconds: 10
periodSeconds: 15
failureThreshold: 5