卷曲：（60）证书类型未批准用于 wget 工作的应用程序
curl: (60) Certificate type not approved for application where wget works
问题
如果这是 curl 的特定问题，或者是否缺少某些选项以使其正常工作，请提出建议。
CURL 显示 60) Certificate type not approved for application 访问 kubelet healthz 端点时出错，而 wget 没有问题。
$ sudo wget -v -S -O - \
> --ca-cert=/var/lib/kubelet/pki/kubelet.crt \
> --certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt \
> --private-key=/etc/kubernetes/pki/apiserver-kubelet-client.key \
> https://ip-172-31-4-117.us-west-1.compute.internal:10250/healthz
--2017-12-27 06:55:53-- https://ip-172-31-4-117.us-west-1.compute.internal:10250/healthz
Resolving ip-172-31-4-117.us-west-1.compute.internal (ip-172-31-4-117.us-west-1.compute.internal)... 172.31.4.117
Connecting to ip-172-31-4-117.us-west-1.compute.internal (ip-172-31-4-117.us-west-1.compute.internal)|172.31.4.117|:10250... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Date: Wed, 27 Dec 2017 06:55:53 GMT
Content-Length: 2
Content-Type: text/plain; charset=utf-8
Length: 2 [text/plain]
Saving to: ‘STDOUT’
$ sudo curl -ivL \
> --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt \
> --key /etc/kubernetes/pki/apiserver-kubelet-client.key \
> --cacert /var/lib/kubelet/pki/kubelet.crt \
> https://ip-172-31-4-117.us-west-1.compute.internal:10250/healthz
* Trying 172.31.4.117...
* TCP_NODELAY set
* Connected to ip-172-31-4-117.us-west-1.compute.internal (172.31.4.117) port 10250 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /var/lib/kubelet/pki/kubelet.crt
CApath: none
* Server certificate:
* subject: CN=ip-172-31-4-117.us-west-1.compute.internal@1514006010
* start date: Dec 23 05:13:30 2017 GMT
* expire date: Dec 23 05:13:30 2018 GMT
* common name: ip-172-31-4-117.us-west-1.compute.internal@1514006010
* issuer: CN=ip-172-31-4-117.us-west-1.compute.internal@1514006010
* NSS error -8101 (SEC_ERROR_INADEQUATE_CERT_TYPE)
* Certificate type not approved for application.
* stopped the pause stream!
* Closing connection 0
curl: (60) Certificate type not approved for application.
More details here: https://curl.haxx.se/docs/sslcerts.html
***curl failed to verify the legitimacy of the server*** and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
如果没有通过选项明确指定 CA，kubelet 会生成一个自签名证书。
--tls-cert-file 字符串
File containing x509 Certificate used for serving HTTPS (with intermediate certs, if any, concatenated after server cert). If --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory passed to --cert-dir.
研究
找到相关的K8S主题。
- Change self signed certificates to not include extended key usage #311
- Switch to wget for integration apiserver checks
The NSS encryption library does not allow a CA to be used with the extended key usage present, at least in the way we are currently doing so. The generated self signed certificates extension section looks like:
...
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment, Certificate Sign
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Subject Alternative Name:
DNS:localhost, IP Address:127.0.0.1, IP Address:127.0.0.1
Testing with the extended key usage removed allows curl to work, but simply switching to wget will allow integration tests to work on Fedora immediately.
导致问题的 kubelet.crt 服务器证书。
$ openssl x509 -noout -text -in /var/lib/kubelet/pki/kubelet.crt
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=ip-172-31-4-117.us-west-1.compute.internal@1514006010
Validity
Not Before: Dec 23 05:13:30 2017 GMT
Not After : Dec 23 05:13:30 2018 GMT
Subject: CN=ip-172-31-4-117.us-west-1.compute.internal@1514006010
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:d5:48:65:86:4e:ec:08:a1:b1:26:4b:da:7c:e0:
d9:d0:16:96:93:9e:c0:f3:78:cb:27:a9:e1:99:d2:
10:73:70:e5:1e:ee:03:1a:55:51:29:c6:2e:62:71:
d9:c2:72:19:d3:36:41:9b:ef:74:ac:20:97:25:1b:
63:55:96:07:4d:02:01:c9:44:7d:f9:63:2b:50:98:
2a:fb:b7:03:0d:96:6d:6a:36:9d:93:ad:2c:6a:87:
7d:b8:aa:22:ca:f2:c4:a6:90:24:2b:4c:94:d1:4a:
3c:0d:c5:fa:48:fb:e5:82:30:17:f9:67:3a:1d:9b:
85:7a:bc:e9:e9:79:48:0e:53:41:fc:64:3a:c3:c1:
7e:ae:51:23:17:8e:db:1e:4c:99:56:a4:77:ad:74:
64:64:ab:a7:84:02:40:ec:c8:b1:51:2b:b3:80:7e:
b2:51:68:5c:d2:40:9d:f3:54:b9:76:2d:47:ea:f4:
51:c6:03:e9:c4:63:5c:5a:a7:7e:9a:97:89:45:99:
e8:a0:9b:5d:1d:15:94:f9:c9:2a:a4:19:a2:07:25:
2b:e6:0e:50:63:c0:6e:f9:a0:a7:ce:4a:ca:97:17:
64:2f:8a:e2:39:d5:e2:c9:c7:c4:53:f1:6e:14:22:
ca:02:cb:8a:0c:47:68:31:f2:0b:20:2c:a3:a5:5f:
cd:95
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment, Certificate Sign
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Subject Alternative Name:
DNS:ip-172-31-4-117.us-west-1.compute.internal
Signature Algorithm: sha256WithRSAEncryption
bc:84:b2:5c:c1:a5:52:b7:03:ae:9c:53:47:4b:ca:8b:4a:90:
7f:41:0f:09:ff:77:5e:cc:7c:2c:0c:bc:37:71:c9:22:4e:7f:
eb:9f:a9:f8:04:1d:8d:39:67:52:46:9c:3b:dc:d8:1f:b0:00:
64:ed:2f:bf:8f:4c:8c:f2:6b:96:ec:99:66:19:39:1d:11:77:
52:6b:a8:f4:88:e1:ad:6b:61:af:bd:c0:fc:f7:c2:37:a2:c2:
7f:cc:de:98:a4:61:25:e2:78:b2:ab:94:31:0d:8f:2f:92:7b:
4a:4f:5b:1c:c2:e8:bd:43:cb:78:0e:f2:4b:b9:a5:54:0c:46:
0f:b0:92:f8:3c:57:08:6a:df:a4:cd:78:63:23:2f:13:12:7f:
89:7f:3d:c0:dd:c7:33:8d:55:76:10:38:47:2b:16:ce:d0:93:
c4:9e:28:42:1e:2b:f4:78:15:dd:89:1e:67:a5:a1:a1:13:30:
9a:2f:60:82:71:db:29:47:af:e7:61:71:1c:d6:72:27:61:17:
e1:31:aa:ee:84:0d:53:f8:66:18:49:34:5d:fb:50:fb:4f:c7:
b5:a1:8e:34:86:81:25:ad:31:d4:5c:9e:da:8d:08:85:a9:91:
c6:f8:83:c7:23:57:11:04:dc:90:5a:c9:5a:bf:dd:3c:9c:6a:
17:d8:d0:1f
环境
# cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)
# curl --version
curl 7.57.0 (x86_64-redhat-linux-gnu) libcurl/7.57.0 NSS/3.28.4 zlib/1.2.7 libpsl/0.7.0 (+libicu/50.1.2) libssh2/1.8.0 nghttp2/1.21.1
Release-Date: 2017-11-29
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz HTTP2 UnixSockets HTTPS-proxy Metalink PSL
类似问题
Link 下面显示了同样的问题，但看不到解决方案。
基于 answer from Steffen Ullrich，curl 的行为是正确的，因为由于 eKU 将证书的用途限制为服务器证书，证书不能是 CA 证书。
X509v3 Extended Key Usage:
TLS Web Server Authentication
问题
如果这是 curl 的特定问题，或者是否缺少某些选项以使其正常工作，请提出建议。
CURL 显示 60) Certificate type not approved for application 访问 kubelet healthz 端点时出错，而 wget 没有问题。
$ sudo wget -v -S -O - \
> --ca-cert=/var/lib/kubelet/pki/kubelet.crt \
> --certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt \
> --private-key=/etc/kubernetes/pki/apiserver-kubelet-client.key \
> https://ip-172-31-4-117.us-west-1.compute.internal:10250/healthz
--2017-12-27 06:55:53-- https://ip-172-31-4-117.us-west-1.compute.internal:10250/healthz
Resolving ip-172-31-4-117.us-west-1.compute.internal (ip-172-31-4-117.us-west-1.compute.internal)... 172.31.4.117
Connecting to ip-172-31-4-117.us-west-1.compute.internal (ip-172-31-4-117.us-west-1.compute.internal)|172.31.4.117|:10250... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Date: Wed, 27 Dec 2017 06:55:53 GMT
Content-Length: 2
Content-Type: text/plain; charset=utf-8
Length: 2 [text/plain]
Saving to: ‘STDOUT’
$ sudo curl -ivL \
> --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt \
> --key /etc/kubernetes/pki/apiserver-kubelet-client.key \
> --cacert /var/lib/kubelet/pki/kubelet.crt \
> https://ip-172-31-4-117.us-west-1.compute.internal:10250/healthz
* Trying 172.31.4.117...
* TCP_NODELAY set
* Connected to ip-172-31-4-117.us-west-1.compute.internal (172.31.4.117) port 10250 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /var/lib/kubelet/pki/kubelet.crt
CApath: none
* Server certificate:
* subject: CN=ip-172-31-4-117.us-west-1.compute.internal@1514006010
* start date: Dec 23 05:13:30 2017 GMT
* expire date: Dec 23 05:13:30 2018 GMT
* common name: ip-172-31-4-117.us-west-1.compute.internal@1514006010
* issuer: CN=ip-172-31-4-117.us-west-1.compute.internal@1514006010
* NSS error -8101 (SEC_ERROR_INADEQUATE_CERT_TYPE)
* Certificate type not approved for application.
* stopped the pause stream!
* Closing connection 0
curl: (60) Certificate type not approved for application.
More details here: https://curl.haxx.se/docs/sslcerts.html
***curl failed to verify the legitimacy of the server*** and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
如果没有通过选项明确指定 CA，kubelet 会生成一个自签名证书。
--tls-cert-file 字符串
File containing x509 Certificate used for serving HTTPS (with intermediate certs, if any, concatenated after server cert). If --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory passed to --cert-dir.
研究
找到相关的K8S主题。
- Change self signed certificates to not include extended key usage #311
- Switch to wget for integration apiserver checks
The NSS encryption library does not allow a CA to be used with the extended key usage present, at least in the way we are currently doing so. The generated self signed certificates extension section looks like:
...
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment, Certificate Sign
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Subject Alternative Name:
DNS:localhost, IP Address:127.0.0.1, IP Address:127.0.0.1
Testing with the extended key usage removed allows curl to work, but simply switching to wget will allow integration tests to work on Fedora immediately.
导致问题的 kubelet.crt 服务器证书。
$ openssl x509 -noout -text -in /var/lib/kubelet/pki/kubelet.crt
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=ip-172-31-4-117.us-west-1.compute.internal@1514006010
Validity
Not Before: Dec 23 05:13:30 2017 GMT
Not After : Dec 23 05:13:30 2018 GMT
Subject: CN=ip-172-31-4-117.us-west-1.compute.internal@1514006010
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:d5:48:65:86:4e:ec:08:a1:b1:26:4b:da:7c:e0:
d9:d0:16:96:93:9e:c0:f3:78:cb:27:a9:e1:99:d2:
10:73:70:e5:1e:ee:03:1a:55:51:29:c6:2e:62:71:
d9:c2:72:19:d3:36:41:9b:ef:74:ac:20:97:25:1b:
63:55:96:07:4d:02:01:c9:44:7d:f9:63:2b:50:98:
2a:fb:b7:03:0d:96:6d:6a:36:9d:93:ad:2c:6a:87:
7d:b8:aa:22:ca:f2:c4:a6:90:24:2b:4c:94:d1:4a:
3c:0d:c5:fa:48:fb:e5:82:30:17:f9:67:3a:1d:9b:
85:7a:bc:e9:e9:79:48:0e:53:41:fc:64:3a:c3:c1:
7e:ae:51:23:17:8e:db:1e:4c:99:56:a4:77:ad:74:
64:64:ab:a7:84:02:40:ec:c8:b1:51:2b:b3:80:7e:
b2:51:68:5c:d2:40:9d:f3:54:b9:76:2d:47:ea:f4:
51:c6:03:e9:c4:63:5c:5a:a7:7e:9a:97:89:45:99:
e8:a0:9b:5d:1d:15:94:f9:c9:2a:a4:19:a2:07:25:
2b:e6:0e:50:63:c0:6e:f9:a0:a7:ce:4a:ca:97:17:
64:2f:8a:e2:39:d5:e2:c9:c7:c4:53:f1:6e:14:22:
ca:02:cb:8a:0c:47:68:31:f2:0b:20:2c:a3:a5:5f:
cd:95
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment, Certificate Sign
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Subject Alternative Name:
DNS:ip-172-31-4-117.us-west-1.compute.internal
Signature Algorithm: sha256WithRSAEncryption
bc:84:b2:5c:c1:a5:52:b7:03:ae:9c:53:47:4b:ca:8b:4a:90:
7f:41:0f:09:ff:77:5e:cc:7c:2c:0c:bc:37:71:c9:22:4e:7f:
eb:9f:a9:f8:04:1d:8d:39:67:52:46:9c:3b:dc:d8:1f:b0:00:
64:ed:2f:bf:8f:4c:8c:f2:6b:96:ec:99:66:19:39:1d:11:77:
52:6b:a8:f4:88:e1:ad:6b:61:af:bd:c0:fc:f7:c2:37:a2:c2:
7f:cc:de:98:a4:61:25:e2:78:b2:ab:94:31:0d:8f:2f:92:7b:
4a:4f:5b:1c:c2:e8:bd:43:cb:78:0e:f2:4b:b9:a5:54:0c:46:
0f:b0:92:f8:3c:57:08:6a:df:a4:cd:78:63:23:2f:13:12:7f:
89:7f:3d:c0:dd:c7:33:8d:55:76:10:38:47:2b:16:ce:d0:93:
c4:9e:28:42:1e:2b:f4:78:15:dd:89:1e:67:a5:a1:a1:13:30:
9a:2f:60:82:71:db:29:47:af:e7:61:71:1c:d6:72:27:61:17:
e1:31:aa:ee:84:0d:53:f8:66:18:49:34:5d:fb:50:fb:4f:c7:
b5:a1:8e:34:86:81:25:ad:31:d4:5c:9e:da:8d:08:85:a9:91:
c6:f8:83:c7:23:57:11:04:dc:90:5a:c9:5a:bf:dd:3c:9c:6a:
17:d8:d0:1f
环境
# cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)
# curl --version
curl 7.57.0 (x86_64-redhat-linux-gnu) libcurl/7.57.0 NSS/3.28.4 zlib/1.2.7 libpsl/0.7.0 (+libicu/50.1.2) libssh2/1.8.0 nghttp2/1.21.1
Release-Date: 2017-11-29
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz HTTP2 UnixSockets HTTPS-proxy Metalink PSL
类似问题
Link 下面显示了同样的问题，但看不到解决方案。
基于 answer from Steffen Ullrich，curl 的行为是正确的，因为由于 eKU 将证书的用途限制为服务器证书，证书不能是 CA 证书。
X509v3 Extended Key Usage:
TLS Web Server Authentication