Kibana 未连接到 ElasticSearch
Kibana is not connecting to ElasticSearch
我正在尝试使用 Kubernetes 1.7.12 fluentd-elasticsearch 插件：
https://github.com/kubernetes/kubernetes/tree/v1.7.12/cluster/addons/fluentd-elasticsearch
ElasticSearch 启动并可以响应：
{
"name" : "0322714ad5b7",
"cluster_name" : "kubernetes-logging",
"cluster_uuid" : "_na_",
"version" : {
"number" : "2.4.1",
"build_hash" : "c67dc32e24162035d18d6fe1e952c4cbcbe79d16",
"build_timestamp" : "2016-09-27T18:57:55Z",
"build_snapshot" : false,
"lucene_version" : "5.5.2"
},
"tagline" : "You Know, for Search"
}
但是 Kibana 仍然无法连接到它。连接错误以：
开头
{"type":"log","@timestamp":"2018-01-23T07:42:06Z","tags":["warning","elasticsearch"],"pid":6,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}
{"type":"log","@timestamp":"2018-01-23T07:42:06Z","tags":["warning","elasticsearch"],"pid":6,"message":"No living connections"}
并且在ElasticSearch up之后，报错变为：
{"type":"log","@timestamp":"2018-01-23T07:42:08Z","tags":["status","plugin:elasticsearch@1.0.0","error"],"pid":6,"state":"red","message":"Status changed from red to red - Service Unavailable","prevState":"red","prevMsg":"Unable to connect to Elasticsearch at http://elasticsearch-logging:9200."}
看来，Kibana 终于可以从 ElasticSearch 得到响应了，但是还是无法建立连接。
Kibana 仪表板如下所示：
我尝试获取日志以输出更多信息，但对 Kibana 和 ElasticSearch 的了解不够，不知道接下来我还能尝试什么。
我可以使用这个 docker-compose.yml:
在本地重现错误
version: '2'
services:
elasticsearch-logging:
image: gcr.io/google_containers/elasticsearch:v2.4.1-2
ports:
- "9200:9200"
- "9300:9300"
kibana-logging:
image: gcr.io/google_containers/kibana:v4.6.1-1
ports:
- "5601:5601"
depends_on:
- elasticsearch-logging
environment:
- ELASTICSEARCH_URL=http://elasticsearch-logging:9200
根据我从这个问题中可以看出的内容，看起来不应该涉及太多：
和这个博客：https://gunith.github.io/docker-kibana-elasticsearch/
但我不知道我错过了什么。
还有什么我可以尝试的想法吗？
感谢您的宝贵时间。 :)
更新 1：
curling http://elasticsearch-logging 在 Kubernetes 集群上产生相同的输出：
{
"name" : "elasticsearch-logging-v1-68km4",
"cluster_name" : "kubernetes-logging",
"cluster_uuid" : "_na_",
"version" : {
"number" : "2.4.1",
"build_hash" : "c67dc32e24162035d18d6fe1e952c4cbcbe79d16",
"build_timestamp" : "2016-09-27T18:57:55Z",
"build_snapshot" : false,
"lucene_version" : "5.5.2"
},
"tagline" : "You Know, for Search"
}
由于代理规则，在 Kubernetes 集群上 curling http://elasticsearch-logging/_cat/indices?pretty 超时。在本地使用 docker-compose.yml 和 curling（例如 curl localhost:9200/_cat/indices?pretty）会导致：
{
"error" : {
"root_cause" : [ {
"type" : "master_not_discovered_exception",
"reason" : null
} ],
"type" : "master_not_discovered_exception",
"reason" : null
},
"status" : 503
}
docker-compose 日志显示：
[2018-01-23 17:04:39,110][DEBUG][action.admin.cluster.state] [ac1f2a13a637] no known master node, scheduling a retry
[2018-01-23 17:05:09,112][DEBUG][action.admin.cluster.state] [ac1f2a13a637] timed out while retrying [cluster:monitor/state] after failure (timeout [30s])
[2018-01-23 17:05:09,116][WARN ][rest.suppressed ] path: /_cat/indices, params: {pretty=}
MasterNotDiscoveredException[null]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.onTimeout(TransportMasterNodeAction.java:234)
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:236)
at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:804)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
更新 2：
运行 kubectl --namespace kube-system logs -c kubedns po/kube-dns-667321983-dt5lz --tail 50 --follow
产量：
I0124 16:43:33.591112 5 dns.go:264] New service: kibana-logging
I0124 16:43:33.591225 5 dns.go:264] New service: nginx
I0124 16:43:33.591251 5 dns.go:264] New service: registry
I0124 16:43:33.591274 5 dns.go:264] New service: sudoe
I0124 16:43:33.591295 5 dns.go:264] New service: default-http-backend
I0124 16:43:33.591317 5 dns.go:264] New service: kube-dns
I0124 16:43:33.591344 5 dns.go:462] Added SRV record &{Host:kube-dns.kube-system.svc.cluster.local. Port:53 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
I0124 16:43:33.591369 5 dns.go:462] Added SRV record &{Host:kube-dns.kube-system.svc.cluster.local. Port:53 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
I0124 16:43:33.591390 5 dns.go:264] New service: kubernetes
I0124 16:43:33.591409 5 dns.go:462] Added SRV record &{Host:kubernetes.default.svc.cluster.local. Port:443 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
I0124 16:43:33.591429 5 dns.go:264] New service: elasticsearch-logging
更新 3：
我仍在努力让一切正常工作，但在其他人的帮助下，我确信这是一个 RBAC 问题。我不完全确定，但由于权限的原因，elasticsearch 节点似乎无法与主节点连接（我什至不知道它是必需的）。
以下是一些有用的步骤，以防它帮助其他人开始：
启用 RBAC：
# kubectl --kubeconfig kubeconfig.yaml --namespace kube-system logs po/elasticsearch-logging-v1-wkwcs
F0119 00:18:44.285773 9 elasticsearch_logging_discovery.go:60] kube-system namespace doesn't exist: User "system:serviceaccount:kube-system:default" cannot get namespaces in the namespace "kube-system". (get namespaces kube-system)
goroutine 1 [running]:
k8s.io/kubernetes/vendor/github.com/golang/glog.stacks(0x1f7f600, 0xc400000000, 0xee, 0x1b2)
vendor/github.com/golang/glog/glog.go:766 +0xa5
k8s.io/kubernetes/vendor/github.com/golang/glog.(*loggingT).output(0x1f5f5c0, 0xc400000003, 0xc42006c300, 0x1ef20c8, 0x22, 0x3c, 0x0)
vendor/github.com/golang/glog/glog.go:717 +0x337
k8s.io/kubernetes/vendor/github.com/golang/glog.(*loggingT).printf(0x1f5f5c0, 0xc400000003, 0x16949d6, 0x1e, 0xc420579ee8, 0x2, 0x2)
vendor/github.com/golang/glog/glog.go:655 +0x14c
k8s.io/kubernetes/vendor/github.com/golang/glog.Fatalf(0x16949d6, 0x1e, 0xc420579ee8, 0x2, 0x2)
vendor/github.com/golang/glog/glog.go:1145 +0x67
main.main()
cluster/addons/fluentd-elasticsearch/es-image/elasticsearch_logging_discovery.go:60 +0xb53
[2018-01-19 00:18:45,273][INFO ][node ] [elasticsearch-logging-v1-wkwcs] version[2.4.1], pid[5], build[c67dc32/2016-09-27T18:57:55Z]
[2018-01-19 00:18:45,275][INFO ][node ] [elasticsearch-logging-v1-wkwcs] initializing ...
# kubectl --kubeconfig kubeconfig.yaml --namespace kube-system exec kibana-logging-2104905774-69wgv curl elasticsearch-logging.kube-system:9200/_cat/indices?pretty
{
"error" : {
"root_cause" : [ {
"type" : "master_not_discovered_exception",
"reason" : null
} ],
"type" : "master_not_discovered_exception",
"reason" : null
},
"status" : 503
}
关闭 RBAC：
# kubectl --kubeconfig kubeconfig.yaml --namespace kube-system log elasticsearch-logging-v1-7shgk
[2018-01-26 01:19:52,294][INFO ][node ] [elasticsearch-logging-v1-7shgk] version[2.4.1], pid[5], build[c67dc32/2016-09-27T18:57:55Z]
[2018-01-26 01:19:52,294][INFO ][node ] [elasticsearch-logging-v1-7shgk] initializing ...
[2018-01-26 01:19:53,077][INFO ][plugins ] [elasticsearch-logging-v1-7shgk] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
# kubectl --kubeconfig kubeconfig.yaml --namespace kube-system exec elasticsearch-logging-v1-7shgk curl http://elasticsearch-logging:9200/_cat/indices?pretty
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 40 100 40 0 0 2 0 0:00:20 0:00:15 0:00:05 10
green open .kibana 1 1 1 0 6.2kb 3.1kb
感谢大家的帮助:)
此异常表示 2 个配置错误
1. Kubernetes 的DNS Addon 无法正常工作。检查你的 DNS 插件日志
2. Pod 2 Pod 通讯不正常。这与您的底层 sdn 插件 cni flannel calico 有关。
您可以通过从另一个 pod ping 一个 pod 来检查。如果它不起作用，请检查您的网络配置，尤其是 kube-proxy 组件。
一些故障排除提示：
1) 确保 ElasticSearch 运行正常。
进入容器运行ning elasticsearch和运行:
curl localhost:9200
你应该得到一个 JSON，其中包含一些关于 elasticsearch 的数据。
2) 确保可以从 kibana 容器访问 ElasticSearch
进入kibana容器和运行:
curl <elasticsearch_service_name>:9200
您应该得到与上面相同的输出。
3) 确保您的 ES 索引正常。
运行 来自 elasticsearch 容器的以下命令：
curl localhost:9200/_cat/indices?pretty
你应该得到一个 table，其中包含你的 ES 集群中的所有索引及其状态（如果你只有一个 ES 副本，它应该是绿色或黄色）。
如果以上任一点失败，请检查您的 ES 容器的日志中是否有任何错误消息并尝试解决它们。
我正在尝试使用 Kubernetes 1.7.12 fluentd-elasticsearch 插件： https://github.com/kubernetes/kubernetes/tree/v1.7.12/cluster/addons/fluentd-elasticsearch
ElasticSearch 启动并可以响应：
{
"name" : "0322714ad5b7",
"cluster_name" : "kubernetes-logging",
"cluster_uuid" : "_na_",
"version" : {
"number" : "2.4.1",
"build_hash" : "c67dc32e24162035d18d6fe1e952c4cbcbe79d16",
"build_timestamp" : "2016-09-27T18:57:55Z",
"build_snapshot" : false,
"lucene_version" : "5.5.2"
},
"tagline" : "You Know, for Search"
}
但是 Kibana 仍然无法连接到它。连接错误以：
开头{"type":"log","@timestamp":"2018-01-23T07:42:06Z","tags":["warning","elasticsearch"],"pid":6,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}
{"type":"log","@timestamp":"2018-01-23T07:42:06Z","tags":["warning","elasticsearch"],"pid":6,"message":"No living connections"}
并且在ElasticSearch up之后，报错变为：
{"type":"log","@timestamp":"2018-01-23T07:42:08Z","tags":["status","plugin:elasticsearch@1.0.0","error"],"pid":6,"state":"red","message":"Status changed from red to red - Service Unavailable","prevState":"red","prevMsg":"Unable to connect to Elasticsearch at http://elasticsearch-logging:9200."}
看来，Kibana 终于可以从 ElasticSearch 得到响应了，但是还是无法建立连接。
Kibana 仪表板如下所示：
我尝试获取日志以输出更多信息，但对 Kibana 和 ElasticSearch 的了解不够，不知道接下来我还能尝试什么。
我可以使用这个 docker-compose.yml:
version: '2'
services:
elasticsearch-logging:
image: gcr.io/google_containers/elasticsearch:v2.4.1-2
ports:
- "9200:9200"
- "9300:9300"
kibana-logging:
image: gcr.io/google_containers/kibana:v4.6.1-1
ports:
- "5601:5601"
depends_on:
- elasticsearch-logging
environment:
- ELASTICSEARCH_URL=http://elasticsearch-logging:9200
根据我从这个问题中可以看出的内容，看起来不应该涉及太多：
但我不知道我错过了什么。
还有什么我可以尝试的想法吗？
感谢您的宝贵时间。 :)
更新 1：
curling http://elasticsearch-logging 在 Kubernetes 集群上产生相同的输出：
{
"name" : "elasticsearch-logging-v1-68km4",
"cluster_name" : "kubernetes-logging",
"cluster_uuid" : "_na_",
"version" : {
"number" : "2.4.1",
"build_hash" : "c67dc32e24162035d18d6fe1e952c4cbcbe79d16",
"build_timestamp" : "2016-09-27T18:57:55Z",
"build_snapshot" : false,
"lucene_version" : "5.5.2"
},
"tagline" : "You Know, for Search"
}
curling http://elasticsearch-logging/_cat/indices?pretty 超时。在本地使用 docker-compose.yml 和 curling（例如 curl localhost:9200/_cat/indices?pretty）会导致：
{
"error" : {
"root_cause" : [ {
"type" : "master_not_discovered_exception",
"reason" : null
} ],
"type" : "master_not_discovered_exception",
"reason" : null
},
"status" : 503
}
docker-compose 日志显示：
[2018-01-23 17:04:39,110][DEBUG][action.admin.cluster.state] [ac1f2a13a637] no known master node, scheduling a retry
[2018-01-23 17:05:09,112][DEBUG][action.admin.cluster.state] [ac1f2a13a637] timed out while retrying [cluster:monitor/state] after failure (timeout [30s])
[2018-01-23 17:05:09,116][WARN ][rest.suppressed ] path: /_cat/indices, params: {pretty=}
MasterNotDiscoveredException[null]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.onTimeout(TransportMasterNodeAction.java:234)
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:236)
at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:804)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
更新 2：
运行 kubectl --namespace kube-system logs -c kubedns po/kube-dns-667321983-dt5lz --tail 50 --follow
产量：
I0124 16:43:33.591112 5 dns.go:264] New service: kibana-logging
I0124 16:43:33.591225 5 dns.go:264] New service: nginx
I0124 16:43:33.591251 5 dns.go:264] New service: registry
I0124 16:43:33.591274 5 dns.go:264] New service: sudoe
I0124 16:43:33.591295 5 dns.go:264] New service: default-http-backend
I0124 16:43:33.591317 5 dns.go:264] New service: kube-dns
I0124 16:43:33.591344 5 dns.go:462] Added SRV record &{Host:kube-dns.kube-system.svc.cluster.local. Port:53 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
I0124 16:43:33.591369 5 dns.go:462] Added SRV record &{Host:kube-dns.kube-system.svc.cluster.local. Port:53 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
I0124 16:43:33.591390 5 dns.go:264] New service: kubernetes
I0124 16:43:33.591409 5 dns.go:462] Added SRV record &{Host:kubernetes.default.svc.cluster.local. Port:443 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
I0124 16:43:33.591429 5 dns.go:264] New service: elasticsearch-logging
更新 3：
我仍在努力让一切正常工作，但在其他人的帮助下，我确信这是一个 RBAC 问题。我不完全确定，但由于权限的原因，elasticsearch 节点似乎无法与主节点连接（我什至不知道它是必需的）。
以下是一些有用的步骤，以防它帮助其他人开始：
启用 RBAC：
# kubectl --kubeconfig kubeconfig.yaml --namespace kube-system logs po/elasticsearch-logging-v1-wkwcs
F0119 00:18:44.285773 9 elasticsearch_logging_discovery.go:60] kube-system namespace doesn't exist: User "system:serviceaccount:kube-system:default" cannot get namespaces in the namespace "kube-system". (get namespaces kube-system)
goroutine 1 [running]:
k8s.io/kubernetes/vendor/github.com/golang/glog.stacks(0x1f7f600, 0xc400000000, 0xee, 0x1b2)
vendor/github.com/golang/glog/glog.go:766 +0xa5
k8s.io/kubernetes/vendor/github.com/golang/glog.(*loggingT).output(0x1f5f5c0, 0xc400000003, 0xc42006c300, 0x1ef20c8, 0x22, 0x3c, 0x0)
vendor/github.com/golang/glog/glog.go:717 +0x337
k8s.io/kubernetes/vendor/github.com/golang/glog.(*loggingT).printf(0x1f5f5c0, 0xc400000003, 0x16949d6, 0x1e, 0xc420579ee8, 0x2, 0x2)
vendor/github.com/golang/glog/glog.go:655 +0x14c
k8s.io/kubernetes/vendor/github.com/golang/glog.Fatalf(0x16949d6, 0x1e, 0xc420579ee8, 0x2, 0x2)
vendor/github.com/golang/glog/glog.go:1145 +0x67
main.main()
cluster/addons/fluentd-elasticsearch/es-image/elasticsearch_logging_discovery.go:60 +0xb53
[2018-01-19 00:18:45,273][INFO ][node ] [elasticsearch-logging-v1-wkwcs] version[2.4.1], pid[5], build[c67dc32/2016-09-27T18:57:55Z]
[2018-01-19 00:18:45,275][INFO ][node ] [elasticsearch-logging-v1-wkwcs] initializing ...
# kubectl --kubeconfig kubeconfig.yaml --namespace kube-system exec kibana-logging-2104905774-69wgv curl elasticsearch-logging.kube-system:9200/_cat/indices?pretty
{
"error" : {
"root_cause" : [ {
"type" : "master_not_discovered_exception",
"reason" : null
} ],
"type" : "master_not_discovered_exception",
"reason" : null
},
"status" : 503
}
关闭 RBAC：
# kubectl --kubeconfig kubeconfig.yaml --namespace kube-system log elasticsearch-logging-v1-7shgk
[2018-01-26 01:19:52,294][INFO ][node ] [elasticsearch-logging-v1-7shgk] version[2.4.1], pid[5], build[c67dc32/2016-09-27T18:57:55Z]
[2018-01-26 01:19:52,294][INFO ][node ] [elasticsearch-logging-v1-7shgk] initializing ...
[2018-01-26 01:19:53,077][INFO ][plugins ] [elasticsearch-logging-v1-7shgk] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
# kubectl --kubeconfig kubeconfig.yaml --namespace kube-system exec elasticsearch-logging-v1-7shgk curl http://elasticsearch-logging:9200/_cat/indices?pretty
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 40 100 40 0 0 2 0 0:00:20 0:00:15 0:00:05 10
green open .kibana 1 1 1 0 6.2kb 3.1kb
感谢大家的帮助:)
此异常表示 2 个配置错误 1. Kubernetes 的DNS Addon 无法正常工作。检查你的 DNS 插件日志 2. Pod 2 Pod 通讯不正常。这与您的底层 sdn 插件 cni flannel calico 有关。
您可以通过从另一个 pod ping 一个 pod 来检查。如果它不起作用，请检查您的网络配置，尤其是 kube-proxy 组件。
一些故障排除提示：
1) 确保 ElasticSearch 运行正常。
进入容器运行ning elasticsearch和运行:
curl localhost:9200
你应该得到一个 JSON，其中包含一些关于 elasticsearch 的数据。
2) 确保可以从 kibana 容器访问 ElasticSearch
进入kibana容器和运行:
curl <elasticsearch_service_name>:9200
您应该得到与上面相同的输出。
3) 确保您的 ES 索引正常。
运行 来自 elasticsearch 容器的以下命令：
curl localhost:9200/_cat/indices?pretty
你应该得到一个 table，其中包含你的 ES 集群中的所有索引及其状态（如果你只有一个 ES 副本，它应该是绿色或黄色）。
如果以上任一点失败，请检查您的 ES 容器的日志中是否有任何错误消息并尝试解决它们。