filebeat 无法连接到 elasticsearch

Question

我在 Kubernetes (EKS) 上有 Elasticsearch 运行，在 Kubernetes 上有 filebeat 运行作为 daemonset。

现在我正在尝试从其他 EC2 机器（EKS 之外）获取日志，因此在 EC2 上安装了 filebeat 的确切版本并将其配置为发送在 Kubernetes 上记录到 Elasticsearch 运行。

但在 Elasticsearch (Kibana) 中看不到任何日志。这是 filebeat

的日志

2019-08-26T18:18:16.005Z    INFO    instance/beat.go:292    Setup Beat: filebeat; Version: 7.2.1
2019-08-26T18:18:16.005Z    INFO    [index-management]  idxmgmt/std.go:178  Set output.elasticsearch.index to 'filebeat-7.2.1' as ILM is enabled.
2019-08-26T18:18:16.005Z    INFO    elasticsearch/client.go:166 Elasticsearch url: http://elasticsearch.dev.domain.net:9200
2019-08-26T18:18:16.005Z    INFO    add_cloud_metadata/add_cloud_metadata.go:351    add_cloud_metadata: hosting provider type detected as aws, metadata={"availability_zone":"us-west-2a","instance":{"id":"i-0185e1d68306f95b4"},"machine":{"type":"t2.medium"},"provider":"aws","region":"us-west-2"}
2019-08-26T18:18:16.005Z    INFO    [publisher] pipeline/module.go:97   Beat name: dev-web1
2019-08-26T18:18:16.006Z    INFO    elasticsearch/client.go:166 Elasticsearch url: http://elasticsearch.dev.domain.net:9200

日志中的信息不多。

然后我注意到:

root@dev-web1:~# sudo systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-08-26 18:18:47 UTC; 18min ago
     Docs: https://www.elastic.co/products/beats/filebeat
 Main PID: 7768 (filebeat)
   CGroup: /system.slice/filebeat.service
           └─7768 /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs

Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.156Z        ERROR        pipeline/output.go:100        Failed to connect to backoff(elasticsearch(http://elasticsear
Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.156Z        INFO        pipeline/output.go:93        Attempting to reconnect to backoff(elasticsearch(http://elastic
Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.156Z        INFO        [publisher]        pipeline/retry.go:189        retryer: send unwait-signal to consumer
Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.157Z        INFO        [publisher]        pipeline/retry.go:191          done
Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.157Z        INFO        [publisher]        pipeline/retry.go:166        retryer: send wait signal to consumer
Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.157Z        INFO        [publisher]        pipeline/retry.go:168          done
Aug 26 18:35:47 dev-web1 filebeat[7768]: 2019-08-26T18:35:47.028Z        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitori
Aug 26 18:36:17 dev-web1 filebeat[7768]: 2019-08-26T18:36:17.028Z        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitori
root@dev-web1:~#

但是我无法阅读上面状态消息中的完整行。

所以我尝试了：

root@dev-web1:~# curl elasticsearch.dev.domain.net/_cat/health
1566844775 18:39:35 dev-eks-logs green 3 3 48 24 0 0 0 0 - 100.0%
root@dev-web1:~#

哪个有效但不适用于端口

root@dev-web1:~# curl elasticsearch.dev.domain.net:9200/_cat/health

filebeat 具有以下配置

output.elasticsearch:
  hosts: ["elasticsearch.dev.domain.net"]
  username: "elastic"
  password: "changeme"

如何在 filebeat 端修复此问题？

远程登录测试：

root@dev-web1:~# telnet <ip> 5044
Trying <ip>...
telnet: Unable to connect to remote host: Connection refused
root@dev-web1:~# telnet localhost 5044
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
root@dev-web1:~#

Answer 1

我认为是网络问题，检查 A telnet to localhost/IP 5044

Answer 2

https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#hosts-option 说：

hosts...If no port is specified, 9200 is used.

在 filbeat 配置中添加 hosts: ["elasticsearch.dev.domain.net:80"] 应该可以解决问题。

filebeat 无法连接到 elasticsearch

filebeat failed to connect to elasticsearch

elasticsearch

kubernetes

filebeat