删除注册表时重复的日志事件

Question

我目前正在安装 PoC ELK，我想重新发送在 Filebeat 中注册的文件的每一行日志以进行测试。

我是这样做的：

我停止 Filebeat
我通过Kibana删除了Logstash中的索引
我删除了 Filebeat 注册表文件
我启动 Filebeat

在 Kibana 中，我可以看到那里的事件是日志行的两倍，而且我还可以看到每个事件都被复制一次。

这是为什么？

Filebeat 日志：

2017-05-05T14:25:16+02:00 INFO Setup Beat: filebeat; Version: 5.2.2
2017-05-05T14:25:16+02:00 INFO Max Retries set to: 3
2017-05-05T14:25:16+02:00 INFO Activated logstash as output plugin.
2017-05-05T14:25:16+02:00 INFO Publisher name: anonymized
2017-05-05T14:25:16+02:00 INFO Flush Interval set to: 1s
2017-05-05T14:25:16+02:00 INFO Max Bulk Size set to: 2048
2017-05-05T14:25:16+02:00 INFO filebeat start running.
2017-05-05T14:25:16+02:00 INFO No registry file found under: /var/lib/filebeat/registry. Creating a new registry file.
2017-05-05T14:25:16+02:00 INFO Loading registrar data from /var/lib/filebeat/registry
2017-05-05T14:25:16+02:00 INFO States Loaded from registrar: 0
2017-05-05T14:25:16+02:00 INFO Loading Prospectors: 1
2017-05-05T14:25:16+02:00 INFO Prospector with previous states loaded: 0
2017-05-05T14:25:16+02:00 INFO Loading Prospectors completed. Number of prospectors: 1
2017-05-05T14:25:16+02:00 INFO All prospectors are initialised and running with 0 states to persist
2017-05-05T14:25:16+02:00 INFO Starting Registrar
2017-05-05T14:25:16+02:00 INFO Start sending events to output
2017-05-05T14:25:16+02:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017-05-05T14:25:16+02:00 INFO Starting prospector of type: log
2017-05-05T14:25:16+02:00 INFO Harvester started for file: /some/where/anonymized.log
2017-05-05T14:25:46+02:00 INFO Non-zero metrics in the last 30s: registrar.writes=2 libbeat.logstash.publish.read_bytes=54 libbeat.logstash.publish.write_bytes=32390 libbeat.logstash.published_and_acked_events=578 filebeat.harvester.running=1 registar.states.current=1 libbeat.logstash.call_count.PublishEvents=1 libbeat.publisher.published_events=578 publish.events=579 filebeat.harvester.started=1 registrar.states.update=579 filebeat.harvester.open_files=1
2017-05-05T14:26:16+02:00 INFO No non-zero metrics in the last 30s

Answer 1

删除注册表文件导致了问题。

Filebeat 管理文件的状态和事件的 ACK 与探矿者（在内存中）和注册表文件（在磁盘中持久化）。

请阅读文档Here

你可以自己管理每个事件的_id字段，这样任何重复的事件（无论什么原因，即使在生产环境中）在elasticsearch中不会有两个，而是更新事件。

在您的 logstash 管道配置文件中创建以下配置。

#if your logs don't have a unique ID, use the following to generate one
fingerprint{
        #with the message field or choose other(s) that can give you a  uniqueID
        source => ["message"]
        target => "LogID"
        key    => "something"
        method => "MD5"
        concatenate_sources => true
           }
 #in your output section 
    elasticsearch{
                     hosts => ["localhost:9200"]
                     document_id => "%{LogID}"
                     index => "yourindex"
        }

删除注册表时重复的日志事件

Duplicated log events when deleting registry

elasticsearch

logstash

filebeat