如何删除 filebeat 标签，如 id、主机名、版本、grok_failure 消息

Question

我是 elk 的新手，我的样本日志看起来像

2017-01-05T14:28:00 INFO zeppelin IDExtractionService transactionId abcdef1234 operation extractOCRData received request duration 12344 exception error occured

我的 filebeat 配置如下

filebeat.prospectors:
- input_type: log
  paths:
    - /opt/apache-tomcat-7.0.82/logs/*.log

document_type: apache-access
fields_under_root: true

output.logstash:
  hosts: ["10.2.3.4:5044"]

还有我的 logstash filter.conf 文件：

filter {
  grok {
    match => [ "message", "transactionId %{WORD:transaction_id} operation %{WORD:otype} received request duration %{NUMBER:duration} exception %{WORD:error}" ]
  }
}
filter {
    if "beats_input_codec_plain_applied" in [tags] {
        mutate {
            remove_tag => ["beats_input_codec_plain_applied"]
        }
    }
}

; 在 kibana 仪表板中，我可以看到如下日志输出

beat.name:
    ebb8a5ec413b
beat.hostname:
    ebb8a5ec413b
host:
    ebb8a5ec413b
tags:
beat.version:
    6.2.2
source:
    /opt/apache-tomcat-7.0.82/logs/IDExtraction.log
otype:
    extractOCRData
duration:
    12344
transaction_id:
    abcdef1234
@timestamp:
    April 9th 2018, 16:20:31.853
offset:
    805,655
@version:
    1
error:
    error
message:
    2017-01-05T14:28:00 INFO zeppelin IDExtractionService transactionId abcdef1234 operation extractOCRData received request duration 12344 exception error occured
_id:
    7X0HqmIBj3MEd9pqhTu9
_type:
    doc
_index:
    filebeat-2018.04.09
_score:
    6.315

1 第一个问题是如何删除 filebeat 标签，如 id、主机名、版本、grok_failure 消息

2 如何根据时间戳对日志进行排序，因为新生成的日志未出现在 kibana 仪表板顶部

3 我的 grok 过滤器是否需要任何更改

Answer 1

您可以通过在 filebeat 配置文件中设置 fields_under_root: false 的值来删除 filebeat 标签。您可以阅读此选项 here.

If this option is set to true, the custom fields are stored as top-level fields in the output document instead of being grouped under a fields sub-dictionary. If the custom field names conflict with other field names added by Filebeat, the custom fields overwrite the other fields.

您可以使用 if "_grokparsefailure" in [tags] 检查 _grokparsefailure 是否在标签中，并使用 remove_tag => ["_grokparsefailure"]

将其删除

你的 grok 过滤器似乎没问题。

希望对您有所帮助。

如何删除 filebeat 标签，如 id、主机名、版本、grok_failure 消息

How to remove filebeat tags like id, hostname, version, grok_failure message

logstash

kibana

logstash-grok

elastic-stack