Logstash 匹配多个值

Question

1)这是我的logstash.conf文件

input {
  beats {
    type => beats
    port => 5044
  }
}

filter {
  grok {
    match => { "message" => "\[(?<logtime>([0-9]|[\-\+\.\:\ ])*)\] \[(?<level>([a-z-A-Z])*)\] \[(?<msg>(.)+)\] (?<exception>(.)+)" }
  }
  mutate {
    add_field => [ "logtime", "level", "msg", "exception" ]
    remove_field => [ "beat", "offset", "source", "prospector", "host", "tags" ]
  }  
}

output {
  if [type] == "beats"{
    elasticsearch {
      hosts => "localhost:9200"
      manage_template => false
      index => "%{+YYYY.MM.dd}-container.api" 
      document_type => "%{[@metadata][type]}" 
      user => "elastic"
      password => "secret"
    }
  }
}

2) 我用调试器测试了我的 grok，如下所示

3)这是logstash写入elasticsearch的内容

  {
    "_index": "2019.01.28-container.api",
    "_type": "doc",
    "_id": "pZctlWgBojxJzDZGWqZz",
    "_score": 1,
    "_source": {
      "type": "beats",
      "level": "Debug",
      "@timestamp": "2019-01-28T15:56:41.295Z",
      "msg": [
        "Hosting starting",
        "exception"
      ],
      "@version": "1",
      "logtime": [
        "2019-01-28 15:23:12.911 +03:00",
        "level"
      ],
      "message": "[2019-01-28 15:23:12.911 +03:00] [Debug] [Hosting starting] exception 2",
      "exception": "exception 2",
      "input": {
        "type": "log"
      }
    }
  }

4)我想看的是

  {
    "_index": "2019.01.28-container.api",
    "_type": "doc",
    "_id": "pZctlWgBojxJzDZGWqZz",
    "_score": 1,
    "_source": {
      "type": "beats",
      "level": "Debug",
      "@timestamp": "2019-01-28T15:56:41.295Z",
      "msg": "Hosting starting",
      "logtime": "2019-01-28 15:23:12.911 +03:00",
      "message": "2019-01-28 15:23:12.911 +03:00 Debug Hosting starting [exception 2]",
      "exception": "exception 2"
    }
  }

Answer 1

问题出在

mutate {
    add_field => [ "logtime", "level", "msg", "exception" ]
}

您要添加的字段已经由 grok 过滤器创建，再次添加是没有用的，它只会转换数组中已经存在的字段并将新值添加到数组中，因为 mutate.addField 使用散列，它将向字段 logtime 添加值 level 并向字段 msg 添加值 exception.

Answer 2

mutate {
    add_field => [ "logtime", "level", "msg", "exception" ]
}

这等同于：

mutate {
        add_field => { 
              "logtime" => "level" 
              "msg" => "exception" 
        }
   }

这就是数组存在的原因，并且具有多个值。由于您在 grok 模式上定义了变量名称，因此您不必再次指定。所以正如 baudsp 所说，你可以删除这个 "add field"。

Logstash 匹配多个值

Logstash matches multiple value

elasticsearch

logstash

logstash-grok

filebeat