使用 dissect 进行日志解析

Question

输入以下消息文本：

{"timestamp":"2018-08-02 15:56:46,569","level":"DEBUG","class":"classname","method":"run","line":"730","thread":"threadname","message":"messagetext"}

这是我的 logstash 配置过滤器部分：

filter {
  if "pattern_09" in [tags] {
    mutate {
      gsub => [ "message" , "{" , "",
                "message" , "}" , "" ] }
    dissect {
      mapping => { "message" => "%{Timestamp},%{+Timestamp/2},%{level},%{class},%{method},%{line},%{thread},%{mymessage}"}
    }
    mutate {
      gsub => [ "Timestamp" , "timestamp", "",
                "Timestamp" , "\"", "" ,
                "level" , "level" , "" ,
                "level" , ":", "" ,
                "level" , "\"", "" ,
                "class" , "class" , "" ,
                "class" , ":", "" ,
                "class" , "\"", "" ,
                "method" , "method" , "" ,
                "method" , ":", "" ,
                "method" , "\"", "" ,
                "line" , "line" , "" ,
                "line" , ":", "" ,
                "line" , "\"", "" ,
                "thread" , "thread" , "" ,
                "thread" , ":", "" ,
                "thread" , "\"", "" ,
                "mymessage" , "mymessage" , "" ,
                "mymessage" , ":", "",
                "mymessage" , "\"", "" ]
    }
    date {
      match => [ "Timestamp", "yyyy-dd-MM HH:mm:ss,SSS" ]
    }
  }
}

然后我的时间戳是这样的：

:2018-08-02 15:56:46,569

而且我无法 remove/erase 第一次出现“:”。我已经用

试过了

date {
  match => [ "Timestamp", "yyyy-dd-MM HH:mm:ss,SSS" ]
}

但这没有效果。

Answer 1

为什么不使用 grok 过滤器？这样的模式匹配就方便多了

对于上面的例子，可以如下做

grok {
  match => ["message" ,'%{GREEDYDATA}"timestamp":"%{DATA:Timestamp}"%{GREEDYDATA}"level":"%{DATA:Level}"%{GREEDYDATA}"class":"%{DATA:Class}"%{GREEDYDATA}"method":"%{DATA:Method}"%{GREEDYDATA}"line":"%{DATA:Line}"%{GREEDYDATA}"thread":"%{DATA:Thread}"%{GREEDYDATA}"message":"%{DATA:Message}"%{GREEDYDATA}']
}

date {
  match => [ "Timestamp", "yyyy-dd-MM HH:mm:ss,SSS" ]
}

DATA 和 GREEDYDATA 只是模式，我们可以在这里使用更多模式：https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns

更多关于 grok 过滤器的信息：https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html and you can test your grok patterns here:https://grokdebug.herokuapp.com/

使用 dissect 进行日志解析

Log parsing with dissect

logstash

logstash-grok