Logstash 在一个日志文件中处理多种格式的文本

Question

我正在使用 ELK(filebeat,logstash,elasticsearch,kibana) 进行日志管理。在一个日志文件中，我有三种格式。我正在使用

在一种格式中，我有日期+参数+json+堆栈跟踪。这种格式文本有多行。在第二种格式中，它只是日期+请求方法（获取或post）+一些文本。它在一条线上。在第三种格式中，它有 date+ modular Name:(在这种情况下，它是 paymentAdmin)+json

我想我可以使用 logStash 通过 if、else if 子句和 3 种匹配器来处理这三种格式。有人可以告诉我如何编写 if、else if 子句以在一个日志文件中处理 3 种格式的示例吗？

2018-04-10 10:14:18 params:{"currentPage":"1","schoolname":"","schoolcode":"","pageSize":"10","pageNumber":"1","ispage":"yes","sortOrder":"asc","_":"1523326435639"} java.lang.NullPointerException: null
    at com.datalook.group.BusinessHandler.queryGroup(BusinessHandler.java:204) ~[classes/:?]
    at com.datalook.group.BusinessGroupController.businessGroup(BusinessGroupController.java:43) [classes/:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]

121.251.19.243 - - [02/Jan/2018:00:00:13 +0800] "GET /eams5-student/static/cryptoJS/sha1.js HTTP/1.1" 404 - "http://jxglstu.hfut.edu.cn/eams5-student/login" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3; rv:11.0) like Gecko" 0.0.0.107 80 0 - "101.226.226.221" "-"


2018-03-28 13:23:01 PaymentAdmin:{"manageamt":"0.00","managefee":0.2,"pcode":"01","txamt":"0.01"}

Answer 1

有多种方法可以实现，您可以定义一个 grok 具有多个匹配模式的过滤器，像这样，

filter {
   grok {
     match => [ "message", "PATTERN1", "PATTERN2" ]
    }
}

或者，

if "SUCCESS" not in [tags]{
  grok {
    match => { "message" => PATTERN1}
    add_tag => ["SUCCESS"]
    remove_tag => ["_grokparsefailure"]
  }
}

if "SUCCESS" not in [tags]{
  grok {
    match => { "message" => PATTERN2}
    add_tag => ["SUCCESS"]
    remove_tag => ["_grokparsefailure"]
  }
}

或者，setting a type through input plugin

的条件方法

if [type] == "syslog" {
  grok {
    match => ["message", "PATTERN1"]
  }
}

else if [type] == "nginx" {
      grok {  
        match => ["message", "PATTERN2"]
      }
  }

上述方法的问题是模式将按顺序应用，直到找到匹配项。

Logstash 在一个日志文件中处理多种格式的文本

Logstash handle more than one format of text in one log file

regex

logstash

logstash-grok