用于选择和格式化某些日志行的 Grok 过滤器

Question

我正在编写一个 grok 过滤器来解析我的非结构化应用程序日志。我需要的是寻找某些行并以特定格式生成输出。例如下面是我的日志

2018-05-07 01:19:40 M :Memory (xivr = 513.2 Mb, system = 3502.0 Mb, physical = 5386.7 Mb), CpuLoad (sys = 0%, xivr = 0%)
2018-05-07 01:29:40 M :Memory (xivr = 513.2 Mb, system = 3495.3 Mb, physical = 5370.1 Mb), CpuLoad (sys = 0%, xivr = 0%)
2018-05-07 05:51:19 1 :Hangup call
***2018-05-07 05:51:22 24 :Answer call from 71840746 for 91783028 [C:\xivr\es\IVR-Dialin.dtx***]
2018-05-07 05:51:30 24 :Hangup call
***2018-05-07 05:51:34 24 :Answer call from 71840746 for 91783028 [C:\xivr\es\IVR-Dialin.dtx]***
2018-05-07 00:31:21 45 :Device Dialogic Digital dxxxB12C1 [gc60.dev - Dialogic (SDK 6.0) ver 3.0.702:11646] (ThreadID: 1FF0, DriverChannel: 44)
2018-05-07 00:31:22 40 :Device Dialogic Digital dxxxB10C4 [gc60.dev - Dialogic (SDK 6.0) ver 3.0.702:11646] (ThreadID: 1B2C, DriverChannel: 39)

我只需要在我的 Kibana 中以以下格式输入用 *** 突出显示的行：其他行应该简单地忽略

Logtimestamp: 2018-05-07 05:51:22
Channel_id: 24
Source_number: 71840746
Destination_Number: 91783028

如何实现？

Answer 1

您可以显式地写出关于该特定模式的任何独特之处，其余的使用 pre-defined grok patterns。

在你的例子中，grok 模式是，

%{TIMESTAMP_ISO8601:Logtimestamp} %{NUMBER:Channel_id} :Answer call from %{NUMBER:Source_number} for %{NUMBER:Destination_Number} %{GREEDYDATA:etc}

它只会匹配以下模式，

2018-05-07 05:51:34 24 :Answer call from 71840746 for 91783028 [C:\xivr\es\IVR-Dialin.dtx]

说明

syntax for a grok pattern 是 %{SYNTAX:SEMANTIC}。

在您的过滤器中，

%{TIMESTAMP_ISO8601:Logtimestamp} 匹配 2018-05-07 05:51:34
%{NUMBER:Channel_id} 匹配 24
:Answer call from 按字面意思匹配字符串
%{NUMBER:Source_number} 匹配 71840746
%{NUMBER:Destination_Number} 匹配 91783028
%{GREEDYDATA:etc} 匹配其余数据，即 [C:\xivr\es\IVR-Dialin.dtx]

按照这个顺序。

输出：

{
  "Logtimestamp": [
    [
      "2018-05-07 05:51:22"
    ]
  ],
  "Channel_id": [
    [
      "24"
    ]
  ],
  "Source_number": [
    [
      "71840746"
    ]
  ],
  "Destination_Number": [
    [
      "91783028"
    ]
  ],
  "etc": [
    [
      "[C:\xivr\es\IVR-Dialin.dtx***]"
    ]
  ]
}

你可以test it here.

希望对您有所帮助。

用于选择和格式化某些日志行的 Grok 过滤器

Grok filter for selecting and formatting certain logs lines

logstash

logstash-grok

说明