自定义日志字符串的 ElasticSearch Grok 模式问题

Question

我正尝试在 Elasticsearch 上为以下基于 JSON 的自定义日志创建完整的 GROK 模式：

------------------------DEBUG----------------------------
Date   : 2019-12-26 12:18:21,498 
METHOD NAME:   xyz
{
  "methodName": "SMS_POOL_IN",
  "Tran_Type": "Response",
  "URL": "xyz.abcL",
  "ApiResult": "Success",
  "Date": "2019/12/26 12:18:21",
  "ErrorCode": "00",
  "ErrorReason": "Success",
  "Msisdn": "9999999",
  "CNIC": "99999999",
  "RequestID": "1111",
  "SR_TranID": "2222",
  "Channel": "abc"
}

但是每当我解析这个时，我只从 grok 得到时间戳。

我正在使用 grok 调试器对此进行测试。每当我使用 greedydata 时，我只得到第一个 json 参数，其余的被忽略，我在这里遗漏了什么吗？我怎样才能从这些日志中生成 grok？任何帮助将不胜感激

我在grok下面创建了

%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}

并得到以下结果。

{
  "GREEDYDATA": [
    [
      "------------------------DEBUG----------------------------",
      "Date   : 2019-12-26 12:18:21,498 ",
      "METHOD NAME:   xyz",
      "{",
      ""methodName": "SMS_POOL_IN",",
      ""Tran_Type": "Response",",
      ""URL": "xyz.abcL",",
      ""ApiResult": "Success",",
      ""Date": "2019/12/26 12:18:21",",
      ""ErrorCode": "00",",
      ""ErrorReason": "Success",",
      ""Msisdn": "9999999",",
      ""CNIC": "99999999",",
      ""RequestID": "1111",",
      ""SR_TranID": "2222",",
      ""Channel": "abc"",
      "} ",
      "",
      ""
    ]
  ],
  "SPACE": [
    [
      "\n",
      "\n",
      "\n",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n",
      "",
      ""
    ]
  ]
}

我需要将所有这些 json 标签分开显示，因为我需要将它们填充到 ELK 中的单独标签中

Answer 1

我自己创建了 grok，唯一的问题是我试图构建 grok 的语法。以下是上面阅读的正确 grok 语法

%{TIMESTAMP_ISO8601:date_time}\s*%{GREEDYDATA:Method}\n%{GREEDYDATA:Bracket}\s*\"methodName\"\:\s\"%{DATA:methodName}\s*\"Tran_Type\"\:\s\"%{DATA:Tran_Type}\s*\"URL\"\:\s\"%{DATA:URL}\s*\"ApiResult\"\:\s\"%{DATA:ApiResult}\s*\"Date\"\:\s\"%{DATA:Date}\s*\"ErrorCode\"\:\s\"%{DATA:ErrorCode}\s*\"ErrorReason\"\:\s\"%{DATA:ErrorReason}\s*\"Msisdn\"\:\s\"%{DATA:Msisdn}\s*\"CNIC\"\:\s\"%{DATA:CNIC}\s*\"RequestID\"\:\s\"%{DATA:RequestID}\s*\"SR_TranID\"\:\s\"%{DATA:SR_TranID}\s*\"Channel\"\:\s\"%{DATA:Channel}\s

首先我选择了时间戳，然后我选择了 GREEDYDATA 中 json 字符串之外的所有内容，然后我用 DATA 关键字分离了 json 标签。

上面的结果是

{
  "date_time": [
    [
      "2019-12-26 12:18:21,498"
    ]
  ],
  "YEAR": [
    [
      "2019"
    ]
  ],
  "MONTHNUM": [
    [
      "12"
    ]
  ],
  "MONTHDAY": [
    [
      "26"
    ]
  ],
  "HOUR": [
    [
      "12",
      null
    ]
  ],
  "MINUTE": [
    [
      "18",
      null
    ]
  ],
  "SECOND": [
    [
      "21,498"
    ]
  ],
  "ISO8601_TIMEZONE": [
    [
      null
    ]
  ],
  "Method": [
    [
      "METHOD NAME:   xyz"
    ]
  ],
  "Bracket": [
    [
      "{"
    ]
  ],
  "methodName": [
    [
      "SMS_POOL_IN","
    ]
  ],
  "Tran_Type": [
    [
      "Response","
    ]
  ],
  "URL": [
    [
      "xyz.abcL","
    ]
  ],
  "ApiResult": [
    [
      "Success","
    ]
  ],
  "Date": [
    [
      "2019/12/26 12:18:21","
    ]
  ],
  "ErrorCode": [
    [
      "00","
    ]
  ],
  "ErrorReason": [
    [
      "Success","
    ]
  ],
  "Msisdn": [
    [
      "9999999","
    ]
  ],
  "CNIC": [
    [
      "99999999","
    ]
  ],
  "RequestID": [
    [
      "1111","
    ]
  ],
  "SR_TranID": [
    [
      "2222","
    ]
  ],
  "Channel": [
    [
      "abc""
    ]
  ]
}

Answer 2

我假设您想将前 3 行分成 3 个单独的字段，并将 JSON 字符串对象的其余部分分成另一个字段。

我从这里复制了您输入的文本，所以每一行都以 \n 结尾。所以这是我的模式匹配元素。

如果 output 需要更多解析等，请告诉我

我的管道配置具有 grok 模式来解析输入

input {                                                                                                                            
           http {

           }

   }

  filter {

          grok {
                  match  => { "message" => "(?<debug-string>[^\n]+)\n(?<date-string>[^\n]+)\n(?<method-name>[^\n]+)\n%{GREEDYDATA:RestOfIt}" }
          }
         mutate {
                 remove_field => ["headers", "host", "@timestamp", "@version"]
         }

  }

  output {
          stdout {

          }
}

输出

{ "消息" => "---------------------DEBUG----------------------------\nDate : 2019-12- 26 12:18:21,498 \n方法名称: xyz\n{\n \"methodName\": \"SMS_POOL_IN\",\n \"Tran_Type\": \"Response\", \n \"URL\": \"xyz.abcL\",\n \"ApiResult\": \"Success\",\n \"Date\": \"2019/12/26 12:18:21\",\n \"ErrorCode\": \"00\",\n \"ErrorReason\": \"Success\",\n \"Msisdn\": \"9999999\ ",\n \"CNIC\": \"99999999\",\n \"RequestID\": \"1111\",\n \"SR_TranID\": \"2222\",\n \ "Channel\": \"abc\"\n} ",

"日期字符串" => "Date : 2019-12-26 12:18:21,498 ",

"方法名称" => "METHOD NAME: xyz",

"RestOfIt" => "{\n \"methodName\": \"SMS_POOL_IN\",\n \"Tran_Type\": \"Response\",\n \"URL\": \"xyz.abcL\",\n \"ApiResult\": \"Success\",\n \"Date\": \"2019/12 /26 12:18:21\",\n \"ErrorCode\": \"00\",\n \"ErrorReason\": \"Success\",\n \"Msisdn\": \"9999999\",\n \"CNIC\": \"99999999\",\n \"RequestID\": \"1111\",\n \"SR_TranID\": \"2222\" ,\n \"Channel\": \"abc\"\n} ",

"调试字符串" => "---------------------DEBUG-- --------------------------” }

自定义日志字符串的 ElasticSearch Grok 模式问题

ElasticSearch Grok Pattern issue for Custom Log String

logging

elasticsearch

logstash

logstash-grok