Apache Catalina 日志的 Grok 模式

Question

我有一些看起来像这样的 apache catalina 日志：

[22/Jul/2016:09:22:37 +0000] 10.10.29.1 - GET GET /static/s/en/providerLayer_ROOT.js HTTP/1.1  200 6298 HTTP/1.1 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 https://wpqa.test.com/app/prov/provSelectAccount.htm
[22/Jul/2016:09:22:37 +0000] 10.10.29.1 - GET GET /static/s/en/gregorian.js HTTP/1.1  200 4987 HTTP/1.1 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 https://wpqa.test.com/app/prov/provSelectAccount.htm

如何编写 grok 模式来匹配这些。我在网上四处看看，但找不到太多。我试过了：

match => [ "message", "%{TOMCATLOG}", "message", "%{CATALINALOG}" ]

但想要更详细地了解细节。

[22/Jul/2016:09:22:37 +0000] --is date time
10.10.29.1                   --is Ip address
GET GET                      --HTTP Method
/static/s/en/providerLayer_ROOT.js -- Request
HTTP/1.1                     --Protocol Version
200                          --HTTP Status
6298                         --Response time
HTTP/1.1 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36  --Browser info
https://wpqa.test.com/app/prov/provSelectAccount.htm -- Called URL

我完全无法解决这个问题，而且无论正则表达式如何，我都会不断收到 _grokparsefailures。我的模式文件中是否缺少某些内容？

谢谢，

Answer 1

我使用了以下 grok 过滤器，它非常适合您的日志：

%{SYSLOG5424SD:timestamp} %{IPV4:IP} - %{CRON_ACTION:HTTPMETHOD}%{URIPATH:request} %{NOTSPACE:protocolVersion}  %{NUMBER:status} %{NUMBER:responseTime} %{NOTSPACE:browserinfo} %{NOTSPACE:browserinfo} (?<browserinfo>(\((.*)\))) %{NOTSPACE:browserinfo} %{NOTSPACE:browserinfo} %{URI:calledURL}

这是输出：

{
  "timestamp": [
    [
      "[22/Jul/2016:09:22:37 +0000]"
    ]
  ],
  "IP": [
    [
      "10.10.29.1"
    ]
  ],
  "HTTPMETHOD": [
    [
      "GET GET "
    ]
  ],
  "request": [
    [
      "/static/s/en/gregorian.js"
    ]
  ],
  "protocolVersion": [
    [
      "HTTP/1.1"
    ]
  ],
  "status": [
    [
      "200"
    ]
  ],
  "responseTime": [
    [
      "4987"
    ]
  ],
  "browserinfo": [
    [
      "HTTP/1.1",
      "Mozilla/5.0",
      "Chrome/51.0.2704.103",
      "Safari/537.36"
    ],
    [
      "(Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)"
    ]
  ],
  "calledURL": [
    [
      "https://wpqa.test.com/app/prov/provSelectAccount.htm"
    ]
  ]
}

您可以使用 grok 调试器来调试您的 grok 过滤器 here
您可以在此 link

上使用 grok 构造函数

Apache Catalina 日志的 Grok 模式

Grok pattern for Apache Catalina logs

regex

pattern-matching

logstash

logstash-grok