Logstash 将 JSON 解析为单个事件

Question

我正在尝试一个 HTTP 轮询器，它 returns 向我提供以下格式的响应（这是一行 JSON）。

{"total":3,"offset":1,"len":50,"workflows":[
{"appName":"test1","createdTime":"Wed, 11 May 2016 13:30:28  GMT","startTime":"Wed, 11 May 2016 13:30:28 GMT","endTime":"Wed, 11 May 2016 13:31:06 GMT","status":"SUCCEEDED"},
{"appName":"test2","createdTime":"Wed, 11 May 2016 13:30:28 GMT","startTime":"Wed, 11 May 2016 13:30:28 GMT","endTime":"Wed, 11 May 2016 13:31:06 GMT","status":"SUCCEEDED"},
{"appName":"test3","createdTime":"Wed, 11 May 2016 13:30:28 GMT","startTime":"Wed, 11 May 2016 13:30:28 GMT","endTime":"Wed, 11 May 2016 13:31:06 GMT","status":"SUCCEEDED"}
]
}

我的要求是将每个工作流项目（数组元素）存储为弹性搜索中的单独事件。具体来说，我想提取每条记录的 appName、createdTime、Status，并将这个单独的事件传递给 ElasticSearch 输出插件。

你能帮忙吗？

logstash 配置文件如下

input {
  http_poller 
  {
    urls => 
    {
      mycall => 
      {
        method => "GET"
        url => "http://myip/url"            
      }
    }
    tags => 'data'
    request_timeout =>60
    interval => 1200
    codec => "json"
    metadata_target => "http_poller_metadata"
  }


  }

output {    
stdout  
{   

 codec => rubydebug }
}

Answer 1

使用 split filter you can split and with mutate 您可以提取字段：

会议：

split {
    field => "workflows"
    terminator => ","
    }
mutate {
   rename => {
    "[workflows][appName]" => "appName"
    "[workflows][createdTime]" => "createdTime"
    "[workflows][startTime]" => "startTime"
    "[workflows][endTime]" => "endTime"
    "[workflows][status]" => "status"
   }
   remove_field => ["workflows", "total", "offset", "len"]
}

结果：

{
       "@version" => "1",
     "@timestamp" => "2016-05-19T16:35:50.177Z",
           "host" => "Alpers-MacBook-Pro.local",
        "appName" => "test1",
    "createdTime" => "Wed, 11 May 2016 13:30:28  GMT",
      "startTime" => "Wed, 11 May 2016 13:30:28 GMT",
        "endTime" => "Wed, 11 May 2016 13:31:06 GMT",
         "status" => "SUCCEEDED"
}
{
       "@version" => "1",
     "@timestamp" => "2016-05-19T16:35:50.177Z",
           "host" => "Alpers-MacBook-Pro.local",
        "appName" => "test2",
    "createdTime" => "Wed, 11 May 2016 13:30:28 GMT",
      "startTime" => "Wed, 11 May 2016 13:30:28 GMT",
        "endTime" => "Wed, 11 May 2016 13:31:06 GMT",
         "status" => "SUCCEEDED"
}
{
       "@version" => "1",
     "@timestamp" => "2016-05-19T16:35:50.177Z",
           "host" => "Alpers-MacBook-Pro.local",
        "appName" => "test3",
    "createdTime" => "Wed, 11 May 2016 13:30:28 GMT",
      "startTime" => "Wed, 11 May 2016 13:30:28 GMT",
        "endTime" => "Wed, 11 May 2016 13:31:06 GMT",
         "status" => "SUCCEEDED"
}

Logstash 将 JSON 解析为单个事件

Logstash parse JSON into individual events

logstash

logstash-grok