LogStash Filter - 聚合重复事件并将其合并

Question

鉴于这三个 LogStash 事件，ruby 哈希

{
    "id" => "1",
    "address_unit" => "1",
    "address_sqft" => 1098,
    "address_city" => "NEW YORK"
}

{
    "id" => "1",
    "address_unit" => "1",
    "address_sqft" => 1090,
    "address_city" => "NEW YORK"
}


{
    "id" => "1",
    "address_unit" => "1",
    "address_sqft" => 2000,
    "address_city" => "NEW YORK"
}

应该将其存储到 ES 中

{
    "id" => "1",
    "address_unit" => "1",
    "address_sqft" => [1098, 1090, 2000],
    "address_city" => "NEW YORK"
}

我想知道是否有一些 built-in/community 过滤器可以实现它，或者如何使用 ruby 代码实现它。

谢谢！

Answer 1

如果您传入的哈希值类似于 a 和 b，您可以使用

[15] pry(main)> a
=> {"id"=>"1", "address_unit"=>"1", "address_sqft"=>1098, "address_city"=>"NEW YORK"}
[16] pry(main)> b
=> {"id"=>"1", "address_unit"=>"1", "address_sqft"=>1090, "address_city"=>"NEW YORK"}
[17] pry(main)> a.merge(b) {|_, old_val, new_val| old_val==new_val ? old_val :  ([old_val] << new_val).flatten}
=> {"id"=>"1", "address_unit"=>"1", "address_sqft"=>[1098, 1090], "address_city"=>"NEW YORK"}

LogStash Filter - 聚合重复事件并将其合并

LogStash Filter - Aggregate duplicated events and merge it

ruby

logstash