Logstash

Question

我希望以不同的方式处理不同的字段。

我有两条管道。一种是处理布尔值，另一种是将字符串转换为数组。

 output {
    stdout {
        codec => rubydebug
    }
    elasticsearch {
        action => "index"
        hosts => ["127.0.0.1:9200"]
        index => "mini_system"
        document_id => "%{mini_system_key}"
        if [source] == "secure_flag" {
            pipeline => "bool-pipeline"
        } else if "partners" == %{FIELD} {
            pipeline => "partners-pipeline"
        }
    }
}

我正在尝试这样做。但是我无法做到这一点，也找不到参考。

示例文档：

key,partners,secure_flag,date_added
5369922730525,"1002300,1009747,12359,2285459",FALSE,2020-03-31T14:00:00Z    
2218100624,,FALSE,2020-03-31T14:00:00Z

这里，

“1002300,1009747,12359,2285459”是伙伴。 FALSE 是 secure_flag。

合作伙伴渠道：

{
  "description": "Converts \"a,b,c\" to [\"a\", \"b\",\"c\"]",
  "processors" : [
    {
      "split" : {
        "field" : "partners",
        "separator": ",",
        "ignore_missing": true
      }
    }
  ]
}

Answer 1

您不能在插件配置中应用逻辑，但您绝对可以使用 if/else 逻辑获得多个输出：

output {
    stdout {
        codec => rubydebug
    }
    if [source] == "secure_flag" {
        elasticsearch {
            action => "index"
            hosts => ["127.0.0.1:9200"]
            index => "mini_system"
            document_id => "%{mini_system_key}"
            pipeline => "bool-pipeline"
        }
    } else if [field_xyz] == "partners" {
        elasticsearch {
            action => "index"
            hosts => ["127.0.0.1:9200"]
            index => "mini_system"
            document_id => "%{mini_system_key}"
            pipeline => "partners-pipeline"
        }
    }
}

更新：

您实际上不需要任何逻辑，只需将您的两个处理器添加到同一个管道中即可：

PUT _ingest/pipeline/mini-pipeline
{
  "processors" : [
    {
      "convert" : {
        "field" : "secure_flag",
        "type": "boolean",
        "ignore_missing": true
      }
    },
    {
      "split" : {
        "field" : "partners",
        "separator": ",",
        "ignore_missing": true
      }
    }
  ]
}

然后简单的使用这个配置

output {
    stdout {
        codec => rubydebug
    }
    elasticsearch {
        action => "index"
        hosts => ["127.0.0.1:9200"]
        index => "mini_system"
        document_id => "%{mini_system_key}"
        pipeline => "mini-pipeline"
    }
}

Logstash - 指定多个管道

Logstash - specify more than one pipeline

elasticsearch