如何使用具有 logstash elasticsearch 输出更新功能的摄取管道

Question

我正在使用 Logstash Elasticsearch 输出将数据发布到 Elasticsearch。合并两个记录以从请求和响应创建单个记录。此代码没有任何问题。

elasticsearch {
  hosts => [ "localhost:9200" ]
  index => "transactions"
  action => "update"
  doc_as_upsert => true
  document_id => "%{tid}"
  script =>'
    if(ctx._source.transaction=="request"){
      ctx._source.status = params.event.get("status");
    }else if(ctx._source.transaction=="response"){                           
      ctx._source.api = params.event.get("api");                            
    }
}

现在我正在尝试使用摄取管道添加一个包含上述记录更新的新字段。

PUT _ingest/pipeline/ingest_pipe2
{
  "description" : "describe pipeline",
  "processors" : [
    {
      "set" : {
        "field": "api-test",
        "value": "new"
      }
    }
  ]
}

这将为传入事件添加一个新字段。它适用于以下代码。

elasticsearch {
  hosts => [ "localhost:9200" ]
  index => "transactions"
  pipeline => "ingest_pipe2"  
}

问题是 logstash 更新和摄取管道更新不能一起工作。

elasticsearch {
  hosts => [ "localhost:9200" ]
  index => "transactions"
  pipeline => "ingest_pipe2"**
  action => "update"
  doc_as_upsert => true
  document_id => "%{tid}"
  script =>'
    if(ctx._source.transaction=="request"){
      ctx._source.status = params.event.get("status");
    }else if(ctx._source.transaction=="response"){                           
      ctx._source.api = params.event.get("api");                            
    }
}

Answer 1

无法将摄取管道与 doc_as_upsert

一起使用

Using ingest pipelines with doc_as_upsert is not supported.

您可以找到更多信息here and here

如何使用具有 logstash elasticsearch 输出更新功能的摄取管道

How to use ingest pipeline with logstash elaticsearch output update feature

elasticsearch

logstash