尝试使用 ElasticSearch 输出创建与请求路径值同名的索引时发生无效 FieldReference

Question

这是我的 logstash.conf 文件：

input {
    http {
        host => "127.0.0.1"
        port => 31311 
    }
}

filter {
    mutate {
        split => ["%{headers.request_path}", "/"]
        add_field => { "index_id" => "%{headers.request_path[0]}" }
        add_field => { "document_id" => "%{headers.request_path[1]}" }
    }
}

output {
  elasticsearch {
    hosts => "http://localhost:9200"
    index => "%{index_id}"
    document_id => "%{document_id}"
  }
  stdout {
    codec => "rubydebug"
  }
}

当我发送 PUT 请求时

C:\Users\BolverkXR\Downloads\curl-7.64.1-win64-mingw\bin> .\curl.exe -XPUT 'http://127.0.0.1:31311/twitter'

我想创建一个名为 twitter 的新索引，而不是使用 ElasticSearch default。

但是，Logstash 立即崩溃并显示以下（截断的）错误消息：

Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: headers.request_path[0]

我确定我在某个地方犯了语法错误，但我看不到它在哪里。我该如何解决这个问题？

编辑：

当我将 filter 段更改为以下内容时，出现同样的错误：

filter {
    mutate {
        split => ["%{[headers][request_path]}", "/"]
        add_field => { "index_id" => "%{[headers][request_path][0]}" }
        add_field => { "document_id" => "%{[headers][request_path][1]}" }
    }
}

Answer 1

要拆分字段，%{foo} 语法 未使用 。另外你应该从数组的位置 [1] 开始，因为在位置 [0] 会有一个空字符串（""），因为第一个分隔符（/).相反，您的过滤器部分应该是这样的：

 filter {
    mutate {
        split => ["[headers][request_path]", "/"]
        add_field => { "index_id" => "%{[headers][request_path][1]}" }
        add_field => { "document_id" => "%{[headers][request_path][2]}" }
    }
}

您现在可以使用 %{index_id} 和 %{document_id} 中的值。我使用 logstash 6.5.3 版本对此进行了测试，并使用 Postman 发送了“http://127.0.0.1:31311/twitter/1”HTTP 请求，控制台中的输出如下：

{
        "message" => "",
       "index_id" => "twitter",
    "document_id" => "1",
       "@version" => "1",
           "host" => "127.0.0.1",
     "@timestamp" => 2019-04-09T12:15:47.098Z,
        "headers" => {
             "connection" => "keep-alive",
           "http_version" => "HTTP/1.1",
            "http_accept" => "*/*",
          "cache_control" => "no-cache",
         "content_length" => "0",
          "postman_token" => "cb81754f-6d1c-4e31-ac94-fde50c0fdbf8",
        "accept_encoding" => "gzip, deflate",
           "request_path" => [
            [0] "",
            [1] "twitter",
            [2] "1"
        ],
              "http_host" => "127.0.0.1:31311",
        "http_user_agent" => "PostmanRuntime/7.6.1",
         "request_method" => "PUT"
    }
}

您的配置的输出部分没有改变。因此，您的最终 logstash.conf 文件将如下所示：

input {
    http {
        host => "127.0.0.1"
        port => 31311 
    }
}

filter {
    mutate {
        split => ["[headers][request_path]", "/"]
        add_field => { "index_id" => "%{[headers][request_path][1]}" }
        add_field => { "document_id" => "%{[headers][request_path][2]}" }
    }
}

output {
  elasticsearch {
    hosts => "http://localhost:9200"
    index => "%{index_id}"
    document_id => "%{document_id}"
  }
  stdout {
    codec => "rubydebug"
  }
}

尝试使用 ElasticSearch 输出创建与请求路径值同名的索引时发生无效 FieldReference

Invalid FieldReference occurred when attempting to create index with the same name as request path value using ElasticSearch output

elasticsearch

elasticsearch-plugin