Logstash 弹性搜索输出不正确 document_id

Question

我正在使用 Logstash 从 Solace 队列读取 json 消息并将其写入弹性 Search.I'm 使用 doc_as_upsert => true 以及 document_id 中的参数output.This 是我的 logstash 配置的样子

logstash.conf

input

    {
        jms {
            include_header => false
            include_properties => false
            include_body => true
            use_jms_timestamp => false
            destination => 'SpringBatchTestQueue'
            pub_sub => false
            jndi_name => '/JMS/CF/MDM'
            jndi_context => {
             'java.naming.factory.initial' => 'com.solacesystems.jndi.SolJNDIInitialContextFactory'
             'java.naming.security.principal' => 'EDM_Test_User@NovartisDevVPN'
             'java.naming.provider.url' => 'tcp://localhost:55555'
             'java.naming.security.credentials' => 'EDM_Test_User'
            }
            require_jars=> ['/app/elasticsearch/jms/commons-lang-2.6.jar',
                            '/app/elasticsearch/jms/sol-jms-10.10.0.jar',
                            '/app/elasticsearch/jms/geronimo-jms_1.1_spec-1.1.1.jar']
        }
    }
    output
    {
        elasticsearch
        {
            hosts => ["https://glchbs-sd220240.eu.novartis.net:9200/"]
            index => "test-%{+YYYY.MM.dd}"
            document_id => "%{customerId}"
            doc_as_upsert => true
            ssl => true
            ssl_certificate_verification => true
            cacert => "/app/elasticsearch/config/ssl/Novartis_Silver_Three_Chain.pem"
        }
    }

Json 留言：

{
    "customerId": "N-CA-Z9II2YJ1YJ",
    "name": "Alan Birch",
    "customerRecordType": "Health Care Professional",
    "country": "CA",
    "language": "EN",
    "privacyLawStatus": false,
    "salutation": "Mr.",
    "firstName": "Alan",
    "lastName": "Birch",
    "customerType": "Non Prescriber",
    "hcpType": "Pharmacist Assistant",
    "isMedicalExpert": false,
    "customerAddresses": [
        {
            "addressType": "Primary Address",
            "addressLine1": "4001 Leslie Street"
        },
        {
            "addressType": "Other",
            "addressLine1": "3004 Center St"
        }
    ],
    "meansOfContact": [
        {
            "type": "Email1",
            "value": "alab@noname.com",
            "status": "Active"
        },
        {
            "type": "Email2",
            "value": "balan@gmail.com",
            "status": "Active"
        }
    ],
    "specialities": [
        {
            "specialtyType": "Primary Specialty",
            "specialty": "Pharmacy Technician",
            "status": "Active"
        }
    ]
}

如您所见，我正在尝试使用 JSON 消息的 customerId 字段作为 elasticsearch 的文档 ID。但这是插入 Elasticsearch 的文档的样子：

如您所见，document_id 字段应映射到 customerId 字段，但事实并非如此。文档被插入为 %{customerId}

如何解决这个问题？感谢您的帮助

Answer 1

这告诉您 [customerId] 字段在该事件中不存在。如果 [message] 字段是 JSON 那么你应该添加一个 json 过滤器来解析它。这将创建 [customerId] 字段，然后您可以将其用作 document_id.

json { source => "message" }

Logstash 弹性搜索输出不正确 document_id

Incorrect document_id for Logstash elastic search output

elasticsearch

logstash

logstash-configuration