无法摄取系统日志-logstash.conf 以进行删除和替换功能

Question

我只是 ELK 的新手，正在尝试对此进行一些测试，我能够运行进行一些测试，但是当我尝试使用 grok & [=14 的过滤器时=] 从我的系统日志输出中删除并替换一些字段我遇到了以下错误..

21:58:47.976 [LogStash::Runner] ERROR logstash.agent - Cannot create pipeline {:reason=>"Expected one of #, {, ,, ] at line 21, column 9 (byte 496) after filter {\n  if [type] == \"syslog\" {\n    grok {\n      match => { \"message\" => \"%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:hostname} %{DATA:program}(?:\[%{POSINT:pid}\])?: %{GREEDYDATA:syslog_message}\" }\n    }\n    date {\n      match => [ \"syslog_timestamp\", \"MMM  d HH:mm:ss\", \"MMM dd HH:mm:ss\" ]\n    }\n    mutate {\n      remove_field => [\n        \"message\",\n        \"pid\",\n        \"port\"\n        "}

Below is my config file ....

# cat logstash-syslog2.conf
input {
  file {
    path => [ "/scratch/rsyslog/*/messages.log" ]
    type => "syslog"
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:hostname} %{DATA:program}(?:\[%{POSINT:pid}\])?: %{GREEDYDATA:syslog_message}" }
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
    mutate {
      remove_field => [
        "message",
        "pid",
        "port"
        "_grokparsefailure"
      ]
    }
    mutate {
      replace => [
        "@source_host", "%{allLogs_hostname}"
        "@message", "%{allLogs_message}"
      ]
    }
    mutate {
      remove => [
        "allLogs_hostname",
        "syslog_message",
        "syslog_timestamp"
      ]
    }
}
output {
  if [type] == "syslog" {
    elasticsearch {
      hosts => "localhost:9200"
      index => "%{type}-%{+YYYY.MM.dd}"
    }
  }
}

请指出我做错了什么并帮助理解 lagstash 的删除和替换功能..

PS: 我的ELK版本是5.4

Answer 1

您必须在您的 logstash 配置文件中 "port" 后添加一个逗号。

   mutate {
      remove_field => [
        "message",
        "pid",
        "port",
        "_grokparsefailure"
      ]
    }

Answer 2

您发布的配置有很多语法错误，logsatsh 有自己的配置语言，希望配置文件遵守规则。此 link 具有完整的 logstash 配置语言参考。

我对您的配置文件进行了一些更正并张贴在这里，添加了我对配置文件本身错误的评论和解释

input 
{
    file 
    {
        path => [ "/scratch/rsyslog/*/messages.log" ]
        type => "syslog"
    }
}

filter 
{
    if [type] == "syslog" 
    {
        grok 
        {
            match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:hostname} %{DATA:program}(?:\[%{POSINT:pid}\])?: %{GREEDYDATA:syslog_message}" }
        }

    date 
    {
        match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }

    # Have merged it with the remove_field option below
    #mutate {
    #  remove_field => [
    #    "message",
    #    "pid",
    #    "port",
    #    "_grokparsefailure"
    #  ]
    #}

    mutate 
    {

        # The replace option only accept hash data type which has a syntax as below 
        # For more details visit the below link
        # https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-replace
        replace => {
            "@source_host" => "%{allLogs_hostname}" 
            "@message" => "%{allLogs_message}"
        }
    }

    mutate 
    {
        # Mutate does not have remove option i guess your intention is to remove the event field
        # hence used remove_field option here
        # The remove_filed option only accepts arary as value type as shown below
        # For details read the below link
        # https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-remove_field
        remove_field => [
            "message",
            "pid",
            "port",
            "_grokparsefailure",
            "allLogs_hostname",
            "syslog_message",
            "syslog_timestamp"
        ]
    }
  }
}

output 
{
    if [type] == "syslog" 
    {
        elasticsearch 
        {
            # The Hosts option only takes uri as a value type , originally you have provided string as it's value type
            # For more info please read the below link
            #https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-hosts
            hosts => ["localhost:9200"]
            index => "%{type}-%{+YYYY.MM.dd}"
        }
    }
}

您可以使用 logstash 命令行选项 -t 来测试配置文件的语法是否正确，此选项将测试并报告配置文件的语法是否正确

bin\logstash -f 'path-to-your-config-file' -t

请让我知道任何澄清

无法摄取系统日志-logstash.conf 以进行删除和替换功能

Unable to ingest the syslog-logstash.conf for remove & replace functions

elastic-stack