logstash 服务器日志中的类型和 max_open_files 错误

Question

我在我的 logstash 服务器上的 logstash 日志文件中收到一些烦人的消息：

的第一个样子

[2019-01-29T21:27:30,230][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"syslog-2019.01.29", :_type=>"doc", :routing=>nil}, #<LogStash::Event:0x7e88287a>], :response=>{"index"=>{"_index"=>"syslog-2019.01.29", "_type"=>"doc", "_id"=>"zsY5nWgB6AmJPdJO_omb", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [syslog-2019.01.29] as the final mapping would have more than 1 type: [messages, doc]"}}}}

第二个 'max_open_files'

[2019-01-29T21:24:57,887][WARN ][filewatch.tailmode.processor] Reached open files limit: 4095, set by the 'max_open_files' option or default, files yet to open: 422

这个max_open_files是否与弹性服务器发送数据有关。

我增加了 /usr/lib/systemd/system/elasticsearch.service 文件中的限制和 /etc/security/limits.conf 但没有任何变化。

我的 logstash 配置文件：

大一：

[root@myelk04 ~]# cat /etc/logstash/conf.d/syslog.conf
input {
  file {
    path => [ "/data/SYSTEMS/*/messages.log" ]
    start_position => beginning
    sincedb_path => "/dev/null"
    type => "syslog"
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp } %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      remove_field => ["@version", "host", "message", "_type", "_index", "_score", "path"]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
 }
}
}
output {
        if [type] == "syslog" {
        elasticsearch {
                hosts => "myelk01:9200"
                manage_template => false
                index => "syslog-%{+YYYY.MM.dd}"
                document_type => "messages"
  }
 }
}
[root@myelk04 ~]#

当前一个：

可能是我刚刚删除了 document_type => "messages"，因为它弹出此消息并现在默认为文档。

[root@myelk04 ~]# cat /etc/logstash/conf.d/syslog.conf
input {
  file {
    path => [ "/data/SYSTEMS/*/messages.log" ]
    start_position => beginning
    sincedb_path => "/dev/null"
    type => "syslog"
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp } %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      remove_field => ["@version", "host", "message", "_type", "_index", "_score", "path"]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
 }
}
}
output {
        if [type] == "syslog" {
        elasticsearch {
                hosts => "myelk01:9200"
                manage_template => false
                index => "syslog-%{+YYYY.MM.dd}"
  }
 }
}
[root@myelk04 ~]#

Answer 1

第一个错误表明，logstash 正在尝试更新特定索引的映射。此更新将为类型 "doc" 添加一个新映射，但已经存在 "messages" 的映射。这将导致同一索引中有两个映射，不再支持这种情况。请检查此索引的映射以及您尝试在 syslog-* 索引中索引的文档类型。也许您已经对类型为 "message"?

的某种文档使用了完全相同的索引

第二个错误表明，已达到打开的文件数。要永久增加它，您需要遵循 this 说明（您已经部分应用的说明）。不仅在您的 elasticsearch 服务器上，而且在 logstash 主机上发出此更改。

要在服务器运行时应用此设置，您需要执行此命令并重新启动服务：

sudo ulimit -n 65535

logstash 服务器日志中的类型和 max_open_files 错误

Errror for type and max_open_files on the logstash server's logs

elasticsearch

logstash

logstash-configuration

elastic-stack