将数据输出到 ElasticSearch 时，Logstash 不起作用

Question

我正在尝试使用 Logstash 将 xml 文件处理到 ES。但是我试了很多次还是不行。非常感谢您的帮助。配置文件如下：

input {
  file {
    path => "/data/logstashtest/*.xml"
    start_position => "beginning"
  }
}
filter {
  multiline {
    pattern => "^\s|</report>|^[A-Za-z].*"
    what => "previous"
  }
  xml {
    store_xml => "false"
    source => "message"
    xpath => [
       "/report/@logtype", "logtype",
       "/report/result/@name", "name",
       "/report/result/@start-epoch", "start-epoch",
       "/report/result/@generated-at","generated-at"
    ]
  }
  date {
    match => [ "generated-at", "ISO8601" ]
  }
}
output {
  elasticsearch {
    protocol => http
    host => localhost
    port => 9200
    cluster => mycluster
    index => mylog
  }
  stdout { codec => rubydebug }
}

xml源文件如下：

<report reportname="" logtype="news">
  <result name="financial news" logtype="news" start-epoch="1433134800" end-epoch="1433149199" generated-at="2015/06/01 04:10:17"/>
</report>

Logstash 与其中一个 ES 节点在同一个节点中。我使用了以下命令：

bin/logstash -f threatlog.conf

它输出：

[2015-09-09 17:55:29.811]  WARN -- Concurrent: [DEPRECATED] Java 7 is deprecated, please use Java 8.
Java 7 support is only best effort, it may not work. It will be removed in next release (1.0).
Logstash startup completed

当我查看ES索引时，什么也没有。我正在使用 logstash-1.5.4。提前致谢！

Answer 1

你看到这个的原因是因为 Logstash 会跟踪文件中的位置，直到它已经处理了内容。第一次启动 Logstash 时，您可能会看到一些输出，然后再看到 none。要摆脱这种情况并继续重新开始，直到您获得正确的配置，您需要将 sincedb_path 设置为 /dev/null 这样 Logstash 就不会跟踪它在处理您的 XML 文件。

所以你的输入过滤器很可能是这样的：

input {
  file {
    path => "/data/logstashtest/*.xml"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

那么你的日期过滤器也有问题，它不期望正确的日期格式，你会得到如下错误：

Failed parsing date from field {:field=>"generated-at", :value=>"2015/06/01 04:10:17", :exception=>"Invalid format: \"2015/06/01 04:10:17\" is malformed at \"/06/01 04:10:17\"", :config_parsers=>"ISO8601", :config_locale=>"default=fr_FR", :level=>:warn}

因此，为了解决这个问题，您只需将日期过滤器更改为正确的日期格式即可：

date {
    match => [ "generated-at", "yyyy/MM/dd HH:mm:ss" ]
}

之后，您将获得一个漂亮且格式正确的 Logstash 事件：

{
         "message" => "<report reportname=\"\" logtype=\"news\">\n  <result name=\"financial news\" logtype=\"news\" start-epoch=\"1433134800\" end-epoch=\"1433149199\" generated-at=\"2015/06/01 04:10:17\"/>\n</report>",
        "@version" => "1",
      "@timestamp" => "2015-06-01T02:10:17.000Z",
            "host" => "localhost",
            "path" => "/data/text.xml",
            "tags" => [
        [0] "multiline"
    ],
         "logtype" => [
        [0] "news"
    ],
            "name" => [
        [0] "financial news"
    ],
     "start-epoch" => [
        [0] "1433134800"
    ],
    "generated-at" => [
        [0] "2015/06/01 04:10:17"
    ]
}

将数据输出到 ElasticSearch 时，Logstash 不起作用

Logstash doesn't work when output data to ElasticSearch

xml

elasticsearch

logstash