测试自定义 Logstash 过滤器

Question

我们正在使用 Ansible 和 Logstash。

如何编写一些测试来覆盖我们的自定义 logstash 过滤器？我想做的是：

给定一个带过滤器的 Logstash 配置，
向其传递日志行（或多行日志记录），
看到它成功解析成碎片。

我知道有这个 – https://github.com/elastic/logstash/wiki/Tips:Testing-your-filters，但我不明白它有什么用 – 它看起来已经过时了。

Answer 1

我找到了 this 并得到了以下工作测试代码：

# simple_filter_spec.rb
#
# run using:
#   bundle exec rspec simple_filter_spec.rb

require "logstash/devutils/rspec/spec_helper"

LogStash::Environment::LOGSTASH_HOME = `gem which logstash-core`
module LogStash::Environment
  unless self.method_defined?(:pattern_path)
    def pattern_path(path)
      ::File.join(LOGSTASH_HOME, "patterns", path)
    end
  end
end


require "logstash/filters/grok"

describe LogStash::Filters::Grok do
  config <<-CONFIG
  filter {
    grok {
      match => { "message" => "%{SYSLOGLINE}" }
      singles => true
      overwrite => [ "message" ]
    }
  }
  CONFIG

  sample "Mar 16 00:01:25 evita postfix/smtpd[1713]: connect from camomile.cloud9.net[168.100.1.3]" do
    insist { subject["tags"] }.nil?
    insist { subject["logsource"] } == "evita"
    insist { subject["timestamp"] } == "Mar 16 00:01:25"
    insist { subject["message"] } == "connect from camomile.cloud9.net[168.100.1.3]"
    insist { subject["program"] } == "postfix/smtpd"
    insist { subject["pid"] } == "1713"
  end
end

我的 Gemfile 看起来像这样：

source 'https://www.rubygems.org'

platform :jruby do
  gem 'pry'
  gem 'rspec'
  gem 'logstash-core'
  gem 'logstash-devutils'
  gem 'logstash-filter-grok'
end

Answer 2

您可以使用 Logstash-Tester - 一个对 logstash 过滤器和模式进行单元测试的小工具。您在 json 中编写测试用例，logstash-tester 使用 docker 容器在 logstash 上运行它们。（免责声明：我写了工具）

Answer 3

来自this blog：

$ git clone https://github.com/elastic/logstash
$ cd logstash
$ git checkout 2.1
$ rake bootstrap
$ rake test:install-core

而不是查看 2.1 分支，您应该查看标记的 logstash 版本，您实际上是运行ning，例如v2.3.2（注意前导"v"）。

运行执行上述命令后，您可以运行 bin/rspec /some/path/your_filter_spec.rb 在logstash 仓库中

重要：我发现编码行# encoding: utf-8是必须的，否则会匹配失败。

示例测试文件：

# encoding: utf-8

require "spec_helper"

describe "simple request log" do
  config (<<-CONFIG)
  filter {
    grok {
      match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }
    }
  }
  CONFIG

  sample '55.3.244.1 GET /index.html 15824 0.043' do
    insist { subject['client']   } == '55.3.244.1'
    insist { subject['method']   } == 'GET'
    insist { subject['request']  } == '/index.html'
    insist { subject['bytes']    } == '15824'
    insist { subject['duration'] } == '0.043'
  end
end

Answer 4

gmile 给出的答案适用于 logstash 1.5 版。从那时起，grok 模式被从核心中移除并放置在 logstash-core-patterns 中。在 logstash 2.2 中对我有用的是：

# encoding: utf-8

require 'spec_helper'
require "logstash/patterns/core"

# solution based on https://github.com/logstash-plugins/logstash-filter-grok/blob/master/spec/filters/grok_spec.rb
module LogStash::Environment
  # running the grok code outside a logstash package means
  # LOGSTASH_HOME will not be defined, so let's set it here
  # before requiring the grok filter

  # the path that is set is the plugin root path
  unless self.const_defined?(:LOGSTASH_HOME)
    LOGSTASH_HOME = File.expand_path("../../../", __FILE__)
  end

  # also :pattern_path method must exist so we define it too

  # method is called by logstash-filter-grok to create patterns_path array
  #
  #   logstash-filter-grok/lib/logstash/filters/grok.rb(line ~230):
  #
  #   @@patterns_path += [
  #     LogStash::Patterns::Core.path,
  #     LogStash::Environment.pattern_path("*")
  #
  # patterns defined in spec/patterns/ will be joined to the array by grok 

  unless self.method_defined?(:pattern_path)
    def pattern_path(path)
      ::File.join(LOGSTASH_HOME, "spec", "patterns", path)
    end
  end
end

require "logstash/filters/grok"
require "logstash/filters/<tested-plugin>"

规范示例的其余部分仍然有效。

随着新的依赖需求的到来，Gemfile 也必须改变。我的 gemspec 依赖项是这样的：

# Gem dependencies
s.add_runtime_dependency "logstash-core-plugin-api", "~> 1.0"
s.add_runtime_dependency "<tested-plugin>"
s.add_development_dependency 'logstash-devutils', '~> 0'
s.add_development_dependency 'logstash-filter-grok', '~> 3.2'
s.add_development_dependency 'logstash-patterns-core', '~> 4.0'

可以找到完整的工作示例here

测试自定义 Logstash 过滤器

Testing custom Logstash filters

testing

rspec

logstash

ansible