測試自定義Logstash過濾器

Question

我們使用Ansible並擁有Logstash。

如何編寫一些測試來覆蓋我們的自定義logstash過濾器？我想要做的是這樣的：

1. 給予Logstash配置有過濾器，
2. 傳一個日誌行（或多行日誌記錄），
3. 看到它成功地解析成片。

我知道這是 - https://github.com/elastic/logstash/wiki/Tips:Testing-your-filters，但我不明白這是如何有用 - 它似乎已過時。

Answer 1

回答行之有效高達logstash的1.5版本。此後，grok模式從核心中移除並放入logstash-core-patterns中。在logstash 2.2中對我有用的是：

# encoding: utf-8 

require 'spec_helper' 
require "logstash/patterns/core" 

# solution based on https://github.com/logstash-plugins/logstash-filter-grok/blob/master/spec/filters/grok_spec.rb 
module LogStash::Environment 
    # running the grok code outside a logstash package means 
    # LOGSTASH_HOME will not be defined, so let's set it here 
    # before requiring the grok filter 

    # the path that is set is the plugin root path 
    unless self.const_defined?(:LOGSTASH_HOME) 
    LOGSTASH_HOME = File.expand_path("../../../", __FILE__) 
    end 

    # also :pattern_path method must exist so we define it too 

    # method is called by logstash-filter-grok to create patterns_path array 
    # 
    # logstash-filter-grok/lib/logstash/filters/grok.rb(line ~230): 
    # 
    # @@patterns_path += [ 
    #  LogStash::Patterns::Core.path, 
    #  LogStash::Environment.pattern_path("*") 
    # 
    # patterns defined in spec/patterns/ will be joined to the array by grok 

    unless self.method_defined?(:pattern_path) 
    def pattern_path(path) 
     ::File.join(LOGSTASH_HOME, "spec", "patterns", path) 
    end 
    end 
end 

require "logstash/filters/grok" 
require "logstash/filters/<tested-plugin>" 

其餘規範示例仍然有效。

隨着新的依賴關係要求的到來，Gemfile也必須更改。我gemspec依賴關係是這樣的：

# Gem dependencies 
s.add_runtime_dependency "logstash-core-plugin-api", "~> 1.0" 
s.add_runtime_dependency "<tested-plugin>" 
s.add_development_dependency 'logstash-devutils', '~> 0' 
s.add_development_dependency 'logstash-filter-grok', '~> 3.2' 
s.add_development_dependency 'logstash-patterns-core', '~> 4.0' 

一個完整的工作exapmle可以發現here

Answer 2

我發現this並結束了以下工作的測試代碼：

# simple_filter_spec.rb 
# 
# run using: 
# bundle exec rspec simple_filter_spec.rb 

require "logstash/devutils/rspec/spec_helper" 

LogStash::Environment::LOGSTASH_HOME = `gem which logstash-core` 
module LogStash::Environment 
    unless self.method_defined?(:pattern_path) 
    def pattern_path(path) 
     ::File.join(LOGSTASH_HOME, "patterns", path) 
    end 
    end 
end 


require "logstash/filters/grok" 

describe LogStash::Filters::Grok do 
    config <<-CONFIG 
    filter { 
    grok { 
     match => { "message" => "%{SYSLOGLINE}" } 
     singles => true 
     overwrite => [ "message" ] 
    } 
    } 
    CONFIG 

    sample "Mar 16 00:01:25 evita postfix/smtpd[1713]: connect from camomile.cloud9.net[168.100.1.3]" do 
    insist { subject["tags"] }.nil? 
    insist { subject["logsource"] } == "evita" 
    insist { subject["timestamp"] } == "Mar 16 00:01:25" 
    insist { subject["message"] } == "connect from camomile.cloud9.net[168.100.1.3]" 
    insist { subject["program"] } == "postfix/smtpd" 
    insist { subject["pid"] } == "1713" 
    end 
end 

而且我的Gemfile看起來像這樣：

source 'https://www.rubygems.org' 

platform :jruby do 
    gem 'pry' 
    gem 'rspec' 
    gem 'logstash-core' 
    gem 'logstash-devutils' 
    gem 'logstash-filter-grok' 
end 

Answer 3

可以使用Logstash-Tester - 一個小工具，單元測試logstash過濾器和模式。您使用json編寫測試用例，logstash-tester使用docker容器在logstash上運行它們。 （免責聲明：我寫的工具）

Answer 4

從this blog：

$ git clone https://github.com/elastic/logstash 
$ cd logstash 
$ git checkout 2.1 
$ rake bootstrap 
$ rake test:install-core 

而不是檢查出來的2.1分支，你應該檢查出logstash的標記的版本，你會實際運行，例如v2.3.2（注意前面的「v」）。

運行上述命令後，可以在logstash回購中運行bin/rspec /some/path/your_filter_spec.rb。

重要：我發現編碼行# encoding: utf-8是必需的，否則匹配將失敗。

樣品測試文件：由gmile給出

# encoding: utf-8 

require "spec_helper" 

describe "simple request log" do 
    config (<<-CONFIG) 
    filter { 
    grok { 
     match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" } 
    } 
    } 
    CONFIG 

    sample '55.3.244.1 GET /index.html 15824 0.043' do 
    insist { subject['client'] } == '55.3.244.1' 
    insist { subject['method'] } == 'GET' 
    insist { subject['request'] } == '/index.html' 
    insist { subject['bytes'] } == '15824' 
    insist { subject['duration'] } == '0.043' 
    end 
end