Overview
As your logstash configuration grows (mine is over 3,000 lines in 40+ files right now), you’ll want a way to make sure you don’t break anything with a new release (and that Elastic doesn’t, either!).
You can run logstash with the ‘–configtest’ option, but that’s only checking for syntax errors.
Logstash uses the rspec harness, so I wanted to start there. All the doc I could find on the web was old and wrong (even postings from elastic). Thanks to the great community in the #logstash IRC channel, I was able to get it working.
Goals
I had three goals when I started:
- to re-use my production configuration files, which are already logically arranged (“apache”, “yum”, etc). I did not want to repeat the configuration in the test files.
- to test my desired output. OK, this probably goes without saying.
- to use logstash 1.4, which is [still] in use by my main cluster. I’ve heard that things change some in 1.5, as you might expect.
Setup
You can either run the test scripts on a production machine (using a short-lived second instance of logstash) or make a VM that would have logstash and your logstash filter{} configurations installed.
Config
As part of your installation, all your configs should be in /etc/logstash/conf.d. We’ll use a small config example, test.conf:
filter { if [type] == "test" { grok { match => [ "message", "%{TIMESTAMP_ISO8601:timestamp} %{WORD:word1} %{INT:int1:int} %{NUMBER:[inner][float1]:float}" ] tag_on_failure => [ "_grokparsefailure_test" ] } } }
This grok pattern would match a string like this:
2015-21-01 12:01:02.003 UTC Hello 42 3.14159
and create four fields of the appropriate type (string, integer, float).
On my logstash machine, the built-in rspec files are in /opt/logstash/spec. I made a parallel directory as /opt/logstash/my-spec that contained this script, test.rb:
# encoding: utf-8 require "test_utils" file = "/etc/logstash/conf.d/test.conf" @@configuration = String.new @@configuration << File.read(file) describe "Test event" do extend LogStash::RSpec config(@@configuration) message = %(2015-21-01 12:01:02.003 UTC Hello 42 3.14159) sample("message" => message, "type" => "test") do insist { subject["type"] } == "test" insist { subject["timestamp"] } == "2015-21-01 12:01:02.003 UTC" insist { subject["word1"] } == "Hello" insist { subject["int1"] } == 42 insist { subject["inner"]["float1"] } == 3.14159 end end
Running
cd /opt/logstash bin/logstash rspec my-spec/test.rb
and you should see
Finished in 0.361 seconds 1 example, 0 failures
Conclusion
In my real-world config, I have a series of filters in one file that do a lot of processing on the events. In the simple example above, you can also imagine wanting to run the date{} filter on the `timestamp` column to update @timestamp. As you add more complexity, update the test cases to match. So, if your config included date{}, you would add this:
insist { subject["@timestamp"] } == Time.iso8601("2015-12-01T12:01:03.003Z").utc
Please note that this example uses the rspec2 “insist” construct. I’ve heard that rspec3 uses “expect”. There may be other syntax issues when using logstash2.
Hi. Great post. I just published a tool to streamline logstash configuration testing, kind of extending what your doing by auto generating tests from json files, describing the input and the expected output. Its available here: https://github.com/gaspaio/logstash-tester