Processing common event information with grok{}

If you’re lucky, most of your log messages in a given input will arrive in a standard format, typically with a set of common fields at the front (date, time, server, etc).

Rather than multiple grok{} patterns that are looking across the entire message, like these:

grok {
    match => ["message", "%{SYSLOGTIMESTAMP:syslogtime} %{HOSTNAME:sysloghost} Save this %{WORD:word1}"]
    tag_on_failure => ["_grokparsefailure_match1"]
}

grok {
    match => ["message", "%{SYSLOGTIMESTAMP:syslogtime} %{HOSTNAME:sysloghost} Save this other %{WORD:word2}"]
    tag_on_failure => ["_grokparsefailure_match2"]
}

I like to split off the common stuff:

grok {
    match => ["message", "%{SYSLOGTIMESTAMP:syslogtime} %{HOSTNAME:sysloghost} %{GREEDYDATA:message}"]
    overwrite => [ "message" ]
    tag_on_failure => ["_grokparsefailure_syslog"]
}

Note that the last pattern puts the results into the field “message”.  Since that field already exists, we have to use the “overwrite” setting to update it.

Then use smaller patterns against this smaller “message” for your application specific info:

grok {
    match => ["message", "Save this %{WORD:word1}"]
    tag_on_failure => ["_grokparsefailure_match1"]
}

This is easier to read, and the later grok{}s will be running smaller regexps
against smaller input, which should be faster.