Logstash best practices

Don’t forget to check out the Elasticsearch best practices, too.

Set @timestamp

If your log entry contains the time at which the event occurred, use the date{} filter to replace @timestamp with this value.  See “Save the Date”, too.

If you’ve already grok{}’ed out a field called ‘apache_timestamp’, your date{} stanza might look like this:

filter {
  if [type] == "access" {
    date {
      match => [ 'apache_timestamp', "dd/MMM/yyyy:HH:mm:ss Z" ]
      remove_field => [ 'apache_timestamp' ]
    }
  }
}

Save the Date

By overwriting @timestamp (as suggested, above), you won’t know when logstash processed the event.  Being able to manage this lag is important.  Cover this by copying @timestamp to another field (using mutate->add_field) before applying the date{} filter.

Use Smart Patterns

You can cast the field to integer or float in the pattern, e.g.:

%{INT:my_field:int}

Use grok’s tag_on_failure

When the grok{} filter fails, it adds a tag called “_grokparsefailure”.  This is helpful, unless you have multiple grok{} filters.  In this case, add a unique tag_on_failure attribute to each grok{}.

grok {
    match => [
        "message", "%{PATTERN_1}"
    ]
    tag_on_failure => [ "failedPattern1" ]
}

Note that the syslog{} input uses grok{} internally, so it can also throw _grokparsefailure messages.  In logstash 1.5, this tag is _grokparsefailure_sysloginput.

Increase the Output Workers

Many outputs (include elasticsearch{}) support using multiple output workers.  Each will maintain their own cache (of flush_size events).  Specifying multiple output workers can increase throughput.

Increase the Filter Workers

If you’re not using the multiline{} filter, you can increase the number of logstash worker threads.  This will better utilize the CPU of your logstash machine.  In your startup script, use the “-w” flag with a value that slightly less than the number of CPUs on the machine, e.g.:

-w 18

 Increase the Open File Descriptors

On centos the init script will run ‘ulimit -n’ if you update /etc/sysconfig/logstash, e.g.:

LS_OPEN_FILES=65535

 

One response to “Logstash best practices

Leave a Reply

Your email address will not be published. Required fields are marked *