Elasticsearch configuration options

This is an attempt at a complete listing of elasticsearch config variables since they’re located all over the elastic.co website.

The list is not complete, and will start to “rot” as soon as it’s published, but… If you know of some variables that aren’t listed, please let me know.

Note that static settings must be set in the config file on every machine in the cluster.

 

Bootstrap

Name Type Notes Doc
bootstrap.mlockall Static Link

Cloud

Name Type Notes Doc
cloud.aws.access_key Unknown Link
cloud.aws.ec2 Unknown Link
cloud.aws.protocol Unknown Link
cloud.aws.proxy Unknown Link
cloud.aws.region Unknown Link
cloud.aws.s3 Unknown Link
cloud.aws.secret_key Unknown Link
cloud.aws.signer Unknown Link

Cluster

Name Type Notes Doc
cluster.blocks.read_only Dynamic Link
cluster.info.update.interval Dynamic Link
cluster.name Unknown Link
cluster.routing.allocation.allow_rebalance Dynamic Link
cluster.routing.allocation.awareness.attributes Dynamic Link
cluster.routing.allocation.awareness.force.zone.values Dynamic Link
cluster.routing.allocation.balance.shard Unknown Link
cluster.routing.allocation.balance.index Unknown Link
cluster.routing.allocation.balance.threshold Unknown Link
cluster.routing.allocation.cluster_concurrent_rebalance Dynamic Link
cluster.routing.allocation.disk.include_relocations Dynamic Link
cluster.routing.allocation.disk.threshold_enabled Dynamic Link
cluster.routing.allocation.disk.watermark.low Dynamic Link
cluster.routing.allocation.disk.watermark.high Dynamic Link
cluster.routing.allocation.enable Dynamic Link
cluster.routing.allocation.exclude Dynamic Link
cluster.routing.allocation.include Dynamic Link
cluster.routing.allocation.node_concurrent_recoveries Dynamic Link
cluster.routing.allocation.node_initial_primaries_recoveries Dynamic Link
cluster.routing.allocation.require Dynamic Link
cluster.routing.allocation.same_shard.host Dynamic Link
cluster.routing.allocation.total_shards_per_node Dynamic Link
cluster.routing.rebalance.enable Dynamic Link

Discovery

ec2 discovery can also have: groups, host_type, availability_zones, any_group, ping_timeout, and node_cache_time. Use this inside discovery.ec2, e.g. discover.ec2.groups.

Name Type Notes Doc
discovery.type Dynamic Link
discovery.zen.minimum_master_nodes Dynamic Link
discovery.zen.ping.multicast.enabled Unknown Removed in ES 2.2 Link
discovery.zen.ping.unicast.hosts Unknown Link

Gateway

Name Type Notes Doc
gateway.expected_nodes Unknw Link
gateway.expected_master_nodes Static Link
gateway.expected_data_nodes Static Link
gateway.recover_after_time Static Link
gateway.recover_after_nodes Static Link
gateway.recover_after_master_nodes Static Link
gateway.recover_after_data_nodes Static Link

HTTP

Name Type Notes Doc
http.port Static Link
http.publish_port Static Link
http.bind_host Static Link
http.publish_host Static Link
http.host Static Link
http.max_content_length Static Link
http.max_initial_line_length Static Link
http.max_header_size Static Link
http.compression Static Link
http.compression_level Static Link
http.cors.enabled Static Link
http.cors.allow-origin Static Link
http.cors.max-age Static Link
http.cors.allow-methods Static Link
http.cors.allow-headers Static Link
http.cors.allow-credentials Static Link
http.detailed_errors.enabled Static Link
http.pipelining Static Link
http.pipelining.max_events Static Link

Index

Name Type Notes Doc
index.analysis.analyzer Static Link
index.analysis.filter Static Link
index.analysis.tokenizer Static Link
index.auto_expand_replicas Dynamic Link
index.blocks.metadata Dynamic Link
index.blocks.read Dynamic Link
index.blocks.read_only Dynamic Link
index.blocks.write Dynamic Link
index.codec Static Link
index.gateway.local.sync Unknown Renamed to index.translog.sync_interval in ES 2.0 Link
index.max_result_window Dynamic Link
index.merge.policy.calibrate_size_by_deletes Unknown Removed in ES 2.0 Link
index.merge.policy.expunge_deletes_allowed Unknown Removed in ES 2.0 Link
index.merge.policy.max_merge_docs Unknown Removed in ES 2.0 Link
index.merge.policy.max_merge_size Unknown Removed in ES 2.0 Link
index.merge.policy.merge_factor Unknown Removed in ES 2.0 Link
index.merge.policy.min_merge_docs Unknown Removed in ES 2.0 Link
index.merge.policy.min_merge_size Unknown Removed in ES 2.0 Link
index.merge.policy.type Unknown Removed in ES 2.0 Link
index.merge.scheduler.max_thread_count Dynamic Link
index.number_of_replicas Dynamic Link
index.number_of_shards Static Link
index.recovery.initial_shards Dynamic Link
index.refresh_interval Dynamic Requires units in ES 2.0 Link
index.routing.allocation.exclude Dynamic Link
index.routing.allocation.include Dynamic Link
index.routing.allocation.require Dynamic Link
index.routing.allocation.total_shards_per_node Dynamic Link
index.search.slowlog.threshold Dynamic Link
index.shard.check_on_startup Static Link
index.similarity.default.type Static Link
index.store.throttle.type Unknown Removed in Es 2.0 Link
index.store.throttle.max_bytes_per_sec Unknown Removed in Es 2.0 Link
index.store.type Static memory and ram types removed in ES 2.0 Link
index.ttl.disable_purge Dynamic Link
index.translog.durability Dynamic Link
index.translog.fs.type Dynamic Link
index.translog.flush_threshold_ops Dynamic Link
index.translog.flush_threshold_period Dynamic Link
index.translog.flush_threshold_size Dynamic Link
index.translog.interval Dynamic Link
index.translog.sync_interval Static Link
index.unassigned.node_left.delayed_timeout Dynamic Link

Indices

indices.store.throttle.max_bytes_per_sec

Name Type Notes Doc
indices.analysis.hunspell.dictionary.location Unknown Removed in ES 2.0 Link
indices.recovery.concurrent_streams Dynamic Link
indices.recovery.concurrent_small_file_streams Dynamic Link
indices.store.throttle.type Unknown Removed in ES 2.0 Link
indices.store.throttle.max_bytes_per_sec Unknown Removed in ES 2.0 Link

Logger

Name Type Notes Doc
logger.indexes.recovery Dynamic Link
logger.transport.tracer Dynamic Link

Network

Name Type Notes Doc
network.bind_host Unknown Link
network.host Dynamic See special values Link
network.publish_host Unknown Link
network.tcp.no_delay Unknown Link
network.tcp.keep_alive Unknown Link
network.tcp.reuse_address Unknown Link
network.tcp.send_buffer_size Unknown Link
network.tcp.receive_buffer_size Unknown Link

Node

Name Type Notes Doc
node.data Unknown Link
node.enable_custom_paths Unknown Removed in ES 2.0 Link
node.master Unknown Link
node.max_local_storage_nodes Static Link
node.name Unknown Link

Path

Name Type Notes Doc
path.conf Static Link
path.data Static Link
path.home Static Link
path.logs Static Link
path.plugins Static Link
path.repo Static Link
path.scripts Static Link
path.shared_data Unknown Link

Plugin

Name Type Notes Doc
plugin.mandatory Static Link

Resource

Name Type Notes Doc
resource.reload.enabled Unknown
resource.reload.interval Unknown Link
resource.reload.interval.low Unknown
resource.reload.interval.medium Unknown
resource.reload.interval.high Unknown

Repositories

Name Type Notes Doc
repositories.url.allowed_urls Unknown Link

Script

Name Type Notes Doc
script.auto_reload_enabled Static Link
script.default_lang Static Link
script.disable_dynamic Unknown Removed in ES 2.0
script.file Static Link
script.index Static Link
script.inline Static Link
script.update Static Link
script.mapping Static Link
script.engine.expression Static Link
script.engine.groovy Static Link
script.engine.javascript Static Link
script.engine.mustache Static Link
script.engine.python Static Link

Thread Pool

There are several thread pools. Elastic lists the “important” ones as including: generic, index, search, suggest, get, bulk, percolate, snapshot, warmer, refresh, listener. Some settings are documented, and are listed below.

You can also control the number of processors for a thread pool, which is briefly documented here.

Name Type Notes Doc
threadpool.generic.keep_alive Dynamic Link
threadpool.index.queue_size Dynamic Link
threadpool.index.size Dynamic Link

Transport

Transport allows you to bing to multiple ports on different interfaces. See the transport profiles doc for more info.

Name Type Notes Doc
transport.bind_host Unknown Link
transport.host Unknown Link
transport.ping_schedule Unknown Link
transport.publish_host Unknown Link
transport.publish_port Unknown Link
transport.tcp.compress Unknown Link
transport.tcp.connect_time Unknown Link
transport.tcp.port Unknown Link
transport.tracer.exclude Dynamic Link
transport.tracer.include Dynamic Link

Tribe

There are a lot of options for tribes that vary based on the tribe name. Some info is presented here.

Watcher

Name Type Notes Doc
tribe.blocks.metadata Unknown Link
tribe.blocks.metadata.indices Unknown Link
tribe.blocks.write Unknown Link
tribe.blocks.write.indices Unknown Link
tribe.t1.cluster.name Unknown Link
Name Type Notes Doc
watcher.enabled Unknown Renamed in ES 2.0 Link
watcher.interval Unknown Renamed in ES 2.0 Link
watcher.interval.low Unknown Renamed in ES 2.0 Link
watcher.interval.medium Unknown Renamed in ES 2.0 Link
watcher.interval.high Unknown Renamed in ES 2.0 Link

Managing Elastic watches with Ansible

If you’re an Ansible convert, you want every part of your deployment to be managed by the tasks in your roles.  I recently wanted to manage my Elastic Watcher config with Ansible.  It was enough of a struggle (“learning process”) that I wanted to document it here.

While I was developing my first few watches, I made stand-alone shell scripts that could be run to create/update them.  This worked fine, and even handled the basic auth configuration in front of the master nodes.  Basically it was:

curl -u username -XPUT 'http://hostname/_watcher/watch/watchname' -d '{ ... }'

It would prompt for the user’s password and send the information along.

Ansible can run local commands, but doesn’t like interactive commands.  As such, the password prompt would block the task from completing.  Looking around for other solutions led me to the ‘uri’ module.  By default, it won’t prompt for the password either, but you can use the prompts system.  I chose to use the Ansible vault to store the password to avoid external sharing/management of the password.

The final config looks like this:

- set_fact:
 watch: "{{ lookup('file','watcher.json') }}"

- name: Install watcher for cluster health
 uri:
 delegate_to: 127.0.0.1
 url: "http://hostname//_watcher/watch/watchname"
 method: PUT
 user: "ansible"
 password: "{{ ansible_password }}"
 force_basic_auth: yes
 body: "{{ watch }}"
 run_once: true

The set_fact loads the contents of the file into a variable (otherwise, you’d have to list the entire json in uri’s body section).

By using delegate_to, the PUT will be run on the local machine.  Coupled with run_once, it contacts the cluster and sets the config one time.

On my desktop mac, it originally gave me errors about httplib2, which I had to install:

sudo pip install httplib2

As mentioned before, the variable that contains the basic auth password comes from an ansible vault.

One final – yet important – gotcha.  Ansible seemingly wants to process all data as templates.  Unfortunately, watcher uses the same variable syntax as Ansible, so your watcher definition may include lines with watcher variables:

{{ ctx.payload.status }}

Ansible will think these are its own variables, and you’ll get errors when running the playbook:

fatal: [localhost] => One or more undefined variables: 'ctx' is undefined

The solution is to tell Ansible to ignore that section:

{% raw %}{{ ctx.payload.status }}{% endraw %}

This can be a little tedious, but I didn’t find a way to prevent Ansible from interpreting the string.

Elasticsearch disk space calculations

Each node provides storage capacity to your cluster.  Elasticsearch will stop indexing if the nodes start to fill up.  This is controlled with the cluster.routing.allocation.disk.watermark.low parameter.  By default, no new shards will be allocated when a machine goes above 85% disk space.

Clearly you must manage the disk space when all of your nodes are running, but what happens when a node fails?

Let’s look at a three-node cluster, setup with three shards and one replica, so data is evenly spread out across the cluster:

Untitled

If each node has 1TB of disk space for data, they would hit the per-node 85% limit at 850GB.  If one node failed, the 6 total shards would need to be distributed across two nodes.   In our example, if we lost node #1, the primary for shard 1 and the replica for shard 3 would be lost.  The replica for shard 1 that is on node #2 would be promoted to primary, but we would then have no replica for either shards 1 or 3.  Elasticsearch would try to rebuild the replicas on the remaining hosts:

Untitled

This is good on paper, except each of the remaining two nodes would need to absorb up to 425GB each.  The remaining nodes would be full, and no new shards would be created.

To plan for a node outage, you need to have enough free disk space on each node to reallocate the primary and replica data from the dead node.

This formula will yield the maximum amount of data a node can safely hold:

(disk per node * .85) * (node count - 1 / node count)

In my example, we would get:

( 1TB * .85 ) * ( 2 / 3 ) = 566GB

If your three nodes contained 566GB of data each and one node failed, 283GB of data would be rebuilt on the remaining two nodes, putting them at 849GB used space.  This is just below the 85% limit of 850GB.

I would pad the number a little, and limit the disk space used to 550GB for each node, with 1.65TB data total across the 3-node cluster.  This number plays a part in your data retention policy and cluster sizing strategies.

If 1.65TB is too low, you either need to add more space to each node, or add more nodes to the cluster.  If you added a 4th similarly-sized node, you’d get

( 1TB * .85 ) * ( 3 /4 ) = 637GB

which would allow 2.5GB of storage across the entire cluster.

The formula shown is based on one replica shard.  If you had configured your cluster with more replicas (to survive the outage of more than one node), note that the formula is really:

(space per node * .85) * (node count - replica count / node count)

If we had two replicas in our example, we’d get:

( 1TB * .85 ) * ( 1 / 3 ) = 283GB

So you would only allow 283GB of data per node if you wanted to survive a 2-node outage in a 3-node cluster.

 

Adventures in ELK pipeline debugging

I recently brought up some machines in a new colo and needed to ship their logs to a local logstash and then on to a centralized elasticsearch server at another site.

The first batch of machines seemed to come up OK.  When I brought the second set up a few weeks later, I started seeing errors in logstash:

Lumberjack input: the pipeline is blocked, temporary refusing new connection.

Lumberjack input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.

The “refusing new connection” part was easily confirmed in the logstash-forwarder logs:

Connecting to [10.1.2.3]:5544 (10.1.2.3)
Connected to 10.1.2.3
Read error looking for ack: EOF

These errors seem to be common understood to mean that your ELK infrastructure is congested.  Perhaps logstash is dead/slow (no); perhaps elasticsearch was dead/slow (no).

Elasticsearch looked healthy, and there were no signs of similar problem in other colos.  We’d see this behavior for most of an hour, and then a small batch of logs would make it through, which just added to the mystery.

Upon further investigation, we found that even the first batch of machines was suffering from this problem.  That they were able to process a backlog of files and keep up to date is surprising.

Turning up debugging in logstash yielded one additional message, but it was basically a repeat of information already seen.

The next step was to see what was going on when logstash talked to elasticsearch.  I grabbed a few minute’s info with tcpdump:

tcpdump -ni eth0 -w tcpdump.out tcp port 9200

This showed three interesting datapoints:

  1. all interactions between logstash and elasticsearch resulted in “200” responses (OK)
  2. the small successful batches were being sent exactly 40 seconds apart
  3. elasticsearch was responding quickly to the indexing requests that made it through.

A google search for issues around “40 seconds” turned up nothing.

To confirm the 40-second problem, I ran another tcpdump but restarted logstash at the same time.  This showed it to work for a while (~2 minutes), and then slow down to 40-second intervals.

There didn’t seem to be much more to learn from tcpdump.  That led to java debugging steps, starting with a thread dump:

kill -3 <pid>

This added a thread dump to logstash’s logfile.  Looking through it, I found about 1/3 of the threads were “BLOCKED”:


java.lang.Thread.State: BLOCKED (on object monitor)
at java.net.InetAddress.getLocalHost(InetAddress.java:1486)
- waiting to lock <0x0000000538a5dfe0> (a java.lang.Object)
at org.jruby.ext.socket.SocketUtils.gethostname(SocketUtils.java:77)
at org.jruby.ext.socket.RubySocket.gethostname(RubySocket.java:257)
at org.jruby.ext.socket.RubySocket$INVOKER$s$0$0$gethostname.call(RubySocket$INVOKER$s$0$0$gethostname.gen)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134) at rubyjit.LogStash::Filters::Metrics$$flush_486027b1b0fe337b5753334309092d65d96630e51028566121.__file__(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-metrics-1.0.0/lib/logstash/filters/metrics.rb:186)
at rubyjit.LogStash::Filters::Metrics$$flush_486027b1b0fe337b5753334309092d65d96630e51028566121.__file__(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-metrics-1.0.0/lib/logstash/filters/metrics.rb)

The great thing here is the stacktrace, which showed that the block was coming from the metrics{} filter, on this line:

event["message"] = Socket.gethostname

This led to the discovery that these logstash instances – in only this colo – didn’t have access to a DNS server.  This is seemingly required so that the metric event can be tagged with the hostname from which it originates.

Adding a line to /etc/hosts fixed the problem:

127.0.0.1 myhost.mydomain.com

Hopefully nobody else will hit this exact problem, but perhaps the debugging techniques described above will be helpful.

Introduction to Elasticsearch Tokenization and Analysis

Elasticsearch is a text engine.  This is usually good if you have text to index, but can cause problems with other types of input (log files).  One of the more confusing elements of elasticsearch is the idea of tokenization and how fields are analyzed.

Tokens

In a text engine, you might want to take a string and search for each “word”.  The rules that are used to convert a string into words are defined in a tokenizer.   A simple string:

The quick brown fox

can easily be processed into a series of tokens:

[“the”, “quick”, “brown”, “fox”]

But what about punctuation?

Half-blood prince

or

/var/log/messages

The default tokenizer in elasticsearch will split those up:

[“half”, “blood”, “prince”]

[“var”, “log”, “messages”]

Unfortunately, this means that searching for “half-blood price” might also find you an article about a royal prince who fell half way to the floor while donating blood.

As of this writing, there are 12 built-in tokenizers.

You can test some input text against a tokenizer on the command line:

curl -XGET 'localhost:9200/_analyze?analyzer=standard&pretty' -d '/var/log/messages'

Analyzers

An analyzer lets you combine a tokenizer with some other rules to determine how the text will be indexed.  This is not something I’ve had to do, so I don’t have examples or caveats yet.

You can test the analyzer rules on the command line as well:

curl -XGET 'localhost:9200/_analyze?tokenizer=keyword&filters=lowercase' -d 'The quick brown fox'

Mappings

When you define the mapping for your index, you can control how each field is analyzed.  First, you can specify *if* the field is even to be analyzed or indexed:

"myField": {
    "index": "not_analyzed"
}

By using “not_analyzed”, the value of the field will not be tokenized in any way and will only be available as a raw string.  Since this is very useful for logs, the default template in logstash uses this to create the “.raw” fields (e.g. myField.raw).

You can also specify “no”, which will prevent the field from being indexed at all.

If you would like to use a different analyzer for your field, you can specify that:

"myField": {
    "analyzer": "spanish"
}

 

Filebeat

It was a nice run, but logstash-forwarder is dead.  In its place comes filebeat, a lightweight (still Java-free and written in Go) log file shipper that is actually supported by Elastic.

If you’re coming from logstash-forwarder, Elastic provides a migration guide.

We hope to migrate our own stuff to filebeat soon, which will certainly yield more postings.  Stay tuned!

 

Testing your logstash configuration

Overview

As your logstash configuration grows (mine is over 3,000 lines in 40+ files right now), you’ll want a way to make sure you don’t break anything with a new release (and that Elastic doesn’t, either!).

You can run logstash with the ‘–configtest’ option, but that’s only checking for syntax errors.

Logstash uses the rspec harness, so I wanted to start there.  All the doc I could find on the web was old and wrong (even postings from elastic).  Thanks to the great community in the #logstash IRC channel, I was able to get it working.

Goals

I had three goals when I started:

  • to re-use my production configuration files, which are already logically arranged (“apache”, “yum”, etc).  I did not want to repeat the configuration in the test files.
  • to test my desired output.  OK, this probably goes without saying.
  • to use logstash 1.4, which is [still] in use by my main cluster.  I’ve heard that things change some in 1.5, as you might expect.

Setup

You can either run the test scripts on a production machine (using a short-lived second instance of logstash) or make a VM that would have logstash and your logstash filter{} configurations installed.

Config

As part of your installation, all your configs should be in /etc/logstash/conf.d.  We’ll use a small config example, test.conf:

filter {
    if [type] == "test" {
        grok { 
            match => [ "message", "%{TIMESTAMP_ISO8601:timestamp} %{WORD:word1} %{INT:int1:int} %{NUMBER:[inner][float1]:float}" ]
            tag_on_failure => [ "_grokparsefailure_test" ]
        }
    } 
}

This grok pattern would match a string like this:

2015-21-01 12:01:02.003 UTC Hello 42 3.14159

and create four fields of the appropriate type (string, integer, float).

On my logstash machine, the built-in rspec files are in /opt/logstash/spec.  I made a parallel directory as /opt/logstash/my-spec that contained this script, test.rb:

# encoding: utf-8
require "test_utils"

file = "/etc/logstash/conf.d/test.conf"
@@configuration = String.new
@@configuration << File.read(file)

describe "Test event" do
  extend LogStash::RSpec

  config(@@configuration)

  message = %(2015-21-01 12:01:02.003 UTC Hello 42 3.14159)

  sample("message" => message, "type" => "test") do
    insist { subject["type"] } == "test"
    insist { subject["timestamp"] } == "2015-21-01 12:01:02.003 UTC"
    insist { subject["word1"] } == "Hello"
    insist { subject["int1"] } == 42
    insist { subject["inner"]["float1"] } == 3.14159
  end
end

Running

cd /opt/logstash
bin/logstash rspec my-spec/test.rb

and you should see

Finished in 0.361 seconds
1 example, 0 failures

Conclusion

In my real-world config, I have a series of filters in one file that do a lot of processing on the events.  In the simple example above, you can also imagine wanting to run the date{} filter on the `timestamp` column to update @timestamp.  As you add more complexity, update the test cases to match.  So, if your config included date{}, you would add this:

insist { subject["@timestamp"] } == Time.iso8601("2015-12-01T12:01:03.003Z").utc

Please note that this example uses the rspec2 “insist” construct.  I’ve heard that rspec3 uses “expect”.  There may be other syntax issues when using logstash2.

SNMP traps with logstash

The Basics

SNMP traps are generally easy to receive and process with logstash.  The snmptrap{} input sets up a listener, which processes each trap and replaces the OIDs with the string representation found in the given mibs.  If the OID can’t be found, logstash will make a new field, using the OID value as the field name, e.g.

"1.3.6.1.4.1.1234.1.2.3.4": "trap value"

(Note that this is currently broken if you use Elasticsearch 2.0).

Forwarding

Probably the biggest issues with most traps is that they are sent to port 162, which is a low-numbered “system” port.  For logstash to listen on that port, it must be run as root, which is not recommended.

The easiest workaround for this is to forward port 162 to a higher-numbered port to which logstash can connect.  iptables is the typical tool to perform the forwarding:

/sbin/iptables -A PREROUTING -t nat -i eth0 -p udp --dport 162 -j REDIRECT --to-port 5000

where ‘5000’ is the port on which logstash is listening.

SNMP Sequences

Some SNMP traps come in with a “sequence number”, which allows the receiver to know if all traps have been received.  In the ones we’ve seen, the sequence is appended to each OID, e.g.

"1.3.6.1.4.1.1234.1.2.3.4.90210": "trap value"

where “90210” is the sequence number.

This seems like a handy feature, but it doesn’t appear to be supported by logstash (or perhaps the underlying SNMP library that is uses).  With the basic snmptrap config, logstash is unable to apply the mib definition and remove the sequence number, so you end up with a new field for each trap value.  That’s not good for you or for elasticsearch/kibana.

Since traps aren’t just simple plain text, you can’t use a “tcp” listener, apply your own filter to remove the sequence, and feed the result back into logstash’s “snmptrap” mechanism.  Without modifying the snmptrap input plugin, you have to fix the problem before it hits logstash.

I was a fan of logstash plugins (and have written a few), but logstash 1.5 requires everything to be done as ruby gems, which has been a painful path.  As such, I’m doing more outside of logstash, like this recommendation.

snmptrapd

We’re now running snmptrapd on our logstash machines.  They listen for traps on port 162 and write them to a regular log file that can then be read by logstash.

Basic config

Update /etc/snmp/snmptrapd.conf to include:

disableAuthorization yes

Put your mib definitions in/usr/share/snmp/mibs.

Trap formatting

To make the traps easier to process by logstash, I format the output as json.  This is done with OPTIONS set in /etc/sysconfig/snmptrapd:

OPTIONS="-A -Lf /var/log/snmptrap -p /var/run/snmptrapd.pid -m ALL -F '{ \"type\": \"snmptrap\", \"timestamp\": \"%04y-%02m-%02l %02h:%02j:%02k\", \"host_ip\":\"%a\", \"trapEnterprise\": \"%N\", \"trapSubtype\": \"%q\", \"trapType\": %w, \"trapVariables\": \"%v\" }\n' "

The flags used are:

  • -A – append to the log file rather than truncating it
  • -Lf – log to a file
  • -m ALL – use all the mibs it can find
  • -F – use this printf-style string for formatting

Then, in logstash, use the json filter:

filter {
    json {
        source => "message"
    }
}

I use a ruby filter to make the separate fields and cast them to the correct type.

Don’t forget to setup a log file rotation on your new /var/log/snmptrap file and setup a process monitor for snmptrapd.

 

Duplicated elasticsearch documents

Intro

The first thing to notice is that the documents probably have different _id values, so the problem then becomes, “who is inserting duplicates??”.

If you’re running logstash, some things to look at include:

  • duplicate input{} stanzas
  • duplicate output{} stanzas
  • two logstash processes running
  • bad file glob patterns
  • bad broker configuration

Duplicate Stanzas

Most people aren’t silly enough to deliberately create duplicate input or output stanzas, but there are still easy ways for them to occur:

  • a logstash config file you’ve forgotten (00-mytest.conf)
  • a backup file (00-input.conf.bak)

Remember that logstash will read in all the files it finds in your configuration directory!

Multiple Processes

Sometimes your shutdown script may not work, leaving you with two copies of your shipper running.  Check it with ‘ps’ and kill off the older one.

File Globs

If your file glob pattern is fairly open (e.g. “*”), you might be picking up files that have been rotated (“foo.log” and “foo.log.00”).

Logstash-forwarder sets a ‘file’ field that you can check in this case.

If you’ve enabled _timestamp in elasticsearch, it will show you when each of the duplicates was indexed, which might give you a clue.

Brokers

As for brokers, if you have multiple logstash indexers trying to read from the same broker without some locking mechanism, it might cause problems.

 

Finding the lag in your ELK cluster

By default, logstash sets @timestamp to the time when it processes the log.  Most people (correctly), use the date filter to set this to the event’s log.

Before you run the date{} filter, tuck away the value of @timestamp with a little bit of ruby:

ruby {
  code => "
    begin
      event['logstash_ts'] = event['@timestamp']
      rescue Exception => e
        event['logstash_ruby_exception'] = '[logstash_ts]: ' + e.message
    end
  "
}

This will create a new field called ‘logstash_ts’ that contains the timestamp when logstash received the event.

Now it’s safe to run your date{} filter to change @timestamp to the event’s timestamp.

After the date{} filter, compute the difference between the event’s time and the received time.  That’s the lag in getting the event processed by logstash:

ruby {
  code => "
    begin
      event['logstash_lag'] = ( ( event['logstash_ts'] - event['@timestamp'] ) ).to_f
    rescue Exception => e
      event['logstash_ruby_exception'] = '[logstash_lag]: ' + e.message
    end
  "
}

If this lag is high, the problem could be:

  • a delay in shipping the logs.  Check the health of your shipper.
  • a delay in processing the logs.  Is your logstash machine powerful enough?  Is your config “economical”?

There’s one more timestamp available, which is when Elasticsearch actually indexes the document.   You can enable the field in your mapping:

curl -XPUT localhost:9200/_template/timestamp -d '
 {
 "template" : "*",
 "order" : 10000,
 "mappings" : {
  "_default_" : {
   "_timestamp" : { "enabled" : true }
  }
 }
 }
 '

Note that this is deprecated in elasticsearch 2.0 and there’s no viable alternative!

Any new indexes created after that template is applied will have _timestamp stored for each document.

If the difference between our logstash_ts field and elasticsearch’s _timestamp is high, then:

  • you have connectivity problems between logstash and elasticsearch
  • elasticsearch is busy doing something else (check garbage collection!)
  • (maybe) you have a high index refresh_interval.

I haven’t found an easy way to compute the difference between these last two timestamps.  Scripted fields should work, but they’re disabled and/or poorly documented.