Elasticsearch configuration options

This is an attempt at a complete listing of elasticsearch config variables since they’re located all over the elastic.co website.

The list is not complete, and will start to “rot” as soon as it’s published, but… If you know of some variables that aren’t listed, please let me know.

Note that static settings must be set in the config file on every machine in the cluster.



Name Type Notes Doc
bootstrap.mlockall Static Link


Name Type Notes Doc
cloud.aws.access_key Unknown Link
cloud.aws.ec2 Unknown Link
cloud.aws.protocol Unknown Link
cloud.aws.proxy Unknown Link
cloud.aws.region Unknown Link
cloud.aws.s3 Unknown Link
cloud.aws.secret_key Unknown Link
cloud.aws.signer Unknown Link


Name Type Notes Doc
cluster.blocks.read_only Dynamic Link
cluster.info.update.interval Dynamic Link
cluster.name Unknown Link
cluster.routing.allocation.allow_rebalance Dynamic Link
cluster.routing.allocation.awareness.attributes Dynamic Link
cluster.routing.allocation.awareness.force.zone.values Dynamic Link
cluster.routing.allocation.balance.shard Unknown Link
cluster.routing.allocation.balance.index Unknown Link
cluster.routing.allocation.balance.threshold Unknown Link
cluster.routing.allocation.cluster_concurrent_rebalance Dynamic Link
cluster.routing.allocation.disk.include_relocations Dynamic Link
cluster.routing.allocation.disk.threshold_enabled Dynamic Link
cluster.routing.allocation.disk.watermark.low Dynamic Link
cluster.routing.allocation.disk.watermark.high Dynamic Link
cluster.routing.allocation.enable Dynamic Link
cluster.routing.allocation.exclude Dynamic Link
cluster.routing.allocation.include Dynamic Link
cluster.routing.allocation.node_concurrent_recoveries Dynamic Link
cluster.routing.allocation.node_initial_primaries_recoveries Dynamic Link
cluster.routing.allocation.require Dynamic Link
cluster.routing.allocation.same_shard.host Dynamic Link
cluster.routing.allocation.total_shards_per_node Dynamic Link
cluster.routing.rebalance.enable Dynamic Link


ec2 discovery can also have: groups, host_type, availability_zones, any_group, ping_timeout, and node_cache_time. Use this inside discovery.ec2, e.g. discover.ec2.groups.

Name Type Notes Doc
discovery.type Dynamic Link
discovery.zen.minimum_master_nodes Dynamic Link
discovery.zen.ping.multicast.enabled Unknown Removed in ES 2.2 Link
discovery.zen.ping.unicast.hosts Unknown Link


Name Type Notes Doc
gateway.expected_nodes Unknw Link
gateway.expected_master_nodes Static Link
gateway.expected_data_nodes Static Link
gateway.recover_after_time Static Link
gateway.recover_after_nodes Static Link
gateway.recover_after_master_nodes Static Link
gateway.recover_after_data_nodes Static Link


Name Type Notes Doc
http.port Static Link
http.publish_port Static Link
http.bind_host Static Link
http.publish_host Static Link
http.host Static Link
http.max_content_length Static Link
http.max_initial_line_length Static Link
http.max_header_size Static Link
http.compression Static Link
http.compression_level Static Link
http.cors.enabled Static Link
http.cors.allow-origin Static Link
http.cors.max-age Static Link
http.cors.allow-methods Static Link
http.cors.allow-headers Static Link
http.cors.allow-credentials Static Link
http.detailed_errors.enabled Static Link
http.pipelining Static Link
http.pipelining.max_events Static Link


Name Type Notes Doc
index.analysis.analyzer Static Link
index.analysis.filter Static Link
index.analysis.tokenizer Static Link
index.auto_expand_replicas Dynamic Link
index.blocks.metadata Dynamic Link
index.blocks.read Dynamic Link
index.blocks.read_only Dynamic Link
index.blocks.write Dynamic Link
index.codec Static Link
index.gateway.local.sync Unknown Renamed to index.translog.sync_interval in ES 2.0 Link
index.max_result_window Dynamic Link
index.merge.policy.calibrate_size_by_deletes Unknown Removed in ES 2.0 Link
index.merge.policy.expunge_deletes_allowed Unknown Removed in ES 2.0 Link
index.merge.policy.max_merge_docs Unknown Removed in ES 2.0 Link
index.merge.policy.max_merge_size Unknown Removed in ES 2.0 Link
index.merge.policy.merge_factor Unknown Removed in ES 2.0 Link
index.merge.policy.min_merge_docs Unknown Removed in ES 2.0 Link
index.merge.policy.min_merge_size Unknown Removed in ES 2.0 Link
index.merge.policy.type Unknown Removed in ES 2.0 Link
index.merge.scheduler.max_thread_count Dynamic Link
index.number_of_replicas Dynamic Link
index.number_of_shards Static Link
index.recovery.initial_shards Dynamic Link
index.refresh_interval Dynamic Requires units in ES 2.0 Link
index.routing.allocation.exclude Dynamic Link
index.routing.allocation.include Dynamic Link
index.routing.allocation.require Dynamic Link
index.routing.allocation.total_shards_per_node Dynamic Link
index.search.slowlog.threshold Dynamic Link
index.shard.check_on_startup Static Link
index.similarity.default.type Static Link
index.store.throttle.type Unknown Removed in Es 2.0 Link
index.store.throttle.max_bytes_per_sec Unknown Removed in Es 2.0 Link
index.store.type Static memory and ram types removed in ES 2.0 Link
index.ttl.disable_purge Dynamic Link
index.translog.durability Dynamic Link
index.translog.fs.type Dynamic Link
index.translog.flush_threshold_ops Dynamic Link
index.translog.flush_threshold_period Dynamic Link
index.translog.flush_threshold_size Dynamic Link
index.translog.interval Dynamic Link
index.translog.sync_interval Static Link
index.unassigned.node_left.delayed_timeout Dynamic Link



Name Type Notes Doc
indices.analysis.hunspell.dictionary.location Unknown Removed in ES 2.0 Link
indices.recovery.concurrent_streams Dynamic Link
indices.recovery.concurrent_small_file_streams Dynamic Link
indices.store.throttle.type Unknown Removed in ES 2.0 Link
indices.store.throttle.max_bytes_per_sec Unknown Removed in ES 2.0 Link


Name Type Notes Doc
logger.indexes.recovery Dynamic Link
logger.transport.tracer Dynamic Link


Name Type Notes Doc
network.bind_host Unknown Link
network.host Dynamic See special values Link
network.publish_host Unknown Link
network.tcp.no_delay Unknown Link
network.tcp.keep_alive Unknown Link
network.tcp.reuse_address Unknown Link
network.tcp.send_buffer_size Unknown Link
network.tcp.receive_buffer_size Unknown Link


Name Type Notes Doc
node.data Unknown Link
node.enable_custom_paths Unknown Removed in ES 2.0 Link
node.master Unknown Link
node.max_local_storage_nodes Static Link
node.name Unknown Link


Name Type Notes Doc
path.conf Static Link
path.data Static Link
path.home Static Link
path.logs Static Link
path.plugins Static Link
path.repo Static Link
path.scripts Static Link
path.shared_data Unknown Link


Name Type Notes Doc
plugin.mandatory Static Link


Name Type Notes Doc
resource.reload.enabled Unknown
resource.reload.interval Unknown Link
resource.reload.interval.low Unknown
resource.reload.interval.medium Unknown
resource.reload.interval.high Unknown


Name Type Notes Doc
repositories.url.allowed_urls Unknown Link


Name Type Notes Doc
script.auto_reload_enabled Static Link
script.default_lang Static Link
script.disable_dynamic Unknown Removed in ES 2.0
script.file Static Link
script.index Static Link
script.inline Static Link
script.update Static Link
script.mapping Static Link
script.engine.expression Static Link
script.engine.groovy Static Link
script.engine.javascript Static Link
script.engine.mustache Static Link
script.engine.python Static Link

Thread Pool

There are several thread pools. Elastic lists the “important” ones as including: generic, index, search, suggest, get, bulk, percolate, snapshot, warmer, refresh, listener. Some settings are documented, and are listed below.

You can also control the number of processors for a thread pool, which is briefly documented here.

Name Type Notes Doc
threadpool.generic.keep_alive Dynamic Link
threadpool.index.queue_size Dynamic Link
threadpool.index.size Dynamic Link


Transport allows you to bing to multiple ports on different interfaces. See the transport profiles doc for more info.

Name Type Notes Doc
transport.bind_host Unknown Link
transport.host Unknown Link
transport.ping_schedule Unknown Link
transport.publish_host Unknown Link
transport.publish_port Unknown Link
transport.tcp.compress Unknown Link
transport.tcp.connect_time Unknown Link
transport.tcp.port Unknown Link
transport.tracer.exclude Dynamic Link
transport.tracer.include Dynamic Link


There are a lot of options for tribes that vary based on the tribe name. Some info is presented here.


Name Type Notes Doc
tribe.blocks.metadata Unknown Link
tribe.blocks.metadata.indices Unknown Link
tribe.blocks.write Unknown Link
tribe.blocks.write.indices Unknown Link
tribe.t1.cluster.name Unknown Link
Name Type Notes Doc
watcher.enabled Unknown Renamed in ES 2.0 Link
watcher.interval Unknown Renamed in ES 2.0 Link
watcher.interval.low Unknown Renamed in ES 2.0 Link
watcher.interval.medium Unknown Renamed in ES 2.0 Link
watcher.interval.high Unknown Renamed in ES 2.0 Link

Managing Elastic watches with Ansible

If you’re an Ansible convert, you want every part of your deployment to be managed by the tasks in your roles.  I recently wanted to manage my Elastic Watcher config with Ansible.  It was enough of a struggle (“learning process”) that I wanted to document it here.

While I was developing my first few watches, I made stand-alone shell scripts that could be run to create/update them.  This worked fine, and even handled the basic auth configuration in front of the master nodes.  Basically it was:

curl -u username -XPUT 'http://hostname/_watcher/watch/watchname' -d '{ ... }'

It would prompt for the user’s password and send the information along.

Ansible can run local commands, but doesn’t like interactive commands.  As such, the password prompt would block the task from completing.  Looking around for other solutions led me to the ‘uri’ module.  By default, it won’t prompt for the password either, but you can use the prompts system.  I chose to use the Ansible vault to store the password to avoid external sharing/management of the password.

The final config looks like this:

- set_fact:
 watch: "{{ lookup('file','watcher.json') }}"

- name: Install watcher for cluster health
 url: "http://hostname//_watcher/watch/watchname"
 method: PUT
 user: "ansible"
 password: "{{ ansible_password }}"
 force_basic_auth: yes
 body: "{{ watch }}"
 run_once: true

The set_fact loads the contents of the file into a variable (otherwise, you’d have to list the entire json in uri’s body section).

By using delegate_to, the PUT will be run on the local machine.  Coupled with run_once, it contacts the cluster and sets the config one time.

On my desktop mac, it originally gave me errors about httplib2, which I had to install:

sudo pip install httplib2

As mentioned before, the variable that contains the basic auth password comes from an ansible vault.

One final – yet important – gotcha.  Ansible seemingly wants to process all data as templates.  Unfortunately, watcher uses the same variable syntax as Ansible, so your watcher definition may include lines with watcher variables:

{{ ctx.payload.status }}

Ansible will think these are its own variables, and you’ll get errors when running the playbook:

fatal: [localhost] => One or more undefined variables: 'ctx' is undefined

The solution is to tell Ansible to ignore that section:

{% raw %}{{ ctx.payload.status }}{% endraw %}

This can be a little tedious, but I didn’t find a way to prevent Ansible from interpreting the string.

Elasticsearch disk space calculations

Each node provides storage capacity to your cluster.  Elasticsearch will stop indexing if the nodes start to fill up.  This is controlled with the cluster.routing.allocation.disk.watermark.low parameter.  By default, no new shards will be allocated when a machine goes above 85% disk space.

Clearly you must manage the disk space when all of your nodes are running, but what happens when a node fails?

Let’s look at a three-node cluster, setup with three shards and one replica, so data is evenly spread out across the cluster:


If each node has 1TB of disk space for data, they would hit the per-node 85% limit at 850GB.  If one node failed, the 6 total shards would need to be distributed across two nodes.   In our example, if we lost node #1, the primary for shard 1 and the replica for shard 3 would be lost.  The replica for shard 1 that is on node #2 would be promoted to primary, but we would then have no replica for either shards 1 or 3.  Elasticsearch would try to rebuild the replicas on the remaining hosts:


This is good on paper, except each of the remaining two nodes would need to absorb up to 425GB each.  The remaining nodes would be full, and no new shards would be created.

To plan for a node outage, you need to have enough free disk space on each node to reallocate the primary and replica data from the dead node.

This formula will yield the maximum amount of data a node can safely hold:

(disk per node * .85) * (node count - 1 / node count)

In my example, we would get:

( 1TB * .85 ) * ( 2 / 3 ) = 566GB

If your three nodes contained 566GB of data each and one node failed, 283GB of data would be rebuilt on the remaining two nodes, putting them at 849GB used space.  This is just below the 85% limit of 850GB.

I would pad the number a little, and limit the disk space used to 550GB for each node, with 1.65TB data total across the 3-node cluster.  This number plays a part in your data retention policy and cluster sizing strategies.

If 1.65TB is too low, you either need to add more space to each node, or add more nodes to the cluster.  If you added a 4th similarly-sized node, you’d get

( 1TB * .85 ) * ( 3 /4 ) = 637GB

which would allow 2.5GB of storage across the entire cluster.

The formula shown is based on one replica shard.  If you had configured your cluster with more replicas (to survive the outage of more than one node), note that the formula is really:

(space per node * .85) * ((node count - replica count) / node count)

If we had two replicas in our example, we’d get:

( 1TB * .85 ) * ( 1 / 3 ) = 283GB

So you would only allow 283GB of data per node if you wanted to survive a 2-node outage in a 3-node cluster.

Adventures in ELK pipeline debugging

I recently brought up some machines in a new colo and needed to ship their logs to a local logstash and then on to a centralized elasticsearch server at another site.

The first batch of machines seemed to come up OK.  When I brought the second set up a few weeks later, I started seeing errors in logstash:

Lumberjack input: the pipeline is blocked, temporary refusing new connection.

Lumberjack input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.

The “refusing new connection” part was easily confirmed in the logstash-forwarder logs:

Connecting to []:5544 (
Connected to
Read error looking for ack: EOF

These errors seem to be common understood to mean that your ELK infrastructure is congested.  Perhaps logstash is dead/slow (no); perhaps elasticsearch was dead/slow (no).

Elasticsearch looked healthy, and there were no signs of similar problem in other colos.  We’d see this behavior for most of an hour, and then a small batch of logs would make it through, which just added to the mystery.

Upon further investigation, we found that even the first batch of machines was suffering from this problem.  That they were able to process a backlog of files and keep up to date is surprising.

Turning up debugging in logstash yielded one additional message, but it was basically a repeat of information already seen.

The next step was to see what was going on when logstash talked to elasticsearch.  I grabbed a few minute’s info with tcpdump:

tcpdump -ni eth0 -w tcpdump.out tcp port 9200

This showed three interesting datapoints:

  1. all interactions between logstash and elasticsearch resulted in “200” responses (OK)
  2. the small successful batches were being sent exactly 40 seconds apart
  3. elasticsearch was responding quickly to the indexing requests that made it through.

A google search for issues around “40 seconds” turned up nothing.

To confirm the 40-second problem, I ran another tcpdump but restarted logstash at the same time.  This showed it to work for a while (~2 minutes), and then slow down to 40-second intervals.

There didn’t seem to be much more to learn from tcpdump.  That led to java debugging steps, starting with a thread dump:

kill -3 <pid>

This added a thread dump to logstash’s logfile.  Looking through it, I found about 1/3 of the threads were “BLOCKED”:

java.lang.Thread.State: BLOCKED (on object monitor)
at java.net.InetAddress.getLocalHost(InetAddress.java:1486)
- waiting to lock <0x0000000538a5dfe0> (a java.lang.Object)
at org.jruby.ext.socket.SocketUtils.gethostname(SocketUtils.java:77)
at org.jruby.ext.socket.RubySocket.gethostname(RubySocket.java:257)
at org.jruby.ext.socket.RubySocket$INVOKER$s$0$0$gethostname.call(RubySocket$INVOKER$s$0$0$gethostname.gen)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134) at rubyjit.LogStash::Filters::Metrics$$flush_486027b1b0fe337b5753334309092d65d96630e51028566121.__file__(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-metrics-1.0.0/lib/logstash/filters/metrics.rb:186)
at rubyjit.LogStash::Filters::Metrics$$flush_486027b1b0fe337b5753334309092d65d96630e51028566121.__file__(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-metrics-1.0.0/lib/logstash/filters/metrics.rb)

The great thing here is the stacktrace, which showed that the block was coming from the metrics{} filter, on this line:

event["message"] = Socket.gethostname

This led to the discovery that these logstash instances – in only this colo – didn’t have access to a DNS server.  This is seemingly required so that the metric event can be tagged with the hostname from which it originates.

Adding a line to /etc/hosts fixed the problem: myhost.mydomain.com

Hopefully nobody else will hit this exact problem, but perhaps the debugging techniques described above will be helpful.

Introduction to Elasticsearch Tokenization and Analysis

Elasticsearch is a text engine.  This is usually good if you have text to index, but can cause problems with other types of input (log files).  One of the more confusing elements of elasticsearch is the idea of tokenization and how fields are analyzed.


In a text engine, you might want to take a string and search for each “word”.  The rules that are used to convert a string into words are defined in a tokenizer.   A simple string:

The quick brown fox

can easily be processed into a series of tokens:

[“the”, “quick”, “brown”, “fox”]

But what about punctuation?

Half-blood prince



The default tokenizer in elasticsearch will split those up:

[“half”, “blood”, “prince”]

[“var”, “log”, “messages”]

Unfortunately, this means that searching for “half-blood price” might also find you an article about a royal prince who fell half way to the floor while donating blood.

As of this writing, there are 12 built-in tokenizers.

You can test some input text against a tokenizer on the command line:

curl -XGET 'localhost:9200/_analyze?analyzer=standard&pretty' -d '/var/log/messages'


An analyzer lets you combine a tokenizer with some other rules to determine how the text will be indexed.  This is not something I’ve had to do, so I don’t have examples or caveats yet.

You can test the analyzer rules on the command line as well:

curl -XGET 'localhost:9200/_analyze?tokenizer=keyword&filters=lowercase' -d 'The quick brown fox'


When you define the mapping for your index, you can control how each field is analyzed.  First, you can specify *if* the field is even to be analyzed or indexed:

"myField": {
    "index": "not_analyzed"

By using “not_analyzed”, the value of the field will not be tokenized in any way and will only be available as a raw string.  Since this is very useful for logs, the default template in logstash uses this to create the “.raw” fields (e.g. myField.raw).

You can also specify “no”, which will prevent the field from being indexed at all.

If you would like to use a different analyzer for your field, you can specify that:

"myField": {
    "analyzer": "spanish"



It was a nice run, but logstash-forwarder is dead.  In its place comes filebeat, a lightweight (still Java-free and written in Go) log file shipper that is actually supported by Elastic.

If you’re coming from logstash-forwarder, Elastic provides a migration guide.

We hope to migrate our own stuff to filebeat soon, which will certainly yield more postings.  Stay tuned!


Testing your logstash configuration