There seem to be a lot of old ELK guides on the internet. \u00a0It’s time to make a new one that can begin its own eventual decay.<\/p>\n
To help minimize the aging process, I’m not going to cover how to install specific packages on specific platforms, but rather discuss the choice of tools and configurations that are available.<\/p>\n
You have to get your log files off the remote machines and eventually into Elasticsearch. \u00a0It’s the shipper’s job to, um, ship the logs to the next step.<\/p>\n
There are a few shippers, some of which are outlined here.<\/p>\n
tl;dr: use filebeat if you're only moving logs around.<\/pre>\nlogstash<\/h2>\n
You can use the full logstash build as your shipper. \u00a0There are almost no reasons to do this. \u00a0Being JVM-based, it’s big and takes memory. \u00a0It also has more features than you’ll typically need for a shipper.<\/p>\n
The only excuse to run logstash as a remote shipper is if you have a ton of logs and need to apply some business logic about which ones to ship. \u00a0For example, maybe DEBUG logging is enabled in your production environment (?); you could use a full logstash to only ship the more important levels for processing.<\/p>\n
<\/h2>\n
logstash-forwarder<\/h2>\n
NOTE: logstash-forwarder is dead. \u00a0See filebeat, below.<\/strong><\/p><\/blockquote>\n
This is the right choice. \u00a0It’s a light-weight program that does nothing other than read log files and send them to logstash. \u00a0Traffic is encrypted with SSL, so certs are required.<\/p>\n
logstash-forwarder speaks the “lumberjack” protocol with logstash.<\/p>\n
filebeat<\/h2>\n
Filebeat<\/a> is the replacement for logstash-forwarder. \u00a0It’s also lightweight, gives you the option of not using encryption, and they’re planning to add some nice client-side features (multiline and a basic ‘grep’).<\/p>\n
Filebeat requires logstash 1.5+.<\/p>\n
beaver<\/h2>\n
If you need a broker (see below), then beaver is a lightweight tool that, not being encrypted, can talk to redis.<\/p>\n
Broker<\/h1>\n
Many guides describe the use of a broker like redis or rabbitmq between the shipper and logstash.<\/p>\n
If you’re using logstash and\/or logstash-forwarder as your shipper, you don’t need a broker. \u00a0Both of these packages keep track of where they are in the local files, and should recover from a logstash outage. \u00a0(If the outage lasts through a file rotation, this may not be true!).<\/p>\n
I only like to use brokers\u00a0when shipping logs from systems that don’t automatically handle logstash failures (e.g. syslog, netflow, etc). \u00a0This covers for unplanned outages, and also lets you release changes to logstash without losing data.<\/p>\n
Logstash Indexer<\/h1>\n
Here’s where the magic happens…. \u00a0Unstructured data is turned into structured information. \u00a0 See our guide to patterns<\/a>\u00a0and\u00a0best practices<\/a>.<\/p>\n
Elasticsearch<\/h1>\n
The storage part of the whole equation.<\/p>\n
See our best practices<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
Overview There seem to be a lot of old ELK guides on the internet. \u00a0It’s time to make a new one that can begin its own eventual decay. To help minimize the aging process, I’m not going to cover how … Continue reading