If you’ve setup your ELK cluster and logs are flowing in from your shippers, you’re now sitting on a goldmine of data. \u00a0The question becomes, “what should I do?!??”<\/p>\n
A first step is to make Kibana dashboards, but they serve little value in a lights-out environment (see\u00a0http:\/\/svops.com\/blog\/?p=11<\/a>).<\/p>\n When you’re ready to actively monitor the information that’s sitting in the cluster, you’ll want to pull it into your monitoring system (Nagios, Zabbix, ScienceLogic, whatever).<\/p>\n There are many benefits to this approach over Logstash’s build-in notifications, including:<\/p>\n (*) Logstash doesn’t provide these features.<\/p>\n This system is also better than using Logstash’s nagios-related plugins, since you’ll be querying all the documents in Elasticsearch, not just one document at a time. \u00a0You’ll also be using Elasticsearch as a database, rather than using Logstash’s metric{} functionality as a poor substitute.<\/p>\n There are two systems that you should build. \u00a0I’ll reference Nagios as the target platform.<\/p>\n If you wanted to query Elasticsearch for the total number of Java exceptions that have occurred, this is a good individual metric.<\/p>\n In Nagios, you would first define a virtual host (e.g. “elasticsearch”, “java”, “my_app”, etc) and a virtual service (e.g. “java exceptions”). \u00a0The service would run a new command (e.g. “run_es_query”). \u00a0Set the check interval to something that makes sense for your organization.<\/p>\n The magic comes in writing the underlying program that is run by the “run_es_query” command. \u00a0This program should take a valid Elasticsearch query_string as a parameter, and run it against the cluster.<\/p>\n In the Nagios world, the script has to return the values to show OK, WARNING, etc. \u00a0The output of the script can also include performance data, which is\u00a0used for charting.<\/p>\n The python elasticsearch module<\/a> makes writing the script pretty easy. \u00a0Write one script for each query type (max, count, most recent document, etc); this will help keep your code from becoming unreadable due to being so generic.<\/p>\n If you wanted to count the Java exceptions, but report them on a machine-by-machine basis, you would not want to launch the “individual metric” command for a set of physical hosts. \u00a0Doing this would result in many queries being run against Elasticsearch, and doesn’t scale well at all.<\/p>\n The better alternative is to run one “bulk” script that pulls the data for all hosts from Elasticsearch and then passes that information to Nagios using the “passive check<\/a>” system. \u00a0Nagios will react to the information as configured.<\/p>\n I’ve written this plugin a few times for different platforms, but always as (unsharable) work-for-hire. \u00a0I hope to rewrite this in my spare time some day, but this outline should get you started.<\/p>\n","protected":false},"excerpt":{"rendered":" Overview If you’ve setup your ELK cluster and logs are flowing in from your shippers, you’re now sitting on a goldmine of data. \u00a0The question becomes, “what should I do?!??” A first step is to make Kibana dashboards, but they … Continue reading \n
Individual Metrics<\/h1>\n
Bulk Metrics<\/h1>\n
\u00a0Where’s the Code?<\/h1>\n