Most logging frameworks include a concept of severity or priority, including tags like “WARNING”, “CRITICAL”, etc.<\/p>\n
Breaking these off into their own fields in logstash makes a lot of sense. \u00a0Unfortunately, when you go to look for problems, you end up with a search like this:<\/p>\n
log_level:(\"EMERG\" OR \"ALERT\" OR \"CRIT\" OR \"ERROR\")<\/pre>\nwhich is both inefficient (5 string comparisons) and unclear (did I miss or misspell one?).<\/p>\n
What I like to do is create an additional,<\/em>\u00a0numeric<\/em> representation of the log level, so that my search looks like this:<\/p>\n
log_code:<=3<\/pre>\nWith the two fields, you can easily query for bad stuff, but still use log_level for display (in aggregations, etc).<\/p>\n
I have standardized on the Apache LogLevel definitions<\/a>:<\/p>\n
\n\n
\n Level<\/strong><\/th>\n Description<\/strong><\/th>\n Example<\/strong><\/th>\n<\/tr>\n \n emerg<\/code><\/td>\nEmergencies – system is unusable.<\/td>\n “Child cannot open lock file. Exiting”<\/td>\n<\/tr>\n \n alert<\/code><\/td>\nAction must be taken immediately.<\/td>\n “getpwuid: couldn’t determine user name from uid”<\/td>\n<\/tr>\n \n crit<\/code><\/td>\nCritical Conditions.<\/td>\n “socket: Failed to get a socket, exiting child”<\/td>\n<\/tr>\n \n error<\/code><\/td>\nError conditions.<\/td>\n “Premature end of script headers”<\/td>\n<\/tr>\n \n warn<\/code><\/td>\nWarning conditions.<\/td>\n “child process 1234 did not exit, sending another SIGHUP”<\/td>\n<\/tr>\n \n notice<\/code><\/td>\nNormal but significant condition.<\/td>\n “httpd: caught SIGBUS, attempting to dump core in …”<\/td>\n<\/tr>\n \n info<\/code><\/td>\nInformational.<\/td>\n “Server seems busy, (you may need to increase StartServers, or Min\/MaxSpareServers)…”<\/td>\n<\/tr>\n \n debug<\/code><\/td>\nDebug-level messages<\/td>\n “Opening config file …”<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n My logstash config is broken into many smaller pieces. \u00a0The filter{} stanza for each type of log is contained in a separate file, and there are generic stanzas that run before and after the business logic.<\/p>\n
If you’re processing a log file whose levels are different, you need to normalize them. \u00a0The translate{} filter is great for this:<\/p>\n
translate {\r\n dictionary => [\r\n \"WRN\", \"warn\",\r\n \"INF\", \"info\",\r\n \"DBG\", \"debug\"\r\n ]\r\n field => \"[loglevel][tmp]\"\r\n destination => \"[loglevel][name]\"\r\n remove_field => [ \"[loglevel][tmp]\" ]\r\n}<\/pre>\nFrom what I can tell, translate{} won’t replace the source field, so the earlier grok{} matches into a temporary variable that is removed here.<\/p>\n
Once [loglevel][name] is normalized, I use a post-processing config file to add the second [loglevel][code] field:<\/p>\n
if [loglevel] and [loglevel][name] {\r\n translate {\r\n dictionary => [\r\n \"emerg\", \"0\",\r\n \"alert\", \"1\",\r\n \"crit\", \"2\",\r\n \"error\", \"3\",\r\n \"warn\", \"4\",\r\n \"notice\", \"5\",\r\n \"info\", \"6\",\r\n \"debug\", \"7\"\r\n ]\r\n field => \"[loglevel][name]\"\r\n destination => \"[loglevel][code]\"\r\n }\r\n\r\n # make it an int\r\n mutate {\r\n convert => [ \"[loglevel][code]\", \"integer\" ]\r\n }<\/pre>\n<\/p>\n","protected":false},"excerpt":{"rendered":"
Most logging frameworks include a concept of severity or priority, including tags like “WARNING”, “CRITICAL”, etc. Breaking these off into their own fields in logstash makes a lot of sense. \u00a0Unfortunately, when you go to look for problems, you end … Continue reading