The grok filter – and its use of patterns – is the truly powerful part of logstash. \u00a0 Grok allows you to turn unstructured log text\u00a0into structured data.<\/p>\n
The grok filter<\/a>\u00a0attempts to match a field with a pattern. \u00a0Think of patterns as a named regular expression. \u00a0Patterns allow for increased readability and reuse. \u00a0If the pattern matches, logstash can create additional fields (similar to a regex capture group).<\/p>\n This example takes the event’s “message” field and attempts to match it with 5 different patterns (e.g. “IP”, “WORD”). \u00a0If it finds a match for the entire expression, it will add fields for the patterns (“IP” will be stored in the “client” field, etc).<\/p>\n If the input doesn’t match the pattern, a tag will be added for “_grokparsefailure”. \u00a0You\u00a0can (and should; see best practices<\/a>) customize this tag.<\/p>\n Logstash ships with lots of predefined patterns. \u00a0You can browse them on github<\/a>.<\/p>\n Patterns consist of a label and a regex, e.g.:<\/p>\n In your grok filter, you would refer to this as %{USERNAME}:<\/p>\n Patterns can contain other patterns, e.g.:<\/p>\n <\/p>\n A pattern can store the matched value in a new field. \u00a0Specify the field name in the grok filter:<\/p>\n If you’re using a regexp, you can make a new field with an\u00a0Oniguruma\u00a0trick:<\/p>\n This would find three lower case letters and create a field called ‘myField’.<\/p>\n By default, grok’ed fields are strings. \u00a0Numeric fields (int and float) can be declared in the pattern:<\/p>\n Note that this is just a hint that logstash will pass along to elasticsearch when it tries to insert the event. \u00a0If the field already exists in the index with a different type, this won’t change the mapping in elasticsearch until a new index is created.<\/p>\n While logstash ships with many patterns, you eventually will need to write a custom pattern for your application’s logs. \u00a0The general strategy is to start slowly, working your way from the left of the input string, parsing one field at a time.<\/p>\n Your pattern does not need to match the entire event message, so you can skip leading and trailing information if you just need something from the middle.<\/p>\n Grok uses\u00a0Oniguruma regular expressions<\/a>.<\/p>\n Be sure to use the debugger (see below) when developing custom patterns.<\/p>\n There is an online grok debugger<\/a> available for building and testing patterns.<\/p>\n","protected":false},"excerpt":{"rendered":" Introduction The grok filter – and its use of patterns – is the truly powerful part of logstash. \u00a0 Grok allows you to turn unstructured log text\u00a0into structured data. grok The grok filter\u00a0attempts to match a field with a pattern. … Continue reading filter {\r\n grok {\r\n match => [ \"message\", \"%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}\" ]\r\n }\r\n}<\/pre>\nPatterns<\/h2>\n
USERNAME [a-zA-Z0-9._-]+<\/pre>\n
filter {\r\n grok {\r\n match => [ \"message\", \"%{USERNAME}<\/span>\" ]\r\n }\r\n}<\/pre>\nSYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}<\/pre>\nTarget Variables<\/h2>\n
filter {\r\n grok {\r\n match => [ \"message\", \"%{USERNAME:user<\/span>}\" ]\r\n }\r\n}<\/pre>\nfilter {\r\n \u00a0grok {\r\n match => [ \"message\", \"(?<myField>[a-z]{3})\" ]\r\n }\r\n}<\/pre>\nCasting<\/h2>\n
filter {\r\n grok {\r\n match => [ \"message\", \"%{USERNAME:user:int<\/span>}\" ]\r\n }\r\n}<\/pre>\nCustom Patterns<\/h2>\n
Debugging<\/h2>\n