Elasticsearch is a text engine. \u00a0This is usually good if you have text to index, but can cause problems with other types of input (log files). \u00a0One of the more confusing elements of elasticsearch is the idea of tokenization and how fields are analyzed.<\/p>\n
In a text engine, you might want to take a string\u00a0and search for each “word”. \u00a0The rules that are used to convert a string into words are defined in a tokenizer<\/a>. \u00a0 A simple string:<\/p>\n The quick brown fox<\/p><\/blockquote>\n can easily\u00a0be processed into a series of tokens:<\/p>\n [“the”, “quick”, “brown”, “fox”]<\/p><\/blockquote>\n But what about punctuation?<\/p>\n Half-blood prince<\/p><\/blockquote>\n or<\/p>\n \/var\/log\/messages<\/p><\/blockquote>\n The default tokenizer in elasticsearch will split those up:<\/p>\n [“half”, “blood”, “prince”]<\/p>\n [“var”, “log”, “messages”]<\/p><\/blockquote>\n Unfortunately, this means that searching for “half-blood price” might\u00a0also find you an article about a royal prince<\/em> who fell half<\/em> way to the floor while donating blood<\/em>.<\/p>\n As of this writing, there are 12 built-in tokenizers<\/a>.<\/p>\n You can test some input text against a tokenizer on the command line:<\/p>\n An analyzer<\/a> lets you combine a tokenizer with some other rules to determine how the text will be indexed. \u00a0This is not something I’ve had to do, so I don’t have examples or caveats yet.<\/p>\n You can test the analyzer rules on the command line as well:<\/p>\n When you define the mapping for your index, you can control how each field is analyzed. \u00a0First, you can specify *if* the field is even to be analyzed or\u00a0indexed:<\/p>\n By using “not_analyzed”, the value of the field will not be tokenized in any way and will only be available as a raw string. \u00a0Since this is very useful for logs, the default template in logstash uses this to create the “.raw” fields (e.g. myField.raw).<\/p>\n You can also specify “no”, which will prevent the field from being indexed at all.<\/p>\n If you would like to use a different analyzer for your field, you can specify that:<\/p>\n <\/p>\n","protected":false},"excerpt":{"rendered":" Elasticsearch is a text engine. \u00a0This is usually good if you have text to index, but can cause problems with other types of input (log files). \u00a0One of the more confusing elements of elasticsearch is the idea of tokenization and … Continue reading curl -XGET 'localhost:9200\/_analyze?analyzer=standard&pretty' -d '\/var\/log\/messages'<\/pre>\n
Analyzers<\/h1>\n
curl <\/span>-<\/span>XGET <\/span>'localhost:9200\/_analyze?tokenizer=keyword&filters=lowercase'<\/span> -<\/span>d <\/span>'The quick brown fox'<\/span><\/pre>\nMappings<\/h1>\n
\"myField\"<\/span>:<\/span> {<\/span>\r\n \"index\"<\/span>:<\/span> \"not_analyzed\"\r\n<\/span>}<\/span><\/pre>\n\"myField\"<\/span>:<\/span> {\r\n<\/span> \"analyzer\"<\/span>:<\/span> \"spanish\"\r\n<\/span>}<\/span><\/pre>\n