The first thing to notice is that the documents probably have different _id values, so the problem then becomes, “who is inserting duplicates??”.<\/p>\n
If you’re running logstash, some things to look at include:<\/p>\n
Most people aren’t silly enough to deliberately create duplicate input or output stanzas, but there are still easy ways for them to occur:<\/p>\n
Remember that logstash will read in all the files it finds in your configuration directory!<\/p>\n
Sometimes your shutdown script may not work, leaving you with two copies of your shipper running. \u00a0Check it with ‘ps’ and kill off the older one.<\/p>\n
If your file glob pattern is fairly open (e.g. “*”), you might be picking up files that have been rotated (“foo.log” and “foo.log.00”).<\/p>\n
Logstash-forwarder sets a ‘file’ field that you can check in this case.<\/p>\n
If you’ve enabled _timestamp in elasticsearch, it will show you when each of the duplicates was indexed, which might give you a clue.<\/p>\n
As for brokers, if you have multiple logstash indexers trying to read from the same broker without some locking mechanism, it might cause problems.<\/p>\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
Intro The first thing to notice is that the documents probably have different _id values, so the problem then becomes, “who is inserting duplicates??”. If you’re running logstash, some things to look at include: duplicate input{} stanzas duplicate output{} stanzas … Continue reading