Changing @timestamp with the date{} filter

By default, elasticsearch sets the @timestamp field to the time that the document was written into elasticsearch.  For log file data, one of the first things you’ll want to do with logstash is to set @timestamp to be the value from the log event you’re processing.

To keep this example simple, we’ll assume that you’ve already grok’ed the event and have a field called “my_timestamp” that contains the correct information from the log event, e.g. “14/Mar/2015:09:26:53 -0500”.

filter {
  date {
    match => [ 'my_timestamp', "dd/MMM/yyyy:HH:mm:ss Z" ]
    remove_field => [ 'my_timestamp' ]
  }
}

The match{} takes the “my_timestamp” field, applies it against the specified pattern, and puts the resulting date object into the @timestamp field.

Once @timestamp has been populated, we no longer need “my_timestamp”, so it is removed.

 

4 responses to “Changing @timestamp with the date{} filter

  1. Rajesh Swarnkar

    Why does the field timestamp is designated with an @ at beginning?
    Is there special meaning the way such fields are treated?

    • There used to be a bunch of “special” fields (to logstash), which they prefaced with the ‘@’ to keep them from interfering with your own fields. Most of them got the ‘@’ removed a while back.

  2. Rajesh Swarnkar

    Hi DevOps, I have a query.
    By default, the logstash has an @timestamp field, which is used by Kibana for x -axis time display.
    I have logs that have different time format than ISO8601 format.
    I am able to ‘extract’ the date from log by custom logstash pattern and Date plugin, to a field name viz. logtimestamp. Pushed it to Elastic Search.
    How do I set this extracted field as Kibana’s x-axis? Also Is this good practice?

    • Use grok{} to get the date information into a field, and then you can pass the format (joda) to the date{} filter to update @timestamp. Then @timestamp will be available for kibana to use on the x-axis as you describe.

Leave a Reply

Your email address will not be published. Required fields are marked *