SNMP traps with logstash

The Basics

SNMP traps are generally easy to receive and process with logstash.  The snmptrap{} input sets up a listener, which processes each trap and replaces the OIDs with the string representation found in the given mibs.  If the OID can’t be found, logstash will make a new field, using the OID value as the field name, e.g.

"1.3.6.1.4.1.1234.1.2.3.4": "trap value"

(Note that this is currently broken if you use Elasticsearch 2.0).

Forwarding

Probably the biggest issues with most traps is that they are sent to port 162, which is a low-numbered “system” port.  For logstash to listen on that port, it must be run as root, which is not recommended.

The easiest workaround for this is to forward port 162 to a higher-numbered port to which logstash can connect.  iptables is the typical tool to perform the forwarding:

/sbin/iptables -A PREROUTING -t nat -i eth0 -p udp --dport 162 -j REDIRECT --to-port 5000

where ‘5000’ is the port on which logstash is listening.

SNMP Sequences

Some SNMP traps come in with a “sequence number”, which allows the receiver to know if all traps have been received.  In the ones we’ve seen, the sequence is appended to each OID, e.g.

"1.3.6.1.4.1.1234.1.2.3.4.90210": "trap value"

where “90210” is the sequence number.

This seems like a handy feature, but it doesn’t appear to be supported by logstash (or perhaps the underlying SNMP library that is uses).  With the basic snmptrap config, logstash is unable to apply the mib definition and remove the sequence number, so you end up with a new field for each trap value.  That’s not good for you or for elasticsearch/kibana.

Since traps aren’t just simple plain text, you can’t use a “tcp” listener, apply your own filter to remove the sequence, and feed the result back into logstash’s “snmptrap” mechanism.  Without modifying the snmptrap input plugin, you have to fix the problem before it hits logstash.

I was a fan of logstash plugins (and have written a few), but logstash 1.5 requires everything to be done as ruby gems, which has been a painful path.  As such, I’m doing more outside of logstash, like this recommendation.

snmptrapd

We’re now running snmptrapd on our logstash machines.  They listen for traps on port 162 and write them to a regular log file that can then be read by logstash.

Basic config

Update /etc/snmp/snmptrapd.conf to include:

disableAuthorization yes

Put your mib definitions in/usr/share/snmp/mibs.

Trap formatting

To make the traps easier to process by logstash, I format the output as json.  This is done with OPTIONS set in /etc/sysconfig/snmptrapd:

OPTIONS="-A -Lf /var/log/snmptrap -p /var/run/snmptrapd.pid -m ALL -F '{ \"type\": \"snmptrap\", \"timestamp\": \"%04y-%02m-%02l %02h:%02j:%02k\", \"host_ip\":\"%a\", \"trapEnterprise\": \"%N\", \"trapSubtype\": \"%q\", \"trapType\": %w, \"trapVariables\": \"%v\" }\n' "

The flags used are:

  • -A – append to the log file rather than truncating it
  • -Lf – log to a file
  • -m ALL – use all the mibs it can find
  • -F – use this printf-style string for formatting

Then, in logstash, use the json filter:

filter {
    json {
        source => "message"
    }
}

I use a ruby filter to make the separate fields and cast them to the correct type.

Don’t forget to setup a log file rotation on your new /var/log/snmptrap file and setup a process monitor for snmptrapd.

 

8 responses to “SNMP traps with logstash

  1. Hello Robert,

    i have done your configuration on my ELK system. But it throws the json parsefailure on every message and the message is not divided into aprociated fields. I use the snmptrapd way of receiving and process the traps at first.

    Testing the json part against jsonlint show several formatting problems.

    It is necessary to have the json codec parameter in the file input and output section that reference to the snmptrap.

    Thank you very much in advance.

    Best regards
    Steffen

    • I’m not using snmp traps anymore, but the old config used the json filter (though the json codec is probably a better goal). You would not want to use both.

      For more specific help, you should jump on IRC.

  2. Good article,
    I get parsing error in logstash.log , can you help me thanks?

    {:timestamp=>”2016-05-24T11:37:28.941000+0200″, :message=>”Error parsing json”, :source=>”message”, :raw=>”{ \”type\”: \”snmptrap\”, \”timestamp\”: \”2016-05-24 11:37:27\”, \”host_ip\”:\”127.0.0.1\”, \”trapEnterprise\”: \”SNMPv2-SMI::internet\”, \”trapSubtype\”: \”.17\”, \”trapType\”: 6, \”trapVariables\”: \”SNMPv2-SMI::internet = STRING: \”Just a test\”\” }”, :exception=>#<LogStash::Json::ParserError: Unexpected character ('J' (code 74)): was expecting comma to separate OBJECT entries

    • The snmptrap plugin wouldn’t have thrown that error. You probably have something else going on. I’d just on IRC or StackOverflow for more interactive help.

    • I have the same issue here, too. When receiving a trap with a string in the Variables I get:
      ….= STRING: \”WWW Server Has Been Restarted\”\” }”, :exception=>#<LogStash::Json::ParserError: Unexpected character ('W' (code 87)): was expecting comma to separate OBJECT entries
      ^^ Only an example

  3. It is not necessary to run as root, to bind to a lower port. Setting CAP_NET_BIND_SERVICE is enough.

  4. Just an update for Ubuntu server, the trap options configuration need to be done in /etc/default/snmpd and change the TRAPDOPTS

  5. Great writeup – thanks a lot for that! Would you mind to share the logstash json part?

Leave a Reply

Your email address will not be published.