Ship logs to Logstash with Lumberjack / Logstash Forwarder

In my previous post, I explained how to set up Logstash instances on your servers, acting as logs data shippers.

However, as you may already have noticed, Logstash instances have a non-negligible memory footprint on your servers, preventing it's use where memory is limited. Furthermore, you must have Java installed on each platform you want to run Logstash.

This is where Logstash Forwarder (formerly Lumberjack) becomes interesting : this small tool, developed in Go, allows to securely ship compressed logs data (to a Logstash "indexer" for instance), with minimal resources usage, using the Lumberjack protocol.

I’ve hence decided to replace all of the Logstash "shipper" instances by Logstash Forwarder. This also means no more using Redis as a logs data broker, as Logstash Forwarder won't talk with Redis (no encryption support). In consequence, if your Logstash indexer stops running, you may lose data once Logstash Forwarder's spool max size is reached.

Installing Logstash Forwarder

To install Logstash Forwarder on your log shippers, we'll need to compile it from sources : the full procedure is very well described in the project's readme. I strongly recommend that you compile it once and make a package (either RPM or DEB) so yo can easily deploy it on all of your other servers.

Init script

Once installed from a package, Logstash Forwarder is located in /opt/logstash-forwarder. We'll use the init script available in LF's repository to handle startup :

$ cd /etc/init.d/
$ sudo wget https://raw.github.com/elasticsearch/logstash-forwarder/master/logstash-forwarder.init -O logstash-forwarder
$ chmod +x logstash-forwarder
$ update-rc.d logstash-forwarder defaults

SSL certificate generation

First of all, we'll need to generate a SSL certificate that will be used to secure communications between your shippers and indexer :

$ openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout logstash-forwarder.key -out logstash-forwarder.crt

Now move freshly created logstash-forwarder.key in /etc/ssl/private/ and logstash-forwarder.crt in /etc/ssl/certs/. Note that you'll need both of these files on each of your shippers and indexer.

Configuration

We're now ready configure Logstash Forwarder : config file is in JSON format, and will preferably be saved in /etc/logstash-forwarder (yes, it's a file), as it's the location defined in the init script we installed above.

Note : if you need to override any of the init script's parameters (ie. config file location), create a file /etc/default/logstash-forwarder and set your custom parameters.

In the following configuration example, we'll assume you want to track your iptables and Apache logs data, and that your indexer's IP is 10.0.0.5 :

{
  "network": {
    "servers": [ "10.0.0.5:5043" ],
    "ssl certificate": "/etc/ssl/certs/logstash-forwarder.crt",
    "ssl key": "/etc/ssl/private/logstash-forwarder.key",
    "ssl ca": "/etc/ssl/certs/logstash-forwarder.crt"
  },
  "files": [
    {
      "paths": [ "/var/log/syslog" ],
      "fields": { "type": "iptables" }
    },
    {
      "paths": [ "/var/log/apache2/*access*.log" ],
      "fields": { "type": "apache" }
    }
  ]
}

iptables logs filtering

To avoid processing and transmitting all of syslog's file data to our indexer, I recommend to filter your iptables log entries to have them in a separate file.

First of all, you need to have a specific criteria to filter on ; you may simply add "IPTABLES" to the log-prefix value of your iptables log rules, so it looks something like :

/sbin/iptables -A LogAndDrop -p tcp -j LOG --log-prefix "IPTABLES RULE 1 -- DROP" --log-level=info

If using rsyslog, you'll have to create an iptables.conf file in /etc/rsyslog.d/ (usually all files in this directory will be read by rsyslog) and set up a very basic filtering rule :

if $programname == 'kernel' and $msg contains 'IPTABLES' then /var/log/iptables.log

Restart rsyslog. You can now replace the iptables log file path in your Logstash Forwarder config file.

Indexer side : Logstash configuration

Next step, edit the config of Logstash on your indexer server, and add the following input :

lumberjack {
  port => 5043
  type => "logs"
  ssl_certificate => "/etc/ssl/certs/logstash-forwarder.crt"
  ssl_key => "/etc/ssl/private/logstash-forwarder.key"
}

Also add these filters to extract fields from logs data :

filter {
  if [type] == "apache" {
    grok {
      pattern => "%{COMBINEDAPACHELOG}"
    }
  }

  if [type] == "iptables" {
    grok {
      patterns_dir => "/usr/share/grok/patterns/iptables"
      pattern => "%{IPTABLES}"
    }
  }
}

You may also want logs data not to be stored in Elasticsearch if Grok patterns didn't match. In this case, add the following in the output section, surrounding your output plugins (elasticsearch for instance) :

if !("_grokparsefailure" in [tags]) {
  elasticsearch { bind_host => "10.0.0.5" }
}