rsyslog: Encrypted Remote Logging with an Official Certificate

Please see Learning rsyslog for the introduction and index to this series of blog posts about rsyslog.

My choice of title is deliberate: I'm working on rsyslog at home with the intention of having a local machine log to another local machine. Neither of these machines has a public-facing network interface, so I would have been using self-signed TLS certificates ... and this has proved more difficult than I had hoped. However, to my surprise, it turns out to be fairly straight-forward with an official certificate from a real Certificate Authority instead of a self-signed certificate. The problem is (just to emphasize this) that you have to have a FQDN (Fully Qualified Domain Name) on the public internet for this to work.

In previous entries I've been using both Debian and Fedora. What I'm writing here applies only to Debian 10 and is untested on any other platform (but from everything I've seen should work elsewhere).

To allow TLS encrypted remote logging with rsyslog, the first step (above and beyond the many previous steps in this series of blog entries!) is to get yourself a certificate. If you're familiar with Let's Encrypt and you have an FQDN, it's easy. If you're not familiar with Let's Encrypt, I suggest you look for a tutorial (I've been using it so long I've never bothered to write one, sorry) and get familiar. So ... we wave our hands and a certificate appears.

# ls --classify /etc/letsencrypt/live/<domain_name>/
cert.pem@  chain.pem@  fullchain.pem@  privkey.pem@  README

Each of those is a link to another file in the letsencrypt/ hierarchy, but they are always maintained and kept current by the letsencrypt scripts / cron jobs. We'll need two of these files.

Logging Server

The receiving server needs the "rsyslog-gnutls" package for rsyslog encryption. You need to update your configuration file:

# /etc/rsyslog.conf
global(
    defaultNetstreamDriver         = "gtls"
    defaultNetstreamDriverCAFile   = "/etc/letsencrypt/live/receiver.gilesorr.com/chain.pem"
    defaultNetstreamDriverCertFile = "/etc/letsencrypt/live/receiver.gilesorr.com/cert.pem"
    defaultNetstreamDriverKeyFile  = "/etc/letsencrypt/live/receiver.gilesorr.com/privkey.pem"
)
module(
    load                  = "imtcp"
    StreamDriver.Name     = "gtls"
    StreamDriver.Mode     = "1"
    StreamDriver.AuthMode = "anon"
)

Assume "receiver.gilesorr.com" is a valid FQDN.

You'll also want some form of filtering for your remote log messages, as discussed in rsyslog: Basic Remote Logging, something like this should work:

# /etc/rsyslog.conf
template (name="DynFile" type="string" string="/var/log/remote/%FROMHOST%/%SYSLOGFACILITY-TEXT%.log")
ruleset(name="remote") {
    action(type="omfile" dynafile="DynFile")
    stop
}
input(type="imtcp" port="6514" ruleset="remote")

Note the change to port="6514" - port 514, which I used previously, is the semi-standard port for syslog (unencrypted), 6514 is the semi-standard port for encrypted syslog. My repeated use of the preface "semi-" may be taken to mean that you can probably change this if you want, keeping in mind you shouldn't conflict with well-known ports: https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers#Well-known_ports . Whichever port you choose, you'll need to punch a new hole in your firewall.

Sending Server

The sending server(s) will also need the "rsyslog-gnutls" package. You also need to copy the Let's Encrypt chain.pem from the previous server "receiver.gilesorr.com" to somewhere on the sending server (I used the folder /etc/rsyslog-keys/) so it can be linked below. The rsyslog configuration will need to be modified:

# /etc/rsyslog.conf
$DefaultNetStreamDriverCAFile /etc/rsyslog-keys/receiver.gilesorr.com.chain.pem  # chain.pem from receiving server
$DefaultNetStreamDriver gtls
$ActionSendStreamDriverMode 1    # run driver in TLS-only mode
$ActionSendStreamDriverAuthMode anon

*.*     @@(o)receiver:6514    # forward everything to remote server

I think the lines that start with $ are old-style directives and should be updated ... but it's working and limited experimentation didn't manage to "fix" anything. A single "@" symbol in the last line would mean "use UDP" (which I'd guess wouldn't work with encryption), and the "@@" means "use TCP." Sadly, I have no idea what the "(o)" is about, I hope to learn someday.

With these pieces in place (and a restart of both rsyslog services), you should have secure log transfer between the sending and receiving servers. See Testing Your TLS-Encrypted Server for (somewhat limited) testing instructions.

Details to Consider

At this point, logs are transmitted securely, encrypted with TLS. With a basic hole in your firewall, your log server is - theoretically at least - still subject to log spamming. That means that my next step is to use the firewall to limit the logs to expected incoming addresses and drop unknown sources. I'll also continue to look into the self-signed cert issue for logging on private networks, and will post if/when I solve that one.

'Learning rsyslog' Index

Bibliography