rsyslog: Basic Remote Logging

Please see Learning rsyslog for the introduction and index to this series of blog posts about rsyslog.

One of rsyslog's most famous functions is the ability to log remotely. Log messages are created on host X and recorded on host Y (they can also be recorded on host X ... it's flexible). Of all the stuff I've been mucking with in rsyslog, this surprised me as being possibly the easiest to enable.

Keep in mind that there are - again - multiple options on how log messages can travel from X to Y: TCP, UDP, or RELP (Reliable Event Logging Protocol). RELP is probably best, but I don't have my head around it so I'm starting with TCP which is what you'll see below. I also hope to eventually encrypt all traffic in flight.

Here's a fairly simple set-up on the receiving host (what I called "host Y" previously):

# Add/update /etc/rsyslog.conf

# provides TCP syslog reception
# (the next line is probably already in your config: uncomment it if necessary)
module(load="imtcp")
ruleset(name="remote") {
    action(type="omfile" file="/var/log/remotefile")
    stop
}
input(type="imtcp" port="514" ruleset="remote")

This is a proof-of-concept we're working on here, not a good final configuration. But let's prove it works before we worry about more extensive message filtering. As mentioned, I prefer to restart the rsyslog service with the command: systemctl restart rsyslog ; systemctl status rsyslog | cat to immediately see errors - if there are any. You should also at this point make sure that messages to port 514 will actually get through your firewall: with UFW that would be ufw allow 514.

To test this new configuration, we again use logger to push a message to our new log server:

$ logger --server hostY.local --port 514 --tcp "GO log test 44"

I recommend running this from a host that isn't hostY - but if you need to, that will work too. Because it's sent to port 514, it's treated as a remotely logged message even if it comes from localhost. But if you send from the same host that's doing the receiving, you're missing out on testing if the port is externally available.

With the above command, and the rsyslog config in place, I get a line in /var/log/remotefile on hostY:

2020-10-24T19:46:57.502338-04:00 hostX giles GO log test 44

Because we used the "stop" statement, the logging appears only there and doesn't fall through to appear in /var/log/messages.

I'd like to have more control over where the logs land on the remote server. To do this, we use a template and the hard-to-discover built-in variables:

template (name="DynFile" type="string" string="/var/log/remote/%FROMHOST%/%SYSLOGFACILITY-TEXT%.log")

ruleset(name="remote") {
    action(type="omfile" dynafile="DynFile")
    stop
}
input(type="imtcp" port="514" ruleset="remote")

My snarky comment about the variables is a result of many hours of fighting with them. rsyslog has a lot of internal variables, but their scope isn't very obvious - they rarely worked as I expected them to. Worse, some of them need to be prefaced with a "$" while still inside surrounding "%" signs as you see above: for example, %$YEAR% is required to get part of the date into a path string. You can look at https://rsyslog.readthedocs.io/en/latest/configuration/properties.html to try to decipher it yourself. I opted out of date-based paths because logrotate is a great command that can still be used on logs, even if they're remote.

Restart the rsyslog service, and send a log message from another host:

$ logger --server hostY.local -p local3.warn --port 514 --tcp "GO log test 45"

The resulting log message appeared in file /var/log/remote/192.168.0.182/local3.log (apparently the host I'm sending the message from doesn't have a properly defined hostname). The message looks like this:

2020-10-24T19:56:07.869765-04:00 hostX giles GO log test 45

It's interesting to notice that the correct hostname (rather than an IP) is used in the log: it must come from the source host's own knowledge of itself rather than the receiving host's (lack of) knowledge of the other host.

Remote logging is working, with the caveat that all log messages are sent in the clear. If you trust your local network (you shouldn't), you may consider this okay. But I'd recommend figuring out how to encrypt your log data in transit, which I hope to do in another blog entry.

'Learning rsyslog' Index