Please see Learning Rsyslog for the introduction and index to this series of blog posts about Rsyslog.
If you're learning Rsyslog, you'll see a lot of references to "Facility" and "Severity". These are both very useful and very limiting. Practical examples may help:
# /etc/rsyslog.conf *.info;mail.none;authpriv.none;cron.none /var/log/messages authpriv.* /var/log/secure mail.* -/var/log/maillog cron.* /var/log/cron *.emerg :omusrmsg:* uucp,news.crit /var/log/spooler local7.* /var/log/boot.log
These are all default filter lines from a Fedora 32 system (Debian's defaults are very close, but not identical). And their meaning should be pretty clear: the second line means that everything that's got a "facility" of "authpriv" goes into the /var/log/secure file, and the first line indicates that all messages with a "severity" of "info" or higher go into /var/log/messages - except we're going to exclude facilities of mail, authpriv, and cron, which are directed to other files shortly after. That should be enough hints to read the whole thing.
Which is all fine and good: it shows basic filtering, and makes some sense of how your log files come into existence. But ... how did the log messages get marked with their facility and severity? Unfortunately, I'm at a point in my understanding of Rsyslog that means I'm better at asking these questions than answering them. I'll do my best, but these won't all be answered when I'm done. This blog entry started because I had one question: can I create my own facility names?
Most log messages that pass through Rsyslog are generated by one of the programs or subsystems that go with the names you're seeing: the kernel, cron jobs, mail. But what about Nginx or Apache? Well ... here's the thing. While Rsyslog is perfectly happy to handle your web server's log messages, under normal circumstances it doesn't. Nginx, Apache, HAProxy - they all log directly to your filesystem without ever communicating with Rsyslog in any way. It's not hard to make them do so (at least it doesn't appear to be for Nginx, and I assume the same is true of the others) - they can write to an Rsyslog-controlled socket instead of the default file, and Rsyslog handles the messages. But now you have a non-standard configuration, and you have to write new filters to tell Rsyslog where to send the new messages. And they DON'T (by default) have a facility attached to them, and may or may not have a severity. I thought it would be reasonable to create your own facility name, so I tried to do that with my Memory log (see Rsyslog: A New Log) when sending the log to my partially set up central logging server. I passed a "facility" value of "memfree." The receiving Rsyslog was supposed to file these under "%SYSLOGFACILITY-TEXT%.log", but saved it to "invld.log". It apparently thought my created facility name was invalid. Further research has shown that "[t]he syslog interface only allows a fixed set of facilities, defined by constants in /usr/include/sys/syslog.h. The only provision for custom facilities are local0 through local7." (Courtesy of StackExchange, see the link in the Bibliography).
"Defined by constants" may sound like numerical values: that's because they are. Each facility has a number, although for most purposes it's easier to deal with the name. (If you need the numbers, see the Tables section of this post).
Having a facility name attached to a log message makes it much easier to sort. Rsyslog provides extensive, powerful filtering - but to use it, you have to write filters. Writing memfree.* /var/log/remote/<host>/memfree.log is simple enough to verge on being intuitive. But Rsyslog's filters are complex and take a lot of time to learn. The simplest solution for for me is to create a consistent key: [ local0 = Nginx , local1 = Apache , local2 = Ruby , local3 = HAProxy ]. But for this to work across dozens or hundreds of machines feeding a centralized logging server, you have to stick with it like it's written in stone ... and hope you don't have more than eight services. Seven if you use Fedora, because if you look at the example rsyslog.conf above, "local7" is already in use.
I think I'm going to need filtering anyway: I'll write about it when I get to it.
Honestly, I'm not sure of a lot of these names. Facility 5 is apparently used for "messages generated internally by syslog." The list I took this from lists "clock daemon" twice (9 and 15). It's unlikely anyone uses line printers anymore ... does that mean that one's available? And if you're not running an FTP server? Etc. The names show the origins of these are very old.
One mystery is answered though: the Priority of a message is calculated by doing the math:
priority = facility-value * 8 + severity-value.
Thus "local1.info" is 17 * 8 + 6 = 142, and so the value used in Testing Your TLS-Encrypted Server with
gnutls-cli is explained.
- https://success.trendmicro.com/solution/TP000086250-What-are-Syslog-Facilities-and-Levels - this was hugely helpful and some material here is a straight lift
- https://unix.stackexchange.com/questions/21041/add-new-syslog-facility - explanation of the limitations of facility's names