Most UNIX implementations (Linux, BSD, etc.) use a program called syslogd to handle most of the logging. syslogd is a system logging daemon and what it does is sit and receive messages from various applications. It will receive log messages from other daemons, programs, and even the kernel. Linux systems also come with a daemon called klogd, the Linux kernel log daemon.
Configuration of syslog is done via the /etc/syslog.conf file. Each Linux distribution, and other systems like FreeBSD, Mac OS X, etc. may have their default syslog configurations slightly different, and each may run a slightly different syslogd that may offer enhanced or reduced functionality. The basic syntax of the logfiles are similar, however.
What syslog does is receive messages from various “facilities” on the system. Each received message will also have an importance level assigned to it. The syslog.conf file tells syslogd what to do with these messages, based upon their facility and level of importance. The basic syntax of syslog.conf is as follows:
facility.level action
The selector (the first field; “facility.level”), must be separated from the action by one or more TAB characters. While some syslog implementations understand using spaces as a separator, others do not so if you’re unsure, use tabs. You can also include blank lines and comments (lines beginning with the # character) in the configuration file. You may also have multiple facilities specified, separated by a comma.
Facilities, Levels, and Actions
The following is a list of syslog facilities:
| Facility | Description |
|---|---|
| auth | Authentication information, but has been deprecated in favor of authpriv, although some programs still use it |
| authpriv | Authentication information which may contain data like usernames and other privileged information |
| console | Messages written to /dev/console by the kernel console output driver [BSD only] |
| cron | Messages from the cron and at daemons |
| daemon | Messages from daemons, such as inetd, xinetd, etc. |
| ftp | Messages from FTP daemons |
| kern | Messages from the kernel; on Linux systems these are normally passed through klogd first |
| lpr | Messages from printer services like lpd, LPRng, etc. |
| Messages from mail-related daemons such as sendmail or postfix | |
| mark | An internal facility for syslog to generate timestamps |
| news | Messages from NNTP daemons such as innd or leafnode |
| security | Messages from the security subsystem, such as ipfw [BSD only] |
| syslog | Messages from syslog itself |
| user | Messages generated by user-space programs; this is the default for unspecified syslog facilities |
| uucp | Messages from UUCP |
| local0-local7 | Facilities used by customized programs (ie. in some programs you can tell it via a configuration file what facility to use, so you may opt to have OpenLDAP log to local0, OpenSSH to log to local1, and so forth). |
| * | All facilities except mark |
The following table describes the various syslog levels:
| Level | Description |
|---|---|
| emerg | The system is unusable |
| alert | A condition exists that needs to corrected immediately |
| crit | An error condition that indicates a program or subsystem may no longer be useable |
| err | An error condition that indicates a component of a program or subsystem may no longer be useable |
| warning | A warning message |
| notice | A normal condition with significance |
| info | An informational message |
| debug | A debug message, usually a message that does not indicate any problems or have any normal significance |
| none | No level, usually used as a facility exemption when using the wildcard |
| * | All levels, except none |
Finally, the following table illustrates the different actions syslog can take. Not all of these actions may be available with all implementations of syslog, however.
| Action | Description |
|---|---|
| file or device | The absolute pathname to a file (ie. /var/log/messages). The message will be written to the specified file. You can also use the full path to a device such as /dev/console, /dev/tty1, or /dev/lp0, etc. This can be useful for writing to a serial port, printer, or specific console. On Linux systems, you can prefix a file with the “-” character to tell syslog not to sync the file after every logging, which can have some performance benefits, but may also result in missing messages in the event of a system crash. |
| @hostname | Messages are sent to specified hostname. The remote syslogd must be able to receive remote messages and must be configured for the selected facility.level being sent. |
| username | Sends the message to the specified user(s) using write, provided they are logged in. Multiple users can be specified by using a comma-separated list (ie. root,vdanen,joe. To write to everyone, use “*”. |
| named pipe | ) symbol. This tells syslog to write into the FIFO (pipe), and other programs like swatch can read from the FIFO and act on the message. [Linux only] |
You can also specify multiple selectors by separating them with a semi-colon (ie. you can use kern.emerg;mail.crit to specify two facilities to log to the same place. Each specified level indicates that level and higher; so if you specifiy “crit”, you will actually be logging “crit”, “alert”, and “emerg”. To specify one level and that level only, prefix it with the “=” character (ie. kern.=crit). To negate a specific level use the “!” character (ie. kern.info;kern.!err) tells syslog to log everything from facility kern with level “info” and higher, except everything with level “err” and higher (basically, log levels “info”, “notice”, and “warning”)).
Configuring syslog
With the above information, you can finetune your /etc/syslog.conf file. Most vendors provide sane defaults for syslog and likely you will not need to change anything. The logfile that usually contains the most “centralized” information and will be the one most people will likely want to monitor closely is /var/log/messages (on Linux) and /var/log/system.log (on BSD). For instance, on Linux, the following is used to determine what to write to /var/log/messages (taken from Mandrakelinux 9.1):
*.info;mail.none;news.none;authpriv.none -/var/log/messages
This tells syslog to log every facility, of level “info” and higher, except the facilities “mail”, “news”, and “authpriv” to the file /var/log/messages, and to not sync the file after every write.
On Mac OS X, the default is as follows:
*.notice;*.info;authpriv,remoteauth,ftp.none;kern.debug;mail.crit /var/log/system.log
This tells syslog to log every facility of level “notice” and “info” except for facilities “authpriv”, “remoteauth”, and “ftp”, as well as debug kernel messages, and critical mail messages to the file /var/log/system.log.
NOTE: I’m not sure where this remoteauth facility is coming from for OS X as Apple neglected to include a manpage for syslog.conf, and the syslog manpage makes no reference to this facility, nor does it look like FreeBSD has it, so I don’t know if this is a valid facility or where it comes from or what it specifically does (although I suspect perhaps telnet logins by the name, as OpenSSH seems to be logging to auth.info, which is it’s default (unless Apple patched it for some reason)). Using logger to test it does indeed show that it is a working facility, but I don’t know what systems use it.
When you make changes to your /etc/syslog.conf, you need to tell syslog to re-read it’s configuration file. This can be done by executing, as root, “killall -HUP syslogd” or restarting the service itself (ie. executing “/etc/init.d/syslog restart” on some Linux systems). Some implementations of syslog will create any newly specified logfiles from the configuration upon startup; others may not so you may want to do a simple “touch /var/log/mynewlogfile” before restarting syslogd.
You can also test your syslog changes quite easily by using the logger tool. For instance, I wanted to test if remoteauth.crit was actually being logged on OS X, so I executed:
$ logger -p remoteauth.crit -t TEST "This is a test"
syslog dutifully wrote it to my /var/log/secure.log file in OS X and the file contained:
Feb 21 14:36:59 tibook TEST: This is a test
You can make use of this in your scripts so you can have, for example, some auditing script or archival script record it’s start and end (and any errors it may have encountered) to a logfile via syslog.
Remote and Central Logging
Logging remotely, to a central loghost, is a great idea, especially if you have machines that are publically accessible, such as web servers in a DMZ. One thing that crackers attempt to do after gaining access to a system is to srub the logfiles to remove as much evidence that they were there as possible. By having your system logger log remotely, they are unlikely to be able to do this without also breaking into your log host.
There are two simple steps to accomplishing this. The first is to configure syslog appropriately, the second is to have the system time synchronized between machines. For the latter, using something like ntp is appropriate; you can use it to match your clock to a specified time server. If all machines use the same time server, the time between machines should match to a high degree.
In your /etc/syslog.conf configuration file, you need to tell it to log data to a log server. For redundancy, have syslog log both locally and remotely, except on the central log server. For instance, you may add the following to the end of your /etc/syslog.conf file:
*.info @loghost.mydomain.com
where loghost.mydomain.com is a fully resolveable domain name that points to your log server. This will log all messages of level “info” and higher to the log server.
On the log server itself, you must start syslog with the “-r” commandline parameter (on Linux) or “-u” on Mac OS X. On FreeBSD, syslog listens to the network with no commandline options, but is configured to run in “secure mode” (with the commandline parameter “-s”) by default. This tells syslog to listen to UDP port 514 for logging messages. It will write to the local system as per the rules defined in /etc/syslog.conf on the host. So if you have, for instance, mail.info being written to /var/log/mail/info on your server, but written to /var/log/mailinfo on the log server, the mail.info messages will be also be logged to /var/log/mailinfo as syslog pays attention to the rules on the host it is running on; it knows nothing of the rules on the remote server.
On a Mandrakelinux system, you would modify the /etc/sysconfig/syslog file and change:
SYSLOGD_OPTIONS="-m 0"
to:
SYSLOGD_OPTIONS="-r -m 0"
and then restart the syslogd daemon.
In Mac OS X, you need to modify the file /System/Library/StartupItems/SystemLog/SystemLog. In the StartService() function is a line that simply reads “syslogd”; replace it with “syslogd -u”, like in the following illustration.
StartService ()
{
ConsoleMessage "Starting system log"
if [ -f /etc/syslog.conf ]; then
if ! pid=$(GetPID syslog); then
rm -f /dev/log
syslogd -u
fi
else
echo "Warning: syslogd was not started"
fi
}
You will either need to reboot to have the change take effect, or kill and start syslogd manually in order to see the immediate change.
On a FreeBSD system,syslog is started in “safe mode” to begin with, so you need to disable it. To do this, edit /etc/rc.conf and add:
syslogd_flags=""
to the end of the file. You will need to restart syslog completely in order for this to take effect, which can be accomplished by doing a reboot or killing syslogd and then starting it again manually.
Perhaps one of the most frustrating things with syslog is the differences across various platforms. It performs the same function, but as you can see with three different, yet similar, operating systems, there are three very different syslog implementations in terms of how it’s run (the configurations are very similar across the board).
On a final note, you will need to modify your firewall to allow inbound traffic on UDP port 514. You will want to limit this to the client systems that will be sending logs to you, to ensure you receive only legitimate log entries. If you leave it open for the world, an attacker can DoS your system by just filling your filesystem full of bogus log entries.
On a Linux system, using iptables, you would use the following rule on the logging client (the machine from which you will receive syslog messages). It assumes a default OUTPUT policy of DENY:
iptables -A OUTPUT -o $IFACE -p udp -s $MYIP -d $LOGHOST --dport 514 -j ACCEPT
This implied $IFACE being equal to your external ethernet interface (ie. eth0), $MYIP being the IP address of the server (the machine you are adding this iptables rule to) and $LOGHOST is the IP address of the machine you wish to send the logs to.
On the receiving system, you would use a similar rule:
iptables -A INPUT -o $IFACE -p udp -s $LOGCLIENT -d $MYIP --dport 514 -j ACCEPT
In this case, $LOGCLIENT is the IP address of the machine sending the logs, and $MYIP is the IP address of the log server.