A very useful tool for monitoring logfiles is Swatch (aka Simple WATCHer of Logfiles). This is a perl program that essentially watches a logfile. You can run as many instances of it as you want, but you can only watch one logfile per instance.
Depending on the log file you want to watch, you may have to run swatch with some additional privilege. For instance, /var/log/messages is likely to be the primary target for anyone interested in using swatch. But this file should be readable by root on most, if not all, systems. So Joe User can't just monitor the logfile.
A good solution here is to use sudo to give root priv to a given swatch command. This reduces the risk of a) opening a root shell and using it to execute swatch and b) giving root access to swatch, thus allowing one to view any file they want. For instance, you may use the following in your /etc/sudoers file:
joe ALL = /usr/local/swatch/swatchmsgs
Then create the /usr/local/swatch/swatchmsgs script which may look like this:
#!/bin/sh /usr/bin/swatch --config-file=/usr/local/swatch/swatchrc \ --script-dir=/usr/local/swatch \ --tail-file=/var/log/messages
And make sure that the swatchmsgs script is mode 700 so that no user other than root can modify it (to change the --tail-file, for example). It is probably a good idea to make the /usr/local/swatch directory accessible only by root as well, and make /usr/local/swatch/swatchrc (the configuration file) mode 600.
So now, user joe can execute:
$ sudo /usr/local/swatch/swatchmsgs
and watch the log messages according to the system configuration in /usr/local/swatch/swatchrc. I suggest taking this sort of precaution because logfiles are sources for a wealth of information and you don't want the wrong person able to browse them.
Configuring swatch is pretty easy. The configuration file has an extremely straightforward syntax. A typical stanza for something to monitor begins with the keywords watchfor or ignore. Each keyword takes a perl regular expression to match the message you wish to act upon. Following that, you can take a number of actions upon the matched line. You can use these actions:
There is one other special action that can be used with any of of the above:
For example, to send an email to a pager Monday through Saturday, from 8:00am to 5:00pm, you would use:
[email protected],when=1-6:8-17
All very straightforward. The big thing is to know some perl regular expressions so you can fine-tune your rules. Let's look at a few examples.
# Bad login attempts watchfor /INVALID|REPEATED|INCOMPLETE|[Ff]ail / echo magenta_h bell 3 mail addresses=sysadmin\@myhost.com,subject=Bad_login_attempt throttle 01:00
This stanza will send an email to the specified email address (note you have to escape the @ sign, like you would in a perl script), with the subject "Bad login attempt". It will echo the message to the screen in highlighted magenta and will throttle the message to display once per minute, and ring a bell three times.
# Logins, password changes and such watchfor /workstation.*(LOGIN|passwd|ruserok|session opened|session closed)/ echo yellow bell 3 watchfor /firewall.*(LOGIN|passwd|ruserok|session opened|session closed)/ echo yellow_h bell 3
This is two different stanzas grouped together. They are almost identical except one uses a highlight yellow, and the regular expression to watchfor is slightly different. In this example, messages coming from the machine named "workstation" are echoed in yellow, while those coming from the machine named "firewall" are yellow highlighted. This illustrates how you can configure swatch on a central loghost to differentiate between messages from different machines. If you wanted, you could have emails sent for these attempts on the firewall machine, and nothing on the other machine. Specifying the machine name (on a machine that receives logs for more than just itself) is useful when trying to write some fine-grained rules.
Rules can be small and simple as well. Remember, swatch only displays what you tell it to, so if you want to see something without making it fancy, you still have to write a rule for it.
watchfor /(su|sudo):/ echo bold watchfor /firewall kernel.*(DROP|REJECT)/ echo blue_h watchfor /rsyncd.*from/ echo
This is three very simple rules. The first just bolds anything having to do with su or sudo. The second shows any iptables DROP or REJECT logs from the firewall machine, and the last simple echos to the screen (with absolutely no highlighting or coloring), messages from rsyncd that contain the word "from" (these lines contain the IP address or reverse-DNS name of connecting systems), so we exclude statistical stuff from rsyncd like how much was written or received.
As you can see, swatch is fairly flexible, particulary due to the use of perl regular expressions. And colorizing the output is nice if you have a virtual desktop you can spare to maximize an xterm (or transparent borderless Eterm which looks very slick) and watch swatch's output. It doesn't prevent you from having to read log files; no program can really do that, but it can reduce the "noise" so that you're only reading what is important (to you).