• Choose a category
• All
• Classic-Fiction

11 Syslog and Log Files

The accounting system, the kernel, and various utilities all emit data that is logged and eventually ends up on your finite-sized disks. Most of that data has a limited useful lifetime and needs to be summarized, compressed, archived, and eventually thrown away.

11.1 LOGGING POLICIES

Logging policies vary from site to site. Common schemes include the following:

• Throw away all data immediately.

• Reset log files at periodic intervals.

• Rotate log files, keeping data for a fixed time.

• Compress and archive logs to tape or other permanent media.

The correct choice for your site depends on how much disk space you have and how security conscious you are. Even sites with an abundance of disk space must deal with the cancerous growth of log files.

Whatever scheme you select, maintenance of log files should be automated with cron. See Chapter 9, Periodic Processes, for more information about this daemon.

Throwing away log files

We do not recommend throwing away all logging information. Sites that are subject to security problems routinely find that accounting data and log files provide important evidence of break-ins. Log files are also helpful for alerting you to hardware and software problems. In general, given a comfortable amount of disk space, data should be kept for at least a month and then discarded. In the real world, it may take this long for you to realize that your site has been compromised by a hacker and that you need to review the logs. If you need to go back further into the past, you can recover older log files from your backup tapes.

Some administrators allow log files to grow until they become bothersome, then restart them from zero. This plan is better than keeping no data at all, but it does not guarantee that log entries will be retained for any particular length of time. Average disk usage may also be higher than with other management schemes.

Rotating log files

Most sites store each day’s log information on disk, sometimes in a compressed format. These daily files are kept for a specific period of time and then deleted. If you have sufficient disk space, it is handy to keep the log files uncompressed so that they can be easily searched with grep.

At our site we dedicate a disk partition on a central logging host (/var/log) to log files. We compress data that’s more than a week old with gzip.

One common way of implementing this policy is called “rotation.” In a rotation system, you keep backup files that are one day old, two days old, and so on. Each day, a script renames the files to push older data toward the end of the chain.

If a log file is called logfile, for example, the backup copies might be called logfile.1, logfile.2, and so on. If you keep a week’s worth of data, there will be a logfile.7 but no logfile.8. Every day, the data in logfile.7 is lost as logfile.6 overwrites it.

Suppose a file needs daily attention and you want to archive its contents for three days (to keep the example short). The following script would implement an appropriate rotation policy:

#!/bin/sh

cd /var/log

mv logfile.2 logfile.3

mv logfile.1 logfile.2

mv logfile logfile.1

cat /dev/null > logfile

chmod 600 logfile

Ownership information is important for some log files. You may need to run your rotation script from cron as the log files’ owner rather than as root, or you may need to add a chown command to the sequence.

Some sites identify log files by date rather than by sequence number; for example, logfile.tues or logfile.aug26. This system is a little harder to implement, but it can be worth the effort if you frequently refer to old log files. It’s much easier to set up in Perl than in sh. One useful idiom that doesn’t require any programming is

mv logfile logfile.`date +%Y.%m.%d`

This scheme has the advantage of making ls sort the log files chronologically.

Some daemons keep their log files open all the time. Because of the way the filesystem works, our example script cannot be used with such daemons. Instead