• Choose a category
• All
• Classic-Fiction

Piped Logging

Piped logging is a mechanism used to offload log manipulation from Apache and onto external programs. Instead of giving a configuration directive the name of the log file, you give it the name of a program that will handle logs in real time. A pipe character is used to specify this mode of operation:

CustomLog "|/usr/local/apache/bin/piped.pl /var/www/logs/piped_log" combined

All logging directives mentioned so far support piped logging. Many third-party modules also try to support this way of logging.

External programs used this way are started by the web server and restarted later if they die. They are started early, while Apache is still running as root, so they are running as root, too. Bugs in these programs can have significant security consequences. If you intend to experiment with piped logging, you will find the following proof-of-concept Perl program helpful to get you started:

#!/usr/bin/perl

use IO::Handle;

# check input parameters

if ((!@ARGV)||($#ARGV != 0)) {

print "Usage: piped.pl \n";

exit;

}

# open the log file for appending, configuring

# autoflush to avoid potential data loss

$logfile = shift(@ARGV);

open(LOGFILE, ">>$logfile") || die "Failed to open $logfile for writing";

LOGFILE->autoflush(1);

# handle log entries until the end

while (my $logline = ) {

print LOGFILE $logline;

}

close(LOGFILE);

If you prefer C to Perl, every Apache distribution comes with C-based piped logging programs in the support/ folder. Use these programs for skeleton source code.

Though the piped logging functionality serves the purpose of off-loading the logging task to an external program, it has some drawbacks:

It increases the complexity of the system since Apache must control external processes.

One process is created for every piped logging instance configured in the configuration. This makes piped logging impractical for virtual hosting systems where there are hundreds, or possibly thousands, of different hosts.

The external programs run as the user that has started the web server, typically root. This makes the logging code a big liability. Special care must be taken to avoid buffer overflows that would lead to exploitation.

Log Rotation

Because no one has unlimited storage space available, logs must be rotated on a regular basis. No matter how large your hard disk, if you do not implement log rotation, your log files will fill the partition.

Log rotation is also very important to ensure no loss of data. Log data loss is one of those things you only notice when you need the data, and then it is too late.

There are two ways to handle log rotation:

Write a script to periodically rotate logs.

Use piped logging and external helper binaries to rotate logs in real time.

Periodic rotation

The correct procedure to rotate a log from a script is:

Move the log file to another location.

Gracefully restart Apache.

Wait a long time.

Continue to manipulate (e.g., compress) the moved log file.

Here is the same procedure given in a shell script, with the added logic to keep several previous log files at the same location:

#!/bin/sh

cd /var/www/logs

mv access_log.3.gz access_log.4.gz

mv access_log.2.gz access_log.3.gz

mv access_log.1.gz access_log.2.gz

mv access_log accesss_log.1

/usr/local/apache/bin/apachectl graceful

sleep 600

gzip access_log.1

Without the use of piped logging, there is no way to get around restarting the server; it has to be done for it to re-open the log files. A graceful restart (that's when Apache patiently waits for a child to finish with the request it is processing before it shuts it down) is recommended because it does not interrupt request processing. But with a graceful restart, the wait in step 3 becomes somewhat tricky. An Apache process doing its best to serve a client may hang around for a long time, especially when the client is slow and the operation is long (e.g., a file download). If you proceed to step 4 too soon, some requests may never be logged. A waiting time