• Choose a category
• All
• Classic-Fiction

Use adequate tools if you decide to go real time

Event correlation tools (one of which is described below) will do the hard work for you, filtering out events you do not care about and only disturbing your peace in real trouble.

Periodic reporting

One way to implement periodic monitoring is to use the concept of Artificial Ignorance invented by Marcus J. Ranum. (The original email message on the subject is at http://www.ranum.com/security/computer_security/papers/ai/.) The process starts with raw logs and goes along the following lines:

Remove "noisy" lines—i.e., the lines you know are safe to ignore.

Remove certain parts that are or may be unique for every entry (e.g., the time/stamp or the remote IP address).

Alphabetically sort the lines.

Replace multiple identical lines with a single copy but prefix each such line with the number of occurrences. Prefix each line that occurs only once with the number 1.

Sort the output in descending order, thereby showing the lines that occurred most frequently first.

The idea is to uncover a specific type of event, but without the specifics. The numerical value is used to assess the seriousness of the situation. Here is the same logic implemented as a Perl script (I call it error_log_ai) that you can use:

#!/usr/bin/perl -w

# loop through the lines that are fed to us

while (defined($line = )) {

# ignore "noisy" lines

if (!( ($line =~ /Processing config/)

|| ($line =~ /Server built/)

|| ($line =~ /suEXEC/) )) {

# remove unique features of log entries

$line =~ s/^\[[^]]*\] //;

$line =~ s/\[client [^]]*\] //;

$line =~ s/\[unique_id [^]]*\]//;

$line =~ s/child pid [0-9]*/child pid X/;

$line =~ s/child process [0-9]*/child process X/;

# add to the list for later

push(@lines, $line);

}

}

@lines = sort @lines;

# replace multiple occurences of the same line

$count = 0;

$prevline = "";

foreach $line (@lines) {

next if ($line =~ /^$/);

if (!($line eq $prevline)) {

if ($count != 0) {

$prefix = sprintf("%5i", $count);

push @outlines, "$prefix $prevline";

}

$count = 1;

$prevline = $line;

} else {

$count++;

}

}

undef @lines;

@outlines = sort @outlines;

print "--httpd begin------\n";

print reverse @outlines;

print "--httpd end--------\n";

The script is designed to take input from stdin and send output to stdout, so it is easy to use it on the command line with any other script:

# cat error_log | error_log_ai.pl | mail ivanr@webkreator.com

From the following example of daily output, you can see how a long error log file was condensed into a few lines that can tell you what happened:

--httpd begin------

38 [notice] child pid X exit signal Segmentation fault (11)

32 [info] read request line timed out

24 [error] File does not exist: /var/www/html/403.php

19 [warn] child process X did not exit, sending another SIGHUP

6 [notice] Microsoft-IIS/5.0 configured -- resuming normal operations

5 [notice] SIGHUP received. Attempting to restart

4 [error] File does not exist: /var/www/html/test/imagetest.GIF

1 [info] read request headers timed out

--httpd end ------

Swatch

Swatch (http://swatch.sourceforge.net) is a program designed around Perl and regular expressions. It monitors log files for events and evaluates them against expressions in its configuration file. Incoming events are evaluated against positive (take action on event) and negative (ignore event) regular expressions. Positive matches result in one or more actions taking place.

A Swatch configuration file designed to detect DoS attacks by examining the error log could look like this:

# Ignore requests with 404 responses

ignore /File not found/

# Notify me by email about mod_security events

# but not more than once every hour

watchfor /mod_security/

throttle 1:00:00

mail ivanr@webkreator.com,subject=Application attack

# Notify me by email whenever the server

# runs out of processes - could be a DoS attack

watchfor /MaxClients reached/

mail ivanr@webkreator.com,subject=DOS attack

Swatch is easy to learn and use. It does not