Apache Security - Ivan Ristic [105]
The purpose of mod_forensics (available since Versions 1.3.31 and 2.0.50) is to reveal the requests that make the server crash. It does that by having a special log file where requests are logged twice: once at the beginning and once at the end. A special utility script is used to process the log file. If a request appears only once in the log file, we know the server crashed before it could log the request for the second time.
To enable mod_forensics you also need to enable mod_unique_id. After you add the module to your configuration, decide where to put the new log file:
ForensicLog /var/www/logs/forensic_log
After restarting the server, the beginning of each request will be marked with a log of the request data (with headers but excluding the request body). Here is an example:
+QOmjHtmgtpkAADFIBBw|GET /cgi-bin/modsec-test.pl
HTTP/1.1|Accept:text/xml,application/xml,application/xhtml+xml,text/html
%3bq=0.9,text/plain%3bq=0.8,image/png,image/jpeg,image/gif%3
bq=0.2,%2a/%2a%3bq=0.1|Accept-Charset:ISO-8859-1,utf-8%3bq=0.7,%2a%3bq=0.7|
Accept-Encoding:gzip,deflate|Accept-Language:en-us,en%3bq=0.5|
Cache-Control:max-age=0|Connection:keep-alive|Host:www.ivanristic.com:8080|
Keep-Alive:300|User-Agent:Mozilla/5.0 %28Windows%3b U%3b Windows NT 5.1%3b
en-US%3b rv:1.7%29 Gecko/20040616
For each request that was properly handled, the unique identifier will be written to the log, too:
-QOmjHtmgtpkAADFIBBw
As you can see, a lot of data is being logged, so implement frequent log rotation for the forensic log. I don't think it is a good idea to leave mod_forensics enabled on a production server because excessive logging decreases performance.
The chances of catching the offending request with mod_forensics are good though in some rare instances this module will fail:
If the segmentation fault occurs before mod_forensics gets to log the request into the log
If the segmentation fault occurs after mod_forensics writes the second log entry to the log, in which case the complete pair will be in the log in spite of a segmentation fault
Once you figure out the request, you should determine which of the active modules causes it. Your goal here is to determine whether to contact the module author (for a third-party module) or the Apache developers at dev@apache.org (for standard modules).
If you have to continue on your own, consider the following tips:
Make Apache dump core. For information on the CoreDumpDirectory directive, see http://httpd.apache.org/docs-2.0/mod/mpm_common.html#coredumpdirectory.
Increase the error log level to learn more about what is happening.
Start Apache in the debug mode (add -X on the command line) and attach strace to it.
Start Apache together with the debugger (requires programming and debugging knowledge).
Read the Apache Debugging Guide (http://httpd.apache.org/dev/debugging.html).
As a final resort, use the exception hook and the two experimental modules, mod_whatkilledus and mod_backtrace. You can find more information about these modules at http://www.apache.org/~trawick/exception_hook.html.
Audit Log
One major disadvantage of Apache's (and most other web servers') logging facilities is that there is no way to observe and log request and response bodies. While most web application attacks take place through GET requests, that is only because they are performed (or programmed) by less capable attackers. The dangerous types will take the extra two minutes to craft a POST request, knowing the chances of the attack being logged are very small.
However, audit logging becomes a possibility with the help of mod_security (http://www.modsecurity.org). This module (described further in Chapter 12) adds audit logging configuration directives that can be placed almost anywhere in the configuration. It works with the main server, virtual servers, or in a directory context. To specify the audit log file and start audit logging, add the following to your configuration:
SecAuditEngine On
SecAuditLog /var/www/logs/audit_log
After the installation and configuration, you will