Squid_ The Definitive Guide - Duane Wessels [66]
An interesting property of threads is that all processes share the same resources, including memory and file descriptors. For example, when a thread process opens a file as descriptor 27, all other threads can then access that file with the same descriptor number. As you probably know, file-descriptor shortage is a common problem with first-time Squid administrators. Unix kernels typically have two file-descriptor limits: per process and systemwide. While you might think that 256 file descriptors per process is plenty (because of all the thread processes), it doesn't work that way. In this case, all threads share that small number of descriptors. Be sure to increase your system's per-process file descriptor limit to 4096 or higher, especially when using aufs.
Tuning the number of threads can be tricky. In some cases, you might see this warning in cache.log:
2003/09/29 13:42:47| squidaio_queue_request: WARNING - Disk I/O overloading
It means that Squid has a large number of I/O operations queued up, waiting for an available thread. Your first instinct may be to increase the number of threads. I would suggest, however, that you decrease the number instead.
Increasing the number of threads also increases the queue size. Past a certain point, it doesn't increase aufs's load capacity. It only means that more operations become queued. Longer queues result in higher response times, which is probably something you'd like to avoid.
Decreasing the number of threads, and the queue size, means that Squid can detect the overload condition faster. When a cache_dir is overloaded, it is removed from the selection algorithm (see Section 7.4). Then, Squid either chooses a different cache_dir or simply doesn't store the response on disk. This may be a better situation for your users. Even though the hit ratio goes down, response time remains relatively low.
Monitoring aufs Operation
The Async IO Counters option in the cache manager menu displays a few statistics relating to aufs. It shows counters for the number of open, close, read, write, stat, and unlink requests received. For example:
% squidclient mgr:squidaio_counts
...
ASYNC IO Counters:
Operation # Requests
open 15318822
close 15318813
cancel 15318813
write 0
read 19237139
stat 0
unlink 2484325
check_callback 311678364
queue 0
The cancel counter is normally equal to the close counter. This is because the close function always calls the cancel function to ensure that any pending I/O operations are ignored.
The write counter is zero because this version of Squid performs writes synchronously, even for aufs.
The check_callback counter shows how many times the main Squid process has checked the done queue for completed operations.
The queue value indicates the current length of the request queue. Normally, the queue length should be less than the number of threads × 5. If you repeatedly observe a queue length larger than this, you may be pushing Squid too hard. Adding more threads may help but only to a certain point.
The diskd Storage Scheme
diskd (short for disk daemons) is similar to aufs in that disk I/Os are executed by external processes. Unlike aufs, however, diskd doesn't use threads. Instead, inter-process communication occurs via message queues and shared memory.
Message queues are a standard feature of modern Unix operating systems. They were invented many years ago in AT&T's Unix System V, Release 1. The messages passed between processes on these queues are relatively small: 32-40 bytes. Each diskd process uses one queue for receiving requests from Squid and another queue for transmitting results back.
How diskd Works
Squid creates one diskd process for each cache_dir. This is different from aufs, which uses a large pool of threads for all cache_dirs. Squid sends a message to the corresponding diskd process for each I/O operation. When that operation is complete, the diskd process sends a status message back to Squid. Squid and the diskd processes preserve the order of messages in the queues. Thus, there is no concern that I/Os might