Managing NFS and NIS, 2nd Edition - Mike Eisler [248]
Large directories also adversely impact NFS performance. Directories are searched linearly during an NFS lookup operation; the time to locate a named directory component is directly proportional to the size of the directory and the position of a name in the directory. Doubling the number of entries in a directory will, on average, double the time required to locate any given entry. Furthermore, reading a large directory from a remote host may require the server to respond with several packets instead of a single packet containing the entire directory structure.
Disk array caching and Prestoserve
As described in Section 16.4.2.1, synchronous NFS Version 2 writes are slow because the server needs to flush the data to disk before an acknowledgment to the client can be generated. One way of speeding up the disk access is by using host-based fast nonvolatile memory. This battery-backed nonvolatile memory serves as temporary cache for the data before it is written to the disk. The server can acknowledge the write request as soon as the request is placed in the cache, since the cache is considered permanent storage (since it's memory-backed and it can survive reboots). Examples of host-based accelerators include the Sun StorEdge Fast Write Cache product from Sun Microsystems, Inc., and the Prestoserve board from Legato Systems, Inc. They both intercept the synchronous filesystem write operations to later flush the data to the disk drive; significantly improving synchronous filesystem write performance.
Newer disk array systems provide similar benefits by placing the data written in the disk array's NVRAM before the data is written to the actual disk platters. In addition, disk arrays provide extra features that increase data availability through the use of mirroring and parity bits, and increased throughput through the use of striping. There are many good books describing the Berkeley RAID[8] concepts. Refer to Brian Wong's Configuration and Capacity Planning for Solaris Servers book, published by Sun Microsystems Press, for a thorough description of disk array caching and Prestoserve boards in the Sun architecture.
Disk load balancing
If you have one or more "hot" disks that receive an unequal share of requests, your NFS performance suffers. To keep requests in fairly even queues, you must balance your NFS load across your disks.
Server response time is improved by balancing the load among all disks and minimizing the average waiting time for disk service. Disk balancing entails putting heavily used filesystems on separate disks so that requests for them may be serviced in parallel. This division of labor is particularly important for diskless client servers. If all clients have their root and swap filesystems on a single disk, requests using that disk may far outnumber those using any other on the server. Performance of each diskless client is degraded, as the single path to the target disk is a bottleneck. Dividing client partitions among several disks improves the overall throughput of the client root and swap filesystem requests.
The average waiting time endured by each request is a function of the random disk transfer rate and of the backlog of requests for that disk. Use the iostat -D utility to check the utilization of each disk, and look for imbalance in the disk queues. The rps and wps values are the number of read and write operations, per second, performed on each disk device, and the util column shows the utilization of the disk's bandwidth:
% iostat -D 5
md10 md11 md12 md13
rps wps util rps wps util rps wps util rps wps util
17 45 33.7 5 4 10.5 3 3 7.5 5 5 11.6
1 5 6.1 17 20 43.7 1 1 2.0 1 0 1.1
2 7 10.4 14 22 42.0 0 0 0.7 0 1 2.3
If the disk queues are grossly uneven, consider shuffling data on the filesystems to spread the load