Managing NFS and NIS, 2nd Edition - Mike Eisler [249]
If all of your disks are more than 75-80% utilized, you are disk bound and either need faster disks, more disks, or an environment that makes fewer disk requests. Tuning kernel and client configurations usually helps to reduce the number of disk requests made by NFS clients.
Kernel configuration
A significant amount of NFS requests require only information in the underlying inode for a file, rather than access to the data blocks composing the file. A bottleneck can be introduced in the inode table, which serves as a cache for recently opened files. If file references from NFS clients frequently require reloading entries in the inode table, then the file server is forced to perform expensive linear searches through disk-based directory structures for the new file pathname requiring an inode table entry.
Recently read directory entries are cached on the NFS server in the directory name lookup cache, better known as the DNLC. A sufficiently large cache speeds NFS lookup operations by eliminating the need to read directories from disk. Taking a directory cache miss is a fairly expensive operation, since the directory must be read from disk and searched linearly for the named component. For simplicity and storage, many implementations only cache pathnames under 30 characters long. Solaris removes this limitation by caching all pathnames regardless of their length. You can check your directory name lookup cache hit rate by running vmstat -s on your NFS server:
% vmstat -s
...Page and swap info...
621833654 total name lookups (cache hits 96%)
...CPU info...
If you are hitting the cache less than 90% of the time, increase ncsize on the NFS server. The ncsize kernel tunable specifies the number of entries cached by the DNLC.
In Solaris, every file currently opened holds an inode cache entry active, making the inode readily available without the need to access the disk. To improve performance, inodes for files recently opened are kept in this cache, anticipating that they may be accessed again in the not too distant future. Furthermore, inodes of files recently closed are maintained in an inactive inode cache, in anticipation that the same files may be reopened again soon. Since NFS does not define an open operation, NFS clients accessing files on the server will not hold the file open during access, causing the inodes for these files to only be cached in the inactive inode cache. This caching greatly improves future accesses by NFS clients, allowing them to benefit from the cached inode information instead of having to go to disk to satisfy the operation. The size of the inactive inode table is determined by the ufs_ninode kernel tunable and is set to the value of ncsize during boot. If you update ncsize during runtime, make sure to also update the value of ufs_ninode accordingly. The default value for ncsize is (maxusers * 68) + 360. Maxusers can be defined as the number of simultaneous users, plus some margin for daemons, and be set to about one user per megabyte of RAM in the system, with a default limit of 4096 in Solaris.
Cross-mounting filesystems
An NFS client may find many of its processes in a high-priority wait state when an NFS server on which it relies stops responding for any reason. If two servers mount filesystems from each other, and the filesystems are hard-mounted, it is possible for processes on each server to wait on NFS responses from the other. To avoid a deadlock, in which processes on two NFS servers go to sleep waiting on each other, cross-mounting of servers should be avoided. This is particularly important in a network that uses hard-mounted NFS filesystems with fairly large timeout and retransmission count parameters, making it hard to interrupt the processes that are waiting on the NFS server.
If filesystem access requires cross-mounted filesystem, they should be mounted with the background (bg) option.[9] This