Managing RAID on Linux - Derek Vadala [123]
Jun 27 18:35:34 apathy kernel: md: recovery thread finished
...
Now the faulty disk can be replaced or repaired.
Remember that the reconstruction process affects overall system performance. In addition, ATA systems typically require longer reconstruction times and more CPU overhead than SCSI systems because intra-disk transfers are slower and more intensive. During reconstruction of ATA arrays, the CPU needs to handle not only the reconstruction process, but also array and disk management. This sometimes can be too much for a system to handle and can result in extremely long reconstruction times.
Sharing Spare Disks
While the md driver can't directly share spare disks between arrays, mdadm does support this feature through its monitor mode. When mdadm is in monitor mode, it will automatically move a spare disk from a working array to an array that encounters a disk failure, provided they are both marked as having the same spare disk group.
Manual Disk Replacement
Even if online spare disks are used, system administrators must physically replace failed drives. Replacement should take place as soon as possible to avoid the potential for a secondary disk failure that might incapacitate an array. A secondary disk failure, when no more spares are available, means that the array will operate in degraded mode until the disk can be physically replaced. It's also advisable to replace dead disks as soon as possible so they can be reallocated as spares in the event of another failure.
Remember that ATA does not technically support any hot-swap capability. Although some newer disk enclosures and controllers support this feature, disk manufacturers and reports from users discourage the use of hot-swap ATA. Therefore, set up hot-swap ATA equipment at your own risk.
Likewise, SCSI supports hot-swap only when working with SCA drives. Although some users have successfully swapped non-SCA SCSI disks out of running systems, this practice is not recommended.
If your system supports SCA disks, you can simply remove the drive and add a new one. The SCSI bus needs to be told that a new disk is present, because the Linux kernel or the hardware disk controller will have already marked the failed disk as nonoperational when it entered the reconstruction phase.
Using the /proc/scsi interface, disks can be added and removed from a running system. To remove a failed disk from the bus (if the kernel hasn't already removed it), use the following command:
# echo "scsi remove-single-device 0 1 0 0" > /proc/scsi/scsi
This instructs the kernel to scan the first SCSI controllers for the disk with ID 0 and activate it. The scsiadd utility, available from http://llg.cubic.org/tools/, provides a command-line wrapper for this and other SCSI management functions. I encourage you to download and experiment with this utility.
There is no ATA equivalent for this function. On ATA RAID systems, you need to restart the computer to facilitate a rescan of the bus in order to activate disks that have been connected since the last time the system was started.
Don't forget that you also need to use either raidhotadd or mdadm -a to add replacement disks back into their respective arrays.
Problems with Hot-Swap and Disk Replacement
One of the biggest problems with hot-swapping or replacing devices under Linux (and many other Unix systems) stems from device naming. On an ATA system, Linux assigns the first drive on the first controller to /dev/hda. Subsequently detected devices are named in the order in which they are detected, using the same naming scheme. Thus, the slave device on the primary channel is assigned /dev/hdb, and the master and slave devices on the secondary channel are assigned /dev/hdc and /dev/hdd, respectively.
ATA is unique because of its legacy master and slave arrangement. If a device that was initially detected during installation suddenly disappears from a system, the naming order isn't affected. If /dev/hdb suddenly dies, and the system reboots, /dev/hda, /dev/hdc, and /dev/hdd will still be assigned the same