Managing RAID on Linux - Derek Vadala [122]
Under software RAID, disk failures are reported at the kern.crit log level. In the following example, /dev/sdc1 has failed, but the array continues operating in degraded mode:
Jun 12 14:49:20 apathy kernel: raid5: Disk failure on sdg1, disabling device.
Operation continuing on 5 devices
If you enabled Verbose SCSI Reporting (as I described earlier in this chapter), syslogd also reports low-level device information:
Jun 27 18:15:53 apathy kernel: SCSI disk error : host 1 channel 0 id 2 lun 0 return
code = 10000
Jun 27 18:15:53 apathy kernel: I/O error: dev 08:61, sector 0
In this case, the entire drive has failed and is no longer detected on the SCSI bus. (In reality, I have disconnected the drive from the SCSI bus to simulate a disk failure.) Note that the I/O error is returned when reading from sector 0. This information will aid in evaluating whether or not the disk failure is the result of data corruption (multiple sector I/O failures) or complete hardware failure. Usually, when the drive fails completely, only one error message, like the one above, is reported. Consequently, the system will slow down as the SCSI bus is rescanned and reset. During that period, I/O on that channel will be interrupted. On the other hand, data corruption would yield multiple errors at either single or multiple nonzero sectors.
Automatic Failover to a Spare Disk
Disk replacement can be handled automatically by the Linux kernel, or by RAID controllers, if hot-spare disks have been allocated to the array. Insertion of these disks should be fairly automatic. The md driver takes action immediately following the first read or write to an array in degraded mode and reports any changes via syslogd. When the hot-spare is introduced, reconstruction commences.
Jun 27 18:15:53 apathy kernel: md: updating md0 RAID superblock on device
Jun 27 18:15:53 apathy kernel: sde1 [events: 00000002](write) sde1's sb offset:
17920384
Jun 27 18:15:53 apathy kernel: md: recovery thread got woken up ...
Jun 27 18:15:53 apathy kernel: md0: resyncing spare disk sde1 to replace failed disk
First, the md driver updates the RAID superblock for /dev/md0 to reflect the fact that the failed disk is no longer a member of the array and that the spare-disk is now an active member. Next, information about the new member's event counter is reported. (The event counter is simply a report of how many RAID configuration changes have been executed on the device.) Next, reconstruction commences and the mdrecoveryd process rebuilds the array. You can also examine /proc/mdstat to monitor the reconstruction process.
When the process is completed, syslogd reports on the new status of the array:
Jun 27 18:35:33 apathy kernel: md: md0: sync done.
Jun 27 18:35:33 apathy kernel: RAID5 conf printout:
Jun 27 18:35:33 apathy kernel: --- rd:3 wd:2 fd:1
Jun 27 18:35:33 apathy kernel: disk 0, s:0, o:1, n:0 rd:0 us:1 dev:sdb1
Jun 27 18:35:33 apathy kernel: disk 1, s:0, o:1, n:1 rd:1 us:1 dev:sdf1
Jun 27 18:35:33 apathy kernel: disk 2, s:0, o:0, n:2 rd:2 us:1 dev:sdg1
Jun 27 18:35:33 apathy kernel: RAID5 conf printout:
Jun 27 18:35:33 apathy kernel: --- rd:3 wd:3 fd:0
Jun 27 18:35:33 apathy kernel: disk 0, s:0, o:1, n:0 rd:0 us:1 dev:sdb1
Jun 27 18:35:33 apathy kernel: disk 1, s:0, o:1, n:1 rd:1 us:1 dev:sdf1
Jun 27 18:35:33 apathy kernel: disk 2, s:0, o:1, n:2 rd:2 us:1 dev:sde1
Jun 27 18:35:33 apathy kernel: md: updating md0 RAID superblock on device
Jun 27 18:35:33 apathy kernel: sde1 [events: 00000003](write) sde1's sb offset:
17920384
Jun 27 18:35:33 apathy kernel: (skipping faulty sdg1 )
Jun 27 18:35:34 apathy kernel: sdf1 [events: 00000003](write) sdf1's sb offset:
17920384
Jun 27 18:35:34 apathy kernel: sdb1 [events: 00000003](write) sdb1's sb offset:
17920384
Jun 27 18:35:34 apathy