Beautiful Code [57]
This variant of debugging—postmortem debugging—can be contrasted to the more traditional in situ debugging by which one is debugging a live and running (albeit stopped) system. Whereas with in situ debugging one can use breakpoints to iteratively test hypotheses about the system, with postmortem debugging one can use only the state of the system at the time of failure to test hypotheses. While this means that postmortem debugging is necessarily a less complete methodology than in situ debugging (as there are bugs for which there simply does not exist enough state at the time of failure to eliminate a significant number of hypotheses), there are many bugs that are not sufficiently reproducible to apply in situ debugging—there is no other way to debug them but postmortem.
Moreover, because the more limited options afforded by postmortem debugging require more rigorous thinking around both hypothesis generation and validation, developing one's ability to debug postmortem makes one much more efficient at in situ debugging.
Finally, because the state of the system is static, one can present specific, rigorous analysis to peers, who can then perform their own analysis—in parallel—and draw their own conclusions. And even if not actively validated by others, this analysis is valuable, for drafting it forces one to address the holes in one's own logic. In short, postmortem debugging is an essential part of our craft—a skill that every serious software engineer should develop.
Given that background on (and plug for) postmortem debugging, and with the caveat that this analysis will not yet be completely (or perhaps not at all) comprehensible, here is my analysis of the bug at hand, as it appeared verbatim in my initial bug report:[]
[] "Beautiful Bug Reports," anyone?
Code View: Scroll / Show All
[ bmc, 12/13/99 ]
The following sequence of events can explain the state in the dump (the arrow
denotes an ordering):
Thread A (300039c8580) Thread B (30003c492a0)
(executing on CPU 10) (executing on CPU 4)
-------------------------------------+ +-------------------------------------+
| Calls lwp_upimutex_lock() on | | |
| lock 0xff350000 | | |
| | | |
| lwp_upimutex_lock() acquires | | |
| upibp->upib_lock | | |
| | | |
| lwp_upimutex_lock(), seeing the | | |
| lock held, calls turnstile_block() | | |
| | | |
| turnstile_block(): | | |
| - Acquires A's thread lock | | |
| - Transitions A into TS_SLEEP | | |
| - Drops A's thread lock | | |
| - Drops upibp->upib_lock | | |
| - Calls swtch() | | |
: : : :
+----------------------------------------------------------------------+
| Holder of 0xff350000 releases the lock, explicitly handing it off to |
| thread A (and thus setting upi_owner to 300039c8580) |
+----------------------------------------------------------------------+
: : : :
| | | |
| Returns from turnstile_block() | | |
| | | Calls lwp_upimutex_lock() on |
| | | lock 0xff350000 |
| | | |
| | | lwp_upimutex_lock() acquires |
| | | upibp->upib_lock |
| | | Seeing the lock held (by A), calls |
| | | turnstile_block() |
| Calls lwp_upimutex_owned() to | | |
| check for lock hand-off | | turnstile_block(): |
| | | - Acquires B's thread lock |
| lwp_upimutex_owned() attempts | | - Transitions B into TS_SLEEP, |
| to acquire upibp->upib_lock | | setting B's wchan to upimutex |
| | | corresponding to 0xff350000 |
| upibp->upib_lock is held by B; | | - Attempts to promote holder of |
| calls into turnstile_block() | | 0xff350000 (Thread A) |
| through mutex_vector_enter() | | - Acquires A's thread lock |
| | | - Adjusts A's priority |
| turnstile_block(): | | - Drops A's thread lock |
| <--------------+ |
| - Acquires A's thread lock | | - Drops B's