bhive's FAIL-role octopus soft-locked in the raid56 RMW park path after
8.5 hours, and the log had no lock-order report to go with it -- because
lockdep had switched itself off six minutes into the run:
Aug 21 15:25:25 ... MAX_LOCKDEP_CHAINS: 65536
Aug 21 15:31:56 BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
MAX_LOCKDEP_CHAIN_HLOCKS is MAX_LOCKDEP_CHAINS * 5 and MAX_LOCKDEP_CHAINS
is 1 << CONFIG_LOCKDEP_CHAINS_BITS, so at the default 16 the whole
validation ran with 327680 chain-hlock slots. Exhausting that array
disables lockdep permanently, which means every long run on this config
has been doing its real work -- the hours of degraded raid56, balance and
ENOSPC churn -- with no deadlock detection at all, while
CONFIG_PROVE_LOCKING=y in the config said otherwise. A lock-order
inversion or a recursive acquisition in those hours would have gone
unreported, and quite possibly did: the lockup we did hit is a spinlock
that never came back, which is exactly what lockdep exists to name.
Raise the tables well past what this workload needs. The distinct-chain
count is bounded by the code paths exercised, not by run length, so the
question is only whether the ceiling is above that bound; 6 minutes to
exhaustion says the old one was far below it.
LOCKDEP_CHAINS_BITS 16 -> 20 (65536 ->
1048576 chains)
LOCKDEP_BITS 15 -> 18 (32768 -> 262144 entries)
LOCKDEP_STACK_TRACE_BITS 19 -> 21 (stack traces scale with chains)
Costs roughly 25MB of static tables on an 8-15GB test VM, which is a
trade worth making to stop running blind.
projects / linux / commitdiff