borrower: make failed batch reacquires ordered, side-effect-free, quiet
At endgame the only remaining work is a handful of highly-shared ref
clusters, and every planner task restart wakes the whole post-exec
waiter herd of the task that finished. Each woken task re-ran the
batch reacquire, taking memoized locks one by one until one failed,
then releasing them all — knocking over the locks the other woken
tasks were about to succeed on. Two tasks whose two-lock memos
overlap in opposite orders can ping-pong this way for minutes,
re-waking their full waiter sets on every iteration. A full-corpus
run logged 4.36M "memo reacquire contention" lines (64% of the log)
in a 13-minute tail during which the dedupe rate collapsed and CPU
burned on retry churn and log formatting.
Three changes to the batch reacquire in the Borrower constructor:
- Sort the memoized set into canonical (pointer) order, so tasks with
overlapping memos contend on their lowest common Exclusion first,
forming one queue on one lock instead of a swap cycle.
- Peek the whole set with Exclusion::locked_by_another before
acquiring anything. A wakeup that cannot proceed now touches only
the failing Exclusion (to establish the restart rendezvous) and
disturbs no other locks.
- Rate-limit the contention log line to 10/s. The
borrower_dead_reacquire counter is unconditional as before; the
pass-2 race path logs under its own limiter as "memo reacquire
race" so the two causes stay distinguishable.
projects / bees / commit