roots: name the temp files behind a temp_transid crawler hold

An integration run livelocked in the endgame: the 128K crawler sat
held below a temp file transid for hours with an empty work queue and
all workers idle.  The hold-release path is already in place -- the
tempfile pool's checkin handler calls reset(), which erases the
context's temp_transid entry when the pool handle is returned -- so a
persistent hold with idle workers means some live reference (a
BeesTempExtent holds a copy of the pool handle) kept the temp file
checked out of the pool, and the entry could never be erased.  No new
dedupe work will ever arrive to reset it: the crawler waits on the
temp file, and the temp file waits on work the crawler cannot
produce.

The log did not identify which temp file held the crawler or what
retained it.  Add BeesContext::temp_transid_report() and log the held
(fid, transid) pairs, rate limited to once a minute, when the hold
actually binds the crawl window.  Debug level: a hold is normal while
dedupe is in flight and needs no operator action; the report exists
so the retainer of a stuck temp file can be identified from the log
the next time the endgame livelock appears.

Assisted-by: Claude-Code:claude-fable-5

diff --git a/src/bees-context.cc b/src/bees-context.cc
index f2405de635c34a04b0297bd5cf7574ee62e0d2ab..fcbff5a4b52aa188b2755b6c578c473b14ddecc1 100644 (file)
--- a/src/bees-context.cc
+++ b/src/bees-context.cc
@@ -1778,6 +1778,17 @@ BeesContext::temp_transid_erase(const BeesFileId &fid)
        m_temp_transid.erase(fid);
 }
 
+string
+BeesContext::temp_transid_report()
+{
+       ostringstream oss;
+       unique_lock<mutex> lock(m_temp_transid_mutex);
+       for (const auto &[fid, transid] : m_temp_transid) {
+               oss << " " << fid << "@" << transid;
+       }
+       return oss.str();
+}
+
 uint64_t
 BeesContext::temp_transid_min()
 {

diff --git a/src/bees-roots.cc b/src/bees-roots.cc
index c3012b6dbf449c5aa1d22bc6cd89f0ca1be40a77..182ab81b1d7ffcdc71c96e1fd72c657771968ff4 100644 (file)
--- a/src/bees-roots.cc
+++ b/src/bees-roots.cc
@@ -1935,6 +1935,19 @@ BeesRoots::effective_transid_max()
                if (limit < rv) {
                        rv = limit;
                        BEESCOUNT(crawl_temp_transid_hold);
+                       // A hold is normal while dedupe is in flight, but a
+                       // hold that persists with idle workers means a temp
+                       // file was never returned to its pool (something still
+                       // holds its BeesTempExtent).  Name the holders, rate
+                       // limited, so the retainer can be identified from the
+                       // log.  Debug level: bees handles the hold itself;
+                       // operators do not need to act on it.
+                       static RateLimiter s_hold_report_limit(1.0 / 60);
+                       if (s_hold_report_limit.is_ready()) {
+                               BEESLOGDEBUG("crawl held below transid "
+                                       << temp_floor << " by temp file(s):"
+                                       << m_ctx->temp_transid_report());
+                       }
                }
        }
        return rv;

diff --git a/src/bees.h b/src/bees.h
index dff60a8344ed85078b6ca9b78011f839490cc314..00914855356e06ea358817be277bdbf31332450e 100644 (file)
--- a/src/bees.h
+++ b/src/bees.h
@@ -1803,6 +1803,10 @@ public:
        void temp_transid_erase(const BeesFileId &fid);
        /// Earliest transid among live temp files, or 0 when none hold data.
        uint64_t temp_transid_min();
+       /// Describe the registered temp-file transid holds (fid and
+       /// transid of each entry) for hold diagnostics.  Empty string
+       /// when no entries are registered.
+       std::string temp_transid_report();
 
        /// Return the per-inode Exclusion mutex for @p inode (creates if absent).
        shared_ptr<Exclusion> get_inode_mutex(uint64_t inode);