From: Zygo Blaxell Date: Sat, 26 Jul 2025 05:42:01 +0000 (-0400) Subject: bees: separate BEES_MAX_EXTENT_TASK_COUNT from BEES_MAX_EXTENT_REF_COUNT X-Git-Url: http://git.hungrycats.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=7eed58f04a09666c035b4bdf2d8d604bd916ca9d;p=bees bees: separate BEES_MAX_EXTENT_TASK_COUNT from BEES_MAX_EXTENT_REF_COUNT They are both roughly the right number, but the number of tasks is not the same as the number of refs: * in extent scan mode, each extent Task has all of the extent's refs * in subvol scan mode, each Task handles inodes, not extents or refs Making the extent task count smaller reduces parallel and out-of-order task execution. This helps prevent a wide gap from forming between the current progress checkpoint (lowest extent in the queue) and the head of the crawl (highest extent in the queue). This makes restarts repeat less work, which may help make forward progress if bees is run for very short intervals between restarts, but it will make the scans take longer due to more sequential processing. Signed-off-by: Zygo Blaxell --- diff --git a/src/bees-roots.cc b/src/bees-roots.cc index 9b8ce689..2bbd1606 100644 --- a/src/bees-roots.cc +++ b/src/bees-roots.cc @@ -697,7 +697,7 @@ should_throttle() // If there's too many entries in the queue, stop adding new ones until workers catch up // If there's not too many entries in the queue, restart the scan task const auto instance_count = Task::instance_count(); - const auto instance_limit = BEES_MAX_EXTENT_REF_COUNT; + const auto instance_limit = BEES_MAX_EXTENT_TASK_COUNT; // Add some hysteresis so that we aren't constantly flipping throttle on and off const bool queue_empty = s_throttled && instance_count < instance_limit * .90; const bool queue_full = !s_throttled && instance_count > instance_limit * .99; diff --git a/src/bees.h b/src/bees.h index 1bbb0098..e1b8db3c 100644 --- a/src/bees.h +++ b/src/bees.h @@ -97,7 +97,13 @@ const double BEES_TOXIC_SYS_DURATION = 5.0; // Maximum number of refs to a single extent before we have other problems // If we have more than 10K refs to an extent, adding another will save 0.01% space -const size_t BEES_MAX_EXTENT_REF_COUNT = 9999; // (16 * 1024 * 1024 / 24); +// The kernel limit is (16 * 1024 * 1024 / 24), but this is far too large for practical dedupe. +const size_t BEES_MAX_EXTENT_REF_COUNT = 9999; + +// Maximum number of extent tasks in the in-memory queue +// More tasks = more memory, but also more parallel execution +// Fewer tasks = less queue delay, so restart after shutdown repeats less work +const size_t BEES_MAX_EXTENT_TASK_COUNT = 9999; // How long between hash table histograms const double BEES_HASH_TABLE_ANALYZE_INTERVAL = BEES_STATS_INTERVAL;