bees: separate BEES_MAX_EXTENT_TASK_COUNT from BEES_MAX_EXTENT_REF_COUNT

They are both roughly the right number, but the number of tasks is not the
same as the number of refs:

 * in extent scan mode, each extent Task has all of the extent's refs
 * in subvol scan mode, each Task handles inodes, not extents or refs

Making the extent task count smaller reduces parallel and out-of-order
task execution.  This helps prevent a wide gap from forming between the
current progress checkpoint (lowest extent in the queue) and the head
of the crawl (highest extent in the queue).  This makes restarts repeat
less work, which may help make forward progress if bees is run for very
short intervals between restarts, but it will make the scans take longer
due to more sequential processing.

Signed-off-by: Zygo Blaxell <bees@furryterror.org>

diff --git a/src/bees-roots.cc b/src/bees-roots.cc
index 9b8ce68946481f073f055bab1154cb2c6eeb1986..2bbd16067f6f42db91feb8c3f01742f910c9f86c 100644 (file)
--- a/src/bees-roots.cc
+++ b/src/bees-roots.cc
@@ -697,7 +697,7 @@ should_throttle()
        // If there's too many entries in the queue, stop adding new ones until workers catch up
        // If there's not too many entries in the queue, restart the scan task
        const auto instance_count = Task::instance_count();
-       const auto instance_limit = BEES_MAX_EXTENT_REF_COUNT;
+       const auto instance_limit = BEES_MAX_EXTENT_TASK_COUNT;
        // Add some hysteresis so that we aren't constantly flipping throttle on and off
        const bool queue_empty = s_throttled && instance_count < instance_limit * .90;
        const bool queue_full = !s_throttled && instance_count > instance_limit * .99;

diff --git a/src/bees.h b/src/bees.h
index 1bbb00982b6cfe7366c2cb569996b662592457fd..e1b8db3cdb79c83501cb618ddf929d4aa4ce03f8 100644 (file)
--- a/src/bees.h
+++ b/src/bees.h
@@ -97,7 +97,13 @@ const double BEES_TOXIC_SYS_DURATION = 5.0;
 
 // Maximum number of refs to a single extent before we have other problems
 // If we have more than 10K refs to an extent, adding another will save 0.01% space
-const size_t BEES_MAX_EXTENT_REF_COUNT = 9999; // (16 * 1024 * 1024 / 24);
+// The kernel limit is (16 * 1024 * 1024 / 24), but this is far too large for practical dedupe.
+const size_t BEES_MAX_EXTENT_REF_COUNT = 9999;
+
+// Maximum number of extent tasks in the in-memory queue
+// More tasks = more memory, but also more parallel execution
+// Fewer tasks = less queue delay, so restart after shutdown repeats less work
+const size_t BEES_MAX_EXTENT_TASK_COUNT = 9999;
 
 // How long between hash table histograms
 const double BEES_HASH_TABLE_ANALYZE_INTERVAL = BEES_STATS_INTERVAL;