* Default: `100`
* Accepts a non-negative integer or `unlimited`.
+* **`refs-max`**
+ Maximum number of references a single extent may accumulate before bees
+ stops adding more. An extent that already holds this many references is not
+ used as a dedupe source or destination, so no extent grows past the limit.
+ Beyond a few thousand references the per-reference space saving becomes
+ negligible (well under 0.01%), while `LOGICAL_INO` resolution of that extent
+ grows more expensive on every future scan.
+ * Default: `9999`.
+ * Accepts a non-negative integer or `unlimited`. `unlimited` resolves to the
+ kernel ioctl ceiling: the `LOGICAL_INO` result buffer is capped at 16 MiB,
+ which at 24 bytes per reference holds roughly 699050 references — the hard
+ maximum bees can resolve for one extent, and the effective cap on this key.
+ * Raising this enlarges the kernel buffer allocated for every `LOGICAL_INO`
+ call; very large allocations can disturb the kernel's balancing of page
+ cache against swap, so increase it with care.
+ * Exposed to filters as the `${REFS_MAX}` substitution variable: the filter
+ `ref-count-max` key defaults to this value, and `unlimited` in a filter
+ means this value (not the kernel ceiling).
+
* **`free-min`**
Minimum portion of an extent that must be freed before bees accepts a
rewrite plan.
* **`ref-count-min`** / **`ref-count-max`**
Match by the number of logical references to the extent. Requires a `LOGICAL_INO` ioctl per extent.
If `LOGICAL_INO` returns no references, the extent is rejected immediately.
- Defaults: `0` / `9999`.
+ Defaults: `0` / `${REFS_MAX}` (the configured [`rewrite.refs-max`](#rewrite));
+ `unlimited` for `ref-count-max` also means that value, not the kernel ceiling.
#### Reference-derived conditions
# Integer count, or unlimited.
total-max = 100
+ # Maximum references a single extent may accumulate before bees stops
+ # adding more. Integer count, or unlimited (= the kernel ioctl ceiling).
+ # Exposed to filters as ${REFS_MAX}.
+ # See docs/config-file.md for the ref-count rationale.
+ refs-max = 9999
+
# Minimum portion of extent which must be freed.
# Percentage with %, or a size value (e.g. 128K, 1M - 4K).
free-min = 50%
# Minimum number of logical references to the extent.
ref-count-min = 0
- # Maximum number of logical references to the extent.
- ref-count-max = 9999
+ # Maximum number of logical references to the extent. Defaults to the
+ # configured rewrite.refs-max; "unlimited" here means the same value
+ # (not the kernel ceiling).
+ ref-count-max = ${REFS_MAX}
# Minimum physical (compressed) size of the extent in bytes.
# Size value: digits with optional K/M/G/T/P/E suffix, and
// return 0 when it cannot detect the count; clamp to at least 1.
insert_map_unique(m_subst_map, "NPROC", to_string(max(1u, thread::hardware_concurrency())));
+ // ${REFS_MAX} is not a system fact: it mirrors the effective
+ // rewrite.refs-max and is populated later, in set_argv(), once the config
+ // is merged (like THREAD_FACTOR).
+
// THREAD_FACTOR is not a system fact: it derives from the merged
// thread.thread-factor config key and is therefore populated later, in
// set_argv(), once all config layers are available.
// (and on REPL apply-config) without spamming the log on every hot-path
// lookup.
load_rewrite_policy();
+
+ // Expose the effective rewrite.refs-max as ${REFS_MAX} so filter ref-count
+ // values default to — and "unlimited" in a filter resolves to — the same
+ // ceiling. Derived from merged config like THREAD_FACTOR, so it is set
+ // here rather than in the constructor.
+ insert_map_unique(m_subst_map, "REFS_MAX", to_string(m_rewrite_policy.m_refs_max));
}
void
// yes/no for consistency with the embedded YAML-style config.
auto load = [&](const string &key, auto parser) {
BEESTRACE("loading " << key);
- auto value = get(key, parser);
+ // Expand ${VAR} before parsing so rewrite.* values can reference
+ // substitution variables (e.g. refs-max = ${REFS_MAX}); the typed
+ // get() reads the raw Innie value, so subst() must be applied here.
+ auto value = get(key, [&](const string &raw) { return parser(subst(raw)); });
if constexpr (std::is_same_v<decltype(value), bool>) {
BEESLOGINFO(key << " = " << (value ? "yes" : "no") << " [" << key << "]");
} else {
rv.m_copy_max = load("rewrite.copy-max", bees_parse_count);
rv.m_hole_max = load("rewrite.hole-max", bees_parse_count);
rv.m_total_max = load("rewrite.total-max", bees_parse_count);
+ // Clamp to the ioctl's hard ceiling: "unlimited" (and any oversized value)
+ // means "as many as the 16 MiB LOGICAL_INO buffer can hold".
+ rv.m_refs_max = min<uint64_t>(load("rewrite.refs-max", bees_parse_count), BEES_MAX_EXTENT_REF_COUNT);
rv.m_candidate_max_count = load("rewrite.candidate-max-count", bees_parse_count);
rv.m_candidate_max_bytes = load("rewrite.candidate-max-bytes", bees_parse_size);
BEESLOGINFO("rewrite.candidate-max-bytes = " << pretty(rv.m_candidate_max_bytes) << " [rewrite.candidate-max-bytes]");
.length = e.size(),
};
static set<BeesSeenRange> s_seen;
- if (s_seen.size() > BEES_MAX_EXTENT_REF_COUNT) {
+ if (s_seen.size() > m_ctx->refs_max()) {
s_seen.clear();
BEESCOUNT(scan_seen_clear);
}
BEESLOGDEBUG("LOGICAL_INO returned 0 refs at " << to_hex(addr));
BEESCOUNT(resolve_empty);
}
- if (rv_count < BEES_MAX_EXTENT_REF_COUNT) {
+ if (rv_count < refs_max()) {
rv.m_biors = vector<BtrfsInodeOffsetRoot>(log_ino.m_iors.begin(), log_ino.m_iors.end());
} else {
- BEESLOGINFO("addr " << addr << " refs " << rv_count << " overflows configured ref limit " << BEES_MAX_EXTENT_REF_COUNT);
+ BEESLOGINFO("addr " << addr << " refs " << rv_count << " overflows configured ref limit " << refs_max());
BEESCOUNT(resolve_overflow);
}
});
});
}
- m_logical_ino_pool.generator([]() {
+ m_logical_ino_pool.generator([this]() {
const auto extent_ref_size = sizeof(uint64_t) * 3;
- return make_shared<BtrfsIoctlLogicalInoArgs>(0, BEES_MAX_EXTENT_REF_COUNT * extent_ref_size + sizeof(btrfs_data_container));
+ // Buffer capacity is the hard compiled ceiling; the configured
+ // refs_max() (a soft policy limit that may be "unlimited") is clamped
+ // to it so it can never overflow the allocation. Evaluated lazily, so
+ // the config is present by the time the pool first produces a buffer.
+ const auto refs_cap = min(refs_max(), BEES_MAX_EXTENT_REF_COUNT);
+ return make_shared<BtrfsIoctlLogicalInoArgs>(0, refs_cap * extent_ref_size + sizeof(btrfs_data_container));
});
m_config = make_shared<BeesConfig>(m_root_path, m_root_fd);
return m_config->rewrite_policy();
}
+size_t
+BeesContext::refs_max() const
+{
+ return get_rewrite_policy().m_refs_max;
+}
+
void
BeesVerificationTracker::on_transid_change(uint64_t transid, shared_ptr<BeesContext> ctx)
{
});
});
}
- m_logical_ino_pool.generator([]() {
+ m_logical_ino_pool.generator([this]() {
const auto extent_ref_size = sizeof(uint64_t) * 3;
- return make_shared<BtrfsIoctlLogicalInoArgs>(0, BEES_MAX_EXTENT_REF_COUNT * extent_ref_size + sizeof(btrfs_data_container));
+ // Buffer capacity is the hard compiled ceiling; the configured
+ // refs_max() (a soft policy limit that may be "unlimited") is clamped
+ // to it so it can never overflow the allocation. Evaluated lazily, so
+ // the config is present by the time the pool first produces a buffer.
+ const auto refs_cap = min(refs_max(), BEES_MAX_EXTENT_REF_COUNT);
+ return make_shared<BtrfsIoctlLogicalInoArgs>(0, refs_cap * extent_ref_size + sizeof(btrfs_data_container));
});
// Parse [hash.NAME] sections and wire the primary domain's hash
BeesFilter::BeesFilter(const shared_ptr<BeesContext> &ctx)
: m_ctx(ctx)
{
- parse_config(ctx->get_config().innie());
+ const auto &cfg = ctx->get_config();
+ parse_config(cfg.innie(), [&cfg](const string &s) { return cfg.subst(s); });
validate();
}
/// Parse one [filter.NAME] section from @p cfg into a BeesFilterRule.
/// The wildcard section "filter.*" provides defaults for each key.
+// Substitutor applied to ${VAR}-bearing filter values. See parse_config.
+using SubstFn = function<string(const string &)>;
+
static BeesFilterRule
-parse_rule(const string &name, const Innie &cfg)
+parse_rule(const string &name, const Innie &cfg, const SubstFn &subst)
{
// Helper: read a key, falling back via the Innie fallback chain.
const auto get = [&](const string &key) {
}
// ── ref-count (LOGICAL_INO ioctl, loads ref list) ─────────────────────
- const auto rc_min = stoull(get("ref-count-min"));
- const auto rc_max = stoull(get("ref-count-max"));
+ // ${VAR} is expanded before parsing (so ref-count-max = ${REFS_MAX}
+ // resolves). "unlimited" in a filter means the configured rewrite.refs-max
+ // — i.e. ${REFS_MAX} — not the kernel ceiling.
+ const auto parse_rc = [&](const string &raw) -> uint64_t {
+ string s = subst(raw);
+ if (s == "unlimited") s = subst("${REFS_MAX}");
+ return stoull(s);
+ };
+ const auto rc_min = parse_rc(get("ref-count-min"));
+ const auto rc_max = parse_rc(get("ref-count-max"));
uint64_t default_rc_max = BEES_MAX_EXTENT_REF_COUNT;
try {
- default_rc_max = stoull(cfg.get("filter.*", "ref-count-max"));
+ default_rc_max = parse_rc(cfg.get("filter.*", "ref-count-max"));
} catch (const Innie::NotFoundException &) {
// Fallback to compiled-in limit if filter.* is missing in test config
}
// ─── BeesFilter::parse_config ────────────────────────────────────────────────
void
-BeesFilter::parse_config(const Innie &cfg)
+BeesFilter::parse_config(const Innie &cfg, const SubstFn &subst)
{
// sections() returns all content-bearing descendants (compound names for
// sections nested under namespace-only parents, e.g. "common.datacow"
// when [filter.common] has no keys but [filter.common.datacow] does).
for (const auto &name : cfg.sections("filter")) {
if (name == "*") continue;
- m_rules.push_back(parse_rule(name, cfg));
+ m_rules.push_back(parse_rule(name, cfg, subst));
}
}
BeesFilter::BeesFilter(const Innie &cfg)
: m_ctx()
{
- parse_config(cfg);
+ // Test/standalone path: no BeesConfig, so ${VAR} substitution is identity.
+ // Such configs must supply literal ref-count values.
+ parse_config(cfg, [](const string &s) { return s; });
validate();
}
list<BeesFilterRule> m_rules; ///< Stable storage; never reallocated.
map<string, shared_ptr<BeesFilterChain>> m_chains; ///< Resolved chain cache.
- /// Parse all [filter.*] sections from @p config into m_rules.
- void parse_config(const Innie &config);
+ /// Parse all [filter.*] sections from @p config into m_rules. @p subst
+ /// expands ${VAR} in the numeric ref-count values (so ref-count-max =
+ /// ${REFS_MAX} resolves); production passes BeesConfig::subst, the
+ /// Innie-only test ctor passes identity. It is deliberately not applied to
+ /// free-text values (e.g. name-pattern regexes, which may legally contain a
+ /// lone '$').
+ void parse_config(const Innie &config,
+ const std::function<std::string(const std::string &)> &subst);
/// Validate the rule graph (cycles, dangling refs, enum values).
/// @throws std::runtime_error on any error.
void validate();
BEESLOGDEBUG("LOGICAL_INO returned 0 refs for " << len << " bytes (" << pretty(len) << ") at " << to_hex(bytenr));
BEESCOUNT(extent_zero);
return;
- } else if (rv_count >= BEES_MAX_EXTENT_REF_COUNT) {
- // If we find any duplicates when there are BEES_MAX_EXTENT_REF_COUNT references, then
- // we'll end up with some extent with at least BEES_MAX_EXTENT_REF_COUNT + 1 references.
+ } else if (rv_count >= m_ctx->refs_max()) {
+ // If we find any duplicates when there are refs_max() references, then
+ // we'll end up with some extent with at least refs_max() + 1 references.
// That's too many, so don't let that happen.
- BEESLOGINFO("bytenr " << to_hex(bytenr) << " refs " << rv_count << " overflows configured ref limit " << BEES_MAX_EXTENT_REF_COUNT);
+ BEESLOGINFO("bytenr " << to_hex(bytenr) << " refs " << rv_count << " overflows configured ref limit " << m_ctx->refs_max());
BEESCOUNT(extent_overflow);
return;
}
// Don't add more links to a src that already has too many
{
auto &layer = Borrower::current().layer();
- if (candidate.refs(layer)->size() >= BEES_MAX_EXTENT_REF_COUNT) {
+ if (candidate.refs(layer)->size() >= m_policy.m_refs_max) {
return;
}
}
{
auto &layer = Borrower::current().layer();
- if (candidate.refs(layer)->size() >= BEES_MAX_EXTENT_REF_COUNT) {
+ if (candidate.refs(layer)->size() >= m_policy.m_refs_max) {
return;
}
}
BEESTRACE("scan_next accept_dst");
m_start_accept_dst = m_filter_cache->accept_dst(m_start);
if (m_start_accept_dst) {
- if (m_start.refs(layer)->size() >= BEES_MAX_EXTENT_REF_COUNT) {
+ if (m_start.refs(layer)->size() >= m_rewrite_policy.m_refs_max) {
m_start_accept_dst = false;
BEESLOGC(INFO, Plan, "scan_next reject truncated refs list for dst "
<< to_hex(m_start.bytenr()) << " (refs size: " << m_start.refs(layer)->size() << ")");
/// marked toxic and skipped to avoid performance degradation.
const double BEES_TOXIC_SYS_DURATION = 5.0;
-/// Maximum number of references to a single extent before bees stops adding more.
-/// Beyond this the per-reference space saving becomes negligible (<0.01%).
-/// The kernel limit is (16 MiB buffer) / (24 bytes per (root, ino, offset) tuple)
-/// = 16 * 1024 * 1024 / (3 * 8) = 699050, but that is far too large for practical
-/// use: memory consumption and performance degrade severely well before that point.
-const size_t BEES_MAX_EXTENT_REF_COUNT = 9999;
+/// Hard ceiling on how many references to a single extent bees can resolve: the
+/// LOGICAL_INO ioctl returns them into a result buffer the kernel caps at 16 MiB,
+/// which at 24 bytes per (root, ino, offset) tuple holds this many (~699050).
+/// This bounds the LOGICAL_INO buffer size and is the maximum value of
+/// rewrite.refs-max ("unlimited" in rewrite context resolves to exactly this).
+///
+/// It is NOT the default: the practical default reference limit (9999) lives in
+/// the config (rewrite.refs-max), and runtime code uses the configured value
+/// (BeesRewritePolicy::m_refs_max / BeesContext::refs_max()), not this constant —
+/// which is only the buffer capacity and the clamp ceiling for that value.
+const size_t BEES_MAX_EXTENT_REF_COUNT =
+ (16 * 1024 * 1024 - sizeof(btrfs_data_container)) / (sizeof(uint64_t) * 3);
/// Number of bytes to prefetch ahead during scanning to improve sequential read performance.
const size_t BEES_READAHEAD_SIZE = 1024 * 1024;
uint64_t m_copy_max = 100;
uint64_t m_hole_max = 100;
uint64_t m_total_max = 100;
+ /// Maximum references a single extent may accumulate before bees stops
+ /// adding more (rewrite.refs-max; default 9999, clamped to the ioctl hard
+ /// ceiling BEES_MAX_EXTENT_REF_COUNT, which "unlimited" resolves to).
+ /// Sentinel 0 = not loaded (a loader bypass rejects every extent).
+ uint64_t m_refs_max = 0;
/// Maximum number of candidate extents per destination extent.
/// Soft cap that prevents runaway plans driven by many tiny
/// extents. Paired with m_candidate_max_bytes (a soft cap on
/// next set_argv() (e.g. REPL apply-config) runs.
const BeesRewritePolicy &get_rewrite_policy() const;
+ /// Configured maximum references per extent (rewrite.refs-max). Convenience
+ /// for the non-planner call sites; equivalent to get_rewrite_policy().m_refs_max.
+ size_t refs_max() const;
+
/// Return the per-extent Exclusion mutex for @p bytenr (creates if absent).
shared_ptr<Exclusion> get_extent_mutex(uint64_t bytenr);
projects / bees / commitdiff