From: Zygo Blaxell Date: Wed, 22 Apr 2026 05:12:02 +0000 (-0400) Subject: docs: add scan_one extent matching explainer X-Git-Url: http://git.hungrycats.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=ac4516a9e508b0f9a9b5d42bf1e7c0b2cad7f373;p=bees docs: add scan_one extent matching explainer Add docs/scan-one-algorithm.md describing the v0.11 scan_one algorithm for bees users — how the hash table size, data extent size, 50% must-free threshold, and random-insertion LRU interact in practice — and link it from README.md and docs/index.md. Co-Authored-By: Claude Opus 4.7 --- diff --git a/README.md b/README.md index ddaea7d6..fbcb0f6d 100644 --- a/README.md +++ b/README.md @@ -54,6 +54,7 @@ More Information ---------------- * [How bees works](docs/how-it-works.md) + * [How scan_one matches extents](docs/scan-one-algorithm.md) * [Missing bees features](docs/missing.md) * [Event counter descriptions](docs/event-counters.md) diff --git a/docs/index.md b/docs/index.md index e97a0f31..e2fdd857 100644 --- a/docs/index.md +++ b/docs/index.md @@ -54,6 +54,7 @@ More Information ---------------- * [How bees works](how-it-works.md) + * [How scan_one matches extents](scan-one-algorithm.md) * [Missing bees features](missing.md) * [Event counter descriptions](event-counters.md) diff --git a/docs/scan-one-algorithm.md b/docs/scan-one-algorithm.md new file mode 100644 index 00000000..ab15b19c --- /dev/null +++ b/docs/scan-one-algorithm.md @@ -0,0 +1,289 @@ +How `scan_one` Matches Extents +============================== + +The `scan_one` algorithm is the core of how bees (v0.11 and earlier) +decides whether two btrfs extents share data that can be replaced with +a reference to a single copy. This page explains the matching process +and, in particular, how the key sizing and eviction parameters affect +whether any two duplicate extents actually get deduplicated: + + * [hash table size](#hash-table-size) + * [data extent size](#data-extent-size) + * [the 50% must-free threshold](#the-50-must-free-threshold) + * [the random-insertion-order hash LRU](#random-insertion-order-hash-lru) + * [the 256 cells per hash bucket](#256-cells-per-bucket) + +This is intended for users who are tuning bees or trying to understand +why some duplicates are found and others are not. For a shorter, +higher-level description of the daemon, see [how bees works](how-it-works.md). + +The `scan_one` Algorithm, in Brief +---------------------------------- + +For each btrfs data extent bees encounters during a scan: + + 1. **Read and hash the extent**, one 4 KiB block at a time. All-zero + blocks are recognized and set aside as "punch a hole" candidates + rather than hashed for dedupe. + + 2. **For each non-zero block, look up the hash in the hash table.** + A hit means some previously-scanned block on the filesystem had + the same hash. + + 3. **For each hit, ask the kernel where that block lives** (via the + `LOGICAL_INO` ioctl) and try to **extend the match** — see how + far forward and backward from the hit the two extents agree. + A single matching block in the new extent can anchor a much + longer run of adjacent duplicate blocks. + + 4. **Pick the longest non-overlapping matched ranges** and check the + nuisance-dedupe filter: the number of bytes that would be *freed* + (by dedupe plus zero-punching) must be at least the number of bytes + that would have to be *rewritten* — the non-matching parts of the + new extent, whose references have to be relocated off the original + extent so that no reference to any part of the original remains + and btrfs can free it. If not, the dedupe is skipped. + + 5. **Perform the dedupes** via the `FILE_EXTENT_SAME` ioctl. + + 6. **Update the hash table.** Hashes that produced a dedupe match + are moved to the front of their LRU list. Hashes from the new + extent that were not deduped are inserted at a random position + in their bucket. + +A duplicate extent is only eliminated when steps 2, 3, and 4 all +succeed. Each parameter below controls one of those steps. + +When Does Dedupe Actually Happen? +--------------------------------- + +Two extents `A` (older, already scanned) and `B` (newer, being scanned +now) get deduplicated only if **all three** of the following are true: + + * **Gate A — at least one hash from `A` is still in the hash table** + when `B` is scanned. + * **Gate B — at least one block of `B` collides with a retained hash + from `A`.** + * **Gate C — the extension pass finds enough contiguous matching + data to pass the nuisance-dedupe filter.** + +Most tuning questions are really questions about one of these three +gates. The rest of this page maps each parameter onto the gate (or +gates) it controls. + +Hash Table Size +--------------- + +The hash table holds a fixed number of (hash, physical-address) pairs, +one pair per 16 bytes of table. A 1 GB hash table holds about 64 M +pairs — one per 4 KiB block it can remember. + +**Effect on Gate A.** Hash table size is the dominant control on how +long a scanned block's hash survives in the table. The intuition: if +your filesystem has much more unique-block data than the table can hold, +each slot is overwritten on average after a number of new inserts equal +to the number of slots, so older data drops out first. + +This is why the bees rule of thumb is "1 GB of hash table per 10 TB of +unique data": it keeps the expected survival time of a hash long enough +to overlap with the scan period for most real workloads, where +duplicates tend to appear relatively near each other in time. + +Under-provisioning the hash table does **not** prevent dedupe — it +reduces the dedupe *rate*, typically most on the oldest data. + +Data Extent Size +---------------- + +btrfs extents can be up to 128 MiB for uncompressed data (128KiB +for compressed data). Extent size affects two gates. + +**Effect on Gate B.** Gate B asks whether *any* block of the new +extent hits a retained hash. If an extent has `n` blocks and each +block has independent probability `p` of having its hash retained, +then the probability of at least one hit is + + P(hit) = 1 − (1 − p)^n + +Larger extents give bees more independent "shots" at finding an +anchor block. A 16-block extent (64 KiB) and a 256-block extent +(1 MiB) behave very differently, even with the same hash table. + +This is why [how bees works](how-it-works.md) says most of the hashes +in the hash table are redundant: you don't need *every* block of an +extent retained, you only need one, because the extension pass in +step 3 will find the rest. + +**Effect on Gate C.** Larger extents also raise the bar for the +nuisance filter: to pass, more of the extent has to dedupe. A small +extent is almost always either fully duplicate or fully unique; a +large extent is more likely to have only a fraction of its content +match. + +Net effect: medium-to-large extents are the sweet spot for `scan_one`. +Very small extents don't benefit from the multi-shot retention. Very +large, partially-duplicate extents get skipped by the nuisance filter. + +The 50% Must-Free Threshold +--------------------------- + +btrfs cannot free part of an extent — an extent is allocated and +freed as a whole. As long as *any* file anywhere in the filesystem +references *any* part of an extent, the whole extent stays +allocated, including blocks that no file references. Recovering +the space held by an extent therefore means eliminating every +reference to every part of it. + +For the matching parts of the new extent, `FILE_EXTENT_SAME` +redirects their references to the duplicate data elsewhere. For +the non-matching parts, bees rewrites the data into new +allocations, so those file offsets stop pointing at the original +extent. Once no reference to the original extent remains anywhere, +btrfs frees it. + +The rewritten data does not persist as a duplicate — the original +extent is freed, and the rewrite replaces it at those offsets — but +the rewrite still costs I/O, allocator churn, and extent +fragmentation, and those costs have to be paid back in recovered +free space for the dedupe to be worth doing. + +bees enforces the rule: + + bytes_freed ≥ bytes_rewritten + +where `bytes_freed = bytes_deduped + bytes_zero_punched` and +`bytes_rewritten` is the size of the non-matching data that must be +rewritten to free the original extent. + +For an extent with no zero blocks, this reduces to the well-known +**"at least 50% of the extent must dedupe"** threshold. When zero +blocks are present — for example, a mostly-empty VM disk image — the +threshold is easier to pass, because punching holes also frees space. + +**Effect on Gate C.** This is the gate itself. Extents that match +only sparsely (say, 20% shared and 80% unique) are skipped entirely; +the hashes from the unique part are still inserted into the table, +so a future extent that shares more with them can still succeed later. + +Random-Insertion-Order Hash LRU +------------------------------- + +Each bucket of the hash table is an LRU list of 256 cells. bees uses +two distinct insertion policies: + + * **Move to front on hit.** When a hash lookup finds a cell that + produces a successful dedupe match, that cell is moved to slot 0. + * **Random position on unique insert.** When a new, previously-unseen + hash is inserted, it goes into a *uniformly random* slot in its + bucket. If the bucket is full, the cell at the tail (slot 255) is + evicted, and the cells between the insertion point and the tail are + shifted back by one. + +**Effect on Gate A.** This is the subtlest of the parameters. +Compared to a plain FIFO (where every cell would live exactly as long +as the bucket depth before eviction), random insertion gives a +*fat-tailed* survival distribution. A cell that happens to be inserted +near the front of the bucket has a much longer expected lifetime than +one inserted near the back, and that difference is amplified by the +move-to-front promotion: any hash that ever matches gets effectively +pinned near the front. + +The practical consequence: bees keeps "useful" hashes (ones that match +real duplicates) much longer than "unlucky" hashes (ones that never +match), without needing an expensive explicit usage counter. It trades +a small probability of missing a rare duplicate for much better +retention of common patterns. + +256 Cells Per Bucket +-------------------- + +Each 4 KiB page of the hash table file is one bucket of 256 cells. +This size is set by the mmap page size and cell width, not a tunable +parameter, but it has two effects worth understanding. + +**Effect on Gate A (collision capacity).** Different physical blocks +with colliding hashes can all coexist in the same bucket, up to 256 of +them. This matters for widely replicated content such as boilerplate +file headers and common fill patterns: bees can keep many addresses +for the "same" hash without any one of them crowding out the others. + +**Effect on Gate A (LRU depth).** A 256-slot LRU is a meaningful +buffer against eviction. If buckets were shallow (say, 4 cells), only +the very-most-recent or very-popular hashes would survive, and +rare-but-real duplicates would almost always miss. The 256-deep +bucket and the random-insert policy together are what let bees find +old and rare duplicates at a useful rate. + +How the Parameters Interact +--------------------------- + +A few interactions are worth singling out: + + * **Hash table size and extent size compound.** If the hash table + is marginal, small extents suffer disproportionately: their Gate B + probability `1 − (1 − p)^n` falls quickly with small `n`. Doubling + the hash table helps small extents more than large ones. + + * **Extent size fights the 50% threshold.** Increasing extent size + makes Gate B easier but Gate C harder. Which effect dominates + depends on the file's *content pattern*, not on btrfs: + + * Large, sequentially-written files (media files, log archives, + installation images) tend to contain long runs where every + block is unique *within* the file, but the whole sequence + recurs whenever the file is duplicated. Match rates of 100% + are common, so the 50% threshold is met with room to spare. + * Files written in many small writes in random order (for + example, databases) tend to contain a mixture of unique and + duplicate blocks in every extent. The matched fraction can + be non-trivial, but the rewrite cost makes the dedupe + unprofitable, and Gate C filters the extent out. + * VM disk images mix both patterns — contiguous regions that + behave like media files and churn regions that behave like + databases — and produce both outcomes depending on the extent. + + The 50% filter lets bees cherry-pick the extents that are worth + deduping and skip the ones that would just burn time and metadata, + without needing to know anything about file type or content + semantics. + + * **Move-to-front is what keeps the hash table effective in steady + state.** Without it, even a well-sized table would slowly turn + over and forget the data that is actually being deduped. With it, + the table quickly learns which hashes are worth keeping and holds + onto them, regardless of how much non-duplicate traffic is flowing + through. + +Practical Tuning Implications +----------------------------- + + * If you are missing dedupes you expect to see, the most likely cause + is hash table size (Gate A) or extent layout (Gates B and C). + + * The nuisance filter is intentional and generally should not be + worked around: dedupes that rewrite more than they recover + still gain some free space (the original extent is freed either + way), but the ratio of recovered space to I/O, allocator churn, + and metadata cost is too low to be worth doing. + + * The random-insertion LRU combined with move-to-front on match + biases the hash table toward the hashes that actually produce + dedupes: matched hashes are effectively pinned near the front, + and hashes that are never used drift back until they are evicted. + Most of the cells in the table at any moment are redundant, but + that is the cost of not knowing in advance which ones will be + useful. + + * Continuous and periodic operation are usually equivalent for + per-extent dedupe probability. bees persists its hash table and + scan position, so a restart does not reset the LRU and does not + skip data, and each extent is scanned in the same order either + way. The important exception is workloads that rapidly churn + the same blocks (rotating logs, continuous rebuilds, + heavily-updated databases): continuous bees will scan every + intermediate version of those blocks and fill the LRU with + hashes that point to extents that are already freed by the time + the next duplicate arrives, crowding out more useful hashes. + Running bees periodically lets short-lived data be overwritten + before bees ever sees it; increasing the hash table size is the + other way to compensate.