| `[reports.progress]` | `${UUID}.progress` | `1s` | `0444` | `progress` |
| `[reports.tasks]` | `${UUID}.tasks` | `1s` | `0400` | `workers queue tasks` |
+## [state] section
+
+The `[state]` section is the master switch for on-disk persistence. When
+persistence is enabled bees stores its hash table in `beeshash.dat` and its
+crawl checkpoint in `beespoint.ini` under the directory named by `paths.home`.
+The `[state.hash]` and `[state.point]` subsections configure the two state
+files in detail.
+
+* **`persistent`**
+ Enable on-disk persistence of both the hash table and the crawl checkpoint.
+ * `yes` (default) — bees reads `beeshash.dat` and `beespoint.ini` at startup
+ (creating them if necessary, subject to `state.hash.create`) and writes
+ them back as the daemon runs.
+ * `no` — bees runs entirely from an in-memory hash table and starts every
+ crawl from the beginning. No state files are read or written. Combine
+ with `loop.exit-when-finished = yes` for a stateless one-shot run; without
+ a loop-exit option the in-memory hash table is discarded on restart and
+ every restart redoes work.
+ * When `persistent = no`, the keys in `[state.hash]` and `[state.point]`
+ have no effect.
+
+## [state.hash] section
+
+The `[state.hash]` section configures `beeshash.dat`: its size, whether bees
+may create or resize it, and how aggressively it is written back to disk.
+
+### Sizing
+
+* **`size`**
+ Hash table size in bytes. Larger tables hold more hashes and find more
+ duplicates; the practical ceiling is the amount of RAM that can be locked.
+ * Default: `min(${RAM_BYTES} / 16, ${FS_BYTES} / 16384)`. This formula
+ reserves at most 1/16 of installed RAM and at most 1/16384 of filesystem
+ capacity, whichever is smaller. The `/ 16384` term assumes a typical
+ average extent size of about 64 KiB (half a maximum compressed extent);
+ it is conservative because identical blocks tend to cluster in runs and
+ neighbour-matching extends each hash hit to cover the entire run.
+ * The result is rounded up to the next 128 KiB hash-table extent boundary
+ (`BLOCK_SIZE_HASHTAB_EXTENT`).
+ * Accepts a [size value](config-format.md#size-values), including
+ arithmetic expressions and the substitution variables `${RAM_BYTES}`
+ (total installed RAM) and `${FS_BYTES}` (filesystem capacity).
+ * Expressions in a config file are recomputed at every startup. If RAM
+ or filesystem size changes between runs, the recomputed `size` may
+ differ from the on-disk file. When that happens bees logs a warning
+ and uses the existing on-disk size unless `state.hash.resize = yes`.
+ For one-time changes, prefer the `--option` command-line flag
+ (combined with `state.hash.resize = yes`) rather than editing the
+ config file in place.
+
+* **`create`**
+ Create `beeshash.dat` if it does not already exist.
+ * `yes` (default) — bees creates the file at startup, sized to
+ `state.hash.size`.
+ * `no` — startup fails if the file is missing. Useful to detect a
+ misconfigured `paths.home` before bees writes a fresh empty hash table
+ to the wrong location.
+ * Has no effect when `state.persistent = no`.
+
+* **`resize`**
+ Resize the on-disk hash table when `state.hash.size` does not match the
+ existing file size.
+ * `no` (default) — a size mismatch is logged as a warning and bees runs
+ against the existing file at its current size.
+ * `yes` — bees rebuilds `beeshash.dat` to match `state.hash.size`,
+ preserving the most-recently-used entries via a reverse-LRU rebuild.
+ The rebuild is written to a temporary file and renamed over the old
+ one, so an interrupted resize never leaves a partial hash table on
+ disk.
+ * Has no effect when `state.persistent = no`.
+
+### Writeback
+
+* **`writeback-time`**
+ Time budget for writing the full hash table to disk. Drives the
+ default writeback rate: `state.hash.size / writeback-time`.
+ * Default: `2h20m` (about 2.3 hours). This is the period over which
+ every dirty extent in the hash table is guaranteed to reach disk.
+ * Accepts a [duration value](config-format.md#duration-values).
+
+* **`writeback-rate-max`**
+ Maximum hash table writeback rate in bytes per second. Caps the rate
+ derived from `writeback-time` so very large hash tables do not generate
+ proportionally large I/O storms.
+ * Default: `128K`.
+ * Accepts a [size value](config-format.md#size-values) per second.
+ * The effective writeback rate is
+ `min(state.hash.size / writeback-time, writeback-rate-max)`.
+
+* **`writeback-fsync`**
+ Issue `fsync` after each hash-table extent write.
+ * `no` (default) — rely on the periodic close-time flush plus the
+ kernel's writeback for durability.
+ * `yes` — flush each extent immediately. Increases durability but
+ raises write latency and I/O amplification; only useful on systems
+ where a crash between writebacks would lose unacceptable progress.
+
+* **`writeback-unreadahead`**
+ Discard hash-table pages from the page cache after each write.
+ * `no` (default) — leave pages in the page cache for reuse.
+ * `yes` — call `posix_fadvise(POSIX_FADV_DONTNEED)` after each write.
+ Reduces page-cache pressure on memory-constrained systems at the
+ cost of having to re-read from disk on the next access.
+
+* **`close-fsync`**
+ `fsync` `beeshash.dat` once on clean exit.
+ * `no` (default) — rely on the kernel's writeback for the closing
+ flush.
+ * `yes` — block on a final `fsync` so bees does not return control
+ until the hash table is durable on disk. Useful when bees is run
+ by a service manager that may immediately power down or unmount
+ the filesystem after bees exits.
+
+## [state.point] section
+
+The `[state.point]` section configures `beespoint.ini`: how often the crawl
+checkpoint is written and whether checkpoint advances are deferred until the
+hash table catches up.
+
+* **`interval`**
+ Time between crawl checkpoint writes.
+ * Default: `900s` (15 minutes).
+ * Accepts a [duration value](config-format.md#duration-values).
+ * Smaller values reduce the amount of work redone after a crash at the
+ cost of slightly more overhead. The interval is independent of the
+ hash-table writeback schedule.
+
+* **`defer`**
+ Defer crawl checkpoint writes until every hash-table extent dirtied
+ since the snapshot has been written back.
+ * `no` (default) — checkpoints advance on every interval regardless
+ of hash-table writeback progress.
+ * `yes` — bees holds back the checkpoint write until the hash-table
+ writeback counter has confirmed that all extents modified since the
+ snapshot are on disk. Prevents `beespoint.ini` from advancing past
+ the durable contents of `beeshash.dat`, which would otherwise cause
+ missed deduplication on restart after a crash.
+ * `defer = yes` depends on the hash-table writeback counter. It has
+ no effect when `state.persistent = no` (no hash table is written
+ back) and may significantly delay checkpoint advances when
+ `state.hash.size` is large and `writeback-rate-max` is small.
+
## [thread] section
The `[thread]` section sets how many worker threads bees runs and how aggressively it ramps up under system load.
projects / bees / commitdiff