manager that may immediately power down or unmount the filesystem
after bees exits.
+### Eviction
+
+* **`randomize-lru`**
+ Insert new (not-yet-duplicate) hashes at a random position in the
+ per-bucket LRU list rather than always at the front.
+ * `yes` (default) — each insert picks a random slot in the bucket.
+ Randomizing the insertion point spreads evictions out so that new
+ reads do not systematically evict older inserts, which keeps hit
+ rates higher when all hashes are inserted from each extent.
+ * `no` — insert at the LRU front only (most-recently-used position),
+ evicting from the back. This removes the per-insert randomness, so
+ that — combined with `thread.thread-max = 1` — hash-table eviction
+ is deterministic from one run to the next. Useful for reproducible
+ performance analysis, where run-to-run variance from randomized
+ eviction would otherwise mask the effect being measured. Also
+ useful when combined with controls on insert-time block counts.
+
## [state.point] section
The `[state.point]` section configures `beespoint.ini`: how often the crawl
# fsync the hash table file when bees exits.
close-fsync = no
+ # Insert new hashes at a random position in the per-bucket
+ # LRU list. Set to no to insert at the LRU front only.
+ randomize-lru = yes
+
# The [state.point] section controls crawl checkpoint options.
[state.point]
Cell *ip = find(er.first, er.second, mv);
bool found = (ip < er.second);
- const auto pos = tl_distribution(bees_generator);
+ const size_t pos = m_randomize_lru ? tl_distribution(bees_generator) : 0;
int case_cond = 0;
#if 0
BEESLOGINFO("state.hash.close-fsync = " << (v ? "yes" : "no") << " [state.hash.close-fsync]");
}
+void
+BeesHashTable::set_randomize_lru(bool v)
+{
+ m_randomize_lru = v;
+ BEESLOGINFO("state.hash.randomize-lru = " << (v ? "yes" : "no") << " [state.hash.randomize-lru]");
+}
+
void
BeesHashTable::resize_file(off_t new_size_signed)
{
BEESLOGINFO("state.hash: effective writeback rate = " << pretty(static_cast<uint64_t>(effective_rate)) << "/s");
ht->set_flush_rate(effective_rate);
}
+ // LRU randomization applies to both file-backed and in-memory
+ // tables (it governs insert position, not file I/O).
+ bc->hash_table()->set_randomize_lru(bc->get_config().get("state.hash.randomize-lru", bees_parse_bool));
bc->start_hash_writeback();
// Crawl checkpoint options
void set_writeback_unreadahead(bool v);
/// fsync the hash table on close (at bees termination only).
void set_close_fsync(bool v);
+ /// If false, push_random_hash_addr* insert at the LRU front instead of
+ /// a random position. Combined with thread.thread-max = 1 this makes
+ /// hash-table eviction deterministic for performance analysis.
+ void set_randomize_lru(bool v);
private:
string m_filename;
bool m_writeback_fsync = false;
bool m_writeback_unreadahead = false;
bool m_close_fsync = false;
+ bool m_randomize_lru = true;
/// Configured state.hash.size, captured in init() before set_resize()
/// has been called. prepare_in_memory() compares this against the