btrfs: raid56: drop cached rbios when their stripes' extents are freed

A cached rbio keeps an in-memory copy of its full stripe's data
sectors so that a later sub-stripe write can steal the pages (see
steal_rbio()) and skip the RMW read.  That is only safe while the
cache and the disk agree.  Nothing invalidates cache entries when the
stripe's extents are freed, so an entry can outlive the data it
describes: the space is reallocated, a sub-stripe write to the reused
stripe steals the stale pages, and parity is generated from the
cache's memory of the stripe's previous life instead of from disk.

If the on-disk content diverged from that memory in the meantime, the
result is silent, latent damage.  The divergence does not require a
second write: a device error that corrupts a column holding only
freed data is invisible (no live extent, no csum to fail, nothing for
scrub to check), but the stale cached copy of that column predates
the corruption.  The poisoned write then produces parity consistent
with the cache's memory and inconsistent with the actual on-disk
column.  Every read of the live data still passes csum and scrub
reports nothing, so the loss surfaces only when a device failure
forces reconstruction -- which XORs the real on-disk garbage against
parity that remembers different bytes, and hands back trash.

This was found and proven byte-exactly on a raid5 test array: a
stripe's parity solved uniquely as the XOR of fresh data with the
PRE-corruption content of a free-space column -- bytes that existed
nowhere on disk at write time and could only have come from the rbio
cache.  The resulting single-column loss was undetectable by scrub
and unrecoverable by reconstruction.

Drop cache entries overlapping freed extents at unpin time, when the
space returns to circulation.  The walk is gated on raid56 block
groups, so filesystems without raid56 never touch it, and the LRU
list is bounded (RBIO_CACHE_SIZE entries).

The cost of dropping entries was measured on an instrumented stock
kernel that marks affected entries instead of dropping them, and
counts a reintroduced RMW full-stripe read whenever a read-skip was
only possible because of a marked entry.  In a worst-case free-heavy
small-file churn on a 5-device raid5 array with raid5 metadata --
conditions chosen to flatter the cache, and it delivers, with 87% of
sub-stripe RMWs skipping their read -- only 1% of those skips (64 of
6430) were enabled by entries this patch drops, costing 0.04-0.12
MB/s of extra reads against the workload's own 0.3-0.7 MB/s of
writes.  93% of the entries the patch drops were evicted without
ever being stolen again.

Assisted-by: Claude:claude-fable-5

diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index a0d5ab03aae264a83f7f05bba15c947cbc40facd..1d39ae318d1b13d84adca65d1cbe451f2d40ab13 100644 (file)
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -2868,6 +2868,16 @@ static int unpin_extent_range(struct btrfs_fs_info *fs_info,
                if (return_free_space)
                        btrfs_add_free_space(cache, start, len);
 
+               /*
+                * The rbio stripe cache may still remember this range's
+                * pre-free contents; a later sub-stripe write into the
+                * reused stripes would steal those pages and generate
+                * parity from them.  Drop the stale entries now that the
+                * extents are gone.
+                */
+               if (cache->flags & BTRFS_BLOCK_GROUP_RAID56_MASK)
+                       btrfs_raid56_uncache_range(fs_info, start, len);
+
                start += len;
                total_unpinned += len;
                space_info = cache->space_info;

diff --git a/fs/btrfs/raid56.c b/fs/btrfs/raid56.c
index d5c30d2e8ecf09c28bb0a2da62e127e7f04dd52b..b5ab71cd36a61c6411acedc0efe406248a7d7c1f 100644 (file)
--- a/fs/btrfs/raid56.c
+++ b/fs/btrfs/raid56.c
@@ -535,6 +535,43 @@ static void remove_rbio_from_cache(struct btrfs_raid_bio *rbio)
        spin_unlock(&table->cache_lock);
 }
 
+/*
+ * Drop cached rbios whose full stripes overlap [start, start + len).
+ *
+ * Called when extents in that range are freed.  A cached rbio holds an
+ * in-memory copy of its stripe's data sectors; once the extents are gone
+ * the space can be reallocated, and a later sub-stripe write to the same
+ * stripe would steal the cached pages (see steal_rbio()) and compute
+ * parity from the cache's memory of the stripe's previous life instead
+ * of reading the disk.  If the on-disk content diverged from that memory
+ * in the meantime -- for example a device error corrupted a column that
+ * no longer had live data, so nothing noticed -- the write produces
+ * parity that disagrees with the on-disk columns, and the stripe is
+ * latently unreconstructible: every read of the live data passes csum,
+ * scrub sees nothing wrong, and the loss only surfaces when a device
+ * failure forces reconstruction.  Freed extents' cache entries have no
+ * value to keep: drop them.
+ */
+void btrfs_raid56_uncache_range(struct btrfs_fs_info *info, u64 start, u64 len)
+{
+       struct btrfs_stripe_hash_table *table = info->stripe_hash_table;
+       struct btrfs_raid_bio *rbio;
+       struct btrfs_raid_bio *tmp;
+
+       if (!table)
+               return;
+
+       spin_lock(&table->cache_lock);
+       list_for_each_entry_safe(rbio, tmp, &table->stripe_cache, stripe_cache) {
+               const u64 rbio_start = rbio->bioc->full_stripe_logical;
+               const u64 rbio_len = (u64)rbio->nr_data << BTRFS_STRIPE_LEN_SHIFT;
+
+               if (rbio_start < start + len && start < rbio_start + rbio_len)
+                       __remove_rbio_from_cache(rbio);
+       }
+       spin_unlock(&table->cache_lock);
+}
+
 /*
  * remove everything in the cache
  */

diff --git a/fs/btrfs/raid56.h b/fs/btrfs/raid56.h
index 47f8e85db2949e73e2007876d1114d8b4278ae30..c7c4b0db61f4c7ede79107d68d7dbd206552cc33 100644 (file)
--- a/fs/btrfs/raid56.h
+++ b/fs/btrfs/raid56.h
@@ -295,5 +295,6 @@ void raid56_parity_cache_data_folios(struct btrfs_raid_bio *rbio,
 
 int btrfs_alloc_stripe_hash_table(struct btrfs_fs_info *info);
 void btrfs_free_stripe_hash_table(struct btrfs_fs_info *info);
+void btrfs_raid56_uncache_range(struct btrfs_fs_info *info, u64 start, u64 len);
 
 #endif