merge_reloc_root() refills the relocation block reserve before each step
of the merge. Since
44d354abf33e ("btrfs: relocation: review the call
sites which can be interrupted by signal") it does so with
BTRFS_RESERVE_FLUSH_LIMIT. FLUSH_LIMIT only flushes delayed items and
tries to allocate a chunk. It does not run delayed refs, does not flush
delalloc, does not commit the transaction to unpin what the merge and
everybody else freed, and never falls back to the global reserve. A
failure here is fatal: merge_reloc_roots() handles it with
btrfs_handle_fs_error() and the filesystem goes read-only.
The merge routinely needs more than prepare_to_merge() reserved for it,
and refills from free space. On a filesystem with every device
allocated, metadata free space is all in other tasks' reservations, and
the refill loses:
BTRFS: error (device dm-6) in merge_reloc_roots:2053: errno=-28 No space left
BTRFS info (device dm-6 state E): forced readonly
BTRFS error (device dm-6 state EA): error relocating chunk
14022541312: -28
This is the reclaim worker relocating a data block group of a 6-device
raid5 filesystem (metadata raid1) with 6 MiB unallocated, under
fsstress. A trace of the failing merge: prepare_to_merge() reserved 704
KiB, the merge used it up and refilled 14 times (560 KiB more), and the
15th refill, for 32 KiB, failed. At that moment the metadata space_info
(256 MiB) had 160 MiB used, 9 MiB pinned and 86 MiB in may_use, of which
27 MiB was the untouched global reserve: 16 KiB free. Background
reclaim is optional work; it took the filesystem down.
Refill with BTRFS_RESERVE_FLUSH_EVICT instead.
Why FLUSH_LIMIT was chosen. This refill used FLUSH_ALL from
08e007d2e577 ("Btrfs: improve the noflush reservation") until
44d354abf33e. That commit changed it only so that a fatal signal cannot
interrupt the merge: FLUSH_ALL waits for its ticket killably and
returns -EINTR, which here would turn a killed balance into a read-only
filesystem. Its message already notes that the change "can be
ENOSPC-prone", and it shrank min_reserved to compensate. No deadlock
was given as a reason.
Why not another mode. No existing mode is "run delayed refs and commit,
but neither flush delalloc nor steal, and do not wait killably":
- FLUSH_LIMIT does not run delayed refs, which is what the merge refills
that needed flushing in testing needed.
- FLUSH_ALL and FLUSH_ALL_STEAL flush delalloc and commit too, through
the async reclaim worker, and wait killably. That brings back the
-EINTR that
44d354abf33e removed.
- FLUSH_EVICT is the priority flusher with the full list: delayed items,
delayed refs, delalloc, chunk allocation and a commit, in that order,
run in the caller's context. It stops at the first state that serves
the ticket, does not wait killably, and steals from the global reserve
only after all of them have failed.
In testing, every merge refill that needed flushing was served by the
delayed-ref states (5 of 6 tickets; the sixth needed only delayed
items). None reached delalloc, the commit or the steal. Those states
run only where the old code forced the filesystem read-only.
Deadlock analysis. At the refill the merging task holds:
- No transaction handle. The previous step ended its transaction with
btrfs_end_transaction_throttle(), and the first step has not started
one, so current->journal_info is NULL, as FLUSH_EVICT requires.
- No tree locks. The path into the reloc tree holds references only
(the reloc tree is walked without locks, and btrfs_unlock_up_safe()
dropped the locks of the resume search). replace_path() unlocks the
subvolume tree before it returns.
- Not reloc_mutex. merge_reloc_roots() takes it only around its list
splices.
- These locks from its callers: cleaner_mutex (do_nonremap_reloc()),
reclaim_bgs_lock, the exclusive operation, and sb_start_write for a
data block group. Writeback, ordered extent completion and
transaction commit take none of them. The other takers of
reclaim_bgs_lock either use trylock (unused block group deletion, the
reclaim worker) or are relocation entry points (balance, system chunk
relocation, device shrink, zoned repair). Those take it with no
transaction handle held, and the exclusive operation serializes them
with this one anyway. The other takers of cleaner_mutex are the
cleaner (trylock), mount and unmount (btrfs_start_pre_rw_mount(),
btrfs_commit_super()) and the quota ioctl, which takes it before
starting a transaction. So no flush or commit can wait on a lock that
the merging task holds.
The same task already does each of these things in the same lock
context:
- prepare_to_merge() reserves with FLUSH_ALL, waiting while the async
reclaim worker flushes delalloc and commits, and then commits itself.
- relocate_block_group() commits again after the merge.
- btrfs_end_transaction_throttle() at the end of every merge step runs
delayed iputs. An evicted inode reserves with FLUSH_EVICT from right
there.
A commit in the middle of the merge is legal. The transaction kthread,
sync and other tasks already commit between merge steps. Each step
saves drop_progress and drop_level before it ends its transaction, and
the commit writes them to the reloc root item through
btrfs_update_reloc_root(). init_reloc_root() creates nothing while
rc->merge_reloc_tree is set. btrfs_end_transaction_throttle() used to
commit from this loop itself, until
302167c50b32 ("btrfs: don't end the
transaction for delayed refs in throttle").
DATA_RELOC inode. In the UPDATE_DATA_PTRS stage its delalloc was
written back and waited for, and its page cache invalidated, before the
stage started (do_nonremap_reloc()), so there is nothing to flush. In
the MOVE_DATA_EXTENTS stage its dirty pages may still be unwritten
during the merge. Flushing them from here is the same writeback that
the flusher threads, sync, or another task's FLUSH_ALL or FLUSH_EVICT
ticket may start at any time. It is also what do_nonremap_reloc() waits
for right after relocate_block_group() returns. Completing its ordered
extents joins a transaction on the data reloc root, which is not
shareable, so it does not take reloc_mutex.
The steal is bounded. steal_from_global_rsv() leaves 10% of the global
reserve, and each steal is at most min_reserved, which is two nodes per
level of the reloc root (64 KiB in the trace above). A merge that fails
here takes the filesystem read-only. Keeping it writable is what the
global reserve is for, as it is for eviction.
If even the global reserve cannot cover the refill, the merge fails as
before.
Reproduced with reloc-merge-enospc-test.sh (raid56-wh-harness): the
octopus P1 sequence on 6 x 3 GiB, fsstress until the reclaim worker's
relocations, plain relocations or balances run at the fill edge. On the
base, 3 of 3 reclaim runs went read-only in merge_reloc_roots(). With
the patch, 6 runs had no merge failure.
Fixes: 44d354abf33e ("btrfs: relocation: review the call sites which can be interrupted by signal")
Assisted-by: Claude:claude-opus-5-5