Commit
9f5315d3f4368590c9fe06d4313b7247470bcf15 upstream.
A task other than the owner can free a request, io_ring_exit_work()
running the local task_work of a DEFER_TASKRUN ring for example, or
io_do_iopoll() reaping completions. io_put_task() then takes the else
branch:
percpu_counter_sub(&tctx->inflight, 1);
if (unlikely(atomic_read(&tctx->in_cancel)))
wake_up(&tctx->wait);
put_task_struct(tctx->task);
The first line is what lets the owner leave io_uring_cancel_generic(),
and on exec it frees the tctx right after. Nothing holds that off for
the remaining three lines, the task reference only covers a tctx freed
along with the task_struct. It used to work out because
io_uring_del_tctx_node() took the uring_lock that the freeing side holds
across io_put_task(), until that moved to tctx_lock.
Free the tctx with kfree_rcu(), and enter an RCU read section in
io_put_task() before the count is dropped.
Fixes: 5623eb1ed035 ("io_uring/tctx: add separate lock for list of tctx's in ctx")
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
/* drains ->task_list once the task can no longer run task_work */
struct work_struct fallback_work;
+ /* the tctx is freed after a grace period, see io_put_task() */
+ struct rcu_head rcu;
+
struct { /* task_work */
struct mpscq task_list;
struct callback_head task_work;
if (likely(tctx->task == current)) {
tctx->cached_refs++;
} else {
+ guard(rcu)();
+
percpu_counter_sub(&tctx->inflight, 1);
if (unlikely(atomic_read(&tctx->in_cancel)))
wake_up(&tctx->wait);
WARN_ON_ONCE(tctx->cached_refs);
percpu_counter_destroy(&tctx->inflight);
- kfree(tctx);
+ kfree_rcu(tctx, rcu);
tsk->io_uring = NULL;
}
}