bpf: Clear NON_OWN_REF after RCU protection ends

A local kptr load of an object containing a graph node is marked MEM_RCU
and NON_OWN_REF while protected by RCU. When the last RCU read-side critical
section ends, invalidate_rcu_protected_refs() removes MEM_RCU and marks the
pointer PTR_UNTRUSTED, but leaves NON_OWN_REF set.

The stale flag lets graph kfunc argument checks continue treating the
pointer as a live borrowed reference. In particular, bpf_rbtree_remove()
can accept a pointer after its protection ended and return it as a new
owning reference, even though the object may already have been freed.

Clear NON_OWN_REF when an RCU-protected pointer is demoted. A spin lock also
provides implicit RCU protection, so invalidate non-owning references before
demoting RCU-protected pointers when releasing the lock. Otherwise the
demotion would clear the flag before invalidate_non_owning_refs() can find
and invalidate those aliases.

The demoted pointer remains available for fault-protected reads. Exempt such
reads from the allocated-object reference-state assertion; writes through a
fault-prone pointer are already rejected, and bpf_may_fault_on_deref() makes
the surviving loads use BPF_PROBE_MEM.

Fixes: 1b12171533a9 ("bpf: Mark direct ld of stashed bpf_{rb,list}_node as non-owning ref")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260904084325.52250-6-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 3af8bd838b823e4776589ce1a64feda4ff5c014b..9c6ad157a61ecf76f85dec49540a87419afa1897 100644 (file)
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -6045,7 +6045,13 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env,
                        return -EACCES;
                }
 
-               if (type_is_alloc(reg->type) && !type_is_non_owning_ref(reg->type) &&
+               /*
+                * A fault-prone allocated object may still be read through a
+                * BPF_PROBE_MEM load after its lifetime protection ends. Writes
+                * through such pointers were rejected above.
+                */
+               if (type_is_alloc(reg->type) && !bpf_may_fault_on_deref(reg->type) &&
+                   !type_is_non_owning_ref(reg->type) &&
                    !(reg->type & MEM_RCU) && !reg_is_referenced(env, reg)) {
                        verifier_bug(env, "allocated object must have a referenced id");
                        return -EFAULT;
@@ -7423,10 +7429,14 @@ static int process_spin_lock(struct bpf_verifier_env *env, struct bpf_reg_state
                                lock);
                        return -EINVAL;
                }
+               /*
+                * Invalidate non-owning refs before RCU demotion clears their
+                * NON_OWN_REF flag.
+                */
+               invalidate_non_owning_refs(env);
+
                if (!in_rcu_cs(env))
                        invalidate_rcu_protected_refs(env);
-
-               invalidate_non_owning_refs(env);
        }
        return 0;
 }
@@ -9526,7 +9536,7 @@ static void invalidate_rcu_protected_refs(struct bpf_verifier_env *env)
        bpf_for_each_reg_in_vstate_mask(env->cur_state, state, reg, stack, clear_mask, ({
                if (reg->type & MEM_RCU) {
                        bpf_diag_mod_begin(env, reg, NULL, BPF_DIAG_MOD_WRITE);
-                       reg->type &= ~(MEM_RCU | PTR_MAYBE_NULL);
+                       reg->type &= ~(MEM_RCU | PTR_MAYBE_NULL | NON_OWN_REF);
                        reg->type |= PTR_UNTRUSTED;
                        bpf_diag_mod_end(env);
                }