Leo Yan [Tue, 2 Jun 2026 14:47:13 +0000 (15:47 +0100)]
libbpf: Initialize CFLAGS before including Makefile.include
tools/scripts/Makefile.include may expand EXTRA_CFLAGS in a future
change. This could alter the initialization of CFLAGS, as the default
options "-g -O2" would never be set once EXTRA_CFLAGS is expanded.
Prepare for this by moving the CFLAGS initialization before including
tools/scripts/Makefile.include, so it is not affected by the extended
EXTRA_CFLAGS.
Append EXTRA_CFLAGS to CFLAGS only after including Makefile.include and
place it last so that the extra flags propagate properly and can
override the default options.
tools/scripts/Makefile.include already appends $(CLANG_CROSS_FLAGS) to
CFLAGS, the Makefile appends $(CLANG_CROSS_FLAGS) again, remove the
redundant append.
Leo Yan [Tue, 2 Jun 2026 14:47:10 +0000 (15:47 +0100)]
bpftool: Pass host flags to bootstrap libbpf
bpftool builds a bootstrap libbpf with HOSTCC, but the libbpf submake can
still inherit target build flags through CFLAGS. This can break cross
builds when host objects are compiled with target-only options.
Since HOST_CFLAGS contains warning options that are not suitable for
building libbpf, use LIBBPF_BOOTSTRAP_CFLAGS with the warning options
removed to build the bootstrap libbpf. Clear EXTRA_CFLAGS so target
extra flags are not mixed into the host bootstrap libbpf build.
====================
bpf: Allow uprobe_multi binary specified by file descriptor
Add ability to open uprobe_multi link on top of binary identified
by file descriptor. This allows us to avoid the race where the binary is
replaced between path resolution and attachment, ensuring we monitor the
intended binary.
v3 changes:
- guard t_user acesss with access_ok [sashiko]
v2 changes:
- move path retrieval in separate function so CLASS(..) is not used in function
with goto-based cleanup [sashiko]
- force zero path_fd in case BPF_F_UPROBE_MULTI_PATH_FD is not set [sashiko]
- add space around | in bpf_uprobe_multi_link_attach [Alexei]
====================
Jiri Olsa [Thu, 11 Jun 2026 11:42:30 +0000 (13:42 +0200)]
selftests/bpf: Fix typo in verify_umulti_link_info
We verify info.uprobe_multi.flags against wrong kprobe-multi flag
(BPF_F_KPROBE_MULTI_RETURN). It's the same value as the correct
flag (BPF_F_UPROBE_MULTI_RETURN), so there's not functional change.
Jiri Olsa [Thu, 11 Jun 2026 11:42:28 +0000 (13:42 +0200)]
selftests/bpf: Add uprobe_multi path_fd test
Add a uprobe_multi link API selftest that opens /proc/self/exe and passes
the resulting descriptor through opts.uprobe_multi.path_fd with
BPF_F_UPROBE_MULTI_PATH_FD set.
Jiri Olsa [Thu, 11 Jun 2026 11:42:26 +0000 (13:42 +0200)]
bpf: Add support to specify uprobe_multi target via file descriptor
Allow uprobe_multi link to identify the target binary by an already
opened file descriptor.
Adding new BPF_F_UPROBE_MULTI_PATH_FD flag and the path_fd field for
the attr.link_create.uprobe_multi struct.
When the flag is set, we resolve the target from path_fd, without the
flag, we keep the existing string path behavior.
I don't see a use case for supporting O_PATH file descriptors, because
we need to read the binary first to get probes offsets, so I'm using
the CLASS(fd, f), which fails for O_PATH fds.
Jiri Olsa [Thu, 11 Jun 2026 11:42:25 +0000 (13:42 +0200)]
bpf: Use user_path_at for path resolution in uprobe_multi
Resolve the uprobe_multi user path with user_path_at() instead of copying
the string with strndup_user() and passing it to kern_path(). This removes
the temporary allocation and keeps the lookup logic in one helper.
Woojin Ji [Fri, 12 Jun 2026 05:26:55 +0000 (14:26 +0900)]
selftests/bpf: Add arena direct-value one-past-end reject test
BPF_MAP_TYPE_ARENA supports direct-value pseudo loads, but unlike array
maps its map value_size is zero and the valid direct-value range is the
arena mmap size, max_entries * PAGE_SIZE.
Commit 3ac1a467e376 ("bpf: Fix off-by-one boundary validation in arena
direct-value access") fixed arena_map_direct_value_addr() to reject an
offset exactly at the end of the arena mapping. Add a regression test
that loads a BPF_PSEUDO_MAP_VALUE with off == arena_size and verifies
that the verifier rejects it with the expected offset in the log.
This is intentionally kept as a userspace raw-instruction test. I tried
expressing the same BPF_PSEUDO_MAP_VALUE + off == arena_size case in
verifier_arena.c with inline assembly. The only form that produces the
desired instruction bytes uses __imm_addr(arena), but that emits
R_BPF_64_NODYLD32, which the libbpf/bpftool link step rejects. Other
register, immediate, and memory constraints either fail in the BPF
backend or lower to a normal R_BPF_64_64 load followed by an ALU add,
which does not exercise arena_map_direct_value_addr() with the boundary
offset in the second ldimm64 slot.
A legacy test_verifier fixture can express the raw instruction directly,
but it needs arena map creation, mmap, and fixup plumbing in the legacy
runner. That is more intrusive than the small prog_tests raw-instruction
test.
Use the userspace raw-instruction test, following the existing selftests
pattern used for direct map-value pseudo loads, so insns[1].imm can be
set to arena_size precisely.
Assisted-by: ChatGPT:gpt-5.5 Signed-off-by: Woojin Ji <random6.xyz@gmail.com> Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com> Cc: Emil Tsalapatis <emil@etsalapatis.com> Cc: Junyoung Jang <graypanda.inzag@gmail.com> Link: https://lore.kernel.org/r/20260612-arena-direct-value-v1-v4-1-b81b642f5277@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Gabriele Monaco [Wed, 10 Jun 2026 09:04:29 +0000 (11:04 +0200)]
rqspinlock: Fix order in raw_res_spin_(un)lock_irq to allow schedule
raw_res_spin_unlock_irqrestore() calls raw_res_spin_unlock() and then
restores interrupts, this means preemption is enabled when interrupts
are still disabled (as part of raw_res_spin_unlock()) so this cannot
trigger an actual preemption.
This is inconsistent with other spinlock implementations
(raw_spin_unlock_irqrestore() and bpf_res_spin_unlock_irqrestore()
itself).
Adjust the macro to ensure interrupts are enabled before enabling
preemption, allowing to schedule at that point. Make the same
modification in the error path of raw_res_spin_lock_irqsave().
====================
bpf: Fix setting retval to -EPERM for cgroup hooks not returning errno
This series fixes the issue reported by sashiko in [1]. The issue is that,
when a cgroup BPF program exits with 0, bpf_prog_run_array_cg() sets
the hook return value to -EPERM if it is not a valid errno. This is
correct for errno-based hooks, which return 0 on success and negative
errno on failure, but wrong for void and boolean LSM hooks. Boolean
LSM hooks should only return true or false, and void LSM hooks have
no return value at all.
Fix it by skipping setting -EPERM for hooks not returning errno.
Xu Kuohai [Wed, 10 Jun 2026 20:17:24 +0000 (20:17 +0000)]
selftests/bpf: Add retval test for bool and errno LSM cgroup hooks
Add test to check the return value when a BPF program exits with 0 for
a boolean and an errno LSM hook.
For each hook, two BPF programs are attached. The first program returns
0 without calling bpf_set_retval() to exercise the return value translation
logic, while the second program reads the retval via bpf_get_retval().
Xu Kuohai [Wed, 10 Jun 2026 20:17:23 +0000 (20:17 +0000)]
bpf: Fix setting retval to -EPERM for cgroup hooks not returning errno
When a cgroup BPF program exits with 0, bpf_prog_run_array_cg() sets
the hook return value to -EPERM if it is not a valid errno. This is
correct for errno-based hooks, which return 0 on success and negative
errno on failure, but wrong for boolean and void LSM hooks. Boolean
LSM hooks should only return true or false, and void LSM hooks have
no return value at all.
Fix it by skipping setting -EPERM for hooks not returning errno.
Generic XDP devmap multi redirect can leave cloned skbs sharing packet
data. When a devmap egress program mutates packet data, another
destination sharing the same data may observe that mutation.
Fix this by making cloned skbs private before running the generic devmap
egress program. The private copy is made in dev_map_generic_redirect()
so dev_map_bpf_prog_run_skb() can keep returning the XDP action directly.
Add selftest coverage for the last-destination case, where the final
destination runs on the original skb while earlier destinations use
cloned skbs. The test records the source MAC observed by an earlier
destination and checks that it is neither the sentinel value left in the
result map nor the MAC written by the final destination.
---
v5:
- Move the skb_copy() check back to dev_map_generic_redirect() to keep
dev_map_bpf_prog_run_skb() returning only the XDP action.
- Preserve mac_len after skb_copy().
- Use __be64 temporary values when updating mac_map from userspace.
- Initialize rx_mac with a sentinel in the last-destination test instead
of relying on -ENOENT for ARRAY map lookups.
- Adjust the last-destination test topology so the checked earlier
destination is not the ingress/source veth.
- Split the last-destination check into two assertions: one for store_mac_1
updating rx_mac and one for detecting last-destination rewrite leakage.
v4: https://lore.kernel.org/bpf/20260611080850.536996-1-sun.jian.kdev@gmail.com/T/#mf830f03d362f33e0941d1b0e425169698fce76e5
- Preserve mac_len after skb_copy().
- Separate errno return from XDP action output in
dev_map_bpf_prog_run_skb().
- Zero-initialize net_config in the new selftest.
v3: https://lore.kernel.org/bpf/20260611043317.512843-1-sun.jian.kdev@gmail.com/
- Split the kernel fix and selftest into separate patches.
- Move the private-copy logic into dev_map_bpf_prog_run_skb().
- Use deterministic DEVMAP_HASH keys in the last-destination selftest.
- Fix the Fixes tag.
v2: https://lore.kernel.org/bpf/08c35c70-a59e-4e0e-91db-22b5ec30b611@linux.dev/
- Move the private-copy step into dev_map_generic_redirect() so the
last-destination path is covered as well.
- Use skb_copy() instead of skb_unshare() to keep caller ownership
unchanged on allocation failure.
- Add a generic XDP last-destination selftest case.
Strengthen xdp_veth_egress to check that each destination observes the
MAC selected for its own egress ifindex, instead of only checking that
the observed MAC differs from a single magic value.
Add a generic XDP last-destination test where an earlier destination does
not have a devmap egress program while the final destination does. This
covers the case where the final destination runs on the original skb and
could otherwise rewrite packet data still shared with an earlier cloned
skb.
Use deterministic DEVMAP_HASH keys for the egress map so the intended
last destination is stable. Initialize the result map with a sentinel
value and check that store_mac_1 overwrites it before checking that the
earlier destination did not observe the MAC written by the final
destination.
Sun Jian [Fri, 12 Jun 2026 11:40:31 +0000 (19:40 +0800)]
bpf: Run generic devmap egress prog on private skb
Generic XDP devmap multi redirect uses skb_clone() for intermediate
destinations and sends the last destination with the original skb. This
can leave multiple destinations sharing the same packet data.
This becomes visible after generic devmap egress-program support was
added: a devmap egress program may mutate packet data, and another
destination sharing the same data can observe that mutation.
Native XDP broadcast redirect does not have this issue because
xdpf_clone() copies the frame data for each destination. Generic XDP
should provide the same per-destination isolation before running a
devmap egress program.
Fix this by making cloned skbs private before running the generic devmap
egress program. Use skb_copy() instead of skb_unshare() so allocation
failure does not consume the skb and the existing caller error paths keep
their ownership semantics.
Add tail-call selftests for prog-array ownership when cgroup storage
is in use. Verify that loading succeeds when callers and callees reuse
the owner's cgroup storage map, and that loading fails for a different
storage map and for the A(storage) -> B(no storage) -> C(storage)
bridge case addressed in the previous commit.
Also verify that a storage-less leaf program which cannot perform tail
calls itself is still allowed to join a storage-owned prog array, while
a storage-less tail-caller is rejected also at map update time.
Signed-off-by: Lin Ma <malin89@huawei.com> Signed-off-by: Rongzhen Cui <cuirongzhen@huawei.com> Signed-off-by: Jingguo Tan <tanjingguo@huawei.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Yonghong Song <yonghong.song@linux.dev> Link: https://lore.kernel.org/r/20260610105539.705887-2-daniel@iogearbox.net Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Daniel Borkmann [Wed, 10 Jun 2026 10:55:38 +0000 (12:55 +0200)]
bpf: Tighten cgroup storage cookie checks for prog arrays
The fix in commit abad3d0bad72 ("bpf: Fix oob access in cgroup local
storage") is still incomplete. The prog-array compatibility check
treats a program with no cgroup storage as compatible with any stored
storage cookie. This allows a storage-less program to bridge a tail
call chain between an entry program and a storage-using callee even
though cgroup local storage at runtime still follows the caller's
context, that is, A -> B(no storage) -> C(storage) path.
Requiring exact cookie equality would break the legitimate case of a
storage-less leaf program being tail called from a storage-using one.
Instead, only accept a zero storage cookie if the program cannot
perform tail calls itself. This keeps A -> B(no storage) working
while rejecting the A -> B(no storage) -> C(storage) bridge.
Fixes: abad3d0bad72 ("bpf: Fix oob access in cgroup local storage") Reported-by: Lin Ma <malin89@huawei.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Yonghong Song <yonghong.song@linux.dev> Link: https://lore.kernel.org/r/20260610105539.705887-1-daniel@iogearbox.net Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Yonghong Song [Wed, 10 Jun 2026 05:18:31 +0000 (22:18 -0700)]
selftests/bpf: Fix bpf_iter/task_vma test
For selftest bpf_iter/task_vma, I got a failure like below on my qemu run:
test_task_vma_common:FAIL:compare_output unexpected compare_output:
actual
'561593546000-561593585000r--p0000000000:241256579534/root/devshare/bpf-next/tools/testing/selftests/bpf/test_progs'
!= expected
'561593546000-561593585000r--p0000000000:245551546830/root/devshare/bpf-next/tools/testing/selftests/bpf/test_progs'
Further debugging found out file->f_inode->i_ino value may exceed 32bit,
e.g., i_ino = 0x14c2eae35, but the format string is '%u'. This caused
inode mismatch between bpf iter and proc result.
Fix the issue by using format string '%llu' to accommodate 64bit i_ino.
Fixes: e8168840e16c ("selftests/bpf: Add test for bpf_iter_task_vma") Signed-off-by: Yonghong Song <yonghong.song@linux.dev> Acked-by: Leon Hwang <leon.hwang@linux.dev> Link: https://lore.kernel.org/r/20260610051831.1346659-1-yonghong.song@linux.dev Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Referenced kptr destruction can run from tracing/NMI contexts through
bpf_obj_drop() and map value update/delete paths, reaching NMI-unsafe
special field teardown and deadlocks. Justin reported the issue and
iterated on fixes in [0]-[2], and also confirmed the bpf_obj_drop()
reproducer in [3].
This series rejects unsafe obj drops from non-iterator tracing programs,
limits map value recycle to NMI-safe field cancellation, and adds
focused selftests for the obj_drop(), NMI delete, and recycle teardown
cases.
Add focused map_kptr coverage for BPF-side map updates that touch values
containing referenced kptrs.
The new syscall programs stash the testmod refcounted object in an array
map, a preallocated hash map, and a no-prealloc hash map, then update the
same map from BPF. The refcount must remain elevated after the update,
while the userspace runner destroys the skeleton and reuses the existing
refcount wait to confirm map teardown releases the kptr.
selftests/bpf: Exercise unsafe obj drops from tracing progs
Add task_kfunc failure cases for bpf_obj_drop() on local objects with
referenced kptr fields from tracing and NMI tracing programs. These programs
must be rejected because dropping the object would run full special-field
destruction synchronously in an unsafe context.
Justin Suess [Tue, 9 Jun 2026 20:25:44 +0000 (22:25 +0200)]
bpf: Cancel special fields on map value recycle
Map update and delete paths currently call bpf_obj_free_fields() when a
value is being replaced or recycled. That makes field destruction depend
on the context of the update/delete operation. For tracing programs this
can include NMI context, where referenced kptr destructors, uptr
unpinning, and graph root destruction are not generally safe.
Introduce bpf_obj_cancel_fields() for the reusable-value path. It only
performs NMI-safe cleanup for timer, workqueue, and task_work fields.
Fields that need full destruction are left attached to the recycled value
and are destroyed by the final cleanup path instead.
Switch array and hashtab update/delete/recycle paths to this cancel
helper. Keep bpf_obj_free_fields() for final map destruction and for
bpf_mem_alloc destructors. Preallocated hashtabs do not have allocator
destructors, so teardown continues to walk the normal and extra elements
and fully destroy their fields.
This deliberately relaxes the eager-free semantics of map update/delete
for special fields. Programs that relied on a recycled map slot becoming
empty immediately after update/delete were relying on behavior that
cannot be implemented safely from every BPF execution context without
offloading arbitrary destructors.
There is a chance this change breaks programs making assumptions
regarding the eager freeing of fields. If so, we can relax semantics to
cancellation only when irqs_disabled() is true in the future. However,
theoretically, map values that get reused eagerly already have weaker
guarantees as parallel users can recreate freed fields before the new
element becomes visible again.
Justin Suess [Tue, 9 Jun 2026 20:25:43 +0000 (22:25 +0200)]
bpf: Reject bpf_obj_drop() from tracing progs
bpf_obj_drop() runs bpf_obj_free_fields() synchronously for
program-allocated objects. When such an object contains NMI unsafe
fields, tracing programs that can run from arbitrary instrumented
context can reach that destruction from unsafe contexts, including NMI.
NMI is likely one instance of this problem, and other instances would
include possible unsafe reentrancy. Deferring bpf_obj_drop() is not
appealing either: it would add delayed-free machinery to a release
operation that otherwise has straightforward synchronous ownership
semantics.
Reject bpf_obj_drop() and bpf_percpu_obj_drop() from tracing programs
that may run from unsafe contexts unless every field in the object's BTF
record is explicitly NMI safe. Do not reject sleepable
BPF_PROG_TYPE_TRACING programs, since they are not the arbitrary/NMI
contexts that motivate the restriction.
Note that while bpf_rb_root and bpf_list_head would be NMI safe on their
own to free, the objects recursively held by them may not be; be
conservative and just mark them as not NMI safe for now.
Use a whitelist for the NMI-safe field set instead of listing only known
NMI unsafe fields. Locks, async fields, unreferenced kptrs, and
refcounts are known to be NMI safe because their destruction is either a
no-op, simple state reset, or async cancellation. Referenced kptrs,
percpu referenced kptrs, uptrs, graph roots, graph nodes, and any future
field type are rejected until audited for arbitrary tracing and NMI
contexts. This is less susceptible to future changes in fields that were
previously safe by exclusion, and to new fields being added without
updating this check.
Convert the existing recursive local-object drop success case to a
syscall program in the same commit, since this verifier change makes the
old tracing program form invalid. The test still exercises
bpf_obj_drop() releasing a referenced task kptr from a safe program
type.
====================
selftests/bpf: Fix tests for llvm23 true signature
LLVM23 ([1]) records the 'true' function signature in BTF, i.e. the
signature inferred after optimization rather than the one written in C.
This caused two kinds of selftest failures (see below).
Case 1: keep int return type for tailcall subprogs
The verifier requires any subprog that issues a bpf_tail_call to return
an 'int' (see check_btf_func() in kernel/bpf/check_btf.c, which rejects
it with "tail_call is only allowed in functions that return 'int'").
Several tailcall subprogs do 'return 0' (or another constant) whose
result no caller uses. With llvm23 the compiler folds the constant and,
since the return value is dead, optimizes the subprog to effectively
return 'void' and records 'void' in BTF, so the program fails to load.
Use barrier_var() and __sink() to prevent returned value from being
optimized.
Case 2: adjust tracing prog ctx layout for the true signature
test_pkt_access_subprog2() has an unused argument that llvm optimizes
away. Before llvm23 the BTF signature did not match the optimized
assembly, so the verifier fell back to MAX_BPF_FUNC_REG_ARGS (5) u64
arguments and the fexit return value sat after args[5]. With llvm23 the
true signature has a single argument, so the return value moves to the
slot after args[1]. Select the matching ctx struct based on __clang_major__
so the test works with both old and new llvm.
Changelogs:
v1 -> v2:
- v1: https://lore.kernel.org/bpf/20260609163947.1717694-1-yonghong.song@linux.dev/
- Do not use bpf array map or bpf global var. Use __sink() instead.
====================
Yonghong Song [Tue, 9 Jun 2026 23:34:12 +0000 (16:34 -0700)]
selftests/bpf: Adjust fexit_bpf2bpf ctx layout for llvm23 true signature
test_pkt_access_subprog2() is defined in C as
int test_pkt_access_subprog2(int val, volatile struct __sk_buff *skb)
but llvm optimizes away the unused 'int val' argument. Before llvm23 the
BTF signature did not match the optimized assembly, so the verifier set
attach_func_proto to NULL and fell back to MAX_BPF_FUNC_REG_ARGS (5) u64
arguments (see btf_ctx_access()). The fexit ctx struct therefore placed
the return value after args[5].
With llvm23 the 'true' signature
int test_pkt_access_subprog2(volatile struct __sk_buff *skb)
is recorded in BTF, so nr_args becomes 1 and the return value moves to
the slot right after args[1]. Select the matching args_subprog2 layout
based on __clang_major__ so the test works with both old and new llvm.
Yonghong Song [Tue, 9 Jun 2026 23:34:07 +0000 (16:34 -0700)]
selftests/bpf: Keep int return type for tailcall subprogs
LLVM23 ([1]) supports 'true' function signature in BTF. The return type
of the caller of a tailcall must be an 'int'. Otherwise, verification will
fail (see check_btf_func() in check_btf.c). So with llvm23, it is possible
that the compiler may change the caller's return type from 'int' to 'void'.
To prevent this, barrier_var() and __sink() are used to avoid returning
a constant prone to be optimized.
====================
bpf, lpm_trie: Allow sleepable BPF programs to use LPM tries
trie_lookup_elem() annotates its rcu_dereference_check() walks with only
rcu_read_lock_bh_held(), so a sleepable BPF program that touches an LPM
trie (e.g. a sleepable LSM hook calling bpf_map_lookup_elem()) trips a
"suspicious RCU usage" lockdep splat on debug kernels: it holds only
rcu_read_lock_trace(), which that annotation does not accept.
Patch 1 relaxes the rcu_dereference annotations in the trie walks so they
no longer trip lockdep from the Tasks Trace context, including the
trie_update_elem()/trie_delete_elem() writer walks (protected by
trie->lock). Patch 2 adds BPF_MAP_TYPE_LPM_TRIE to the verifier's
sleepable map whitelist so sleepable programs can reference an LPM trie
directly, not just as the inner map of a map-of-maps. LPM trie nodes are
reclaimed via bpf_mem_cache_free_rcu(), which chains a regular RCU grace
period into a Tasks Trace grace period before freeing -- the same
discipline BPF_MAP_TYPE_HASH relies on for sleepable access.
Changes since v1:
- Split into a 2-patch series.
- Patch 1 now also converts the trie_update_elem()/trie_delete_elem()
walks from rcu_dereference() to rcu_dereference_protected(*p, 1),
addressing review feedback that v1 only fixed the lookup path and left
the same splat on the writer paths.
- New patch 2 adds the verifier whitelist entry so the fix is actually
reachable for directly-referenced LPM tries.
- Retitled v1 ("Allow lookups from sleepable BPF programs").
Vlad Poenaru [Tue, 9 Jun 2026 13:55:58 +0000 (06:55 -0700)]
bpf: Allow sleepable programs to use LPM trie maps directly
The previous change relaxed the rcu_dereference annotations in
lpm_trie.c so the trie walks no longer trip lockdep when reached from a
sleepable BPF program holding only rcu_read_lock_trace(). By itself
that only helps tries reached as the inner map of a map-of-maps, or
from the classic-RCU syscall path: a sleepable program that references
an LPM trie directly is still rejected at load time by
check_map_prog_compatibility(), whose sleepable whitelist omits
BPF_MAP_TYPE_LPM_TRIE:
Sleepable programs can only use array, hash, ringbuf and local storage maps
LPM trie nodes are allocated from a bpf_mem_alloc (trie->ma) and freed
with bpf_mem_cache_free_rcu(), which chains a regular RCU grace period
into a Tasks Trace grace period before the node -- and the value
embedded in it that trie_lookup_elem() returns to the program -- is
released. That is the same reclaim discipline BPF_MAP_TYPE_HASH relies
on for sleepable access, so a value handed to a sleepable reader cannot
be freed while the program is still running under rcu_read_lock_trace().
The writer paths take trie->lock across the walk and never relied on the
RCU read-side lock to keep nodes alive.
Add BPF_MAP_TYPE_LPM_TRIE to the sleepable map whitelist so these
programs can use LPM tries directly.
Vlad Poenaru [Tue, 9 Jun 2026 13:55:57 +0000 (06:55 -0700)]
bpf: Allow LPM map access from sleepable BPF programs
trie_lookup_elem() annotates its rcu_dereference_check() walks with
only rcu_read_lock_bh_held(). Because rcu_dereference_check(p, c)
resolves to "c || rcu_read_lock_held()", this passes for XDP/NAPI and
classic RCU readers but fails for sleepable BPF programs, which enter
via __bpf_prog_enter_sleepable() and hold only rcu_read_lock_trace().
trie_update_elem() and trie_delete_elem() have the same problem in a
different form: they walk the trie with plain rcu_dereference(), which
asserts rcu_read_lock_held() unconditionally. Both are reachable from
sleepable BPF programs via the bpf_map_update_elem / bpf_map_delete_elem
helpers, and from the syscall path under classic rcu_read_lock(). In
the writer paths the trie is actually protected by trie->lock (an
rqspinlock taken across the walk); we never relied on the RCU read-side
lock to keep nodes alive there.
A sleepable LSM hook that ends up touching an LPM trie therefore
triggers lockdep on debug kernels:
=============================
WARNING: suspicious RCU usage
7.1.0-... Tainted: G E
-----------------------------
kernel/bpf/lpm_trie.c:249 suspicious rcu_dereference_check() usage!
1 lock held by net_tests/540:
#0: (rcu_tasks_trace_srcu_struct){....}-{0:0},
at: __bpf_prog_enter_sleepable+0x26/0x280
Call Trace:
dump_stack_lvl
lockdep_rcu_suspicious
trie_lookup_elem
bpf_prog_..._enforce_security_socket_connect
bpf_trampoline_...
security_socket_connect
__sys_connect
do_syscall_64
This is lockdep-only -- no UAF, since Tasks Trace RCU does serialize
against the trie's reclaim path -- but it spams the console once per
distinct callsite on every debug kernel running a sleepable BPF LSM
that touches an LPM trie, which is increasingly common.
For the lookup path, switch the rcu_dereference_check() annotation
from rcu_read_lock_bh_held() to bpf_rcu_lock_held(), which accepts all
three contexts (classic, BH, Tasks Trace). Other map types already
follow this convention.
For trie_update_elem() and trie_delete_elem(), annotate the walks as
rcu_dereference_protected(*p, 1) -- matching trie_free() in the same
file -- since trie->lock is held across the walk. rqspinlock has no
lockdep_map, so the predicate degenerates to '1' rather than
lockdep_is_held(&trie->lock); the protection is real but not
machine-verifiable. trie_get_next_key() also uses bare
rcu_dereference() but is reachable only from the BPF syscall, which
holds classic rcu_read_lock() before dispatching, so it is left
untouched.
Fixes: 694cea395fde ("bpf: Allow RCU-protected lookups to happen from bh context") Cc: stable@vger.kernel.org Signed-off-by: Vlad Poenaru <vlad.wing@gmail.com> Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com> Link: https://lore.kernel.org/r/20260609135558.193287-2-vlad.wing@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Emil Tsalapatis [Tue, 9 Jun 2026 06:36:30 +0000 (02:36 -0400)]
selftests/bpf: Avoid spurious spmc parallel selftest errors in libarena
The libarena parallel spmc selftest is nondeterministic by design.
As a result it depends up to a point on the relative timing between the
producer and consumer threads. This introduces the possibility for two
kinds of spurious failures that this patch addresses.
1) Spurious timeouts. The test proceeds in phases, and threads use a
common counter as a barrier to avoid proceeding to the next phase
until all threads are ready to do so. If a thread takes too long to
reach the barrier, the already waiting threads may time out.
Increase the current timeout. The timeout's value is a balance
between the maximum amount of time spent on the test and the
possibility of spurious failures. Right now the timeout is too short.
Err on the side of caution and significantly increase it to avoid
spurious failures.
2) Spurious resize failures. Some selftests require the spmc queue to
resize itself. This in turn requires for the producer side to be
materially faster than the consumer side so that the queue gets full
enough for a resize. However, in the benchmark the spmc queue's producer
is outnumbered 3:1. To offset it we add busy waits for consume
queues. However, we still see occasional failures due to the queue
never resizing.
Minimize the possibility for this in two ways: First, remove one of
the consumers. The 2 consumers still exercise the "race between
consumers" scenario. Second, increase the busy wait duration to
decrease the rate by which the consumers act on the queue.
While at it, also replace a stray invalid error value "153" with EINVAL.
Fixes: 42998f819256 ("selftests/bpf: libarena: parallel test harness and spmc parallel selftest") Reported-by: Jakub Kicinski <kuba@kernel.org> Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com> Link: https://lore.kernel.org/r/20260609063630.10245-1-emil@etsalapatis.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
====================
bpf: Enforce BTF pointer write checks for global args
check_mem_reg() verifies both read and write access when a caller passes
memory into a global subprogram. For PTR_TO_BTF_ID callers,
check_helper_mem_access() currently always checks the access as BPF_READ.
That lets a tracing program pass a task_struct field pointer to a global
subprogram argument typed as writable memory. The direct field store is rejected
with "only read is supported", but the callee is validated with a generic
writable PTR_TO_MEM argument and can store through it.
Forward the requested access type into the PTR_TO_BTF_ID helper-access path and
add verifier coverage for the global-subprogram argument case.
Without this series:
direct BTF field store rejected with "only read is supported";
global-subprogram candidate loaded, attached, and runtime-confirmed.
With this series applied:
direct BTF field store rejected with "only read is supported";
global-subprogram candidate rejected with "only read is supported".
Nuoqi Gui [Tue, 9 Jun 2026 14:43:51 +0000 (22:43 +0800)]
selftests/bpf: Cover writable BTF field global subprog args
Add a verifier test for passing a BTF-backed task_struct field pointer to a
global subprogram argument typed as writable memory.
The direct field store is already rejected.
The global subprogram path should be rejected too.
The callee must not lose the BTF pointer's read-only provenance.
It must not validate the argument as ordinary writable memory.
Nuoqi Gui [Tue, 9 Jun 2026 14:43:50 +0000 (22:43 +0800)]
bpf: Enforce write checks for BTF pointer helper access
check_mem_reg() verifies both read and write access for global subprogram
memory arguments. When the caller register is PTR_TO_BTF_ID,
check_helper_mem_access() currently forwards the access to
check_ptr_to_btf_access() as BPF_READ regardless of the requested access
type.
This lets a BTF-backed kernel object field pointer pass the caller-side
writable memory check for a global subprogram argument. The callee is then
validated with a generic writable PTR_TO_MEM argument and can store through
it, even though an equivalent direct BTF field store is rejected with "only
read is supported".
Forward the requested access type to check_ptr_to_btf_access().
This enforces existing BTF write restrictions for global subprogram memory
arguments as well.
Paul Moses [Tue, 9 Jun 2026 10:08:54 +0000 (05:08 -0500)]
selftests/bpf: Add BTF repeated field count overflow test
Add a raw BTF test that exercises repeated special-field expansion with a
large array count. The compact element layout keeps the array byte size
representable while the repeated field count overflows the old u32 capacity
calculation in btf_repeat_fields().
Paul Moses [Fri, 5 Jun 2026 23:43:09 +0000 (23:43 +0000)]
bpf: Validate BTF repeated field counts before expansion
btf_parse_struct_metas() walks user-supplied BTF during BPF_BTF_LOAD,
and btf_repeat_fields() expands repeatable fields from array elements
into the fixed BTF_FIELDS_MAX scratch array used by btf_parse_fields().
The remaining-capacity check performs the expanded field count calculation
in u32. A malformed BTF can wrap that calculation, causing the check to
pass even when the expanded field count exceeds the scratch array
capacity. The following memcpy() can then write past the end of the
array.
Use checked addition and multiplication before copying repeated fields
and reject impossible counts.
Fixes: 797d73ee232d ("bpf: Check the remaining info_cnt before repeating btf fields") Cc: stable@vger.kernel.org Signed-off-by: Paul Moses <p@1g4.org> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://lore.kernel.org/bpf/20260605234301.1109063-1-p@1g4.org Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
cgroup_skb programs run in softirq, which may interrupt a task that is
itself in do_exit(). Once that task has passed
exit_notify() -> release_task() -> __unhash_process(), its thread_pid is
cleared, so task_active_pid_ns(current) returns NULL and find_pid_ns()
dereferences &NULL->idr:
An ARRAY_OF_MAPS can use an array created with BPF_F_INNER_MAP as its
inner map template. The flag allows a concrete inner array with a
different max_entries value to replace the template.
The verifier currently uses the template's max_entries to elide
nullness for a constant-key lookup through the inner map pointer. At
runtime, the lookup uses the concrete inner array's max_entries instead.
The verifier can therefore accept an unchecked dereference even though
the runtime helper returns NULL.
Patch 1 keeps lookups through BPF_F_INNER_MAP array templates nullable.
Patch 2 adds a verifier regression test for the unchecked dereference.
Before the fix, the regression program is accepted and the runtime
reproducer triggers a NULL dereference. With the fix, both programs are
rejected with an invalid map_value_or_null access.
Tested by compiling kernel/bpf/verifier.o and
verifier_map_in_map.bpf.o, and by running the regression program and
runtime reproducer in QEMU before and after the fix.
Signed-off-by: Nuoqi Gui <gnq25@mails.tsinghua.edu.cn> Acked-by: Jiri Olsa <jolsa@kernel.org>
---
v1->v2:
- Update the can_elide_value_nullness() comment to match the changed
parameter (const struct bpf_map *map).
To: Alexei Starovoitov <ast@kernel.org>
To: Daniel Borkmann <daniel@iogearbox.net>
To: Andrii Nakryiko <andrii@kernel.org> Cc: Daniel Xu <dxu@dxuuu.xyz> Cc: Eduard Zingerman <eddyz87@gmail.com> Cc: John Fastabend <john.fastabend@gmail.com> Cc: Martin KaFai Lau <martin.lau@linux.dev> Cc: Kumar Kartikeya Dwivedi <memxor@gmail.com> Cc: Song Liu <song@kernel.org> Cc: Yonghong Song <yonghong.song@linux.dev> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Ihor Solodrai <isolodrai@meta.com> Cc: bpf@vger.kernel.org Cc: linux-kernel@vger.kernel.org Cc: linux-kselftest@vger.kernel.org
---
====================
Nuoqi Gui [Sun, 7 Jun 2026 13:24:13 +0000 (21:24 +0800)]
bpf: Keep dynamic inner array lookups nullable
An ARRAY_OF_MAPS can use an array created with BPF_F_INNER_MAP as its
inner map template. A concrete inner array with a different max_entries
value can then replace the template.
After a successful outer map lookup, the verifier represents the
resulting map pointer using the inner map template. Const-key lookup
nullness elision consequently uses the template max_entries even though
the runtime helper uses the concrete inner map max_entries.
Do not elide lookup result nullness for maps marked with BPF_F_INNER_MAP,
because the template max_entries does not prove that the key is in bounds
for the concrete runtime map.
Fixes: d2102f2f5d75 ("bpf: verifier: Support eliding map lookup nullness") Signed-off-by: Nuoqi Gui <gnq25@mails.tsinghua.edu.cn> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Acked-by: Jiri Olsa <jolsa@kernel.org> Cc: stable@vger.kernel.org Link: https://lore.kernel.org/bpf/20260607-f01-v2-v2-1-da48453146e8@mails.tsinghua.edu.cn Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Tejun Heo [Mon, 8 Jun 2026 07:25:47 +0000 (21:25 -1000)]
arm64: mm: Complete the PTE store in ptep_try_set()
ptep_try_set() installs a kernel PTE with try_cmpxchg() but, unlike
__set_pte(), skips the barriers that arm64 requires after writing a valid
kernel PTE. Without them a subsequent access can fault instead of seeing
the new mapping.
Issue them with emit_pte_barriers() rather than __set_pte_complete().
ptep_try_set() must finish the store before it returns, but
__set_pte_complete() would defer the barriers when the calling context is in
lazy MMU mode.
v2: Emit the barriers directly instead of __set_pte_complete(). (Catalin)
This series fixes AA-deadlocks where NMI and tracepoint BPF programs
re-enter the per-CPU or global LRU lock already held on the same CPU
(syzbot c69a0a2c816716f1e0d5, 18b26edb69b2e19f3b33).
Patch 1 converts every LRU lock site to rqspinlock_t
and adds explicit recovery for some failures so no node leaks.
Patch 2 refreshes Documentation/bpf/map_lru_hash_update.dot to show
the new rqspinlock failure exits and recovery routes.
Patch 3 introduces a stress test.
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
---
Changes in v3:
- Removed RFC tag
- Link to v2: https://patch.msgid.link/20260603-lru_map_spin-v2-0-7060cfb6cdac@meta.com
Changes in v2:
- Patch 1: __bpf_lru_node_move_in() now clears pending_free only when
moving to the FREE list.
- Patch 3: address sashiko's feedback.
- Link to v1: https://patch.msgid.link/20260528-lru_map_spin-v1-0-4f52223170cf@meta.com
====================
Introduces stress test for bpf_lru_list that exercises
lock-failures and orphan-recovery, added by the LRU rqspinlock
conversion.
Runs three subtests: common LRU, per-CPU LRU lists (BPF_F_NO_COMMON_LRU),
and per-CPU LRU map. Each pins one userspace hammer per CPU and attaches
the perf_event NMI BPF prog (update+delete mix) on every online CPU.
Pre-fix, lockdep fires the "INITIAL USE -> IN-NMI" splat during stress.
After stress test, drain_then_verify_capacity() drains every key
and refills the lru map.
A stranded node on any CPU's pool would have forced eviction of
a just-inserted key on that CPU, surfacing here as a missing lookup.
Marked serial_ because per-CPU pinning and high-rate HW perf events
would perturb parallel tests.
Mykyta Yatsenko [Sun, 7 Jun 2026 20:30:42 +0000 (13:30 -0700)]
Documentation/bpf: Refresh map_lru_hash_update.dot for rqspinlock
Reflect the rqspinlock conversion and orphan-recovery paths added in
the previous commit:
- All LRU locks are rqspinlock_t; any acquire can fail (AA or
timeout). A shared "rqspinlock acquire failed" terminal collapses
to the existing -ENOMEM exit. Dashed arrows from each acquire site
mark the failure paths.
- The per-CPU local freelist is now lockless (free_llist).
- Post-steal: re-acquiring loc_l->lock to insert the stolen node
into the local pending list can fail; on failure the node is
published to free_llist instead of being orphaned, and the call
returns -ENOMEM.
- Steal-loop victim lock failure is silent: skip the victim and try
the next CPU.
Mykyta Yatsenko [Sun, 7 Jun 2026 20:30:41 +0000 (13:30 -0700)]
bpf: Fix NMI/tracepoint re-entry deadlock on lru locks
NMI and tracepoint BPF programs can re-enter the per-CPU or global
LRU lock that bpf_lru_pop_free()/push_free() already hold on the
same CPU, AA-deadlocking. Lockdep reports "inconsistent
{INITIAL USE} -> {IN-NMI}" on &l->lock (syzbot c69a0a2c816716f1e0d5)
and "possible recursive locking detected" on &loc_l->lock (syzbot 18b26edb69b2e19f3b33).
Prior trylock and rqspinlock based fixes (see links) were nacked
because compromised on reliability.
This patch converts every LRU lock site to rqspinlock_t and adds a
recovery path for some failure windows to avoid node leaks.
Failure recovery:
- *_pop_free top-level: return NULL; prealloc_lru_pop() already
treats that as no-free-element (-ENOMEM).
- Cross-CPU steal: skip the victim's locked loc_l, try next CPU.
- Post-steal local lock fail: publish stolen node to lockless
per-CPU free_llist; next pop on this CPU picks it up.
- push_free fail: mark node pending_free=1. __local_list_flush(),
__local_list_pop_pending() reclaim the node from pending_list.
__bpf_lru_list_shrink_inactive() reclaims the node from inactive
list. Nodes from active list are reclaimed by __bpf_lru_list_shrink()
or after __bpf_lru_list_rotate_active() demotes it to the inactive.
v8 changes:
- add back the btf_is_union check to btf_get_type_size [sashiko]
v7 changes:
- added ftrace_hash_count stub for !CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS cade [sashiko]
- selftests fixes [sashiko]
- use hash_ptr in select_trampoline_lock [sashiko]
- changed the check duplicate logic in check_dup_ids [sashiko]
- use sort_r_nonatomic in check_dup_ids [sashiko]
- added BPF_TRACE_FSESSION_MULTI to can_be_sleepable,
plus added testcase for sleepable fsession
- make bpf_tracing_multi_opts pointer fields as const
- add ___migrate_enable to trace_blacklist
v6 changes:
- move ftrace_hash_count declaration under CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS [sashiko]
- fix ftrace_hash_remove check/deref [sashiko]
- disable context access for multi programs by using stub function with no arguments
for verification [sashiko]
- add __used for bpf_multi_func, and removed arguments, we do not allow direct access [sashiko]
- rebased on latest loongarch changes, fix ppc build
- guard update_ftrace_direct_del with ftrace_hash_count on rollback [sashiko]
- fix noreturn attachment condition in bpf_check_attach_btf_id_multi [sashiko]
- fail early on multiple same IDs provided by user [sashiko]
- fix selftests error paths [sashiko]
- add MAX_RESOLVE_DEPTH check to btf_get_type_size [sashiko]
- use btf__pointer_size [sashiko]
- fixed compilation on powerpc [sashiko]
- added verifier fails selftest
- after discussing with Song, it was determined that cleaning up FTRACE_OPS_CMD_DISABLE_SHARE_IPMODIFY_PEER
is not strictly necessary — keeping the trampoline in the ipmodify_enabled state is acceptable.
The race condition this introduces remains unlikely, so the concern raised in [1] will not be
addressed at this time.
[1] https://lore.kernel.org/bpf/aec7bAbGlnEo3R1g@krava/
v5 changes:
- add dedicated hashes used for detach, so there's no need to allocate
them on detach [sashiko]
- safely release old trampoline images [sashiko]
- add cond_resched() to couple of loops [sashiko]
- validate attr->link_create.target_fd [sashiko]
- allow only bpf_get_func_ret() for return value retrieval [sashiko]
- do not allow attachment of fexit/fsession_multi for noreturn functions [sashiko]
- fixed double free/close in libbpf btf cleanup, in separate patch [sashiko]
- make btf_type_is_traceable_func closer to btf_distill_func_proto [sashiko]
- add prog->attach_btf_obj_fd check to collect_func_ids_by_glob,
to check we don't load module programs for kernel [sashiko]
- make sure program is loaded in bpf_program__attach_tracing_multi [sashiko]
- several selftests fixes [sashiko]
- add attach_type to fdinfo output [Leon Hwang]
- selftests cleanup fixes [Leon Hwang]
v4 changes:
- unlink rollback fix (added ftrace_hash_count) [bot]
- use const for some bpf_link_create_opts tracing_multi members [bot]
- adding missing comment for lockdep keys [bot]
- selftest error path fixes (leaks) and other assorted test fixes [Leon Hwang]
- several compile fixes wrt CONFIG_BPF_SYSCALL and CONFIG_BPF_JIT [kernel test robot]
- make ftrace_hash_clear global, because it's needed in rollback
v3 changes:
- fix module parsing [Leon Hwang]
- use function traceable check from libbpf [Leon Hwang]
- use ptr_to_u64 and fix/updated few comments [ci]
- display cookies as decimal numbers [ci]
- added link_create.flags check [ci]
- fix error path in bpf_trampoline_multi_detach [ci]
- make fentry/fexit.multi not extendable [ci]
- add missing OPTS_VALID to bpf_program__attach_tracing_multi [ci]
v2 changes:
- allocate data.unreg in bpf_trampoline_multi_attach for rollback path [ci]
and fixed link count setup in rollback path [ci]
- several small assorted fixes [ci]
- added loongarch and powerpc changes for struct bpf_tramp_node change
- added support to attach functions from modules
- added tests for sleepable programs
- added rollback tests
v1 changes:
- added ftrace_hash_count as wrapper for hash_count [Steven]
- added trampoline mutex pool [Andrii]
- reworked 'struct bpf_tramp_node' separatoin [Andrii]
- the 'struct bpf_tramp_node' now holds pointer to bpf_link,
which is similar to what we do for uprobe_multi;
I understand it's not a fundamental change compared to previous
version which used bpf_prog pointer instead, but I don't see better
way of doing this.. I'm happy to discuss this further if there's
better idea
- reworked 'struct bpf_fsession_link' based on bpf_tramp_node
- made btf__find_by_glob_kind function internal helper [Andrii]
- many small assorted fixes [Andrii,CI]
- added session support [Leon Hwang]
- added cookies support
- added more tests
Note I plan to send linkinfo support separately, the patchset is big enough.
Jiri Olsa [Sat, 6 Jun 2026 12:39:54 +0000 (14:39 +0200)]
selftests/bpf: Add tracing multi attach rollback tests
Adding tests for the rollback code when the tracing_multi
link won't get attached, covering 2 reasons:
- wrong btf id passed by user, where all previously allocated
trampolines will be released
- trampoline for requested function is fully attached (has already
maximum programs attached) and the link fails, the rollback code
needs to release all previously link-ed trampolines and release
them
We need the bpf_fentry_test* unattached for the tests to pass,
so the rollback tests are serial.
Jiri Olsa [Sat, 6 Jun 2026 12:39:47 +0000 (14:39 +0200)]
selftests/bpf: Add tracing multi skel/pattern/ids module attach tests
Adding tests for tracing_multi link attachment via all possible
libbpf apis - skeleton, function pattern and btf ids on top of
bpf_testmod kernel module.
User can specify functions to attach with 'pattern' argument that
allows wildcards (*?' supported) or provide BTF ids of functions
in array directly via opts argument. These options are mutually
exclusive.
When using BTF ids, user can also provide cookie value for each
provided id/function, that can be retrieved later in bpf program
with bpf_get_attach_cookie helper. Each cookie value is paired with
provided BTF id with the same array index.
Adding support to auto attach programs with following sections:
The provided <pattern> is used as 'pattern' argument in
bpf_program__attach_kprobe_multi_opts function.
The <pattern> allows to specify optional kernel module name with
following syntax:
<module>:<function_pattern>
In order to attach tracing_multi link to a module functions:
- program must be loaded with 'module' btf fd
(in attr::attach_btf_obj_fd)
- bpf_program__attach_tracing_multi must either have
pattern with module spec or BTF ids from the module
Jiri Olsa [Sat, 6 Jun 2026 12:39:44 +0000 (14:39 +0200)]
libbpf: Add btf_type_is_traceable_func function
Adding btf_type_is_traceable_func function to perform same checks
as the kernel's btf_distill_func_proto function to prevent attachment
on some of the functions.
Exporting the function via libbpf_internal.h because it will be used
by benchmark test in following changes.
Adding bpf_trampoline_multi_attach/detach functions that allows to
attach/detach tracing program to multiple functions/trampolines.
The attachment is defined with bpf_program and array of BTF ids of
functions to attach the bpf program to.
Adding bpf_tracing_multi_link object that holds all the attached
trampolines and is initialized in attach and used in detach.
The attachment allocates or uses currently existing trampoline
for each function to attach and links it with the bpf program.
The attach works as follows:
- we get all the needed trampolines
- lock them and add the bpf program to each (__bpf_trampoline_link_prog)
- the trampoline_multi_ops passed in __bpf_trampoline_link_prog gathers
ftrace_hash (ip -> trampoline) objects
- we call update_ftrace_direct_add/mod to update needed locations
- we unlock all the trampolines
The detach works as follows:
- we lock all the needed trampolines
- remove the program from each (__bpf_trampoline_unlink_prog)
- the trampoline_multi_ops passed in __bpf_trampoline_unlink_prog gathers
ftrace_hash (ip -> trampoline) objects
- we call update_ftrace_direct_del/mod to update needed locations
- we unlock and put all the trampolines
We store the old image/flags in the trampoline before the update
and use it in case we need to rollback the attachment.
We keep the ftrace_hash objects allocated during attach in the link
so they can be used for detach as well.
Adding trampoline_(un)lock_all functions to (un)lock all trampolines
to gate the tracing_multi attachment.
Note this is supported only for archs (x86_64) with ftrace direct and
have single ops support.
Jiri Olsa [Sat, 6 Jun 2026 12:39:36 +0000 (14:39 +0200)]
bpf: Move sleepable verification code to btf_id_allow_sleepable
Move sleepable verification code to btf_id_allow_sleepable function.
It will be used in following changes.
Adding code to retrieve type's name instead of passing it from
bpf_check_attach_target function, because this function will be
called from another place in following changes and it's easier
to retrieve the name directly in here.
Jiri Olsa [Sat, 6 Jun 2026 12:39:35 +0000 (14:39 +0200)]
bpf: Add multi tracing attach types
Adding new program attach types multi tracing attachment:
BPF_TRACE_FENTRY_MULTI
BPF_TRACE_FEXIT_MULTI
and their base support in verifier code.
Programs with such attach type will use specific link attachment
interface coming in following changes.
This was suggested by Andrii some (long) time ago and turned out
to be easier than having special program flag for that.
Bpf programs with such types have 'bpf_multi_func' function set as
their attach_btf_id and keep module reference when it's specified
by attach_prog_fd.
They are also accepted as sleepable programs during verification,
and the real validation for specific BTF_IDs/functions will happen
during the multi link attachment in following changes.
Jiri Olsa [Sat, 6 Jun 2026 12:39:34 +0000 (14:39 +0200)]
bpf: Factor fsession link to use struct bpf_tramp_node
Now that we split trampoline attachment object (bpf_tramp_node) from
the link object (bpf_tramp_link) we can use bpf_tramp_node as fsession's
fexit attachment object and get rid of the bpf_fsession_link object.
The link holds the bpf_prog pointer and forces one link - one program
binding logic. In following changes we want to attach program to multiple
trampolines but we want to keep just one bpf_link object.
The 'struct bpf_tramp_link' defines standard single trampoline link
and 'struct bpf_tramp_node' is the attachment trampoline object with
pointer to the bpf_link object.
This will allow us to define link for multiple trampolines, like:
Jiri Olsa [Sat, 6 Jun 2026 12:39:32 +0000 (14:39 +0200)]
bpf: Add bpf_trampoline_add/remove_prog functions
Separate bpf_trampoline_add/remove_prog functions from
__bpf_trampoline_link/unlink functions to be able to add/remove
trampoline programs without the image being updated in following
changes. No functional change is intended.
Jiri Olsa [Sat, 6 Jun 2026 12:39:31 +0000 (14:39 +0200)]
bpf: Move trampoline image setup into bpf_trampoline_ops callbacks
Moving trampoline image setup into bpf_trampoline_ops callbacks,
so we can have different image handling for multi attachment which
is coming in following changes.
Jiri Olsa [Sat, 6 Jun 2026 12:39:30 +0000 (14:39 +0200)]
bpf: Add struct bpf_trampoline_ops object
In following changes we will need to override ftrace direct attachment
behaviour. In order to do that we are adding struct bpf_trampoline_ops
object that defines callbacks for ftrace direct attachment:
register_fentry
unregister_fentry
modify_fentry
The new struct bpf_trampoline_ops object is passed as an argument to
__bpf_trampoline_link/unlink_prog functions.
At the moment the default trampoline_ops is set to the current ftrace
direct attachment functions, so there's no functional change for the
current code.
Jiri Olsa [Sat, 6 Jun 2026 12:39:29 +0000 (14:39 +0200)]
bpf: Use mutex lock pool for bpf trampolines
Adding mutex lock pool that replaces bpf trampolines mutex.
For tracing_multi link coming in following changes we need to lock all
the involved trampolines during the attachment. This could mean thousands
of mutex locks, which is not convenient.
As suggested by Andrii we can replace bpf trampolines mutex with mutex
pool, where each trampoline is hash-ed to one of the locks from the pool.
It's better to lock all the pool mutexes (32 at the moment) than
thousands of them.
There is 48 (MAX_LOCK_DEPTH) lock limit allowed to be simultaneously
held by task, so we need to keep 32 mutexes (5 bits) in the pool, so
when we lock them all in following changes the lockdep won't scream.
Removing the mutex_is_locked in bpf_trampoline_put, because we removed
the mutex from bpf_trampoline.
David Windsor [Fri, 5 Jun 2026 14:57:07 +0000 (10:57 -0400)]
bpf: Reject sleepable BPF_LSM_CGROUP programs at load time
The cgroup shim runs under rcu_read_lock_dont_migrate(), so we should
not attach any sleepable BPF programs there. Add support to the verifier
to explicitly reject attempts to load sleepable BPF programs destined
for LSM cgroup attachment.
Without this, we get the following splat from a BPF_LSM_CGROUP
program marked BPF_F_SLEEPABLE attached to file_open when it calls
bpf_get_dentry_xattr():
libbpf provides a .s named section variant for every sleepable
program type except lsm_cgroup, reflecting that per-cgroup LSM programs
are intended to only run in a non-sleepable context.
The above splat was obtained by bypassing libbpf by using bpf(2)
directly.
Fixes: 69fd337a975c ("bpf: per-cgroup lsm flavor") Signed-off-by: David Windsor <dwindsor@gmail.com> Acked-by: Yonghong Song <yonghong.song@linux.dev> Acked-by: Song Liu <song@kernel.org> Link: https://lore.kernel.org/bpf/20260605145707.608579-1-dwindsor@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
A constant offset added to a PTR_TO_FLOW_KEYS register lands in
reg->var_off, but check_flow_keys_access() bounds-checks only insn->off
and never folds reg->var_off.value. A BPF_PROG_TYPE_FLOW_DISSECTOR
program can therefore do "flow_keys += 0x1000; *(flow_keys + 0)" and have
it accepted, then read/write kernel stack past struct bpf_flow_keys at
runtime. Patch 1 folds reg->var_off.value into the offset (and rejects
non-constant offsets), mirroring check_ctx_access(); patch 2 adds verifier
selftests.
This is a regression introduced in the 7.1 development cycle by commit 022ac0750883 ("bpf: use reg->var_off instead of reg->off for pointers"),
which moved the constant offset from reg->off (folded generically before 022ac0750883) into reg->var_off without updating the flow_keys path. No
released kernel is affected: v7.0.x rejects the program above, and the bug
reproduces only on v7.1-rc1..rc5, so no stable backport is needed.
It was first reported privately to security@kernel.org; per their guidance
it is handled in the open as a normal regression fix. Found by manual
verifier audit and confirmed dynamically in a disposable QEMU/KVM guest:
the load above is accepted, a runtime read leaked a kernel-stack pointer
0x1000 past bpf_flow_keys, and a runtime write of a marker faulted the
guest in net_rx_action.
An alternative -- forbidding pointer arithmetic on PTR_TO_FLOW_KEYS
outright by dropping "if (known) break;" in adjust_ptr_min_max_vals() --
was rejected because v7.0.x accepted (and correctly bounds-checked)
constant arithmetic on the keys pointer; restoring the fold preserves that
behaviour while closing the divergence.
Signed-off-by: Nuoqi Gui <gnq25@mails.tsinghua.edu.cn>
---
v2 -> v3:
- Pass existing reg/argno context into check_flow_keys_access(), avoiding
a stale regno reference in check_mem_access().
- Add a variable-offset selftest using bpf_get_prandom_u32().
v1 -> v2:
- Target bpf-next instead of bpf (per reviewer feedback).
- Base-commit updated to bpf-next/master.
Nuoqi Gui [Sat, 6 Jun 2026 10:50:38 +0000 (18:50 +0800)]
selftests/bpf: add tests for PTR_TO_FLOW_KEYS offset bounds
Add verifier tests covering pointer arithmetic on a PTR_TO_FLOW_KEYS
register. This covers the bpf-next regression where an out-of-bounds
constant offset introduced as flow_keys += K and then dereferenced at
insn->off 0 was accepted, while the equivalent flow_keys + K direct offset
was rejected.
The tests check that in-bounds constant arithmetic on the keys pointer is
still accepted, out-of-bounds constant arithmetic is rejected for both read
and write, and a truly varying offset from bpf_get_prandom_u32() remains
rejected by the existing PTR_TO_FLOW_KEYS pointer arithmetic rules.
Nuoqi Gui [Sat, 6 Jun 2026 10:50:37 +0000 (18:50 +0800)]
bpf: Fold reg->var_off into PTR_TO_FLOW_KEYS bounds check
Constant pointer arithmetic on a PTR_TO_FLOW_KEYS register lands the
constant in reg->var_off (e.g. flow_keys(imm=4096)), but the
PTR_TO_FLOW_KEYS path in check_mem_access() passes only insn->off to
check_flow_keys_access() and never folds reg->var_off.value. The
verifier therefore accepts an access that, at runtime, dereferences past
struct bpf_flow_keys -- a verifier/runtime divergence that yields an
out-of-bounds read and write of kernel stack memory.
Commit 022ac0750883 ("bpf: use reg->var_off instead of reg->off for
pointers") removed the generic "off += reg->off" that check_mem_access()
applied before the per-type dispatch and replaced it with per-path
folding of reg->var_off.value (for example the ctx path now folds the
register offset via check_ctx_access()). The PTR_TO_FLOW_KEYS path was
not given the equivalent fold, so a constant offset that used to be
folded and rejected is now silently accepted:
before 022ac0750883: the offset stays in reg->off and is folded
generically, so the access is checked with off=4096 and rejected.
after 022ac0750883: the offset lands in reg->var_off, the flow_keys
path checks off=0 and accepts; at runtime the access dereferences
base + 0x1000.
For a BPF_PROG_TYPE_FLOW_DISSECTOR program the following is accepted:
has the same effective offset but is correctly rejected with
"invalid access to flow keys off=4096 size=8", which isolates the defect
to the missing var_off fold. Once attached as a flow dissector, the
accepted program reads kernel stack past struct bpf_flow_keys (a
kernel-stack / KASLR information leak) and can likewise write past it,
corrupting kernel memory.
Fix it by folding reg->var_off.value into the offset before the bounds
check and rejecting non-constant offsets, mirroring the other pointer
types (e.g. check_ctx_access()).
Daniel Borkmann [Fri, 5 Jun 2026 21:35:18 +0000 (23:35 +0200)]
selftests/bpf: Inspect the signature verdict exposed to BPF LSM
Add a minimal BPF LSM program on lsm/bpf_prog_load that, for loads on
the monitored thread, reads back prog->aux->sig.{verdict,keyring_type,
keyring_serial}, and a signed_loader subtest that drives the same
gen_loader loader through the hook twice: i) /unsigned/ where the LSM
must observe UNSIGNED, no keyring and serial 0; ii) /signed/ where the
very same insns signed against the session keyring must be observed as
VERIFIED with a user keyring, and the recorded keyring_serial must be
equal to the resolved session keyring serial. Loading (not running) the
loader is sufficient since the verdict is attached at load time.
KP Singh [Fri, 5 Jun 2026 21:35:17 +0000 (23:35 +0200)]
bpf: Expose signature verdict via bpf_prog_aux
BPF_PROG_LOAD verifies the loader signature but does not record the
outcome on the BPF program. [BPF] LSMs and audit can read attr->signature
and attr->keyring_id to infer "was this signed, and if so, against which
keyring".
Add prog->aux->sig (verdict + keyring_{type,serial}), populated by
bpf_prog_load before the LSM hook. keyring_type classifies the keyring
the load referenced (builtin, secondary, platform or user), while
keyring_serial records the serial of the keyring the signature was
actually validated against. System keyrings carry a pseudo key pointer
with no user-visible serial and are reported as 0, as are unsigned loads.
Failed verifications reject the load before the hook runs, so it observes
only either UNSIGNED or VERIFIED.
Signed-off-by: KP Singh <kpsingh@kernel.org> Co-developed-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/r/20260605213518.544262-1-daniel@iogearbox.net Signed-off-by: Alexei Starovoitov <ast@kernel.org>
====================
selftests/bpf: libarena: Add initial data structures
Add two new data structures to libarena. These data structures initially
resided in the sched-ext repo (https://github.com/sched-ext/scx) and
have been adapted to the internal libarena build system. The data
structures are:
- Red black tree: Fundamental tree data structure that can also serve
as a base for more domain-specific data structures.
- Lev-Chase deque: Queue data structure that allows efficient work
stealing, useful in scheduling scenarios.
The data structures are accompanied by selftests that are automatically
discovered by the existing libarena test_progs selftest and incorporated
in the CI.
CHANGELOG
=========
v3 -> v4 (https://lore.kernel.org/bpf/20260604235016.20856-1-emil@etsalapatis.com/)
- Turn off load_acquire/store_relesase - dependent selftests for s390 (CI)
- Various style/non-functional nits (AI)
- Add workaround to handle LLVM 21 and GCC 15 assignment-to-memset promotions
that are causing verification failures for arena programs (CI)
- Incorporate Sashiko feedback for cleanup edge cases (Sashiko)
- Simplify some of the ordering semantics in spmc
- Rename tests from st_ to test_ (Alexei)
- Removed the freelist caches from the rbtrees, previously used to defer freeing (Alexei)
- Moved the type and function definitions to use the __arena identifier
- Removed the typecasts during function return and directly return __arena
pointers (Alexei)
- Renamed queues to spmc queues to abstract away the algorithm (Alexei)
- Adjusted the memory barriers in the spmc queue
- Added multithreaded testing harness for libarena programs (Alexei)
- Added parallel selftest for queues (Alexei)
- Split least upper bound and exact find operations back into separate
functions to prevent RB_DUPLICATE-related bug (AI)
====================
Emil Tsalapatis [Fri, 5 Jun 2026 22:20:20 +0000 (18:20 -0400)]
selftests/bpf: libarena: parallel test harness and spmc parallel selftest
Add a parallel test for the SPMC Lev-Chase workstealing queue. The queue
is built to be wait-free even when there are multiple consumers, and
the parallel selftest provides a signal on whether the queue behaves
correctly when stress tested.
To support the test, this patch includes a test harness for parallel
selftests. The spmc selftest acts as an example of the naming and other
conventions expected by the harness.
Emil Tsalapatis [Fri, 5 Jun 2026 22:20:19 +0000 (18:20 -0400)]
selftests/bpf: libarena: Add spmc queue data structure
Expand libarena with a single producer multiple consumer deque data
structure. This is a single producer, multiple consumer lockless structure
that permits efficient work stealing. The structure is a Lev-Chase queue,
so it is lock-free and wait-free.
The data structure exposes three main calls. two of them are available to
the thread owning the queue and one available to all threads in the program:
spmc_owner_push(): Push an item to the top of the queue.
spmc_owner_pop(): Pop an item from the top of the queue.
spmc_steal(): Steal a thread from the bottom of the queue from
any thread.
Note that the queue is not really FIFO for all consumers, since
non-owners of the queue can only work steal from the bottom.
Emil Tsalapatis [Fri, 5 Jun 2026 22:20:18 +0000 (18:20 -0400)]
selftests/bpf: libarena: Add rbtree data structure
Add a native red-black tree data structure to libarena.
The data structure supports multiple APIs (key-value based,
node based) with which users can query and modify it. The
tree uses the libarena memory allocator to manage its data.
Sean Young [Fri, 5 Jun 2026 15:14:16 +0000 (16:14 +0100)]
selftests/bpf: Fix test_lirc test
Since commit 68a99f6a0ebf ("media: lirc: report ir receiver overflow"),
the rc-loopback driver does not accept edges over 50ms, as these are
never seen in real life ir protocols. Fix this.
====================
Add validation for bpf_set_retval helper
From: Xu Kuohai <xukuohai@huawei.com>
The bpf_set_retval() helper is used by cgroup BPF programs to set the
return value of the kernel hook. The argument type for this helper is
ARG_ANYTHING. This allows setting a positive value, which no cgroup
hook expects and can cause issues, such as the kernel panic reported
in [1].
This series adds validation for the argument of the bpf_set_retval()
helper.
For BPF_LSM_CGROUP, the same validation as BPF_LSM_MAC is enforced,
i.e. validate the argument against the LSM hook specific range, which
is returned by bpf_lsm_get_retval_range().
For all other cgroup program types, restrict the argument to
[-MAX_ERRNO, 0], which matches the kernel convention of 0 for success
and negative errno for error.
BPF_CGROUP_GETSOCKOPT is an exception from this restriction, since valid
getsockopt implementations may return positive values (e.g. optlen), as
allowed by commit c4dcfdd406aa ("bpf: Move getsockopt retval to struct
bpf_cg_run_ctx").
v5:
- Use resolve_prog_type(env->prog) instead of env->prog->type for prog type checks
- Target bpf-next tree
v4: https://lore.kernel.org/bpf/20260604130458.617765-1-xukuohai@huaweicloud.com
- Remove the return value limit for BPF_CGROUP_GETSOCKOPT type
- Refine the range of return value of bpf_get_retval helper
v3: https://lore.kernel.org/bpf/20260530101239.590395-1-xukuohai@huaweicloud.com/
- Mark R1 as precise to prevent validation bypass via branch pruning (sashiko)
v2: https://lore.kernel.org/bpf/20260530055557.549474-1-xukuohai@huaweicloud.com/
- Extend validation from LSM cgroup BPF type to all cgroup BPF types (sashiko)
Xu Kuohai [Fri, 5 Jun 2026 14:02:42 +0000 (14:02 +0000)]
bpf: Add validation for bpf_set_retval argument
The bpf_set_retval() helper is used by cgroup BPF programs to set the
return value of the target hook. The argument type for this helper is
ARG_ANYTHING. This allows setting a positive value, which no cgroup
hook expects and can cause issues, such as:
- BPF_LSM_CGROUP: a positive value from bpf_lsm_socket_create bypasses
the err < 0 check in __sock_create(), leaving the socket object
unallocated. The positive return value is then propagated to the
syscall entry __sys_socket(), which also bypasses the IS_ERR() guard
and ultimately causes a NULL pointer dereference.
- BPF_CGROUP_DEVICE: a positive value can be returned through cgroup
device bpf prog -> devcgroup_check_permission() -> bdev_permission()
-> bdev_file_open_by_dev(), where ERR_PTR(positive) produces a pointer
that IS_ERR() does not catch, leading to a wild pointer dereference.
- BPF_CGROUP_SOCK: a positive value can be returned through cgroup sock
bpf prog -> __cgroup_bpf_run_filter_sk() -> inet_create() ->
__sock_create(), where inet_create() frees the newly allocated sk
via sk_common_release() and sets sock->sk = NULL on the non-zero
return, but __sock_create() only checks err < 0 for cleanup, so a
positive retval bypasses cleanup and returns a socket with NULL sk
to userspace, triggering a NULL pointer dereference on subsequent
socket operations.
- BPF_CGROUP_SYSCTL: a positive value can be returned through the cgroup
bpf prog -> __cgroup_bpf_run_filter_sysctl() -> proc_sys_call_handler(),
where a non-zero return bypasses the normal sysctl proc_handler and is
returned directly to userspace as return value of read() or write()
syscall.
So add validation for the argument of the bpf_set_retval() helper.
For BPF_LSM_CGROUP, enforce the LSM hook specific range returned by
bpf_lsm_get_retval_range().
For all other cgroup program types, restrict the argument to
[-MAX_ERRNO, 0], which matches the kernel convention of 0 for success
and negative errno for error.
BPF_CGROUP_GETSOCKOPT is an exception, since valid getsockopt
implementations may return positive values, as allowed by commit c4dcfdd406aa ("bpf: Move getsockopt retval to struct bpf_cg_run_ctx").
Also refine the return value range of bpf_get_retval() so that
values returned by bpf_get_retval() can be passed directly to
bpf_set_retval() without extra manual bounds checking.
Xu Kuohai [Fri, 5 Jun 2026 14:02:41 +0000 (14:02 +0000)]
selftests/bpf: Restrict bpf_set_retval argument in sk_bypass_prot_mem
Test sk_bypass_prot_mem passes an unchecked value as argument to helper
bpf_set_retval(). The argument can be outside the valid range enforced
by the strict retval validation added in the next patch.
Restrict the argument to -EFAULT when it is outside the valid range, so
the test will not be rejected by the verifier when retval validation
is enforced.
====================
bpf: fix sysctl new-value handling in __cgroup_bpf_run_filter_sysctl
This series fixes three bugs in the sysctl write-buffer replacement path
of __cgroup_bpf_run_filter_sysctl(). It resolves a kvzalloc()/kfree()
mismatch, adds a missing NUL terminator to the replacement string, and
updates a stale return value check to safely restore the replacement
functionality.
Patch Summary:
- patch 1 NUL-terminates the replaced sysctl value
- patch 2 uses kvfree() for the replaced sysctl write buffer
- patch 3 restores sysctl new-value replacement
Changelog:
v2 -> v3:
- reordered patches 1 and 2
- added the missing Reviewed-by/Acked-by tags to patches 2 and 3
- fixed the incorrect Fixes tag in patch 3
- simplified the dynamic test logs in patch 1 and 2, and updated
titles
====================
projects / linux / log