summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2025-03-18bpf: clarify a misleading verifier error messageAndrea Terzolo
The current verifier error message states that tail_calls are not allowed in non-JITed programs with BPF-to-BPF calls. While this is accurate, it is not the only scenario where this restriction applies. Some architectures do not support this feature combination even when programs are JITed. This update improves the error message to better reflect these limitations. Suggested-by: Shung-Hsi Yu <shung-hsi.yu@suse.com> Signed-off-by: Andrea Terzolo <andreaterzolo3@gmail.com> Acked-by: Shung-Hsi Yu <shung-hsi.yu@suse.com> Link: https://lore.kernel.org/r/20250318083551.8192-1-andreaterzolo3@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-18Merge branch 'bpf-reject-attaching-fexit-fmod_ret-to-noreturn-functions'Alexei Starovoitov
Yafang Shao says: ==================== Attaching fexit probes to functions marked with __noreturn may lead to unpredictable behavior. To avoid this, we will reject attaching probes to such functions. Currently, there is no ideal solution, so we will hardcode a check for all __noreturn functions. Once a more robust solution is implemented, this workaround can be removed. v4->v5: - Remove unnecessary functions (Alexei) - Use BTF_ID directly (Alexei) v3->v4: https://lore.kernel.org/bpf/20250317121735.86515-1-laoar.shao@gmail.com/ - Reject also fmod_ret (Alexei) - Fix build warnings and remove unnecessary functions (Alexei) v1->v2: https://lore.kernel.org/bpf/20250223062735.3341-1-laoar.shao@gmail.com/ - keep tools/objtool/noreturns.h as is (Josh) - Add noreturns.h to objtool/sync-check.sh (Josh) - Add verbose for the reject and simplify the test case (Song) v1: https://lore.kernel.org/bpf/20250211023359.1570-1-laoar.shao@gmail.com/ ==================== Link: https://patch.msgid.link/20250318114447.75484-1-laoar.shao@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-18selftests/bpf: Add selftest for attaching fexit to __noreturn functionsYafang Shao
The reuslt: $ tools/testing/selftests/bpf/test_progs --name=fexit_noreturns #99/1 fexit_noreturns/noreturns:OK #99 fexit_noreturns:OK Summary: 1/1 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Yafang Shao <laoar.shao@gmail.com> Link: https://lore.kernel.org/r/20250318114447.75484-3-laoar.shao@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-18bpf: Reject attaching fexit/fmod_ret to __noreturn functionsYafang Shao
If we attach fexit/fmod_ret to __noreturn functions, it will cause an issue that the bpf trampoline image will be left over even if the bpf link has been destroyed. Take attaching do_exit() with fexit for example. The fexit works as follows, bpf_trampoline + __bpf_tramp_enter + percpu_ref_get(&tr->pcref); + call do_exit() + __bpf_tramp_exit + percpu_ref_put(&tr->pcref); Since do_exit() never returns, the refcnt of the trampoline image is never decremented, preventing it from being freed. That can be verified with as follows, $ bpftool link show <<<< nothing output $ grep "bpf_trampoline_[0-9]" /proc/kallsyms ffffffffc04cb000 t bpf_trampoline_6442526459 [bpf] <<<< leftover In this patch, all functions annotated with __noreturn are rejected, except for the following cases: - Functions that result in a system reboot, such as panic, machine_real_restart and rust_begin_unwind - Functions that are never executed by tasks, such as rest_init and cpu_startup_entry - Functions implemented in assembly, such as rewind_stack_and_make_dead and xen_cpu_bringup_again, lack an associated BTF ID. With this change, attaching fexit probes to functions like do_exit() will be rejected. $ ./fexit libbpf: prog 'fexit': BPF program load failed: -EINVAL libbpf: prog 'fexit': -- BEGIN PROG LOAD LOG -- Attaching fexit/fmod_ret to __noreturn functions is rejected. Signed-off-by: Yafang Shao <laoar.shao@gmail.com> Link: https://lore.kernel.org/r/20250318114447.75484-2-laoar.shao@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-18bpf: Only fails the busy counter check in bpf_cgrp_storage_get if it creates ↵Martin KaFai Lau
storage The current cgrp storage has a percpu counter, bpf_cgrp_storage_busy, to detect potential deadlock at a spin_lock that the local storage acquires during new storage creation. There are false positives. It turns out to be too noisy in production. For example, a bpf prog may be doing a bpf_cgrp_storage_get on map_a. An IRQ comes in and triggers another bpf_cgrp_storage_get on a different map_b. It will then trigger the false positive deadlock check in the percpu counter. On top of that, both are doing lookup only and no need to create new storage, so practically it does not need to acquire the spin_lock. The bpf_task_storage_get already has a strategy to minimize this false positive by only failing if the bpf_task_storage_get needs to create a new storage and the percpu counter is busy. Creating a new storage is the only time it must acquire the spin_lock. This patch borrows the same idea. Unlike task storage that has a separate variant for tracing (_recur) and non-tracing, this patch stays with one bpf_cgrp_storage_get helper to keep it simple for now in light of the upcoming res_spin_lock. The variable could potentially use a better name noTbusy instead of nobusy. This patch follows the same naming in bpf_task_storage_get for now. I have tested it by temporarily adding noinline to the cgroup_storage_lookup(), traced it by fentry, and the fentry program succeeded in calling bpf_cgrp_storage_get(). Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org> Link: https://lore.kernel.org/r/20250318182759.3676094-1-martin.lau@linux.dev Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-18bpf: Make perf_event_read_output accessible in all program types.Emil Tsalapatis
The perf_event_read_event_output helper is currently only available to tracing protrams, but is useful for other BPF programs like sched_ext schedulers. When the helper is available, provide its bpf_func_proto directly from the bpf base_proto. Signed-off-by: Emil Tsalapatis (Meta) <emil@etsalapatis.com> Acked-by: Jiri Olsa <jolsa@kernel.org> Link: https://lore.kernel.org/r/20250318030753.10949-1-emil@etsalapatis.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-17Merge branch 'bpftool-using-the-right-format-specifiers'Andrii Nakryiko
Jiayuan Chen says: ==================== bpftool: Using the right format specifiers This patch adds the -Wformat-signedness compiler flag to detect and prevent format string errors, where signed or unsigned types are mismatched with format specifiers. Additionally, it fixes some format string errors that were not fully addressed by the previous patch [1]. [1] https://lore.kernel.org/bpf/20250207123706.727928-1-mrpre@163.com/T/#u --- v1->v2: https://lore.kernel.org/bpf/20250310142037.45932-1-jiayuan.chen@linux.dev/ --- ==================== Link: https://patch.msgid.link/20250311112809.81901-1-jiayuan.chen@linux.dev Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
2025-03-17bpftool: Using the right format specifiersJiayuan Chen
Fixed some formatting specifiers errors, such as using %d for int and %u for unsigned int, as well as other byte-length types. Perform type cast using the type derived from the data type itself, for example, if it's originally an int, it will be cast to unsigned int if forced to unsigned. Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20250311112809.81901-3-jiayuan.chen@linux.dev
2025-03-17bpftool: Add -Wformat-signedness flag to detect format errorsJiayuan Chen
This commit adds the -Wformat-signedness compiler flag to detect and prevent printf format errors, where signed or unsigned types are mismatched with format specifiers. This helps to catch potential issues at compile-time, ensuring that our code is more robust and reliable. With this flag, the compiler will now warn about incorrect format strings, such as using %d with unsigned types or %u with signed types. Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20250311112809.81901-2-jiayuan.chen@linux.dev
2025-03-17Merge branch 'support-freplace-prog-from-user-namespace'Andrii Nakryiko
Mykyta Yatsenko says: ==================== Support freplace prog from user namespace From: Mykyta Yatsenko <yatsenko@meta.com> Freplace programs can't be loaded from user namespace, as bpf_program__set_attach_target() requires searching for target prog BTF, which is locked under CAP_SYS_ADMIN. This patch set enables this use case by: 1. Relaxing capable check in bpf's BPF_BTF_GET_FD_BY_ID, check for CAP_BPF instead of CAP_SYS_ADMIN, support BPF token in attr argument. 2. Pass BPF token around libbpf from bpf_program__set_attach_target() to bpf syscall where capable check is. 3. Validate positive/negative scenarios in selftests This patch set is enabled by the recent libbpf change[1], that introduced bpf_object__prepare() API. Calling bpf_object__prepare() for freplace program before bpf_program__set_attach_target() initializes BPF token, which is then passed to bpf syscall by libbpf. [1] https://lore.kernel.org/all/20250303135752.158343-1-mykyta.yatsenko5@gmail.com/ ==================== Link: https://patch.msgid.link/20250317174039.161275-1-mykyta.yatsenko5@gmail.com Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
2025-03-17selftests/bpf: Test freplace from user namespaceMykyta Yatsenko
Add selftests to verify that it is possible to load freplace program from user namespace if BPF token is initialized by bpf_object__prepare before calling bpf_program__set_attach_target. Negative test is added as well. Modified type of the priv_prog to xdp, as kprobe did not work on aarch64 and s390x. Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Acked-by: Yonghong Song <yonghong.song@linux.dev> Link: https://lore.kernel.org/bpf/20250317174039.161275-5-mykyta.yatsenko5@gmail.com
2025-03-17libbpf: Pass BPF token from find_prog_btf_id to BPF_BTF_GET_FD_BY_IDMykyta Yatsenko
Pass BPF token from bpf_program__set_attach_target to BPF_BTF_GET_FD_BY_ID bpf command. When freplace program attaches to target program, it needs to look up for BTF of the target, this may require BPF token, if, for example, running from user namespace. Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Acked-by: Yonghong Song <yonghong.song@linux.dev> Link: https://lore.kernel.org/bpf/20250317174039.161275-4-mykyta.yatsenko5@gmail.com
2025-03-17bpf: Return prog btf_id without capable checkMykyta Yatsenko
Return prog's btf_id from bpf_prog_get_info_by_fd regardless of capable check. This patch enables scenario, when freplace program, running from user namespace, requires to query target prog's btf. Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Acked-by: Yonghong Song <yonghong.song@linux.dev> Link: https://lore.kernel.org/bpf/20250317174039.161275-3-mykyta.yatsenko5@gmail.com
2025-03-17bpf: BPF token support for BPF_BTF_GET_FD_BY_IDMykyta Yatsenko
Currently BPF_BTF_GET_FD_BY_ID requires CAP_SYS_ADMIN, which does not allow running it from user namespace. This creates a problem when freplace program running from user namespace needs to query target program BTF. This patch relaxes capable check from CAP_SYS_ADMIN to CAP_BPF and adds support for BPF token that can be passed in attributes to syscall. Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20250317174039.161275-2-mykyta.yatsenko5@gmail.com
2025-03-15bpf, x86: Fix objtool warning for timed may_gotoKumar Kartikeya Dwivedi
Kernel test robot reported "call without frame pointer save/setup" warning in objtool. This will make stack traces unreliable on CONFIG_UNWINDER_FRAME_POINTER=y, however it works on CONFIG_UNWINDER_ORC=y. Fix this by creating a stack frame for the function. Fixes: 2fb761823ead ("bpf, x86: Add x86 JIT support for timed may_goto") Reported-by: kernel test robot <lkp@intel.com> Closes: https://lore.kernel.org/oe-kbuild-all/202503071350.QOhsHVaW-lkp@intel.com/ Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Link: https://lore.kernel.org/r/20250315013039.1625048-1-memxor@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15bpf: Check map->record at the beginning of check_and_free_fields()Hou Tao
When there are no special fields in the map value, there is no need to invoke bpf_obj_free_fields(). Therefore, checking the validity of map->record in advance. After the change, the benchmark result of the per-cpu update case in map_perf_test increased by 40% under a 16-CPU VM. Signed-off-by: Hou Tao <houtao1@huawei.com> Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Link: https://lore.kernel.org/r/20250315150930.1511727-1-houtao@huaweicloud.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15selftests/bpf: Fix sockopt selftest failure on powerpcSaket Kumar Bhaskar
The SO_RCVLOWAT option is defined as 18 in the selftest header, which matches the generic definition. However, on powerpc, SO_RCVLOWAT is defined as 16. This discrepancy causes sol_socket_sockopt() to fail with the default switch case on powerpc. This commit fixes by defining SO_RCVLOWAT as 16 for powerpc. Signed-off-by: Saket Kumar Bhaskar <skb99@linux.ibm.com> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Tested-by: Venkat Rao Bagalkote <venkat88@linux.ibm.com> Link: https://lore.kernel.org/bpf/20250311084647.3686544-1-skb99@linux.ibm.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15selftests/bpf: Fix string read in strncmp benchmarkViktor Malik
The strncmp benchmark uses the bpf_strncmp helper and a hand-written loop to compare two strings. The values of the strings are filled from userspace. One of the strings is non-const (in .bss) while the other is const (in .rodata) since that is the requirement of bpf_strncmp. The problem is that in the hand-written loop, Clang optimizes the reads from the const string to always return 0 which breaks the benchmark. Use barrier_var to prevent the optimization. The effect can be seen on the strncmp-no-helper variant. Before this change: # ./bench strncmp-no-helper Setting up benchmark 'strncmp-no-helper'... Benchmark 'strncmp-no-helper' started. Iter 0 (112.309us): hits 0.000M/s ( 0.000M/prod), drops 0.000M/s, total operations 0.000M/s Iter 1 (-23.238us): hits 0.000M/s ( 0.000M/prod), drops 0.000M/s, total operations 0.000M/s Iter 2 ( 58.994us): hits 0.000M/s ( 0.000M/prod), drops 0.000M/s, total operations 0.000M/s Iter 3 (-30.466us): hits 0.000M/s ( 0.000M/prod), drops 0.000M/s, total operations 0.000M/s Iter 4 ( 29.996us): hits 0.000M/s ( 0.000M/prod), drops 0.000M/s, total operations 0.000M/s Iter 5 ( 16.949us): hits 0.000M/s ( 0.000M/prod), drops 0.000M/s, total operations 0.000M/s Iter 6 (-60.035us): hits 0.000M/s ( 0.000M/prod), drops 0.000M/s, total operations 0.000M/s Summary: hits 0.000 ± 0.000M/s ( 0.000M/prod), drops 0.000 ± 0.000M/s, total operations 0.000 ± 0.000M/s After this change: # ./bench strncmp-no-helper Setting up benchmark 'strncmp-no-helper'... Benchmark 'strncmp-no-helper' started. Iter 0 ( 77.711us): hits 5.534M/s ( 5.534M/prod), drops 0.000M/s, total operations 5.534M/s Iter 1 ( 11.215us): hits 6.006M/s ( 6.006M/prod), drops 0.000M/s, total operations 6.006M/s Iter 2 (-14.253us): hits 5.931M/s ( 5.931M/prod), drops 0.000M/s, total operations 5.931M/s Iter 3 ( 59.087us): hits 6.005M/s ( 6.005M/prod), drops 0.000M/s, total operations 6.005M/s Iter 4 (-21.379us): hits 6.010M/s ( 6.010M/prod), drops 0.000M/s, total operations 6.010M/s Iter 5 (-20.310us): hits 5.861M/s ( 5.861M/prod), drops 0.000M/s, total operations 5.861M/s Iter 6 ( 53.937us): hits 6.004M/s ( 6.004M/prod), drops 0.000M/s, total operations 6.004M/s Summary: hits 5.969 ± 0.061M/s ( 5.969M/prod), drops 0.000 ± 0.000M/s, total operations 5.969 ± 0.061M/s Fixes: 9c42652f8be3 ("selftests/bpf: Add benchmark for bpf_strncmp() helper") Suggested-by: Andrii Nakryiko <andrii@kernel.org> Signed-off-by: Viktor Malik <vmalik@redhat.com> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Acked-by: Hou Tao <houtao1@huawei.com> Link: https://lore.kernel.org/bpf/20250313122852.1365202-1-vmalik@redhat.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15selftests/bpf: Fix arena_spin_lock compilation on PowerPCKumar Kartikeya Dwivedi
Venkat reported a compilation error for BPF selftests on PowerPC [0]. The crux of the error is the following message: In file included from progs/arena_spin_lock.c:7: /root/bpf-next/tools/testing/selftests/bpf/bpf_arena_spin_lock.h:122:8: error: member reference base type '__attribute__((address_space(1))) u32' (aka '__attribute__((address_space(1))) unsigned int') is not a structure or union 122 | old = atomic_read(&lock->val); This is because PowerPC overrides the qspinlock type changing the lock->val member's type from atomic_t to u32. To remedy this, import the asm-generic version in the arena spin lock header, name it __qspinlock (since it's aliased to arena_spinlock_t, the actual name hardly matters), and adjust the selftest to not depend on the type in vmlinux.h. [0]: https://lore.kernel.org/bpf/7bc80a3b-d708-4735-aa3b-6a8c21720f9d@linux.ibm.com Fixes: 88d706ba7cc5 ("selftests/bpf: Introduce arena spin lock") Reported-by: Venkat Rao Bagalkote <venkat88@linux.ibm.com> Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Tested-by: Venkat Rao Bagalkote <venkat88@linux.ibm.com> Link: https://lore.kernel.org/bpf/20250311154244.3775505-1-memxor@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15bpf: preload: Add MODULE_DESCRIPTIONArnd Bergmann
Modpost complains when extra warnings are enabled: WARNING: modpost: missing MODULE_DESCRIPTION() in kernel/bpf/preload/bpf_preload.o Add a description from the Kconfig help text. Signed-off-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20250310134920.4123633-1-arnd@kernel.org ---- Not sure if that description actually fits what the module does. If not, please add a different description instead. Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15bpf: bpftool: Setting error code in do_loader()Sewon Nam
We are missing setting error code in do_loader() when bpf_object__open_file() fails. This means the command's exit status code will be successful, even though the operation failed. So make sure to return the correct error code. To maintain consistency with other locations where bpf_object__open_file() is called, return -1. [0] Closes: https://github.com/libbpf/bpftool/issues/156 Reported-by: Dan Carpenter <dan.carpenter@linaro.org> Signed-off-by: Sewon Nam <swnam0729@gmail.com> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Tested-by: Quentin Monnet <qmo@kernel.org> Reviewed-by: Quentin Monnet <qmo@kernel.org> Link: https://lore.kernel.org/bpf/d3b5b4b4-19bb-4619-b4dd-86c958c4a367@stanley.mountain/t/#u Link: https://lore.kernel.org/bpf/20250311031238.14865-1-swnam0729@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15Merge branch 'security-propagate-caller-information-in-bpf-hooks'Alexei Starovoitov
Blaise Boscaccy says: ==================== While trying to implement an eBPF gatekeeper program, we ran into an issue whereas the LSM hooks are missing some relevant data. Certain subcommands passed to the bpf() syscall can be invoked from either the kernel or userspace. Additionally, some fields in the bpf_attr struct contain pointers, and depending on where the subcommand was invoked, they could point to either user or kernel memory. One example of this is the bpf_prog_load subcommand and its fd_array. This data is made available and used by the verifier but not made available to the LSM subsystem. This patchset simply exposes that information to applicable LSM hooks. Change list: - v6 -> v7 - use gettid/pid in lieu of getpid/tgid in test condition - v5 -> v6 - fix regression caused by is_kernel renaming - simplify test logic - v4 -> v5 - merge v4 selftest breakout patch back into a single patch - change "is_kernel" to "kernel" - add selftest using new kernel flag - v3 -> v4 - split out selftest changes into a separate patch - v2 -> v3 - reorder params so that the new boolean flag is the last param - fixup function signatures in bpf selftests - v1 -> v2 - Pass a boolean flag in lieu of bpfptr_t Revisions: - v6 https://lore.kernel.org/bpf/20250308013314.719150-1-bboscaccy@linux.microsoft.com/ - v5 https://lore.kernel.org/bpf/20250307213651.3065714-1-bboscaccy@linux.microsoft.com/ - v4 https://lore.kernel.org/bpf/20250304203123.3935371-1-bboscaccy@linux.microsoft.com/ - v3 https://lore.kernel.org/bpf/20250303222416.3909228-1-bboscaccy@linux.microsoft.com/ - v2 https://lore.kernel.org/bpf/20250228165322.3121535-1-bboscaccy@linux.microsoft.com/ - v1 https://lore.kernel.org/bpf/20250226003055.1654837-1-bboscaccy@linux.microsoft.com/ ==================== Link: https://patch.msgid.link/20250310221737.821889-1-bboscaccy@linux.microsoft.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15selftests/bpf: Convert comma to semicolonChen Ni
Replace comma between expressions with semicolons. Using a ',' in place of a ';' can have unintended side effects. Although that is not the case here, it is seems best to use ';' unless ',' is intended. Found by inspection. No functional change intended. Compile tested only. Signed-off-by: Chen Ni <nichen@iscas.ac.cn> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Acked-by: Anton Protopopov <aspsk@isovalent.com> Link: https://lore.kernel.org/bpf/20250310032045.651068-1-nichen@iscas.ac.cn Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15selftests/bpf: Add a kernel flag test for LSM bpf hookBlaise Boscaccy
This test exercises the kernel flag added to security_bpf by effectively blocking light-skeletons from loading while allowing normal skeletons to function as-is. Since this should work with any arbitrary BPF program, an existing program from LSKELS_EXTRA was used as a test payload. Signed-off-by: Blaise Boscaccy <bboscaccy@linux.microsoft.com> Acked-by: Song Liu <song@kernel.org> Link: https://lore.kernel.org/r/20250310221737.821889-3-bboscaccy@linux.microsoft.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15selftests/bpf: Fix selection of static vs. dynamic LLVMAnton Protopopov
The Makefile uses the exit code of the `llvm-config --link-static --libs` command to choose between statically-linked and dynamically-linked LLVMs. The stdout and stderr of that command are redirected to /dev/null. To redirect the output the "&>" construction is used, which might not be supported by /bin/sh, which is executed by make for $(shell ...) commands. On such systems the test will fail even if static LLVM is actually supported. Replace "&>" by ">/dev/null 2>&1" to fix this. Fixes: 2a9d30fac818 ("selftests/bpf: Support dynamically linking LLVM if static is not available") Signed-off-by: Anton Protopopov <aspsk@isovalent.com> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Acked-by: Daniel Xu <dxu@dxuuu.xyz> Link: https://lore.kernel.org/bpf/20250310145112.1261241-1-aspsk@isovalent.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15security: Propagate caller information in bpf hooksBlaise Boscaccy
Certain bpf syscall subcommands are available for usage from both userspace and the kernel. LSM modules or eBPF gatekeeper programs may need to take a different course of action depending on whether or not a BPF syscall originated from the kernel or userspace. Additionally, some of the bpf_attr struct fields contain pointers to arbitrary memory. Currently the functionality to determine whether or not a pointer refers to kernel memory or userspace memory is exposed to the bpf verifier, but that information is missing from various LSM hooks. Here we augment the LSM hooks to provide this data, by simply passing a boolean flag indicating whether or not the call originated in the kernel, in any hook that contains a bpf_attr struct that corresponds to a subcommand that may be called from the kernel. Signed-off-by: Blaise Boscaccy <bboscaccy@linux.microsoft.com> Acked-by: Song Liu <song@kernel.org> Acked-by: Paul Moore <paul@paul-moore.com> Link: https://lore.kernel.org/r/20250310221737.821889-2-bboscaccy@linux.microsoft.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15Merge branch 'bpf-introduce-helper-for-populating-bpf_cpumask'Alexei Starovoitov
Emil Tsalapatis says: ==================== bpf: introduce helper for populating bpf_cpumask Some BPF programs like scx schedulers have their own internal CPU mask types, mask types, which they must transform into struct bpf_cpumask instances before passing them to scheduling-related kfuncs. There is currently no way to efficiently populate the bitfield of a bpf_cpumask from BPF memory, and programs must use multiple bpf_cpumask_[set, clear] calls to do so. Introduce a kfunc helper to populate the bitfield of a bpf_cpumask from valid BPF memory with a single call. Changelog : ----------- v6->v7 v6:https://lore.kernel.org/bpf/20250307153847.8530-1-emil@etsalapatis.com/ Addressed feedback by Hou Tao: * Removed RUN_TESTS invocation causing tests to run twice * Added is_test_task guard to new selftests * Removed extraneous __success attribute from existing selftests v5->v6 v5:https://lore.kernel.org/bpf/20250307041738.6665-1-emil@etsalapatis.com/ Addressed feedback by Hou Tao: * Removed __success attributes from cpumask selftests * Fixed stale patch description that used old function name v4->v5 v4: https://lore.kernel.org/bpf/20250305211235.368399-1-emil@etsalapatis.com/ Addressed feedback by Hou Tao: * Readded the tests in tools/selftests/bpf/prog_tests/cpumask.c, turns out the selftest entries were not duplicates. * Removed stray whitespace in selftest. * Add patch the missing selftest to prog_tests/cpumask.c * Explicitly annotate all cpumask selftests with __success The last patch could very well be its own cleanup patch, but I rolled it into this series because it came up in the discussion. If the last patch in the series has any issues I'd be fine with applying the first 3 patches and dealing with it separately. v3->v4 v3: https://lore.kernel.org/bpf/20250305161327.203396-1-emil@etsalapatis.com/ * Removed new tests from tools/selftests/bpf/prog_tests/cpumask.c because they were being run twice. Addressed feedback by Alexei Starovoitov: * Added missing return value in function kdoc * Added an additional patch fixing some missing kdoc fields in kernel/bpf/cpumask.c Addressed feedback by Tejun Heo: * Renamed the kfunc to bpf_cpumask_populate to avoid confusion w/ bitmap_fill() v2->v3 v2: https://lore.kernel.org/bpf/20250305021020.1004858-1-emil@etsalapatis.com/ Addressed feedback by Alexei Starovoitov: * Added back patch descriptions dropped from v1->v2 * Elide the alignment check for archs with efficient unaligned accesses v1->v2 v1: https://lore.kernel.org/bpf/20250228003321.1409285-1-emil@etsalapatis.com/ Addressed feedback by Hou Tao: * Add check that the input buffer is aligned to sizeof(long) * Adjust input buffer size check to use bitmap_size() * Add selftest for checking the bit pattern of the bpf_cpumask * Moved all selftests into existing files Signed-off-by: Emil Tsalapatis (Meta) <emil@etsalapatis.com> ==================== Link: https://patch.msgid.link/20250309230427.26603-1-emil@etsalapatis.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15selftests: bpf: fix duplicate selftests in cpumask_success.Emil Tsalapatis
The BPF cpumask selftests are currently run twice in test_progs/cpumask.c, once by traversing cpumask_success_testcases, and once by invoking RUN_TESTS(cpumask_success). Remove the invocation of RUN_TESTS to properly run the selftests only once. Now that the tests are run only through cpumask_success_testscases, add to it the missing test_refcount_null_tracking testcase. Also remove the __success annotation from it, since it is now loaded and invoked by the runner. Signed-off-by: Emil Tsalapatis (Meta) <emil@etsalapatis.com> Acked-by: Hou Tao <houtao1@huawei.com> Link: https://lore.kernel.org/r/20250309230427.26603-5-emil@etsalapatis.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15Merge branch 'selftests-bpf-move-test_lwt_seg6local-to-test_progs'Alexei Starovoitov
Bastien Curutchet says: ==================== This patch series continues the work to migrate the script tests into prog_tests. test_lwt_seg6local.sh tests some bpf_lwt_* helpers. It contains only one test that uses a network topology quite different than the ones that can be found in others prog_tests/lwt_*.c files so I add a new prog_tests/lwt_seg6local.c file. While working on the migration I noticed that some routes present in the script weren't needed so PATCH 1 deletes them and then PATCH 2 migrates the test into the test_progs framework. ==================== Acked-by: Stanislav Fomichev <sdf@fomichev.me> Link: https://patch.msgid.link/20250307-seg6local-v1-0-990fff8f180d@bootlin.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15bpf: fix missing kdoc string fields in cpumask.cEmil Tsalapatis
Some bpf_cpumask-related kfuncs have kdoc strings that are missing return values. Add a the missing descriptions for the return values. Reported-by: Alexei Starovoitov <ast@kernel.org> Signed-off-by: Emil Tsalapatis (Meta) <emil@etsalapatis.com> Acked-by: Hou Tao <houtao1@huawei.com> Link: https://lore.kernel.org/r/20250309230427.26603-4-emil@etsalapatis.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15selftests/bpf: Fix cap_enable_effective() return codeFeng Yang
The caller of cap_enable_effective() expects negative error code. Fix it. Before: failed to restore CAP_SYS_ADMIN: -1, Unknown error -1 After: failed to restore CAP_SYS_ADMIN: -3, No such process failed to restore CAP_SYS_ADMIN: -22, Invalid argument Signed-off-by: Feng Yang <yangfeng@kylinos.cn> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://lore.kernel.org/r/20250305022234.44932-1-yangfeng59949@163.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15selftests/bpf: lwt_seg6local: Move test to test_progsBastien Curutchet (eBPF Foundation)
test_lwt_seg6local.sh isn't used by the BPF CI. Add a new file in the test_progs framework to migrate the tests done by test_lwt_seg6local.sh. It uses the same network topology and the same BPF programs located in progs/test_lwt_seg6local.c. Use the network helpers instead of `nc` to exchange the final packet. Remove test_lwt_seg6local.sh and its Makefile entry. Signed-off-by: Bastien Curutchet (eBPF Foundation) <bastien.curutchet@bootlin.com> Link: https://lore.kernel.org/r/20250307-seg6local-v1-2-990fff8f180d@bootlin.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15selftests: bpf: add bpf_cpumask_populate selftestsEmil Tsalapatis
Add selftests for the bpf_cpumask_populate helper that sets a bpf_cpumask to a bit pattern provided by a BPF program. Signed-off-by: Emil Tsalapatis (Meta) <emil@etsalapatis.com> Acked-by: Hou Tao <houtao1@huawei.com> Link: https://lore.kernel.org/r/20250309230427.26603-3-emil@etsalapatis.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15selftests/bpf: Fix dangling stdout seen by traffic monitor threadAmery Hung
Traffic monitor thread may see dangling stdout as the main thread closes and reassigns stdout without protection. This happens when the main thread finishes one subtest and moves to another one in the same netns_new() scope. The issue can be reproduced by running test_progs repeatedly with traffic monitor enabled: for ((i=1;i<=100;i++)); do ./test_progs -a flow_dissector_skb* -m '*' done For restoring stdout in crash_handler(), since it does not really care about closing stdout, simlpy flush stdout and restore it to the original one. Then, Fix the issue by consolidating stdio_restore_cleanup() and stdio_restore(), and protecting the use/close/assignment of stdout with a lock. The locking in the main thread is always performed regradless of whether traffic monitor is running or not for simplicity. It won't have any side-effect. Signed-off-by: Amery Hung <ameryhung@gmail.com> Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://patch.msgid.link/20250305182057.2802606-3-ameryhung@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15selftests/bpf: lwt_seg6local: Remove unused routesBastien Curutchet (eBPF Foundation)
Some routes in fb00:: are initialized during setup, even though they aren't needed by the test as the UDP packets will travel through the lightweight tunnels. Remove these unnecessary routes. Signed-off-by: Bastien Curutchet (eBPF Foundation) <bastien.curutchet@bootlin.com> Link: https://lore.kernel.org/r/20250307-seg6local-v1-1-990fff8f180d@bootlin.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15bpf: add kfunc for populating cpumask bitsEmil Tsalapatis
Add a helper kfunc that sets the bitmap of a bpf_cpumask from BPF memory. Signed-off-by: Emil Tsalapatis (Meta) <emil@etsalapatis.com> Acked-by: Hou Tao <houtao1@huawei.com> Acked-by: Tejun Heo <tj@kernel.org> Link: https://lore.kernel.org/r/20250309230427.26603-2-emil@etsalapatis.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15selftests/bpf: Allow assigning traffic monitor print functionAmery Hung
Allow users to change traffic monitor's print function. If not provided, traffic monitor will print to stdout by default. Signed-off-by: Amery Hung <ameryhung@gmail.com> Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org> Link: https://patch.msgid.link/20250305182057.2802606-2-ameryhung@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15selftests/bpf: Clean up call sites of stdio_restore()Amery Hung
reset_affinity() and save_ns() are only called in run_one_test(). There is no need to call stdio_restore() in reset_affinity() and save_ns() if stdio_restore() is moved right after a test finishes in run_one_test(). Also remove an unnecessary check of env.stdout_saved in crash_handler() by moving env.stdout_saved assignment to the beginning of main(). Signed-off-by: Amery Hung <ameryhung@gmail.com> Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://patch.msgid.link/20250305182057.2802606-1-ameryhung@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15selftests/bpf: Move test_lwt_ip_encap to test_progsBastien Curutchet (eBPF Foundation)
test_lwt_ip_encap.sh isn't used by the BPF CI. Add a new file in the test_progs framework to migrate the tests done by test_lwt_ip_encap.sh. It uses the same network topology and the same BPF programs located in progs/test_lwt_ip_encap.c. Rework the GSO part to avoid using nc and dd. Remove test_lwt_ip_encap.sh and its Makefile entry. Signed-off-by: Bastien Curutchet (eBPF Foundation) <bastien.curutchet@bootlin.com> Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org> Link: https://patch.msgid.link/20250304-lwt_ip-v1-1-8fdeb9e79a56@bootlin.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15Merge branch 'arena-spin-lock'Alexei Starovoitov
Kumar Kartikeya Dwivedi says: ==================== Arena Spin Lock This set provides an implementation of queued spin lock for arena. There is no support for resiliency and recovering from deadlocks yet. We will wait for the rqspinlock patch set to land before incorporating support. One minor change compared to the qspinlock algorithm in the kernel is that we don't have the trylock fallback when nesting count exceeds 4. The maximum number of supported CPUs is 1024, but this can be increased in the future if necessary. The API supports returning an error, so resiliency support can be added in the future. Callers are still expected to check for and handle any potential errors. Errors are returned when the spin loops time out, when the number of CPUs is greater than 1024, or when the extreme edge case of NMI interrupting NMI interrupting HardIRQ interrupting SoftIRQ interrupting task, all of them simultaneously in slow path, occurs, which is unsupported. Changelog: ---------- v4 -> v5 v4: https://lore.kernel.org/bpf/20250305045136.2614132-1-memxor@gmail.com * Add better comment and document LLVM bug for __unqual_typeof. * Switch to precise counting in the selftest and simplify test. * Add comment about return value handling. * Reduce size for 100k to 50k to cap test runtime. v3 -> v4 v3: https://lore.kernel.org/bpf/20250305011849.1168917-1-memxor@gmail.com * Drop extra corruption handling case in decode_tail. * Stick to 1, 1k, 100k critical section sizes. * Fix unqual_typeof to not cast away arena tag for pointers. * Remove hack to skip first qnode. * Choose 100 as repeat count, 1000 is too much for 100k size. * Use pthread_barrier in test. v2 -> v3 v2: https://lore.kernel.org/bpf/20250118162238.2621311-1-memxor@gmail.com * Rename to arena_spin_lock * Introduce cond_break_label macro to jump to label from cond_break. * Drop trylock fallback when nesting count exceeds 4. * Fix bug in try_cmpxchg implementation. * Add tests with critical sections of varying lengths. * Add comments for _Generic trick to drop __arena tag. * Fix bug due to qnodes being placed on first page, leading to CPU 0's node being indistinguishable from NULL. v1 -> v2 v1: https://lore.kernel.org/bpf/20250117223754.1020174-1-memxor@gmail.com * Fix definition of lock in selftest ==================== Link: https://patch.msgid.link/20250306035431.2186189-1-memxor@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15bpf, docs: Fix broken link to renamed bpf_iter_task_vmas.cT.J. Mercier
This file was renamed from bpf_iter_task_vma.c. Fixes: 45b38941c81f ("selftests/bpf: Rename bpf_iter_task_vma.c to bpf_iter_task_vmas.c") Signed-off-by: T.J. Mercier <tjmercier@google.com> Acked-by: Song Liu <song@kernel.org> Link: https://lore.kernel.org/r/20250304204520.201115-1-tjmercier@google.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15selftests/bpf: Add tests for arena spin lockKumar Kartikeya Dwivedi
Add some basic selftests for qspinlock built over BPF arena using cond_break_label macro. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Link: https://lore.kernel.org/r/20250306035431.2186189-4-memxor@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15selftests/bpf: Introduce arena spin lockKumar Kartikeya Dwivedi
Implement queued spin lock algorithm as BPF program for lock words living in BPF arena. The algorithm is copied from kernel/locking/qspinlock.c and adapted for BPF use. We first implement abstract helpers for portable atomics and acquire/release load instructions, by relying on X86_64 presence to elide expensive barriers and rely on implementation details of the JIT, and fall back to slow but correct implementations elsewhere. When support for acquire/release load/stores lands, we can improve this state. Then, the qspinlock algorithm is adapted to remove dependence on multi-word atomics due to lack of support in BPF ISA. For instance, xchg_tail cannot use 16-bit xchg, and needs to be a implemented as a 32-bit try_cmpxchg loop. Loops which are seemingly infinite from verifier PoV are annotated with cond_break_label macro to return an error. Only 1024 NR_CPUs are supported. Note that the slow path is a global function, hence the verifier doesn't know the return value's precision. The recommended way of usage is to always test against zero for success, and not ret < 0 for error, as the verifier would assume ret > 0 has not been accounted for. Add comments in the function documentation about this quirk. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Link: https://lore.kernel.org/r/20250306035431.2186189-3-memxor@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15selftests/bpf: Introduce cond_break_labelKumar Kartikeya Dwivedi
Add a new cond_break_label macro that jumps to the specified label when the cond_break termination check fires, and allows us to better handle the uncontrolled termination of the loop. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Link: https://lore.kernel.org/r/20250306035431.2186189-2-memxor@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15bpf: correct use/def for may_goto instructionEduard Zingerman
may_goto instruction does not use any registers, but in compute_insn_live_regs() it was treated as a regular conditional jump of kind BPF_K with r0 as source register. Thus unnecessarily marking r0 as used. Fixes: 14c8552db644 ("bpf: simple DFA-based live registers analysis") Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://lore.kernel.org/r/20250305085436.2731464-1-eddyz87@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15Merge branch 'bpf-simple-dfa-based-live-registers-analysis'Alexei Starovoitov
Eduard Zingerman says: ==================== bpf: simple DFA-based live registers analysis This patch-set introduces a simple live registers DFA analysis. Analysis is done as a separate step before main verification pass. Results are stored in the env->insn_aux_data for each instruction. The change helps with iterator/callback based loops handling, as regular register liveness marks are not finalized while loops are processed. See veristat results in patch #2. Note: for regular subprogram calls analysis conservatively assumes that r1-r5 are used, and r0 is used at each 'exit' instruction. Experiments show that adding logic handling these cases precisely has no impact on verification performance. The patch set was tested by disabling the current register parentage chain liveness computation, using DFA-based liveness for registers while assuming all stack slots as live. See discussion in [1]. Changes v2 -> v3: - added support for BPF_LOAD_ACQ, BPF_STORE_REL atomics (Alexei); - correct use marks for r0 for BPF_CMPXCHG. Changes v1 -> v2: - added a refactoring commit extracting utility functions: jmp_offset(), verbose_insn() (Alexei); - added a refactoring commit extracting utility function get_call_summary() in order to share helper/kfunc related code with mark_fastcall_pattern_for_call() (Alexei); - comment in the compute_insn_live_regs() extended (Alexei). Changes RFC -> v1: - parameter count for helpers and kfuncs is taken into account; - copy_verifier_state() bugfix had been merged as a separate patch-set and is no longer a part of this patch set. RFC: https://lore.kernel.org/bpf/20250122120442.3536298-1-eddyz87@gmail.com/ v1: https://lore.kernel.org/bpf/20250228060032.1425870-1-eddyz87@gmail.com/ v2: https://lore.kernel.org/bpf/20250304074239.2328752-1-eddyz87@gmail.com/ [1] https://lore.kernel.org/bpf/cc29975fbaf163d0c2ed904a9a4d6d9452177542.camel@gmail.com/ ==================== Link: https://patch.msgid.link/20250304195024.2478889-1-eddyz87@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15selftests/bpf: test cases for compute_live_registers()Eduard Zingerman
Cover instructions from each kind: - assignment - arithmetic - store/load - endian conversion - atomics - branches, conditional branches, may_goto, calls - LD_ABS/LD_IND - address_space_cast Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://lore.kernel.org/r/20250304195024.2478889-6-eddyz87@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15Merge branch 'introduce-load-acquire-and-store-release-bpf-instructions'Alexei Starovoitov
Peilin Ye says: ==================== Introduce load-acquire and store-release BPF instructions This patchset adds kernel support for BPF load-acquire and store-release instructions (for background, please see [1]), including core/verifier and arm64/x86-64 JIT compiler changes, as well as selftests. riscv64 is also planned to be supported. The corresponding LLVM changes can be found at: https://github.com/llvm/llvm-project/pull/108636 The first 3 patches from v4 have already been applied: - [bpf-next,v4,01/10] bpf/verifier: Factor out atomic_ptr_type_ok() https://git.kernel.org/bpf/bpf-next/c/b2d9ef71d4c9 - [bpf-next,v4,02/10] bpf/verifier: Factor out check_atomic_rmw() https://git.kernel.org/bpf/bpf-next/c/d430c46c7580 - [bpf-next,v4,03/10] bpf/verifier: Factor out check_load_mem() and check_store_reg() https://git.kernel.org/bpf/bpf-next/c/d38ad248fb7a Please refer to the LLVM PR and individual kernel patches for details. Thanks! v5: https://lore.kernel.org/all/cover.1741046028.git.yepeilin@google.com/ v5..v6 change: o (Alexei) avoid using #ifndef in verifier.c v4: https://lore.kernel.org/bpf/cover.1740978603.git.yepeilin@google.com/ v4..v5 notable changes: o (kernel test robot) for 32-bit arches: make the verifier reject 64-bit load-acquires/store-releases, and fix build error in interpreter changes * tested ARCH=arc build following instructions from kernel test robot o (Alexei) drop Documentation/ patch (v4 10/10) for now v3: https://lore.kernel.org/bpf/cover.1740009184.git.yepeilin@google.com/ v3..v4 notable changes: o (Alexei) add x86-64 JIT support (including arena) o add Acked-by: tags from Xu v2: https://lore.kernel.org/bpf/cover.1738888641.git.yepeilin@google.com/ v2..v3 notable changes: o (Alexei) change encoding to BPF_LOAD_ACQ=0x100, BPF_STORE_REL=0x110 o add Acked-by: tags from Ilya and Eduard o make new selftests depend on: * __clang_major__ >= 18, and * ENABLE_ATOMICS_TESTS is defined (currently this means -mcpu=v3 or v4), and * JIT supports load_acq/store_rel (currenty only arm64) o work around llvm-17 CI job failure by conditionally define __arena_global variables as 64-bit if __clang_major__ < 18, to make sure .addr_space.1 has no holes o add Google copyright notice in new files v1: https://lore.kernel.org/all/cover.1737763916.git.yepeilin@google.com/ v1..v2 notable changes: o (Eduard) for x86 and s390, make bpf_jit_supports_insn(..., /*in_arena=*/true) return false for load_acq/store_rel o add Eduard's Acked-by: tag o (Eduard) extract LDX and non-ATOMIC STX handling into helpers, see PATCH v2 3/9 o allow unpriv programs to store-release pointers to stack o (Alexei) make it clearer in the interpreter code (PATCH v2 4/9) that only W and DW are supported for atomic RMW o test misaligned load_acq/store_rel o (Eduard) other selftests/ changes: * test load_acq/store_rel with !atomic_ptr_type_ok() pointers: - PTR_TO_CTX, for is_ctx_reg() - PTR_TO_PACKET, for is_pkt_reg() - PTR_TO_FLOW_KEYS, for is_flow_key_reg() - PTR_TO_SOCKET, for is_sk_reg() * drop atomics/ tests * delete unnecessary 'pid' checks from arena_atomics/ tests * avoid depending on __BPF_FEATURE_LOAD_ACQ_STORE_REL, use __imm_insn() and inline asm macros instead RFC v1: https://lore.kernel.org/all/cover.1734742802.git.yepeilin@google.com RFC v1..v1 notable changes: o 1-2/8: minor verifier.c refactoring patches o 3/8: core/verifier changes * (Eduard) handle load-acquire properly in backtrack_insn() * (Eduard) avoid skipping checks (e.g., bpf_jit_supports_insn()) for load-acquires * track the value stored by store-releases, just like how non-atomic STX instructions are handled * (Eduard) add missing link in commit message * (Eduard) always print 'r' for disasm.c changes o 4/8: arm64/insn: avoid treating load_acq/store_rel as load_ex/store_ex o 5/8: arm64/insn: add load_acq/store_rel * (Xu) include Should-Be-One (SBO) bits in "mask" and "value", to avoid setting fixed bits during runtime (JIT-compile time) o 6/8: arm64 JIT compiler changes * (Xu) use emit_a64_add_i() for "pointer + offset" to optimize code emission o 7/8: selftests * (Eduard) avoid adding new tests to the 'test_verifier' runner * add more tests, e.g., checking mark_precise logic o 8/8: instruction-set.rst changes [1] https://lore.kernel.org/all/20240729183246.4110549-1-yepeilin@google.com/ Thanks, ==================== Link: https://patch.msgid.link/cover.1741049567.git.yepeilin@google.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15bpf: use register liveness information for func_states_equalEduard Zingerman
Liveness analysis DFA computes a set of registers live before each instruction. Leverage this information to skip comparison of dead registers in func_states_equal(). This helps with convergance of iterator processing loops, as bpf_reg_state->live marks can't be used when loops are processed. This has certain performance impact for selftests, here is a veristat listing using `-f "insns_pct>5" -f "!insns<200"` selftests: File Program States (A) States (B) States (DIFF) -------------------- ----------------------------- ---------- ---------- -------------- arena_htab.bpf.o arena_htab_llvm 37 35 -2 (-5.41%) arena_htab_asm.bpf.o arena_htab_asm 37 33 -4 (-10.81%) arena_list.bpf.o arena_list_add 37 22 -15 (-40.54%) dynptr_success.bpf.o test_dynptr_copy 22 16 -6 (-27.27%) dynptr_success.bpf.o test_dynptr_copy_xdp 68 58 -10 (-14.71%) iters.bpf.o checkpoint_states_deletion 918 40 -878 (-95.64%) iters.bpf.o clean_live_states 136 66 -70 (-51.47%) iters.bpf.o iter_nested_deeply_iters 43 37 -6 (-13.95%) iters.bpf.o iter_nested_iters 72 62 -10 (-13.89%) iters.bpf.o iter_pass_iter_ptr_to_subprog 30 26 -4 (-13.33%) iters.bpf.o iter_subprog_iters 68 59 -9 (-13.24%) iters.bpf.o loop_state_deps2 35 32 -3 (-8.57%) iters_css.bpf.o iter_css_for_each 32 29 -3 (-9.38%) pyperf600_iter.bpf.o on_event 286 192 -94 (-32.87%) Total progs: 3578 Old success: 2061 New success: 2061 States diff min: -95.64% States diff max: 0.00% -100 .. -90 %: 1 -55 .. -45 %: 3 -45 .. -35 %: 2 -35 .. -25 %: 5 -20 .. -10 %: 12 -10 .. 0 %: 6 sched_ext: File Program States (A) States (B) States (DIFF) ----------------- ---------------------- ---------- ---------- --------------- bpf.bpf.o lavd_dispatch 8950 7065 -1885 (-21.06%) bpf.bpf.o lavd_init 516 480 -36 (-6.98%) bpf.bpf.o layered_dispatch 662 501 -161 (-24.32%) bpf.bpf.o layered_dump 298 237 -61 (-20.47%) bpf.bpf.o layered_init 523 423 -100 (-19.12%) bpf.bpf.o layered_init_task 24 22 -2 (-8.33%) bpf.bpf.o layered_runnable 151 125 -26 (-17.22%) bpf.bpf.o p2dq_dispatch 66 53 -13 (-19.70%) bpf.bpf.o p2dq_init 170 142 -28 (-16.47%) bpf.bpf.o refresh_layer_cpumasks 120 78 -42 (-35.00%) bpf.bpf.o rustland_init 37 34 -3 (-8.11%) bpf.bpf.o rustland_init 37 34 -3 (-8.11%) bpf.bpf.o rusty_select_cpu 125 108 -17 (-13.60%) scx_central.bpf.o central_dispatch 59 43 -16 (-27.12%) scx_central.bpf.o central_init 39 28 -11 (-28.21%) scx_nest.bpf.o nest_init 58 51 -7 (-12.07%) scx_pair.bpf.o pair_dispatch 142 111 -31 (-21.83%) scx_qmap.bpf.o qmap_dispatch 174 141 -33 (-18.97%) scx_qmap.bpf.o qmap_init 768 654 -114 (-14.84%) Total progs: 216 Old success: 186 New success: 186 States diff min: -35.00% States diff max: 0.00% -35 .. -25 %: 3 -25 .. -20 %: 6 -20 .. -15 %: 6 -15 .. -5 %: 7 -5 .. 0 %: 6 Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://lore.kernel.org/r/20250304195024.2478889-5-eddyz87@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2025-03-15selftests/bpf: Add selftests for load-acquire and store-release instructionsPeilin Ye
Add several ./test_progs tests: - arena_atomics/load_acquire - arena_atomics/store_release - verifier_load_acquire/* - verifier_store_release/* - verifier_precision/bpf_load_acquire - verifier_precision/bpf_store_release The last two tests are added to check if backtrack_insn() handles the new instructions correctly. Additionally, the last test also makes sure that the verifier "remembers" the value (in src_reg) we store-release into e.g. a stack slot. For example, if we take a look at the test program: #0: r1 = 8; /* store_release((u64 *)(r10 - 8), r1); */ #1: .8byte %[store_release]; #2: r1 = *(u64 *)(r10 - 8); #3: r2 = r10; #4: r2 += r1; #5: r0 = 0; #6: exit; At #1, if the verifier doesn't remember that we wrote 8 to the stack, then later at #4 we would be adding an unbounded scalar value to the stack pointer, which would cause the program to be rejected: VERIFIER LOG: ============= ... math between fp pointer and register with unbounded min value is not allowed For easier CI integration, instead of using built-ins like __atomic_{load,store}_n() which depend on the new __BPF_FEATURE_LOAD_ACQ_STORE_REL pre-defined macro, manually craft load-acquire/store-release instructions using __imm_insn(), as suggested by Eduard. All new tests depend on: (1) Clang major version >= 18, and (2) ENABLE_ATOMICS_TESTS is defined (currently implies -mcpu=v3 or v4), and (3) JIT supports load-acquire/store-release (currently arm64 and x86-64) In .../progs/arena_atomics.c: /* 8-byte-aligned */ __u8 __arena_global load_acquire8_value = 0x12; /* 1-byte hole */ __u16 __arena_global load_acquire16_value = 0x1234; That 1-byte hole in the .addr_space.1 ELF section caused clang-17 to crash: fatal error: error in backend: unable to write nop sequence of 1 bytes To work around such llvm-17 CI job failures, conditionally define __arena_global variables as 64-bit if __clang_major__ < 18, to make sure .addr_space.1 has no holes. Ideally we should avoid compiling this file using clang-17 at all (arena tests depend on __BPF_FEATURE_ADDR_SPACE_CAST, and are skipped for llvm-17 anyway), but that is a separate topic. Acked-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Peilin Ye <yepeilin@google.com> Link: https://lore.kernel.org/r/1b46c6feaf0f1b6984d9ec80e500cc7383e9da1a.1741049567.git.yepeilin@google.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
generated by cgit v1.2.3 (git 2.39.1) at 2025-07-30 12:55:34 +0000