Start with clang, libbpf, and the target kernel configuration. eBPF is a way to run small, verified programs inside the Linux kernel without rebuilding the kernel for every change. The CPU matters less than the running kernel, its enabled features, its headers, and its available hook points. On ARM and OpenWrt, build on a host, load on the device, and check the kernel before you blame the compiler.

Last updated: 2026-08-21

The common mistake is treating eBPF as a portable binary format that works everywhere. The bytecode is portable across CPU types, but the verifier, helper set, kernel configuration, BTF data, and attachment points still belong to the target kernel. That is where builds fail.

What is eBPF?

eBPF is a kernel execution system for loading small programs at defined hook points. The kernel checks each program with a verifier before it runs. When just-in-time compilation is available, the kernel converts the verified eBPF instructions into native CPU instructions.

The program usually works with a user-space loader:

• The eBPF object contains programs and maps.
• The kernel verifier checks safety and allowed operations.
• The loader attaches programs to tracepoints, kprobes, network hooks, or other supported locations.
• Maps carry data between kernel code and user space.
• User space reads those maps or consumes events through a ring buffer or perf buffer.

That split matters. Your C source is not a normal application, and the eBPF program is not a kernel module. You cannot call arbitrary library functions, allocate memory whenever you want, or walk kernel memory without permission. The verifier rejects unsafe control flow, invalid pointer use, and unsupported operations before the program loads.

The official eBPF documentation is useful for the broad model. For actual behavior, read the target kernel's configuration and inspect the load error. Marketing diagrams do not tell you why BPF_PROG_LOAD returned EINVAL.

What makes eBPF useful?

eBPF is useful when you need kernel visibility or enforcement without maintaining a custom kernel patch. It fits several jobs well:

• Trace system calls, scheduler activity, block I/O, and network events.
• Measure latency and identify where time disappears.
• Observe processes without adding logging to every application.
• Enforce network or security policy at supported hooks.
• Collect data from production kernels with less disruption than a kernel rebuild.
• Filter packets close to the network path.

The right use is narrow instrumentation with a clear question. “Watch everything” is not a design. It produces too much data, adds overhead, and leaves you debugging your observability tool instead of the original fault.

For a first program, use a tracepoint. Tracepoints expose named events with a documented format and tend to survive kernel changes better than private function symbols. Kprobes are useful when no tracepoint exists, but they depend on symbols and function details that vendors change without asking you.

Which eBPF tools should you use?

Use bpftrace to answer a small question quickly. Use libbpf when the program needs to become a maintained application. Use BCC when its runtime and language bindings are acceptable on the target. Do not install the largest framework because a tutorial did.

For ARM devices with limited storage, I choose libbpf and a small cross-compiled loader. BCC on ARM can work when the device has a package manager, Python, headers, and enough room for its dependencies. That is not the usual embedded image.

The practical split looks like this:

• Build and inspect on an x86 host.
• Copy the eBPF object and loader to the ARM device.
• Let the target kernel verify and JIT the program.
• Read the device logs and trace output there.

The libbpf loader guide covers the full loader path. If you need the library itself, use the libbpf source build guide rather than copying random objects from another system.

How does the eBPF compiler path work?

Use clang with the BPF backend to compile restricted C into an ELF object. The compiler emits eBPF instructions. The target kernel then verifies and loads those instructions.

The LLVM project documents the BPF target as a compiler target, but that does not mean the generated object fits every kernel. The compiler knows the instruction set. It does not know which helpers, map types, attach points, or BTF types your device kernel exposes.

Check the host toolchain before building:

clang --version
llc --version
bpftool version
clang -print-targets | grep -w bpf

The output tells you whether the host has the required compiler target and inspection tools. If bpf is missing from the target list, stop there. Changing C flags will not make a compiler emit a backend it does not have.

A basic compile command may look like this:

clang -O2 -g -target bpf -c trace.c -o trace.bpf.o
bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

The exact include paths depend on the target kernel source and generated headers. Do not use headers from the host distribution and hope the layouts match. That works until a field moves, a helper disappears, or the verifier notices the difference.

For CO-RE, generate vmlinux.h from the target's BTF when possible. CO-RE means Compile Once, Run Everywhere, but the name is not a warranty. It adapts type relocations when the kernel exposes the metadata and the program uses supported access patterns. It cannot invent missing helpers or attach to a hook that the kernel does not have.

What does libbpf add?

libbpf handles the difficult user-space parts of an eBPF application. It opens the ELF object, creates maps, applies relocations, loads programs, and attaches them to hooks. It also supports generated skeletons that give your C loader typed functions and map access.

A typical workflow uses bpftool to generate the skeleton:

bpftool gen skeleton trace.bpf.o > trace.skel.h

Your loader then includes the generated header, opens the object, loads it, and attaches the selected program. The exact API depends on the libbpf version in your build tree, so compile against the same library and headers you plan to ship.

This is where many guides go wrong. They describe CO-RE as if it removes the need to understand the kernel. It does not. Keep the target vmlinux, BTF data, kernel configuration, and build source with the project. When the loader fails, those artifacts let you tell a missing relocation from a missing kernel feature.

How do you check if eBPF is enabled?

Check the running kernel first. Do not check the host kernel and assume the router or board matches it.

Start with:

uname -a
test -r /sys/kernel/btf/vmlinux && echo "BTF available" || echo "BTF missing"
bpftool feature probe kernel

uname identifies the kernel that will actually load the program. The BTF test tells you whether the kernel exposes type metadata through the usual path. bpftool feature probe kernel reports supported program types, map types, helpers, and other kernel capabilities.

Now inspect the configuration:

zcat /proc/config.gz 2>/dev/null | grep -E 'CONFIG_(BPF|BPF_SYSCALL|BPF_JIT|HAVE_EBPF_JIT)'
grep -E 'CONFIG_(BPF|BPF_SYSCALL|BPF_JIT|HAVE_EBPF_JIT)' /boot/config-$(uname -r) 2>/dev/null

You may have only one of those configuration paths. Small systems often omit /boot, and some kernels do not expose /proc/config.gz.

The important symbols are different kinds of checks:

• CONFIG_BPF enables the kernel BPF subsystem.
• CONFIG_BPF_SYSCALL enables the system call used by user-space loaders.
• CONFIG_BPF_JIT enables just-in-time compilation when the architecture supports it.
• CONFIG_HAVE_EBPF_JIT indicates architecture support for an eBPF JIT. It is usually a capability symbol selected by the architecture, not the switch you turn on by itself.

If CONFIG_BPF_SYSCALL is missing, your loader cannot use the normal BPF system call path. If CONFIG_BPF_JIT is missing, programs may still load through the interpreter, but performance and available behavior can differ. If CONFIG_HAVE_EBPF_JIT is absent, enabling CONFIG_BPF_JIT alone will not create an architecture backend.

Check the kernel log after a failed load:

dmesg -T | tail -40
journalctl -k -b --no-pager | tail -40

The verifier often writes the useful reason there. Invalid argument is only the exit code. It is not a diagnosis.

What should you enable in an OpenWrt kernel?

For OpenWrt, edit the kernel configuration through the OpenWrt build system. Do not modify a generated kernel .config and assume the change will survive the next build.

Start from the target configuration, then inspect the resulting kernel config:

make menuconfig
make kernel_menuconfig

The menu names can vary by OpenWrt branch and target. Search for BPF rather than following an old screenshot. OpenWrt 24 builds also differ by target profile, so a setting available on one device may not exist on another.

For a loader-based eBPF setup, check these areas:

CONFIG_BPF
CONFIG_BPF_SYSCALL
CONFIG_BPF_JIT
CONFIG_DEBUG_INFO_BTF
CONFIG_DEBUG_FS
CONFIG_TRACEPOINTS

Not every program needs every setting. CONFIG_DEBUG_INFO_BTF matters for BTF and CO-RE workflows. CONFIG_DEBUG_FS matters when your tracing workflow reads trace_pipe. Tracepoints must exist in the kernel and be enabled by the relevant subsystem.

After building the image, check the device rather than trusting the build menu:

zcat /proc/config.gz 2>/dev/null | grep -E 'CONFIG_(BPF|BPF_SYSCALL|BPF_JIT|DEBUG_INFO_BTF|DEBUG_FS|TRACEPOINTS)'
bpftool feature probe kernel

OpenWrt's small root filesystem changes the tool choice. Shipping LLVM, Python, BCC, kernel headers, and a full development environment to the router is cargo culting. Build on the host and ship the loader, object file, and only the runtime pieces you need.

The OpenWrt eBPF build guide covers the target toolchain and kernel configuration path. Use it with the actual config from your device. “OpenWrt support” is not a single switch across every target.

Where does einat-ebpf fit?

einat-ebpf belongs to the embedded networking side of eBPF rather than ordinary tracing. Treat it as a target-specific application that depends on kernel hooks, map support, verifier behavior, and the network path available on the device.

Do not install it first and diagnose later. Check the device kernel and package architecture before building:

uname -a
opkg print-architecture
bpftool feature probe kernel

Then inspect the project’s required program types, helpers, map types, and attach points. A package name does not prove that the running OpenWrt kernel supports the package.

For NAT or packet-processing workloads, the hook choice matters. XDP, traffic control, and socket-level hooks have different data paths and permissions. If the project expects a hook that your target kernel lacks, changing the compiler flags will not fix it. Rebuild the kernel or choose a supported hook.

What changes on ARM?

The eBPF instruction set is separate from the CPU instruction set. The target ARM kernel performs verification and, where supported, JIT compilation into ARM instructions.

The user-space loader is different. A 32-bit ARM hard-float system needs a different compiler setup from an ARM64 system, and a loader built for the wrong ABI will fail before eBPF gets involved.

Identify the target before choosing the cross compiler:

uname -m
file /bin/busybox
readelf -h /bin/busybox

uname -m reports the kernel architecture. file and readelf show the user-space ABI that the loader must match. Build the loader with the OpenWrt SDK or the exact cross toolchain used for the image.

Check the result before copying it:

file ./loader
readelf -h ./loader

If the loader is dynamically linked, inspect its interpreter:

readelf -l ./loader | grep 'Requesting program interpreter'

A BusyBox-only image may not have the expected dynamic linker or shared libraries. Static linking can avoid that problem, but it also increases the binary and may expose library licensing or resolver issues. Check the target constraints instead of treating static linking as a cure for every deployment problem.

What fails most often?

Kernel mismatch causes more trouble than CPU mismatch. Preserve the exact inputs used for every build:

mkdir -p build-inputs
uname -r > build-inputs/kernel-release
cp /path/to/target/.config build-inputs/
cp /path/to/vmlinux build-inputs/

Then check the object and kernel artifacts:

file trace.bpf.o
readelf -S trace.bpf.o
readelf -S vmlinux | grep -E 'BTF|symtab'

Common failures have distinct causes:

• Operation not permitted often means missing privileges, a locked-down kernel, or a security policy.
• Invalid argument usually needs the verifier log, not another compile attempt.
• No such file or directory may mean a missing BTF file, loader dependency, or attach path.
• An unknown helper means the target kernel does not expose that helper to the selected program type.
• A rejected map means the map type, flags, key size, value size, or kernel limits do not match.
• A missing kprobe symbol means the function was renamed, inlined, hidden, or removed from the vendor tree.
• A program that loads but produces no data may be attached to the wrong hook or reading the context incorrectly.

Load with verbose output where your loader supports it. Then read the kernel messages:

dmesg -T | tail -60
bpftool prog show
bpftool map show

For a tracepoint test, use a small program and watch the output:

cat /sys/kernel/debug/tracing/trace_pipe

If trace_pipe is absent, check whether debugfs and tracefs are mounted. Do not add random mount commands to startup until you know which filesystem the image uses.

How should you choose between BCC and libbpf on ARM?

Choose BCC when you need an existing script and the target can carry its runtime. Choose libbpf when you control the application, need a small image, or plan to deploy the loader across devices.

BCC on ARM is not wrong. It is just a poor default for a constrained router. Python, compiler support, headers, and runtime packages turn a small kernel probe into a larger software deployment. That may be acceptable on a development board. It is harder to justify on an immutable OpenWrt image.

libbpf keeps the runtime in a C library and lets you generate a skeleton during the host build. That makes the target smaller and the deployment reproducible. The trade-off is that you must understand object loading, map lifetimes, attachment errors, and kernel compatibility. There is no framework to hide those details, which is why it is the better long-term choice.

What should you keep in a reproducible build?

Keep the target kernel release, OpenWrt commit or source tree, kernel .config, generated headers, vmlinux, BTF data, compiler versions, and cross-toolchain details.

Run the same checks in CI:

clang --version
bpftool version
file trace.bpf.o
bpftool btf dump file vmlinux format raw >/dev/null

Preverify the object where possible, but do not confuse host validation with a target load test. The target kernel still makes the final decision. A program can pass local inspection and fail on the board because the board has different helpers, limits, BTF, or security policy.

The eBPF preverification guide covers the CI side. Keep the device test too. The kernel is the authority that matters.

FAQ

Can eBPF run without a JIT?

Yes. The kernel can interpret eBPF instructions when JIT support is unavailable. Expect different performance, and confirm the target behavior with bpftool feature probe kernel.

Do eBPF programs need root access?

Usually, loading and attaching programs requires elevated privileges. Containers and service managers can also restrict the needed capabilities. Check the process capabilities and the kernel log before changing file permissions.

Can I use eBPF on an old OpenWrt image?

You can only use the program types, helpers, maps, and hooks that the image kernel provides. Check /proc/config.gz, BTF, and bpftool feature probe kernel; the OpenWrt release name alone does not answer that.

Why does a program compile but fail on the router?

Compilation proves that LLVM accepted the source. Loading proves that the target verifier accepted the resulting object. Compare the target kernel artifacts, read the verifier log, and check the selected hook before changing source code.

Is eBPF suitable for every packet-processing task?

No. It is a good fit when the required hook and map behavior match the kernel. A dedicated kernel feature, firewall path, or ordinary user-space daemon may be easier to operate when eBPF adds more moving parts than value.