Read the errno before you edit a config file. When you add runtime enforcement with eBPF, two things fail before your policy ever loads: the bpffs mount at /sys/fs/bpf, and the bpf() syscall itself. Both need CAP_SYS_ADMIN, or a properly scoped BPF token on newer kernels. Without a persistent bpffs mount, your pinned maps and programs vanish on reboot or process exit. Which fix you need depends on which of those two failures you hit, so read the return codes and dmesg first, then touch fstab.

Last updated: 2026-07-21

eBPF runtime enforcement gives you a practical way to add inline controls without patching the kernel or loading modules. You set up a small, safe policy that attaches programs close to events: check prerequisites, pick hooks, load a minimal program, pin programs and maps, and watch state with bpftool. The verifier and JIT keep that fast and safe. But none of it survives a reboot if the mount and the pins are wrong, so start there.

What does mounting bpffs at /sys/fs/bpf actually do?

bpffs is a special in-memory filesystem whose only job is to hold references to BPF objects by path. The default mount location is /sys/fs/bpf. When you pin a map or program there, the kernel creates a file that holds a reference to that object.

That reference is the whole point. A map or program normally dies the moment the last file descriptor closes, usually when your loader process exits. Pin it under bpffs and the reference lives in the filesystem instead, so the object survives after your program returns.

So the mount is a namespace for pins, not a disk. Think of it as a way to hand a map or program a name that other processes can open by path. Cilium, Tetragon, and hand-rolled loaders all rely on this to keep policy alive between runs.

Why does mounting or using bpffs require CAP_SYS_ADMIN?

Because two separate kernel checks both demand it, and people confuse them. Mounting any filesystem, bpffs included, goes through mount(2), which requires CAP_SYS_ADMIN in the mount namespace. That is the first wall. The bpf() syscall is the second.

Historically every bpf() command was gated behind CAP_SYS_ADMIN. Since kernel 5.8 the check was split, so loading and map work can run with CAP_BPF plus CAP_PERFMON or CAP_NET_ADMIN, depending on the program type. The bpf() syscall reference spells out which command needs which capability. Read it instead of guessing, because a socket filter and an LSM hook do not need the same privileges.

Here is what trips people up. You can grant CAP_BPF to a loader and still fail the mount, because mounting bpffs never cared about CAP_BPF. It wanted CAP_SYS_ADMIN. That is why one process can load programs fine but a fresh node cannot create the mount at all.

How do you diagnose a failed bpffs mount or bpf() call?

Run the failing command under strace and read the errno, not the wrapper's friendly message. For a mount, strace -f mount -t bpf bpffs /sys/fs/bpf shows the exact syscall and its return. For a loader, strace -e bpf ./yourloader narrows output to just the bpf() calls.

The errno tells you which failure mode you are in:

• EPERM on the mount means you lack CAP_SYS_ADMIN, or a mount namespace or user namespace is blocking you.
• EPERM on bpf() means missing CAP_BPF/CAP_SYS_ADMIN, or unprivileged_bpf_disabled is set and you are not root.
• ENODEV on the mount means the kernel has no bpffs support compiled in. Check CONFIG_BPF_SYSCALL.
• EINVAL on bpf() usually means the verifier rejected your program, so fix the code before you touch permissions.

Then read the kernel side. dmesg -T | tail -30 often carries the verifier log or an LSM denial that never reached your terminal. If AppArmor or SELinux is blocking the mount, that is where it says so. Tie the log line to the moment you ran the command and the cause stops being a mystery.

Making the bpffs mount persistent across reboots

Add one line to /etc/fstab and the mount comes back on every boot:

bpffs /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime 0 0

Run mount -a to prove it now, then confirm with findmnt /sys/fs/bpf. If findmnt shows the mount with type bpf, you are done. If it errors, you fixed nothing and the next reboot loses it again.

On a systemd box you can use a mount unit instead, named for the path: sys-fs-bpf.mount. Most distributions already ship one, and systemctl status sys-fs-bpf.mount tells you whether it is active. Prefer the unit when other services must order themselves after the mount, because After=sys-fs-bpf.mount gives you that ordering for free. fstab does not.

Either way, mount early. A loader that starts before bpffs is mounted will pin into the underlying /sys/fs/ directory, and those pins disappear the instant bpffs mounts over the top.

Why do eBPF maps or programs disappear after a reboot even with bpffs mounted?

Because a persistent mount and a persistent object are different things, and this is the gotcha nobody mentions. bpffs lives in memory. A reboot wipes all kernel state, so every map and program is gone regardless of how correct your fstab line is.

The mount coming back does not bring your pins back. It brings back an empty directory. Your pinned files were references to live kernel objects, and those objects no longer exist after a cold start. So the pin file cannot survive either.

What actually persists is your loading process. Something has to reload the program and re-pin it on every boot, which is exactly what a systemd unit or a CNI agent does for you. If you expected the pins alone to carry your policy across a reboot, that is the real problem, and no mount option fixes it.

Harden the bpffs mount so it is not a soft target

Mount it with nosuid, nodev, and noexec, the way the fstab line above already does. bpffs never needs to run executables or honor device nodes or setuid bits, so denying all three costs you nothing and removes an obvious foothold.

Lock down the directory ownership next. Pins under /sys/fs/bpf are readable and openable by anyone with path access, so a world-writable pin directory lets an unprivileged process attach to your maps. Create scoped subdirectories owned by the service that uses them, for example /sys/fs/bpf/cilium at mode 0700, rather than dropping everything in the root of the mount.

Watch for stray pins as a persistence signal. An attacker who gains CAP_BPF can pin a program to survive their session. Enumerating pins regularly, which the bpftool section covers, is how you spot a hook you did not put there.

What do bpf_jit_enable and net.core.bpf_jit_harden actually control?

net.core.bpf_jit_enable turns the JIT compiler on or off. With it on, the kernel compiles verified bytecode into native instructions, which lowers per-event latency. Check it with cat /proc/sys/net/core/bpf_jit_enable and enable it with echo 1 | sudo tee /proc/sys/net/core/bpf_jit_enable. On most modern distributions it is already on.

net.core.bpf_jit_harden is a separate control, and people misread it as an on/off twin. It is not. It sets how aggressively the JIT defends against attacks that abuse compiled BPF, mainly by blinding constants. It takes three values:

Hardening adds overhead, which is why it defaults to off. If you already set unprivileged_bpf_disabled, unprivileged loading is gone anyway, so 1 protects a surface you have closed. Do not burn a debugging afternoon chasing a JIT sysctl when your real problem is a failed mount or a missing capability. These two knobs change speed and attack surface. Neither decides whether your program loads.

How bpf_prog_load fits the mount and capability picture

bpf_prog_load is the libbpf wrapper around the BPF_PROG_LOAD command of the bpf() syscall, and BPF_PROG_LOAD is what actually loads eBPF programs into the kernel. It runs the verifier, and on success hands you a file descriptor for the loaded program.

That file descriptor is the link back to pinning. The program lives only as long as some descriptor references it, so right after load you either attach it to a hook or pin it under bpffs. Skip both and the program is collected the moment your loader exits.

Capabilities gate this call. The mount has nothing to do with it. BPF_PROG_LOAD needs CAP_BPF and, for certain program types like LSM or tracing, extra capabilities on top. A clean mount does nothing for you here if the load itself lacks privilege, which is why the strace -e bpf split earlier matters: it tells you whether you failed at mount time or at load time.

Can eBPF programs be loaded without full root?

Yes, and you should aim for it rather than handing everything CAP_SYS_ADMIN. On kernels from 5.8 on, grant a loader CAP_BPF plus whatever its program type needs, and drop the rest. That covers most network and observability work without a fully privileged process.

Set the guardrail for everyone else. echo 1 | sudo tee /proc/sys/kernel/unprivileged_bpf_disabled blocks unprivileged bpf() loads entirely, which is the sane default on a production host. Persist it with sysctl -w kernel.unprivileged_bpf_disabled=1.

Newer kernels add BPF tokens for delegated loading. A privileged process creates a token that carries a scoped subset of BPF permissions, then hands it to a less privileged process, so a container can load a specific program class without the host granting it blanket CAP_SYS_ADMIN. That is the modern answer to "how do I let this workload load eBPF without making it root." Check your kernel actually supports tokens before you design around them, because older LTS kernels do not.

How do you install bpftool?

Install it from your distribution before you debug anything. bpftool is the reference utility to quickly inspect and manage BPF objects on a Linux system, and the package is named bpftool:

sudo apt install bpftool

On some distributions it ships inside the linux-tools package for your running kernel instead. Once installed, it shows you what is actually loaded, so you stop trusting what you think you loaded.

Using bpftool to inspect loaded programs and pins

Three commands cover most of it. bpftool prog show lists loaded programs with their IDs and types. bpftool map show does the same for maps. To walk the pins under your mount, bpftool prog show pinned /sys/fs/bpf/policy/prog confirms a specific pin resolves to a live program.

Pinning through bpftool is one line each:

bpftool prog pin id <id> /sys/fs/bpf/policy/prog
bpftool map pin id <id> /sys/fs/bpf/policy/map

If a pin path exists but prog show pinned errors on it, that is a stale file from a mount that happened after the pin, exactly the ordering trap from the persistence section. For a fuller walkthrough of loading and attaching, see the guide on building an eBPF IDS on Linux, and for verifier internals the JIT and verification offload tutorial.

How does this apply to Cilium and other CNI plugins on fresh nodes?

Cilium and similar agents mount bpffs themselves at startup, but they need it to stay mounted, or the agent redoes work and can lose pins on restart. On a fresh node the first symptom is usually a Cilium pod that crash-loops or logs that it cannot reach /sys/fs/bpf.

Run the mount check before you blame Cilium. findmnt /sys/fs/bpf on the node tells you whether the mount exists at all. If it is missing, the agent either lacked CAP_SYS_ADMIN in its security context or the host mount propagation blocked it. Both show up as EPERM under strace, exactly as they do anywhere else.

Give the node a persistent mount so the agent is not the only thing holding it. A host-level fstab line or systemd mount unit means bpffs is up before any pod starts, and pod restarts no longer race the mount. That single split, host owns the mount and agent owns the pins, is the difference between a node that reboots cleanly and one that needs a manual kick every time.

FAQ

Does bpffs need CONFIG_BPF_SYSCALL, or is CONFIG_BPF enough?

You need CONFIG_BPF_SYSCALL. CONFIG_BPF alone gives you the core bytecode support but not the bpf() syscall your loaders and the bpffs mount depend on. Confirm with zcat /proc/config.gz | grep CONFIG_BPF_SYSCALL, or grep the same symbol under /boot/config-$(uname -r) if /proc/config.gz is absent.

Can I mount bpffs somewhere other than /sys/fs/bpf?

Technically yes, the mount type does not care about the path. In practice, do not. Nearly every tool, including bpftool and Cilium, assumes the default location, and a nonstandard path means every other program looks in the wrong place and reports missing pins. Keep it at /sys/fs/bpf unless you have a hard reason not to.

Why does my pin work in a container but the host cannot see it?

Mount namespaces. A container with its own bpffs mount pins into a filesystem the host never sees, so bpftool prog show pinned on the host finds nothing. Either share the host bpffs mount into the container with the right propagation, or accept that pins are namespace-local. Check which namespace you are in with ls -l /proc/self/ns/mnt.

Is bpf_jit_harden worth enabling on a normal server?

Only if you allow unprivileged loading, and most hardened servers do not. If you already set unprivileged_bpf_disabled=1, a value of 1 protects a path you have closed off, so the overhead buys little. Set it to 2 only when you have a real threat model that includes privileged JIT abuse and you accept the performance cost.

How do I tell whether a program failed to load or failed to attach?

Split the two with strace. strace -e bpf shows the BPF_PROG_LOAD call and its return, so an EINVAL there points at the verifier. Attachment happens through later calls like BPF_LINK_CREATE or a perf_event_open, so a load that succeeds while the hook never fires points at attachment. Reading the two apart saves you from tuning the wrong thing.

Related on this blog

• Minimal Kernel for eBPF: A Step-by-Step Ultimate Guide