The reliable way to learn how to write, compile, and load a CO-RE eBPF program with libbpf is to trace the whole lifecycle. Write the kernel-facing contract, compile a BPF Type Format object, generate a skeleton, load it with debug logs enabled, attach it, and then inspect what the kernel accepted. A command that exits cleanly proves little. The verifier, the loaded links, and the emitted data are the evidence.

CO-RE means Compile Once, Run Everywhere. It relocates type and field accesses against the target kernel's BPF Type Format data. It does not make missing hooks, helpers, permissions, or kernel features appear.

How to Write, Compile, and Load a CO-RE eBPF Program with libbpf?

Treat the program as a sequence with evidence at every step. If you skip straight from C source to sudo ./loader, every failure looks like a permissions problem. Usually it isn't.

1. Choose the kernel hook. Decide what event you need and which hook owns that event.
2. Write the BPF C source. Define maps, global configuration, program sections, and CO-RE field reads.
3. Generate vmlinux.h. Build the kernel type view from the target or build kernel's BTF data.
4. Compile the BPF object. Use Clang's BPF target with debug information and optimization enabled.
5. Inspect the object. Confirm that program, map, BTF, and relocation sections exist.
6. Generate the skeleton. Let bpftool turn the object into typed loader handles.
7. Configure before loading. Set read-only globals and map properties while changes are still allowed.
8. Load and verify. Read libbpf and verifier output instead of reducing every error to EPERM.
9. Attach and confirm. Keep the returned link alive and inspect the loaded program with bpftool.
10. Run the event loop. Poll buffers or wait for shutdown without doing unsafe work in signal handlers.
11. Clean up. Destroy links, buffers, and the skeleton in reverse order.

That is the complete lifecycle. The rest of this guide shows where each phase fails and what its output tells you.

What makes CO-RE portable across kernels?

CO-RE portability comes from type information, not from compiling with a clever flag. The BPF object contains BTF types and relocation records. During loading, libbpf matches those records against the target kernel's BTF data.

For example, this field access creates a relocation:

struct task_struct *task;
__u32 tgid;

task = (struct task_struct *)bpf_get_current_task_btf();
tgid = BPF_CORE_READ(task, tgid);

The compiled object records that the program wants task_struct.tgid. Libbpf then finds that field in the target kernel's type layout. If the field moved, CO-RE adjusts the access.

The kernel BTF documentation describes the data carried in the object and exposed by the kernel. Check for target BTF before blaming Clang:

test -r /sys/kernel/btf/vmlinux
bpftool btf show

The first command proves that the kernel exposes its type data. The second shows which BTF objects the kernel knows about.

CO-RE does not remove kernel requirements. The target still needs the BPF system call, the selected program type, the helpers you call, and the hook you attach to. A relocation can fix a moved field. It cannot add fentry support to a kernel that lacks it.

For optional fields and types, use existence checks such as bpf_core_field_exists() and bpf_core_type_exists(). That lets one object choose a supported path. It is feature detection, which beats guessing from the kernel release string.

How should you split the BPF program and loader?

Keep kernel code and user-space policy separate. The BPF source should collect or enforce the smallest useful fact. The loader should configure it, attach it, process output, and own its lifetime.

A small project can use this layout:

execs/
├── execs.bpf.c
├── execs.c
├── execs.skel.h
├── vmlinux.h
└── Makefile

execs.bpf.c is compiled for the BPF virtual machine. execs.c is compiled for the host CPU. Mixing those include paths is a reliable way to get errors that mention types you never wrote.

The BPF file normally contains:

• SEC() annotations for programs, maps, and the license
• Map declarations or global data
• Kernel context handling
• Helper calls
• CO-RE reads through macros such as BPF_CORE_READ
• A license string in the license section

Modern CO-RE programs generally do not need a kernel version section. Adding a hard-coded kernel version to look complete works against the point of CO-RE. Keep it only when an actual legacy loading path requires it.

The loader owns:

• Libbpf logging
• Skeleton open, load, and attach calls
• Pre-load configuration
• Ring buffer or perf buffer polling
• Signal handling
• Cleanup and exit codes

If you need to build the library rather than use a packaged development copy, follow the separate guide to compile libbpf from source. Do not bury that build inside the first program until the program itself works.

How do you write BPF C that survives real kernels?

Start with a stable hook and a small payload. Tracepoints are a sensible first target because their meaning is clearer than an arbitrary kprobe on an internal function. Kprobes are useful, but internal function names and arguments can change under you.

This BPF program counts process execution events and records the last thread group identifier. It uses CO-RE to read task_struct.tgid.

// execs.bpf.c
#include "vmlinux.h"

#include <bpf/bpf_core_read.h>
#include <bpf/bpf_helpers.h>

const volatile bool enabled = true;

__u64 exec_count;
__u32 last_tgid;

SEC("tp/sched/sched_process_exec")
int handle_exec(void *ctx)
{
 struct task_struct *task;
 __u32 tgid;

 if (!enabled)
 return 0;

 task = (struct task_struct *)bpf_get_current_task_btf();
 tgid = BPF_CORE_READ(task, tgid);

 __sync_fetch_and_add(&exec_count, 1);
 last_tgid = tgid;

 return 0;
}

char LICENSE[] SEC("license") = "GPL";

The SEC("tp/sched/sched_process_exec") annotation tells libbpf the program type and expected attachment point. The section name is part of the contract, not decoration.

The volatile const global becomes read-only configuration that the loader can set before loading. The mutable globals live in data maps managed through the skeleton.

Keep BPF control flow plain. Loops must be bounded in a way the verifier can prove. Pointer arithmetic must remain inside known objects, and every helper return value that affects safety needs checking.

For kernel memory, use CO-RE access macros or the appropriate probe-read helper. Do not cast a kernel pointer and dereference fields as if this were ordinary C. Clang may accept it while the verifier rejects the path, which is a rather expensive spelling lesson.

How do you compile the BPF object correctly?

Generate vmlinux.h from kernel BTF first. This gives the BPF source the target kernel type definitions without dragging normal kernel headers into the build.

bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

Now compile the BPF translation unit with Clang:

clang \
 -g \
 -O2 \
 -target bpf \
 -D__TARGET_ARCH_x86 \
 -I. \
 -c execs.bpf.c \
 -o execs.bpf.o

The architecture define must match the target ABI. Do not leave x86 in a copied Makefile when building for ARM. It affects register and calling convention details used by tracing macros.

The important flags have different jobs:

Do not remove optimization because you want easier debugging. Unoptimized BPF often leaves stack and branch patterns the verifier cannot reason about cleanly. Debug information and optimization are not opposites here.

Inspect the object before generating anything from it:

llvm-objdump -h execs.bpf.o
readelf -S execs.bpf.o

Check the output for the tracepoint program section, license, .BTF, and .BTF.ext. If the BTF sections are absent, skeleton generation cannot repair the object later.

A single BPF C file normally needs no separate linker step. If the project grows into several BPF translation units, add linking deliberately and inspect the resulting BTF again. Do not assume host linker habits transfer cleanly to BPF objects.

How do you generate and use a libbpf skeleton?

Generate the skeleton from the finished object:

bpftool gen skeleton execs.bpf.o > execs.skel.h

That header contains typed handles for programs, maps, global data, and lifecycle functions. For this object, the main calls follow this shape:

struct execs_bpf *skel;

skel = execs_bpf__open();
err = execs_bpf__load(skel);
err = execs_bpf__attach(skel);

/* Program runs while skel remains alive. */

execs_bpf__destroy(skel);

The open phase parses the object but does not ask the kernel to verify it. This is where you set configuration that must exist before loading.

The load phase applies CO-RE relocations, creates maps, and sends programs to the kernel verifier. The attach phase creates links between loaded programs and their hooks.

Never hand-edit execs.skel.h. Change the BPF source, rebuild the object, and regenerate the header. A generated file that contains local fixes is no longer reproducible, and the next build will erase those fixes without apology.

The libbpf API documentation is the reference for object, program, map, and link calls. Read it when a generated helper hides a lifecycle detail you need to control.

How do you configure maps and globals before loading?

Set read-only configuration after opening the skeleton and before loading it:

skel = execs_bpf__open();
if (!skel) {
 fprintf(stderr, "failed to open BPF skeleton\n");
 return 1;
}

skel->rodata->enabled = true;

err = execs_bpf__load(skel);
if (err) {
 fprintf(stderr, "failed to load BPF skeleton: %d\n", err);
 goto cleanup;
}

That timing matters because the kernel freezes read-only data during loading. Changing it afterward is not runtime configuration. It is an error.

Map geometry also belongs before load. If a skeleton exposes an events map, configure it through the map handle before calling the load helper:

err = bpf_map__set_max_entries(skel->maps.events, desired_entries);
if (err) {
 fprintf(stderr, "failed to resize events map: %d\n", err);
 goto cleanup;
}

Use compile-time constants for values that change code generation. Use read-only globals for deployment settings that must be fixed at load time. Use normal maps or mutable globals for values that need to change while the program runs.

Do not confuse the three. Recompiling a BPF object to change an operational setting is needless, while trying to mutate frozen read-only data is too late.

How do you load the object and read verifier failures?

Enable libbpf debug output before opening the skeleton. Otherwise, you will often get a negative error code after libbpf already discarded the useful context from your terminal.

#include <stdarg.h>
#include <stdio.h>
#include <bpf/libbpf.h>

static int libbpf_print_fn(enum libbpf_print_level level,
 const char *format,
 va_list args)
{
 return vfprintf(stderr, format, args);
}

int main(void)
{
 libbpf_set_print(libbpf_print_fn);

 /* Open, configure, load, and attach here. */
}

On load, check the first real verifier complaint. The final line often says only that verification failed. Earlier lines identify the unsafe register, invalid access, missing helper, or unsupported program type.

Run the loader under strace when libbpf's summary is not enough:

sudo strace -f -e trace=bpf,perf_event_open ./execs

That shows the actual BPF and perf event system calls, their arguments, and their returned errno. An EINVAL at map creation is a different failure from an EPERM at program load. Treating both as "BPF needs root" wastes the evidence.

Permissions also vary by kernel and security policy. Root may work during diagnosis, but it is not the final design. Check capabilities, lockdown settings, seccomp policy, and container restrictions before granting broad privilege.

For repeatable builds, add a kernel-load check to continuous integration. The guide to preverifying eBPF programs before deployment covers that boundary. Compilation alone never invokes the target verifier.

How do you attach to the right kernel hook?

Choose the hook by semantics first, then by convenience. A program that attaches successfully to the wrong event is still wrong.

Skeleton auto-attach reads the section name and chooses the matching libbpf path. Use it while the attachment is static. Reach for explicit calls such as bpf_program__attach_tracepoint() when the hook name or target is selected at runtime.

After attachment, inspect kernel state:

sudo bpftool prog list
sudo bpftool link list
sudo bpftool map list

Check the program name, type, tag, map references, and link target. That tells you whether the intended program exists and whether a link still owns its attachment.

Keep the returned bpf_link or the skeleton alive. Destroying it detaches the program unless you intentionally pinned the link. Many "the program loaded but does nothing" reports are loaders that returned from main() and cleaned up exactly as requested.

For deeper inspection, the bpftool program debugging guide shows how to connect the object you built with what the kernel retained.

How do you process events and keep the program alive?

Use a ring buffer for variable event traffic on kernels that support it. Use a perf buffer where compatibility or the existing design requires it. The choice does not change the lifetime rule: create the consumer after loading, poll while links remain alive, and destroy the consumer before the skeleton.

The example program stores counters in global data, so it can wait for shutdown and read skel->bss. A production event tracer would poll its buffer in the same loop.

Install signal handlers with sigaction. The handler should set a flag and return.

#include <signal.h>

static volatile sig_atomic_t exiting;

static void handle_signal(int signo)
{
 exiting = 1;
}

Do not print, free buffers, destroy links, or call libbpf from the handler. A signal can interrupt those libraries while they already hold internal state.

The main loop owns the work:

while (!exiting) {
 err = ring_buffer__poll(rb, poll_timeout);
 if (err < 0 && err != -EINTR) {
 fprintf(stderr, "ring buffer poll failed: %d\n", err);
 break;
 }
}

Then clean up in reverse order:

ring_buffer__free(rb);
execs_bpf__destroy(skel);

Pin maps or links only when they must outlive the loader. Pinning everything during development leaves stale objects in the BPF filesystem and makes the next run inspect yesterday's state. That is not persistence. It is debris.

How do you diagnose CO-RE, verifier, and loader failures?

Start at the phase that failed. Do not rebuild Clang because an attach call returned ENOENT.

CO-RE relocation failures

Dump the target BTF and search for the type or field named in the libbpf message:

bpftool btf dump file /sys/kernel/btf/vmlinux format c | less
bpftool btf dump file execs.bpf.o format c | less

Compare the object contract with the target type data. If a field is optional, guard it with a CO-RE existence check. If the type does not exist, choose another hook or build a supported fallback.

Verifier rejection

Keep full libbpf debug output and find the first rejected instruction path. Look for:

• Invalid pointer arithmetic
• Out-of-bounds stack or packet access
• Unchecked helper results
• Unbounded control flow
• Unsupported helpers for that program type
• Pointer state lost across a helper call

Then inspect the generated instructions:

llvm-objdump -S execs.bpf.o | less

That connects verifier instruction offsets to the C source. Fix the program the verifier saw, not the line you hoped Clang emitted.

Loader and attach failures

Trace the system calls:

sudo strace -f -e trace=bpf,perf_event_open,ioctl ./execs

Then read kernel evidence:

dmesg -T | tail
journalctl -k -b

The system call tells you which phase failed. The kernel ring buffer may explain a security policy denial, unsupported attach type, or verifier decision.

If the hook targets an internal function, read the target kernel source and BTF. Copying a function name from another kernel's example is cargo-cult tracing. The function may be renamed, inlined, absent, or carrying different arguments.

How do you confirm the program works on the target kernel?

Prove each layer separately. A clean loader exit is only one item on the list.

• Object: .BTF and .BTF.ext exist in the compiled file.
• Relocations: Libbpf reports successful CO-RE matching with debug output enabled.
• Load: bpftool prog list shows the expected program type and name.
• Maps: bpftool map list shows the maps referenced by that program.
• Attachment: bpftool link list shows a live link to the intended hook.
• Workload: You trigger the exact event the hook observes.
• Output: Counters change or events arrive with fields that match the workload.
• Permissions: The loader works under the privilege model you intend to deploy.
• Lifetime: The program remains attached for as long as required, and no longer.
• Cleanup: A normal exit removes unpinned links and maps.

For the process execution example, start the loader and run a known command from another shell. Then stop the loader and inspect its counter and last recorded identifier. If the count does not move, check attachment before touching the event code.

The reliable way to write, compile, and debug CO-RE programs is to keep this test reproducible. Use the same object, loader arguments, workload, and inspection commands on each target kernel. Otherwise, portability failures turn into anecdotes.

FAQ

Can I cross-compile a CO-RE object for another machine?

Yes. The BPF object is not compiled for the target machine's normal instruction set, but architecture-specific tracing definitions still matter. Set the correct __TARGET_ARCH value, use compatible libbpf headers, and test loading against the target kernel's BTF and verifier.

The user-space loader is a separate binary. Cross-compile that for the target CPU and its C library.

Should I commit vmlinux.h and the generated skeleton?

Commit them when reproducible builds matter and your project treats generated files as fixed build inputs. Otherwise, generate both during the build and record the exact tools used.

Do not regenerate them silently on one developer's workstation and commit unrelated changes. Generated type churn can hide the BPF source change you meant to review.

Can the loader run inside a container?

Only if the container can reach the required kernel interfaces and has the needed permissions. The program still loads into the host kernel, not a private container kernel.

Check access to the BPF filesystem, target network namespace, tracing files, and required capabilities. Broad privileged mode can prove the diagnosis, but it should not become the deployment plan.

Should I statically link libbpf?

Static linking can reduce target dependency drift, but it makes updates your responsibility. You must rebuild when libbpf or its dependencies need a security or compatibility fix.

Dynamic linking is easier on systems with a controlled package set. Pick one model deliberately and record the linked version in build output rather than discovering it after a loader behaves differently.

Why does the program work with bpftool but fail in my loader?

Your loader may configure a different map, omit an attach argument, close a link early, or run under another security context. Trace both paths with strace and compare the BPF system calls.

Also compare libbpf debug output. The kernel does not care that both commands started from the same object file if they send different attributes.