You want a clear path for bpf development that doesn’t waste time on guessing. I’ll show how practical workflows, kernel setup, and the right tools help you get a program running reliably across machines.

We’ve learned the hard way: keeping kernels, headers, and Clang in sync matters more than clever hacks. I’ll demystify how libbpf and bpftool play different roles—one as a library you link into user space, the other as a utility for inspection, header generation, and quick prototyping.

The steps you’ll see are repeatable: install a fresh kernel and headers, use Clang/LLVM and libelf, build examples, and move from prototype to a durable loader. Expect concrete commands, file locations, and common gotchas so you can spend less time troubleshooting and more time building.

Key Takeaways

• Focus on reliable, repeatable bpf workflows that survive kernel changes.
• Set up a fresh kernel and headers before you compile or load programs.
• Use bpftool for quick inspection and header generation; use the library for production loaders.
• Modern bpf stacks rely on Clang/LLVM, libelf, and proper build steps.
• Expect concrete commands and file locations to reduce setup friction.

What libbpf and bpftool are—and when each makes sense

Knowing which tool to reach for speeds up development and prevents subtle kernel issues. I’ll define each and give simple guidance so you can pick the right path.

libbpf: a C library for loading, verifying, and attaching programs

I treat libbpf as the runtime you link into a user process. The library wraps the low-level syscalls to load and verify a bpf object and attach it to a hook like a tracepoint or kprobe.

It parses BTF, reads section annotations, and exposes helper prototypes in bpf_helpers.h. That support powers skeleton-based flows and CO-RE relocations for portable programs.

bpftool: a command-line utility for inspecting and generating artifacts

bpftool is the Swiss-army CLI to gather quick information. Use it to generate vmlinux.h from /sys/kernel/btf/vmlinux, emit skeleton headers from an object file, and list maps and programs in the running kernel.

• Use the library when you need a production loader and stable attach flows.
• Use the CLI for fast inspection, header generation, and early prototyping.
• Combine both: generate type info with the CLI, ship with the library.

libbpf vs bpftool: key differences mapped to your BPF development workflow

From source to attach, I map where compilation, inspection, and API calls fit into a typical build loop.

We start with a C source file and compile an object file using clang -target bpf.
Include bpf_helpers.h and the generated vmlinux.h (dumped from /sys/kernel/btf/vmlinux) so kernel types resolve at compile time.

For loading, the library path uses bpf_object__open, bpf_object__load, and bpf_program__attach.
Skeletons simplify that to open(), load(), attach(), destroy() in user space.

The CLI complements this: run bpftool gen skeleton hello_world.o > hello_world.skel.h to embed the object into a header.
Use the same CLI to list programs, enumerate bpf maps, and confirm BTF presence when you test CO-RE builds.

Section names via SEC(“…”) determine attach kind.
Inside BPF code, bpf_printk and BPF_CORE_READ help debug and portably read kernel structs for a tracepoint or other hooks.

I rely on the CLI for quick checks and on the library for lifecycle and typed access to bpf maps.
That mix keeps iteration fast and production runs reliable.

Prerequisites and environment setup in the Linux kernel

A reliable bpf workflow starts with the right kernel configs and a working toolchain. I recommend building a fresh linux kernel, installing its headers, and rebooting so the running kernel matches the files you compile against.

Kernel configs and BTF support

Enable CONFIG_DEBUG_INFO_BTF=y when you build the kernel. That flag produces BTF data so /sys/kernel/btf/vmlinux exists.

If the vmlinux file is present, you can dump a header file and use CO-RE to avoid type mismatches across kernels.

Headers, header file paths, and vmlinux.h availability

After make; make install and make headers_install, reboot and confirm with uname -r. Check include paths in your build system and point them at the installed headers and the generated vmlinux.h.

Toolchain dependencies

Install clang/LLVM (v10+), libelf, and zlib. Build the library from tools/lib/bpf with make; make install and verify via pkg-config so your loader code finds the right headers.

On Debian/Ubuntu, install linux-tools-$(uname -r) for the CLI; otherwise build the tool from kernel sources. A quick verification: bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h — this confirms the pipeline works.

Using bpftool the practical way

I keep a short command set I run every time I prepare a build or debug a running system. These steps turn kernel BTF into a header file, embed an object into a skeleton, and let you inspect the loaded kernel state without guesswork.

Generate vmlinux.h from BTF to compile and run BPF

Dump the kernel types with this one-liner:

bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

This header file gives compile-time type safety and stops mismatches across targets.

Create a skeleton header from an object file

After you compile with clang -target bpf, run:

bpftool gen skeleton hello_world.o > hello_world.skel.h

The skeleton embeds the object and provides open/load/attach helpers so a tiny C main can run the programs without extra plumbing.

Inspecting programs, maps, and the loaded kernel

• List active programs: bpftool prog show
• List maps and contents: bpftool map show
• Explore type info: bpftool btf dump

Typical commands and options you’ll care about

Install via linux-tools or build from tools/bpf/bpftool and place the binary in /usr/local/sbin to match common examples.

Use these steps early to validate files and environment, then move to a production loader using libbpf once your target and attach strategy are settled.

Building with libbpf: from object to running BPF program

I’ll walk you through turning a compiled object into a running BPF workload on a live system. The key is a tiny, reliable loader that parses ELF sections and lets the kernel host your program.

Linking, sections, and the SEC macro

Link your user binary with the library, libelf, and zlib. The loader reads each ELF section and uses the SEC(“…”) macro to learn attach type. In the BPF C file, include bpf_helpers.h so compiler prototypes match kernel helpers like bpf_printk.

Classic loader pattern

The minimal API is straightforward: call bpf_object__open(“./hello_world.o”), then bpf_object__load(obj), iterate programs and call bpf_program__attach(prog). Pause for testing (getchar()) and then clean up.

Skeleton-based flow

For less boilerplate, use generated skeletons. The pattern becomes <app>__open(), <app>__load(), <app>__attach(), and <app>__destroy(). Skeletons simplify CO-RE relocations at load time and make the loader predictable across kernels with BTF.

Tip: consider static linking and llvm-strip to shrink the object for distribution. Use maps or ring buffers instead of bpf_printk in production to move data off the loaded kernel efficiently.

BPF CO-RE and portability: producing the final executable

Packaging a single binary that runs across kernels is the goal I aim for when shipping BPF tools.

We embed the compiled BPF object into a small user-space loader so the final executable holds both pieces. At load time the loader uses CO-RE and BTF to remap types for the running kernel. This requires CONFIG_DEBUG_INFO_BTF=y on build kernels and a vmlinux.h dump when needed.

Globals give a simple, low-overhead way to configure behavior. Set global variables from user space and read them inside the program—no extra syscalls for common flags. For event delivery, I favor ring buffer support on modern kernels; it streams data far more efficiently than printk debugging.

• Embed object + loader to ship one final executable.
• Use CO-RE and BTF for portable bpf programs across linux kernel versions.
• Prefer ring buffers and maps for production observability; use bpf helper calls only for local debug.

Compile Once – Run Everywhere: how libbpf and bpftool glue logic helps

The toolchain generates vmlinux.h and skeleton headers that bind the object into your app. I use one run of bpftool to dump BTF and one link step to produce the executable. At load time the library resolves relocations so the same file adapts to the target kernel layout.

Trade-offs across kernels and features over time

Kernel internals will change—signatures like sched_switch have evolved. CO-RE helps by remapping fields, but if a hook disappears you must adapt and re-test.

I balance modern features with conservative fallbacks so the bpf application stays useful across distributions. In practice, bootstrap examples and prebuilt vmlinux.h speed delivery while keeping portability wide.

Choosing the right tool for the job: libbpf vs bpftool

Pick the tool that matches your current goal: quick inspection or a hardened production loader. I often frame the decision around iteration speed, visibility, and what will ship to users.

Fast iteration, inspection, and information: favor bpftool

If you need instant feedback, use bpftool to dump BTF, list bpf programs, and peek into bpf maps. It helps validate attach assumptions and kernel types before you write a loader.

Run a few commands, fix the source, and repeat. That loop speeds development and reduces guesswork when kernel internals shift.

Shipping a BPF application and loader: favor libbpf (and skeletons)

When it’s time to ship a bpf application, I move to a small loader that uses skeletons. That gives clear lifecycle control and typed access to maps and events.

Skeleton flows make attach bpf and map handling predictable across kernels, so the final binary behaves the same in production.

Case-by-case examples: tracepoint demos, maps, and user-space integration

For tracepoint demos I generate vmlinux.h, craft a tiny program, and verify attach. Once stable, I embed the object into a skeleton-driven loader.

We use bpftool to peek at map state in development, then rely on runtime handles in a loader for updates and ring buffer streaming in production.

Practical takeaways and next steps for BPF development

To finish, I’ll give a focused set of steps to help you move from prototype to a reliable bpf application.

If you’re short on time, start with libbpf-bootstrap. Its minimal examples and prebuilt files get you compiling and running quickly.

Follow a repeatable way: verify BTF, dump vmlinux.h, compile with clang, then use the CLI for inspection and the library for a loader that produces a final executable.

Keep a small toolkit of commands—btf dump, gen skeleton, prog show, map show—so you fix issues in minutes, not hours.

Plan maps and event paths for production, use skeletons to cut boilerplate, and validate on multiple kernels with CO-RE before rollout. The bpf community moves fast and usually takes care to improve these flows.

Next step: pick a tracepoint, wire a tiny program, stream a few metrics via bpf maps or a ring buffer, and iterate—each small example saves you time later.