Reach for BCC, the eBPF compiler collection, when you need deep, ad-hoc kernel tracing on Linux and can live with its cost. It compiles your eBPF C at runtime with LLVM and clang, so the box needs kernel headers that match the running kernel. That buys you syscall, scheduler, block I/O, and network visibility with a huge library of ready-made tools. If you want a program you compile once and run everywhere, use CO-RE with libbpf instead. If you just want a quick probe, run bpftrace. BCC drives from Python, and there is no mature Rust binding, so Rust folks should use libbpf-rs or Aya.

Last updated: 2026-07-21

So you want a practical bcc tools eBPF tutorial to fix slow services and blind spots without risky kernel hacks. Below I lay out what BCC is, which tools earn their place, how the Python frontend works, and when I'd pick something else.

What is BCC in the eBPF toolchain?

BCC is a compiler collection for eBPF, and that name is worth taking literally. It is two things stacked together. First, a framework that turns C source into eBPF bytecode and loads it. Second, a big set of prebuilt command-line tools written on top of that framework.

People conflate the two and get confused when advice about "BCC" doesn't match what they typed. When someone says "just run execsnoop," they mean the tool suite. When someone says "write a BCC script," they mean the framework and its Python bindings. Both live in the same project, but you use them differently.

The safety story is the reason any of this runs in the kernel at all. Every eBPF program must pass the kernel verifier before it executes, which rejects unbounded loops and out-of-range memory access. That is what lets you attach code to a live production kernel without the panic risk of a bad module.

How does BCC compile eBPF programs at runtime?

BCC embeds LLVM and clang and compiles your C on the machine, every time you run the tool. That is the whole design, and it explains both its strength and its main annoyance. Because it compiles against the local kernel headers, it adapts to whatever struct layout your kernel actually has. So a script that reads a task field works even if that field moved between kernel releases.

The annoyance is the flip side. You need clang, llvm, and linux-headers-$(uname -r) present, and compilation eats CPU and a second or two of startup on each launch. On a fleet, that is real overhead you pay repeatedly.

CO-RE (Compile Once, Run Everywhere) with libbpf takes the opposite approach. You compile the program ahead of time into one small binary, and BTF (BPF Type Format) relocations fix up struct offsets at load time on each target. No clang on the production host, no per-run compile. That is why production tooling is drifting toward libbpf and CO-RE, while BCC stays king of ad-hoc work and learning.

The bcc-tools I reach for first

Before writing anything, run the tools that already exist. The bcc-tools package ships dozens of them, and picking the right one is most of the job. Here is the short map I use.

Each one answers a failure you'd actually hit. Slow service with high iowait points you at biolatency. Mystery short-lived processes point you at execsnoop. Learn six of these well and you'll answer most "why is this box acting weird" questions before touching a compiler.

Tracing CPU, syscalls, and kernel latency

Start from the hook that matches your question. A script you copied off the web rarely matches your kernel. Kprobes enable dynamic tracing of kernel functions, which means you can attach to almost any kernel symbol without changing the kernel. That flexibility is also the catch: kprobe targets are internal function names that can rename or vanish between versions, so a script that worked last release may miss.

When stability matters, prefer a tracepoint. Tracepoints provide a very stable API in the Linux kernel, with fields the maintainers commit to keeping. So tracepoint:syscalls:sys_enter_openat is a safer long-term anchor than a kprobe on some internal helper.

The habit that actually pays off is reading kernel source before you trace. Open the function in the tree, see what arguments it takes, then read those exact arguments in your probe. Guessing at argument order is how you get plausible numbers that are quietly wrong. For a full walk-through of the syscall path, see my guide on tracing system calls with eBPF, and for CPU work the notes on profiling kernel performance with eBPF.

Watching I/O and network paths

For storage, latency histograms beat averages every time. biolatency buckets block I/O completion times so you see the tail, and a bimodal histogram usually means two devices or a cache boundary, not one slow disk. An average would hide that entirely.

On the network side, BCC covers both directions. tcplife and tcpconnect trace socket lifecycles for connection-level questions, while XDP (eXpress Data Path) programs count and filter packets on the driver fast path. XDP runs in offloaded, native, or generic mode depending on your NIC and driver, and generic mode is a slow fallback, so check which one attached.

When BCC output surprises you, confirm it against a second tool before you act. Cross-check a suspected slow open with strace -T on the process, and read dmesg for driver or link errors. If BCC says the latency is in the kernel and strace agrees the syscall is the one stalling, now you have a reproducible root cause you can fix.

Setting up a safe environment to run BCC

Pick an environment that keeps experiments repeatable before you load a single probe. Native Ubuntu works well when your kernel headers match the running kernel, which is what avoids compile errors at load time. A lightweight VM costs a little I/O and setup but isolates the mess, which I prefer when I'm trying things that might wedge a driver.

On Apple Silicon, Lima is the easy path. brew install lima, then limactl start an Ubuntu instance, mount your work directory, and provision bpfcc-tools, linux-headers-$(uname -r), build-essential, and clang/llvm in one pass. Keep the work folder mounted so files survive the VM.

Pin images or snapshot the instance for repeatability over time. Do a quick sanity check with a sample BCC example to confirm headers, compiler collection, and permissions are correct. If that one example loads clean, the rest will too; if it fails, fix the environment now instead of blaming your script later.

Two labs: a kprobe hello world and an XDP UDP counter

The fastest way to make the write-load-observe loop click is to run two small programs. Both need sudo, because attaching probes needs privilege.

Kprobe hello world. Write a C snippet with a function syscall__clone that calls bpf_trace_printk("Hello, World!\n"). Load it from Python with BPF(text=...), then build the event name from b.get_syscall_prefix().decode() + "clone" so it works across architectures. Attach with b.attach_kprobe(event=event_name, fn_name="syscall__clone") and stream with b.trace_print(). Every new process prints a line. It's noisy, but it proves the pipeline.

XDP UDP counter. In C, define KBUILD_MODNAME, include linux/bpf.h, if_ether.h, ip.h, and udp.h, and declare BPF_HISTOGRAM(counter, u64). The function udp_counter(struct xdp_md *ctx) walks packet bounds with ctx->data and ctx->data_end, checking each header and returning early on malformed packets. When the IP protocol is UDP, read the destination port, convert with htons, and bump the histogram. From Python, b.load_func("udp_counter", BPF.XDP), attach_xdp(device, fn, 0), and read the map on a timer.

Notice the split. The kernel side stays tiny and bounds-checked so the verifier accepts it. All the aggregation and printing lives in user space, where it's easy to change.

How the Python frontend actually works

Most BCC tools and custom scripts are Python, and the whole frontend hangs off one class. You create a BPF object with your C source, and BCC compiles and loads it. From there, BPF exposes helpers to attach probes, look up map tables by name, and open perf buffers.

Maps are the bridge between the two worlds. The kernel program writes counts, arrays, or histograms into a map, and your Python reads the same map by name and turns it into output. For low-rate events you subscribe to a perf buffer and get a callback per event; for high-rate metrics you aggregate in a map and poll it, which avoids drowning user space.

That is why Python is the default for eBPF work with BCC: the loader, the map access, and the presentation are all a few lines. When you see b["counter"].items() in a script, that is a Python program reading a kernel-populated map. Keep the heavy logic on the Python side and the kernel side stays small enough to pass the verifier without a fight.

Can you write BCC tools in Rust?

Not really, and I'd stop looking for a way. BCC has no first-class Rust frontend, only unmaintained or thin wrappers that lag the C and Python paths. If you fight to bolt Rust onto BCC, you inherit the runtime-compile and header dependency and get none of the ecosystem support.

Rust developers have a better road anyway. Use libbpf-rs, which wraps libbpf and CO-RE so you compile once and ship a portable binary, or use Aya, a pure-Rust eBPF library with no libbpf dependency at all. Both give you a real Rust toolchain, ahead-of-time builds, and low per-run overhead.

Put simply: BCC is a Python story. If your team standardizes on Rust, skip BCC for anything you plan to deploy and pick the CO-RE-based Rust tools from the start.

The Lua frontend and why it faded

BCC once shipped a Lua frontend, and it still exists in the tree, but almost nobody reaches for it now. Python won on tooling, examples, and the sheer number of prebuilt scripts written against it. Nearly every tutorial and reference tool you'll find is Python.

Lua might still matter in one narrow case: a constrained environment where you want a smaller runtime than CPython and can accept fewer examples. For most people that trade isn't worth it. Learn the Python interface, and treat Lua as a footnote.

Installing BCC and matching it to your kernel

Install the distro package and confirm your headers first. A random tarball build just adds variables you'll have to rule out later. On Debian and Ubuntu that's bpfcc-tools plus linux-headers-$(uname -r); on Fedora it's the bcc and bcc-tools packages. The header package must match the exact running kernel, or scripts fail to compile at load with a missing-header error.

Version compatibility is where people lose an evening. A tool that uses a newer kprobe target, tracepoint, or BPF helper simply won't load on an older kernel, and the error can be cryptic rather than "your kernel is too old." Read the BCC project's documentation for the per-tool kernel requirements before you assume the tool is broken.

If you want to understand exactly which kernel features gate which programs, my write-up on building a minimal kernel for eBPF covers the config options that decide whether a probe attaches at all. Check those before blaming BCC.

Where to learn BCC and eBPF tracing properly

Go to the source, not the tenth Stack Overflow copy. Brendan Gregg's books and blog are the reference for BCC-based performance tracing, and reading them beats collecting one-liners you don't understand. When a tool misbehaves, the fastest fix is usually reading the tool's own source in the repo.

Two repos are worth cloning. The bcc repo holds the Python tools and their C source, which teach the write-load-observe pattern by example. The libbpf-tools directory shows the same tools rebuilt as CO-RE programs, which is the clearest before-and-after for understanding why the ecosystem is moving that way.

The point of reading source over forums is that you learn the failure, so the next weird result is one you can diagnose instead of guess at.

When to pick BCC over bpftrace or CO-RE

Match the tool to the job, and most of the confusion disappears. Here's how I split them:

1. bpftrace for a fast, throwaway question. One line, no Python program, ideal for "does this function even get called?"
2. BCC for ad-hoc work that outgrows a one-liner: custom aggregation, structured output, or a script you'll rerun during an incident. It's also the best place to learn, because the prebuilt tools and Python API are so approachable.
3. libbpf + CO-RE for anything you deploy across a fleet. You compile once, drop the clang dependency, and get low per-run overhead across kernel versions.
4. libbpf-rs or Aya when your shop writes Rust, since BCC won't serve you there.

If you're already thinking about production overhead, the notes on reducing kernel overhead with eBPF explain why the runtime-compile model is fine for debugging and wrong for a permanent agent.

FAQ

Why does my BCC script fail with a missing header like "No such file or directory"?

The linux-headers package doesn't match your running kernel. Run uname -r, then confirm linux-headers-$(uname -r) is installed for that exact string. After a kernel upgrade you often boot the new kernel while only the old headers are present, so reinstall the matching package and rerun.

Does BCC work inside a container?

It can, but it needs privilege and access to the host kernel's headers and tracefs. Run the container with the right capabilities and mount the header path, or better, run BCC on the host and trace the container's processes from outside. Trying to compile inside a stripped image usually dies on missing clang or headers.

Is it safe to attach a kprobe on a production box?

Generally yes, because the verifier blocks unsafe programs before they run, and probe overhead is low for most functions. The real risk is rate: tracing an extremely hot function and printing per event can flood user space. Aggregate into a map and poll it rather than streaming every hit.

How do I turn a BCC prototype into a low-overhead production tool?

Rewrite it as a CO-RE program with libbpf, or with libbpf-rs if you're in Rust. You keep the same probes and logic but drop the runtime clang compile and the header dependency. Compile once, ship the small binary, and it loads across kernel versions using BTF relocations.

Related: eBPF for Windows: Hooks, HVCI, and Safe Deployment