Profile-bpfcc: Low-Overhead CPU Stack Sampling
profile-bpfcc is the bcc tool I reach for when I need to know exactly where the CPU is burning cycles, in the kernel and in user space, without recompiling anything or paying the overhead of a heavy tracer. Run it with -p PID to sample one process, or with no PID to sample the whole box. Feed its folded output straight into a flame graph and trust the stacks, not the man page summary, to show you what is actually hot. Expect concrete examples, short code notes, and clear outputs you can read with flame graphs and folded stacks.
Last updated: 2026-07-21
What is profile-bpfcc and what problem does it solve?
profile-bpfcc is bcc's eBPF-based CPU stack sampler. It wakes up on a timer, grabs the current stack trace on every CPU, counts how often each stack shows up, and prints the totals. The stack that appears most is where your time goes. That is the whole idea.
It solves the question every performance problem eventually becomes: which function is eating the CPU right now? Classic /proc sampling and top-style tools tell you a process is busy. They do not tell you which call path inside it is busy. profile-bpfcc does, kernel and userspace, in one pass.
The catch nobody mentions up front: it needs a recent kernel, so anything older and you are back to perf. On modern distros the tool ships in the bpfcc-tools package and lives at /usr/share/bcc/tools/profile.
How profile-bpfcc actually samples stacks
Here is what happens under the hood. profile-bpfcc attaches an eBPF program to a perf_event that fires at a fixed frequency, say 49 times a second per CPU. Each time it fires, the program walks the current stack and bumps a counter in a BPF map keyed by that stack. No stack is ever printed twice; identical stacks just increment the same count.
This is sampling, not tracing. It does not see every function call. It sees a statistical picture: the more CPU a path uses, the more samples land on it. That is exactly what you want for CPU hotspots, and it is why the overhead stays low even under load.
The counting happens in-kernel, so almost nothing crosses into user space until the run ends. That summarize-in-kernel design is the reason profile-bpfcc is cheap enough to leave running while a real workload is under stress.
Profile a single process with -p pid
Point it at one PID and let it run for a few seconds:
sudo /usr/share/bcc/tools/profile -f -p 1234 10
That samples PID 1234 for ten seconds and prints folded stacks. The trailing 10 is the duration. Drop it and the tool runs until you hit Ctrl-C. The -f flag gives you folded output, one stack per line with a sample count, ready for a flame graph.
Scoping to a PID is the right first move when you already know the guilty process. It cuts noise from every other task on the machine and shrinks the flame graph to something you can read. If you are chasing a specific service, get its PID from systemctl status or pgrep and pass it straight in.
One gotcha: -p follows the process, but if it spawns children doing the real work, those live under different PIDs. For a whole process tree, profile the system and filter later, or profile by cgroup.
When to drop -p and profile the whole system
Leave -p off and profile-bpfcc samples every CPU across every process:
sudo /usr/share/bcc/tools/profile -f 30
Reach for system-wide sampling when you do not yet know who the offender is, or when the pain lives in the kernel itself and no single process owns it. Scheduler contention, softirq storms, a driver spinning, a lock everyone fights over: these show up in a system-wide profile and hide in a single-process one.
System-wide costs more, because now you are walking stacks on every CPU for every task. On a busy machine that is more samples and a bigger flame graph. Start wide to find the layer, then re-run with -p on the process the wide view fingered. That two-step saves you from squinting at a flame graph with two hundred towers in it.
Splitting kernel stacks from user stacks
By default profile-bpfcc captures both kernel and user stacks and stitches them together, which is usually what you want. When the picture gets muddy, split them:
-Kshows kernel stacks only.-Ushows user stacks only.
Use -U when a userspace app is clearly the problem and kernel frames are just noise around syscalls. Use -K when you suspect the kernel itself, a driver or the scheduler, and the userspace side is boring. Mixing both is fine for a first look, but a combined flame graph can blur the boundary and make you chase a syscall entry when the real cost is three frames deeper in the kernel.
If you are already comfortable watching syscalls with eBPF, this filter is the same instinct applied to CPU time. My rule: capture both first, then narrow with -K or -U once you know which side of the fence to stand on.
Tuning sampling frequency without lying to yourself
profile-bpfcc defaults to 49 Hz per CPU, and that odd number is deliberate. Change it with -F:
sudo /usr/share/bcc/tools/profile -f -F 99 -p 1234 10
Higher frequency means more samples, so short-lived hot paths show up that a slow rate would miss. It also means more overhead and more data. Lower frequency is what you want for continuous profiling where the tool runs for minutes and you care about the big picture, not the twitchy detail.
Why 49 instead of a round 50? Timer aliasing. If your sample rate lines the up with a periodic workload running at 50 or 100 Hz, you keep catching the same phase and your profile lies. An off number like 49 or 99 breaks that lockstep. This is the one setting people get wrong by picking a "nice" value.
The kernel caps sampling too. kernel.perf_event_max_sample_rate sits at 50000 Hz by default, so you cannot push -F past that without raising the sysctl, and you almost never should.
Turning profile-bpfcc output into a flame graph
Folded output is the point of -f. Each line is a semicolon-joined stack followed by its sample count, which is exactly the format Brendan Gregg's flame graph scripts eat:
sudo /usr/share/bcc/tools/profile -f -p 1234 30 > out.folded
./flamegraph.pl out.folded > profile.svg
Open the SVG in a browser. Width is time: the wider a box, the more samples landed on that function, the more CPU it burned. Height is stack depth, not cost, so a tall thin tower is a deep call chain that barely runs. You read for width, then click to zoom.
Do not skip the -f. Without it you get human-readable stacks that the flame graph tooling will not parse, and you will waste ten minutes wondering why the SVG is empty. Folded in, flame graph out, every time.
How does profile-bpfcc compare to perf and other tools?
perf is the old reliable and it works on kernels far older than eBPF requires. It samples with hardware performance counters and produces flame graphs the same way. The difference is where the work happens: perf writes a fat perf.data file of raw samples and you post-process it, while profile-bpfcc counts stacks in-kernel and hands you a small summary. For long or high-rate runs, the eBPF approach moves far less data.
Here is how I choose:
| Tool | Best for | Kernel needed | Output |
|---|---|---|---|
| profile-bpfcc | Low-overhead CPU sampling, folded stacks in-kernel | Requires eBPF support | Folded stacks, count summary |
| perf | Portable sampling, hardware counters, older kernels | Almost any | perf.data, needs post-processing |
| bpftrace | Quick ad-hoc stack counts as one-liners | Requires eBPF support | stdout, live |
bpftrace is the one-liner front end when you want a fast answer and no files. For a repeatable CPU profile you archive and diff, profile-bpfcc is the sharper tool. If your goal is trimming the tracing itself, that is a different job covered in cutting kernel tracing overhead.
What the profile-bpfcc man page options actually mean
The bcc profile source and docs spell these out, but here is the short version you will use ninety percent of the time:
-p PIDscopes to one process. Leave it off for the whole system.-F FREQsets samples per second per CPU. Default 49.-fprints folded stacks for flame graphs.-dadds a delimiter between kernel and user frames so you can tell them apart.-U/-Kcapture user-only or kernel-only stacks.--stack-storage-size Ngrows the map when you see warnings about dropped stacks.- The trailing number is the run duration in seconds.
A tired reader mixes up -f and -F: lowercase is folded output, uppercase is frequency. Read the flag, not the shape. When in doubt, run profile -h and match the letter to the description before you build a command from memory.
How do you read errors and missing symbols in profile-bpfcc output?
When stacks come back as bare hex addresses or [unknown], the tool is working and the symbols are missing. Install the debug symbols for the binary or the kernel, dbgsym packages on Debian and Ubuntu, and the names come back. A JIT language like Java or Node needs its own symbol map for user frames; without it you get addresses no flame graph can name.
A permission error usually reads as Operation not permitted or a failure to attach the perf event. profile-bpfcc needs privilege to open perf events and load eBPF. Run it under sudo, or grant the process CAP_BPF and CAP_PERFMON so it can use perf events and eBPF tracing without full root. That capability split is the modern, safer path.
If you see a warning about dropped stacks, the in-kernel map filled up. Raise --stack-storage-size and re-run. The real problem there is volume, not a bug, and a bigger map fixes it. Missing kernel frames specifically mean CONFIG_BPF_SYSCALL or frame-pointer support is off in your kernel build, which is worth checking before you blame the tool. For the underlying attach mechanics, getting started with bcc tools walks the same ground.
FAQ
Does profile-bpfcc work inside a container?
Yes, if the container has the privilege and a shared kernel view. You need CAP_BPF and CAP_PERFMON or root, plus access to the host's tracing filesystem. Sampling from the host and filtering by cgroup is often cleaner than running the tool inside each container, because one host-side run sees every process at once.
How much overhead does it add at 49 Hz?
Low enough to leave running on a busy production box for a short window. Because stacks are counted in-kernel and only a small summary crosses to user space, the cost scales with sample rate and CPU count, not with how much work the target does. Push -F up toward the 50000 Hz kernel cap and overhead climbs fast, which is exactly why you keep the frequency modest.
Why is one CPU showing far more samples than the others?
That is usually a single thread pinned or a workload that will not spread across cores. profile-bpfcc samples every CPU, so a lopsided count is a real signal, not a tool artifact. Look at whether the hot stack is a spinlock or a busy loop, and check the affinity of the offending thread.
Can profile-bpfcc profile memory or network activity?
No, it is a CPU stack sampler and nothing else. For allocations, reach for memleak or the alloc tracers; for network paths, tcpconnect and tcpretrans in the same bcc collection. profile-bpfcc answers "where is the CPU going," and you pair it with those other tools when the question changes.
Is bpftrace a full replacement for it?
Not quite. bpftrace can count stacks in a one-liner and is faster to type, and yes, keep an eye on its version since it moves quickly, so check the current release on its project page rather than trusting a number in a blog. For a saved, repeatable CPU profile you diff across runs, profile-bpfcc's folded output is the format the flame graph tooling expects with no extra glue.
