profile-bpfcc is the right first tool for a broad CPU profile: it samples user-space and kernel-space stack traces while Linux keeps running. Use bpftrace for a quick question, BCC for repeatable scripts, perf for traditional sampling, and Rust with Aya only when existing tools cannot express the policy. Start with the target, kernel support, and symbols, because a noisy profile will not diagnose itself.

Last updated: 2026-07-31

Table of Contents Toggle

Key takeaways

• profile-bpfcc samples user-space and kernel-space stack traces to show where CPU time goes.
• The BCC tool samples at 49 Hertz across all CPUs by default.
• profile-bpfcc requires CONFIG_BPF, BCC, and Linux 4.9 or newer with BPF_PROG_TYPE_PERF_EVENT support.
• Use -p PID to focus on one process instead of profiling the whole machine.
• Use bpftrace for quick questions, BCC for repeatable tools, and perf for sampling workflows.
• Use kernel stacks to find syscall, scheduler, and driver time.
• Use user-space stacks to find application functions and library hotspots.
• Folded stacks and flame graphs make sampled data readable.
• Check kernel support, permissions, symbols, and tool paths before blaming the profiler.
• Start with a short capture. A long noisy trace is still noisy.

What you gain by profiling the Linux kernel with eBPF

Linux kernel profiling works best when the data points to a function, process, or stack instead of one vague CPU percentage. eBPF lets you collect that detail while the system keeps running. You can see which kernel paths consume CPU and which user processes caused the work.

The useful split is simple:

What can you connect to a stack?

The kernel can also attach process and thread context to events. That matters when several services share one host and the top-level CPU graph only says “something is busy.”

You can link network packets to the process that sent them. You can follow allocation activity over time. You can inspect scheduler, syscall, and driver paths without adding logging statements to the kernel.

The trade-off is that eBPF shows what you asked it to show. A narrow filter gives clean data. A wide filter gives a pile of stacks that still needs interpretation. Verification and sandboxing reduce the risk of loading bad programs, but they do not make a poorly scoped trace useful.

Keep kernel versions consistent across nodes when you compare profiles. Use filters, maps, and in-kernel summaries to reduce export volume. Continuous agents can collect the same signals across a cluster, but start on one host first. That is where you find bad symbols, missing permissions, and wrong assumptions.

Set up your system before loading a profiler

Begin with the kernel and the access path. Do not install three tracing tools and hope one of them explains the failure.

Check kernel support and verifier constraints

Check the running kernel first:

uname -r

Then check whether the kernel exposes the needed BPF configuration:

grep -E 'CONFIG_BPF|CONFIG_BPF_SYSCALL' /boot/config-$(uname -r)

Some distributions expose the configuration through /proc/config.gz instead:

zgrep -E 'CONFIG_BPF|CONFIG_BPF_SYSCALL' /proc/config.gz

The profile-bpfcc tool needs CONFIG_BPF, BCC, and BPF_PROG_TYPE_PERF_EVENT support. If the kernel lacks that support, changing the command will not fix it. Install a supported kernel or move the profiling task to a host that has one.

The verifier checks every eBPF program before it loads. It rejects unsafe memory access, unbounded loops, invalid helper calls, and paths it cannot prove safe. A program that looks correct in C can still fail because one branch makes the verifier lose track of a pointer.

Read the verifier output from the first rejected instruction. Do not scroll to the final “permission denied” line and stop there. The first error usually names the real problem.

Install BCC, bpftrace, and perf

On Ubuntu, install the BCC tools and bpftrace from the package manager:

sudo apt update
sudo apt install bpfcc-tools bpftrace linux-tools-common

The exact perf package can depend on the running kernel. However, if the perf command is missing, install the matching Linux tools package for that kernel.

BCC commands commonly live under this path:

ls -l /usr/share/bcc/tools

Look for tools such as profile, execsnoop, tcpconnect, tcpretrans, biolatency, and runqlat.

The package name and command name can differ across distributions. If profile-bpfcc is not found, check the installed files instead of guessing:

command -v profile-bpfcc
command -v profile
dpkg -L bpfcc-tools | grep '/profile'

A missing command is often a package layout issue. It is not automatically a kernel problem.

Confirm capabilities and access to perf events

Run a small tracing tool before you start CPU profiling:

sudo /usr/share/bcc/tools/execsnoop

In another shell, run a short command:

date

If the tool prints the command, basic BCC loading and event access work. Stop it with Ctrl-C.

Some tools need elevated privileges or capabilities such as CAP_BPF, CAP_PERFMON, or CAP_SYS_ADMIN, depending on the kernel and the probe type. A container may also block access through its seccomp profile, mount namespace, or device policy.

Check the process limits and kernel perf policy:

ulimit -l
cat /proc/sys/kernel/perf_event_paranoid

Debug symbols matter too. Without them, a profile may show addresses, unknown symbols, or incomplete kernel names. That output is not useless, but it is harder to act on.

Write down the kernel version, package versions, command path, flags, and permissions. Reproducible profiling starts with a reproducible setup.

Which eBPF front end should you choose?

Choose the front end based on the question, not the tool you installed first. A one-line question and a long-running profiler have different needs.

When to use BCC versus bpftrace versus perf

bpftrace shines when you need a quick answer. It has little boilerplate and lets you test a hypothesis while the problem is happening.

BCC suits longer-lived tools and daemons. Use it when you need argument parsing, structured output, filters, or a command you will run repeatedly. The BCC tools are also useful examples when you want to build your own program.

Meanwhile, perf remains the direct choice for traditional CPU sampling and flame graph workflows. It works outside the BCC tool family and has a mature data path through perf events.

Do not choose Rust and Aya for a five-minute question. A custom loader makes sense when you need a service, a fixed policy, long-term storage, or behavior that existing tools cannot express. Otherwise, you are building infrastructure to avoid answering the original question.

Understand kernel-space and user-space components

The eBPF program runs in the kernel and captures events or stack identifiers. A user-space process reads maps or event streams and formats the result.

The data path usually looks like this:

1. A perf event or tracepoint fires.
2. The eBPF program records counters, timestamps, stack IDs, or event data.
3. A map or ring buffer carries the result to user space.
4. The loader resolves symbols and writes readable output.
5. You fold or visualize the stacks for analysis.

Use arrays and histograms for summaries. Use hash maps for counters keyed by process, thread, or stack. Use ring buffers when each event carries its own data.

Maps are not free. A map with an uncontrolled key space grows until the program hits a limit or the output becomes useless. Add filters and cleanup paths before you run it on a busy host.

What is profile-bpfcc, and how do you use it?

profile-bpfcc is the BCC CPU profiling tool. It samples stack traces rather than recording every instruction or event. That keeps the overhead lower than exhaustive tracing while still showing the paths where CPU time accumulates.

The profile-bpfcc BCC tool profiles CPU usage by sampling user-space and kernel-space stack traces. Its default sampling rate is 49 Hertz across all CPUs. That default gives a broad view of the machine. It is not a guarantee that every short function appears in the output.

Start with a short system-wide capture:

sudo /usr/share/bcc/tools/profile -fd 10

How should you read the first capture?

The -f output is folded stack data. Each line represents a call path followed by a count. The count tells you how often the sampler observed that stack, not how many times the function ran exactly.

To profile one process, use the BCC profile tool with -p PID:

sudo /usr/share/bcc/tools/profile -p <PID> -fd 10

Replace <PID> with the process ID you care about. This is the command behind searches such as bcc profile -p pid. The filter cuts unrelated stacks from the capture, which usually makes the result more useful than raising the sampling rate.

You can find the process ID with:

pgrep -x process-name

For a command that starts later, find its PID after launch or use a wrapper that records it. Do not profile the wrong process and then tune the wrong function. That mistake survives many flame graphs.

Read the tool's installed help and man page before adding flags:

/usr/share/bcc/tools/profile --help
man 8 profile-bpfcc

The profile-bpfcc man page tells you which options your packaged version supports. BCC options can differ across distributions, so copying a flag from an old post is a good way to waste an evening.

The output may include both user and kernel frames. If you only need application code, use the user-stack option supported by your installed tool. If you need scheduler, syscall, or driver time, keep kernel stacks enabled.

How do you profile kernel performance with eBPF?

Start with one metric and one time window. Kernel profiling becomes hard to read when you collect CPU, memory, and network events in one unfiltered session.

CPU sampling with BCC profile and perf events

For CPU hotspots, run profile-bpfcc first. It gives you stack samples without requiring a custom program.

sudo /usr/share/bcc/tools/profile -fd 10

If the host is busy, scope the capture to one process:

sudo /usr/share/bcc/tools/profile -p <PID> -fd 10

For a perf-based capture, use:

sudo perf record -F 99 -a -- sleep 10
sudo perf script

The first command records samples. The second expands them into stack data. Pipe that output through stack-collapsing and flame graph scripts when you need a visual summary:

sudo perf script | ./stackcollapse-perf.pl | ./flamegraph.pl > cpu-flamegraph.svg

The flame graph is not the diagnosis. It shows where samples landed. You still need to connect the stack to a request, workload, lock, syscall, or configuration change.

A wide kernel frame may indicate scheduler time, interrupt work, filesystem activity, or a driver path. Read the callers above it. The leaf function is often only where the work became visible.

Memory allocation and leak signals with eBPF tracing

Memory profiling needs a different question. CPU sampling tells you where the process runs. It does not prove which allocation caused resident memory to grow.

Trace allocation and free paths with kprobes or uprobes where the symbols are available. Kernel allocations may involve kmalloc and kfree. User-space allocations may involve malloc and free, but optimized allocators can hide or redirect those calls.

Look for growing counts without matching frees. Then repeat the capture over time. However, one burst of allocations after startup is not a leak.

A useful workflow is:

1. Identify the process or kernel subsystem.
2. Trace allocation and free events.
3. Group events by stack, size, or owner.
4. Compare growth across repeated intervals.
5. Confirm the suspect path with a second tool.

Use maps for counters and histograms. Use a ring buffer when you need individual allocation records. Do not export every event from a busy allocator unless you enjoy filling disks.

For user-space heap analysis, DHAT and Firefox Profiler can provide a different view. Use them when the question is object lifetime rather than kernel allocation activity.

Network hotspots with tcpconnect, tcpretrans, and stack traces

Start with connection creation:

sudo /usr/share/bcc/tools/tcpconnect

Then check retransmits:

sudo /usr/share/bcc/tools/tcpretrans

tcpconnect shows which processes open TCP connections. tcpretrans shows retransmission activity. Together, they separate connection churn from packet loss or congestion.

Add stack capture when the event alone is not enough. stackcount can group kernel paths, and BCC programs can request user stacks with BPF_F_USER_STACK where supported.

If retransmits rise, inspect the calling stacks for blocking syscalls, slow application code, queue pressure, or a path through the network stack. Do not blame the network because a request is slow. The process may be spending its time waiting before it sends anything.

For deeper syscall timing, use trace system calls with eBPF and tie the network event to the delay.

User-space versus kernel-space stacks

Choose the stack type based on where the time can be spent.

• Use kernel stacks for scheduler, syscall, filesystem, interrupt, and driver work.
• Use user-space stacks for application functions and library calls.
• Capture both when a user request crosses into the kernel.
• Keep symbols available so addresses resolve into names.

A user-space-only profile can make a blocking syscall look cheap. A kernel-only profile can hide the application function that made the call. That is why mixed stacks matter for real incidents.

Missing frames do not always mean missing work. The process may lack unwind information, the binary may be stripped, or the stack walker may not support that path. Check symbols before changing the workload.

Hands-on examples with common tools

Use the smallest tool that can answer the question. Larger tools produce more output, not better reasoning.

Use bpftrace one-liners for fast insights

This command prints file opens by process:

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'

It runs immediately and needs no compile step. That makes it useful for checking whether a deeper profiler is justified.

You can also inspect process execution:

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s\n", comm); }'

Use one-liners to test a hypothesis. Move to BCC or a custom program when you need filtering, aggregation, symbol resolution, or output that must survive a long run.

Run BCC tools for deeper, scriptable output

BCC tools cover common kernel profiling questions:

sudo /usr/share/bcc/tools/execsnoop
sudo /usr/share/bcc/tools/tcpconnect
sudo /usr/share/bcc/tools/biolatency
sudo /usr/share/bcc/tools/runqlat

execsnoop shows process launches. tcpconnect shows connection attempts. biolatency groups block I/O latency. runqlat shows scheduler run-queue delay.

For CPU stacks, use:

sudo /usr/share/bcc/tools/profile -p <PID> -fd 10

The BCC profile tool is a good middle ground. It is more repeatable than a one-liner and less work than writing a loader.

Generate flame graphs from perf data

Flame graphs need folded stacks. Perf creates the raw sample data, and the stack-collapse script turns it into one stack per line.

sudo perf record -F 99 -a -- sleep 10
sudo perf script > perf.out
./stackcollapse-perf.pl perf.out > perf.folded
./flamegraph.pl perf.folded > perf.svg

Open the resulting SVG in a browser. Wider blocks represent more samples. The vertical direction shows the call path.

Do not read the graph as a timeline. It is an aggregate view of samples. If you need ordering, latency, or causality, use trace events instead.

Build a simple eBPF sampling profiler in Rust with Aya

Write a custom Aya profiler only after the existing tools show what you need. The point of a custom program is control, not ceremony.

An Aya-based profiler normally has two parts:

• A kernel-side eBPF program attached to a perf event.
• A user-space loader that configures the program and reads the results.

The kernel program should do as little work as possible. Capture the stack ID, process information, and a small amount of context. Let user space resolve symbols, apply policy, and write files.

Program layout: perf event, ring buffer, and stack capture

A practical layout includes:

1. A perf-event program that runs on samples.
2. A stack-trace map for kernel and user stacks.
3. A counter map keyed by stack identifiers.
4. A ring buffer for records that need process metadata.
5. A user-space reader that resolves and aggregates the data.

Use a map for high-volume counters. Use a ring buffer for records that need timestamps, process names, or command-line context.

Do not send a full stack through the ring buffer for every sample if a stack ID will do. Store the stack once and count references to it. That keeps the export path smaller.

User-space loader: attach policy, frequency, and PID scoping

The loader decides which process, cgroup, or namespace to profile. It also chooses the sampling policy and output format.

Keep the policy outside the eBPF program where possible. A loader can change the target without rebuilding the program. It can also reject unsafe combinations before attaching.

A useful loader should report:

• The selected target.
• The attach result.
• The active maps.
• The number of samples read.
• The number of stack lookups that failed.
• The output path.

If a custom profiler says only “attached,” you still do not know whether it collected useful data. Count samples and dropped records.

Verifier-friendly patterns and common pitfalls

Keep pointer checks close to the dereference. Bound every loop. Keep stack usage small. Use helpers the running kernel supports.

Common failures include:

• Reading packet or task data without checking its size.
• Walking a variable-length structure without a hard bound.
• Passing an invalid map value to a helper.
• Building a key that grows without limit.
• Doing symbol work inside the kernel program.
• Sending too much data through the ring buffer.

The verifier is not being difficult for sport. It needs a proof that every path is safe. Rewrite the program so that proof is obvious.

Make sense of the data: stacks, heat maps, and allocations

A profile is evidence, not an explanation. Start by finding the widest repeated paths, then connect them to the workload.

Read and interpret folded stacks and flame graphs

A folded stack looks like this:

main;worker;handle_request;read 42
main;worker;handle_request;parse 17

The semicolon separates callers from callees. The final count is the number of samples for that path.

In a flame graph, wider blocks mean more samples. Taller stacks show deeper call paths. A wide block near a syscall may indicate blocking or kernel work. A wide application function may indicate compute or repeated calls.

Do not compare two flame graphs unless the capture windows and workloads match. A profile from an idle host tells you almost nothing about a loaded request path.

Track heap allocations with DHAT and view them in Firefox Profiler

Use DHAT when you need allocation lifetime and heap behavior from user space. It can show where allocations remain live and which paths create churn.

Firefox Profiler can help inspect time and allocation data in a more navigable interface. Use it when a text report is too flat to show relationships between threads, events, and call paths.

Heap tools have their own overhead and collection model. Do not compare their numbers directly with a kernel allocation counter. They answer different questions.

Run continuous profiling across nodes and Kubernetes

Continuous profiling needs a smaller data set than an incident capture. Store folded stacks, counts, and selected metadata instead of every raw event.

Run one agent per node or workload boundary. Keep the target filters explicit. A cluster-wide agent with no scope will collect a cluster-wide mess.

Agents, secure paths, and storage

A profiling agent needs access to the kernel interfaces and perf events. Containers may lack the required mounts, capabilities, or namespace visibility.

Decide where the agent runs:

• On the host for broad kernel visibility.
• In a privileged container for controlled deployment.
• Beside one workload when you need application-only stacks.
• In a lab namespace while you validate the policy.

Protect the output. Folded stacks can expose process names, file paths, service names, and user-space symbols. Treat them as operational data, not anonymous metrics.

Use retention rules and size limits. A continuous profiler that fills /var is not observability. It is a delayed outage.

Rollout steps for Ubuntu and Kubernetes

Roll out in stages:

1. Validate BPF support on one node.
2. Run a short system-wide profile.
3. Scope the profile to one process or cgroup.
4. Confirm symbols and stack quality.
5. Compare overhead during normal load.
6. Deploy to a small node group.
7. Review storage, permissions, and dropped samples.
8. Expand only after the output stays useful.

Kernel version differences can change helper support, stack behavior, and permissions. Record the node image and kernel release with each profile.

For Kubernetes, decide whether the profile follows the pod, container, cgroup, or node. Those are different scopes. A node profile can show work from every workload, while a pod profile may miss host-side network or filesystem time.

Security, cost control, and operations

Grant the narrowest access that works. Avoid running a long-lived profiler with more privileges than its attach points need.

Use PID, cgroup, namespace, and duration filters. Summarize in the kernel when possible. Sample instead of tracing every event unless you need exact ordering.

Watch these operational signals:

• Failed program loads.
• Missing stack symbols.
• Dropped ring-buffer records.
• Map growth.
• Agent restarts.
• Output volume.
• CPU overhead during peak load.

If the agent fails, keep the workload running. Profiling is diagnostic plumbing, not a reason to take production down.

What should you do after you have the profile?

Start with the widest stack that matches the incident window. Then form one change that should reduce that path.

For CPU, compare the profile before and after the change. For memory, compare allocation and free behavior over time. For network issues, connect retransmits or socket activity to the process and call path.

Keep the raw folded output with the command, kernel release, workload version, and capture window. A flame graph without context is a picture you cannot reproduce.

If the result is unclear, narrow the target instead of collecting longer. Profile one PID, one cgroup, one syscall family, or one node. The real problem is usually hidden by scope, not by a lack of data.

FAQ

Does profile-bpfcc trace every CPU instruction?

No. It samples stack traces at intervals. That makes it useful for finding broad CPU hotspots, but it will miss work that runs too briefly or too rarely to appear in the samples.

Why do some profile stacks show unknown symbols?

Unknown frames usually point to missing debug symbols, stripped binaries, unsupported unwind information, or an incomplete symbol path. Check symbol packages and the profiler's stack options before changing the workload.

Can I run BCC profiling inside a container?

Sometimes, but the container needs access to the required kernel interfaces and permissions. A restricted seccomp profile, missing capabilities, or a separate mount namespace can block the attach even when the host supports BPF.

Does a higher sampling rate always improve the profile?

No. A higher rate can expose short-lived paths, but it also creates more overhead and output. First make the target and capture window correct. Then adjust the rate only if the profile still misses the behavior you need.

How do I know if a hotspot is caused by the application or the kernel?

Capture both user-space and kernel-space stacks, then follow the call path across the boundary. If the application spends time in a syscall, scheduler path, filesystem function, or driver, the kernel frames explain the cost. If the samples stay in application functions, start there.

Related on this blog

• Track System Call Latency