Use an eBPF trace to answer one narrow question, not to collect everything the kernel can expose. Start with logs and strace, identify the real code path, then attach the smallest useful probe. eBPF gives you focused kernel and application evidence with low overhead. However, a clever program on the wrong hook produces fast misinformation, which is worse than having no trace at all.

Last updated: 2026-08-11

What does an eBPF trace actually prove?

An eBPF trace proves that a specific event reached a specific hook while your program was attached. It can record arguments, return values, timing, process identity, cgroup identity, and selected kernel state. It does not prove why the event happened.

That distinction matters. If you see openat return EACCES, you know the kernel denied the request. You still need the path, credentials, mount namespace, and security logs before blaming file permissions.

I choose hooks from three concrete sources:

• Application and service logs pin down the symptom and time window.
• strace shows which system calls the process made and what they returned.
• Kernel logs and source show where the request went after syscall entry.

Run this first on a systemd host:

journalctl -b -p err
dmesg -T | tail

The first command shows errors from the current boot. The second reads the newest messages in the kernel ring buffer. Check the output for denials, filesystem errors, out-of-memory events, and driver failures before adding probes.

Then reproduce the failure under strace:

strace -ff -ttt -T -yy -o /tmp/app.strace ./app

-ff separates child processes, -ttt adds timestamps, -T records syscall duration, and -yy resolves file descriptor details. That tells you which calls failed and what arguments reached the kernel.

Use eBPF after that split. It should answer what strace cannot, such as scheduler delays across the host, kernel function timing, or activity from related processes.

Choosing the hook that matches the failure

Prefer a named tracepoint when it exposes the fields you need. Tracepoints are less tied to internal kernel function names than kprobes, so they usually survive updates with less repair work.

For syscall errors, start with typed sys_enter_* and sys_exit_* tracepoints. Entry gives you the submitted arguments. Exit gives you the result the application received.

Kprobes come later. If a named tracepoint already answers the question, attaching to an internal function buys you maintenance work and little else. For CPU-wide questions, use sampling instead of tracing every syscall; this guide to CPU stack sampling with eBPF covers that path.

Use uprobes when the delay happens before the kernel sees anything. A slow library call that never reaches a syscall is not a kernel problem, no matter how long you stare at sys_enter.

Container syscall visibility and attribution

A process in a container still executes the host kernel’s syscall path. Containers isolate views of processes, mounts, networks, and other resources. They do not bring a private kernel with them.

That means a host-attached syscall tracepoint can observe container workloads. However, attribution is where traces become misleading. The process ID seen by a host-side eBPF program may not match the process ID printed inside the container’s PID namespace.

I normally put these fields in every event record that needs container attribution:

• Host process and thread identifiers
• Process name
• Cgroup identifier
• User identifier
• Mount namespace identifier when file paths matter
• Network namespace information when tracing sockets
• Syscall result and timestamp

The cgroup identifier is usually the cleanest workload key. Userspace can map it back to container or pod metadata without stuffing orchestration logic into the kernel program.

Host visibility also requires host authority. A tracing tool running inside an ordinary container cannot inspect the whole machine merely because it contains an eBPF binary. The host must expose the required kernel interfaces and grant the loader enough privilege under its security policy.

Entry and exit events also need careful pairing. Key temporary state by thread identifier, not process name. Several threads can issue the same syscall under the same name, because Linux enjoys making ambiguous shortcuts expensive.

Raw syscall entry with trace_event_raw_sys_enter

Use trace_event_raw_sys_enter when you need one raw entry point for many system calls. The context exposes a syscall identifier and its argument slots, which lets one dispatcher handle broad syscall coverage.

A minimal libbpf program looks like this:

SEC("tracepoint/raw_syscalls/sys_enter")
int handle_sys_enter(struct trace_event_raw_sys_enter *ctx)
{
 long syscall_id = ctx->id;
 unsigned long first_arg = ctx->args[0];

 /* Filter and dispatch here. */
 return 0;
}

Generate or include the correct kernel type definitions instead of copying the structure from a random header online. Architecture and kernel details matter. Argument slots are raw machine values, so decode them according to the syscall and target architecture.

Pointers need even more care. A filename argument is a userspace address, not a kernel string you can dereference directly. Use the proper eBPF helper to copy bounded userspace data, and record failed reads instead of quietly dropping them.

Typed syscall tracepoints are better when they already expose named fields. They make the program easier to review and harder to decode incorrectly.

An entry event also tells you nothing about success. Attach the matching exit hook when you need return values, errno results, or duration. For a focused latency workflow, see Linux syscall tracing.

The bpf syscall before a trace program runs

The bpf() syscall is the control path userspace uses to ask the kernel to create and manage eBPF objects. It is separate from the application syscall you are tracing.

A loader uses bpf() commands to create maps, load programs, inspect objects, and create links for supported attachment types. The Linux BPF_PROG_LOAD command of the bpf(2) system call loads a BPF program into the kernel. On success, it returns a new file descriptor associated with that eBPF program.

Before accepting the program, the verifier checks its instructions and control flow. It rejects invalid pointer use, unsafe memory access, and paths it cannot prove safe. Passing verification means the program met the verifier’s rules. Your filtering or attribution can still be wrong.

Capabilities and policy decide who can load programs. The exact requirements vary with kernel configuration, distribution policy, lockdown mode, and program type. Granting a container broad host privileges may silence the loader error, but it also replaces a tracing problem with a security problem.

The kernel’s BPF syscall documentation describes the commands and object model. Read it when loader behavior and a tool’s error message disagree. The kernel gets the final vote.

Once loaded, the program runs only when its hook fires. If you trace an application’s connect call, that event is separate from the loader’s earlier use of bpf().

Tail calls for explicit dispatch

Use an eBPF tail call when one verified program needs to dispatch into another without returning. The common case is a broad syscall entry program that selects a decoder from a program-array map.

Here is what happens. The first program reads the syscall identifier, applies shared filters, and calls bpf_tail_call. If the map contains a matching program, execution continues there rather than returning to the caller.

If lookup fails or the kernel-enforced chain limit is reached, execution continues after the helper call. You must handle that path. Otherwise, one missing program-array entry quietly drops the decoder you expected to run.

Tail-called programs do not share ordinary local stack state like normal C functions. Pass durable state through maps, or derive it again from the hook context. Keep the dispatch table explicit so you can tell which decoder ran.

Tail calls earn their place in a few concrete designs:

• Splitting syscall families into separate programs
• Loading optional decoders at runtime
• Keeping each verified program small
• Reusing one filtering and attribution stage

Do not use them to imitate a large application framework inside the kernel. Deep dispatch chains are hard to reason about, and verifier-safe code can still be operationally unreadable.

bpf_trace_printk is a debugging crutch

Use bpf_trace_printk to prove that a probe fired. Stop using it once you need reliable data.

Most modern code reaches it through the bpf_printk macro. Its output appears in the tracing pipe:

sudo cat /sys/kernel/tracing/trace_pipe

That command streams trace messages as they arrive. It is useful during development because you can confirm the hook, inspect a small value, and then remove the print.

However, the tracing pipe is a shared debugging channel. Formatting work happens on the event path, output can be lost under load, and another tracing user may already depend on the same facility. It has no useful schema, lifecycle, or backpressure plan.

Pick the transport from the data you need to keep:

• Maps for counters, histograms, lookup tables, and temporary state
• Ring buffers for structured events sent to userspace
• Perf buffers when supporting code already depends on them
• Userspace reporting for formatting, storage, and export

A ring-buffer event should carry fixed fields such as timestamps, identifiers, return values, and bounded strings. Userspace can enrich those records with service names and container metadata.

Printf tracing feels productive because text appears immediately. So does adding debug prints to a race condition. Neither is an observability pipeline.

How do you trace a syscall from a symptom to a kernel answer?

Start with the failing process, not a catalogue of available probes. I use this order because each step removes a layer of guesswork.

1. Anchor the failure in time. Read the application log, journalctl, and the kernel ring buffer.
2. Reproduce it under strace. Confirm the syscall, arguments, file descriptors, result, and duration.
3. Read the relevant kernel path. Find where the syscall validates inputs, performs permission checks, or waits for a resource.
4. List available tracepoints. Prefer the narrowest named event that exposes the required fields.
5. Attach entry and exit probes. Correlate them by thread identifier and clean up temporary state.
6. Filter before emitting. Select the target process, user, cgroup, or workload in the kernel.
7. Reproduce the same failure. Compare trace timestamps with the original symptom.
8. Remove the probe. A diagnostic attachment is not permanent monitoring by default.

For a quick hypothesis, use bpftrace. On Ubuntu 24.04 LTS, the package name is bpftrace.

List the available hooks before inventing one:

sudo bpftrace -l 'tracepoint:syscalls:sys_*openat*'

That output tells you which matching syscall tracepoints this host exposes. Then trace entry, exit, and duration:

tracepoint:syscalls:sys_enter_openat
/comm == "myapp"/
{
 @start[tid] = nsecs;
 printf("%s path=%s\n", comm, str(args->filename));
}

tracepoint:syscalls:sys_exit_openat
/@start[tid]/
{
 printf("%s ret=%d latency_ns=%llu\n",
 comm, args->ret, nsecs - @start[tid]);
 delete(@start[tid]);
}

This script answers whether myapp reached openat, which path it supplied, what the kernel returned, and how long the call took. It does not tell you why access failed.

The official bpftrace documentation covers its probe syntax and built-ins. Read the host’s listed tracepoint fields as well, because copied scripts often assume fields that are not present.

Comparing strace, bpftrace, and libbpf

Start with strace. Move to bpftrace when you need host-wide context or a kernel hook. Write a libbpf-based program only after the question survives both steps.

strace has a clear advantage: it shows what one process actually asked the kernel to do. However, it does not explain scheduler delays across the host or activity outside the traced process tree.

bpftrace is the right next step for a focused investigation. It lets you test a tracepoint or kprobe without building a loader, object file, event decoder, and deployment path.

A libbpf program earns its cost when you need strict filtering, structured events, map state, version-aware loading, and controlled attachment cleanup. This is also where Build Once, Run Everywhere, usually shortened to CO-RE, and BPF Type Format data become useful.

Do not start with the largest toolchain because the incident sounds serious. Tooling weight does not improve the question.

Keeping tracing low-overhead and safe in production

Filter at the top of the eBPF program. If an event does not belong to the target workload, return before reading strings, updating several maps, or sending data to userspace.

Put the cheapest filters first:

• Process or thread identifier
• Cgroup identifier
• User identifier
• Process name
• Target network address or port
• Selected syscall identifier

Aggregate in maps when you need counts or latency distributions. Emitting every event creates userspace work, memory pressure, and storage noise. For rare failures, structured event output makes sense. For frequent success paths, count in the kernel and report summaries.

Sampling is acceptable for broad performance work. It fails for a rare error where dropping the one bad event ruins the investigation. Pick the method from the failure you need to catch.

Also scope attachments narrowly. Prefer one tracepoint over a wildcard, one cgroup over the host, and one incident window over an attachment nobody remembers owning.

Measure the cost while the probe runs. Compare application latency, CPU use, event loss, and map growth with the trace disabled and enabled. Low overhead is a property you verify, not a sticker attached to eBPF.

For the broader cost model, see how eBPF affects kernel overhead.

Turning trace output into a real fix

Tie every event back to application behavior and kernel source. A timestamp, syscall identifier, and return value are evidence. The explanation still needs a reproducible path.

For latency, compare entry and exit times and check what happened between them. A slow syscall may be waiting on storage, a socket, memory reclaim, a lock, or a remote peer. The syscall name alone does not identify the wait.

For errors, decode the return value and inspect the relevant state. A failed file open can come from permissions, path lookup, a read-only mount, a security module, or a vanished parent directory. Collect the path, mount namespace, credentials, and return value needed to separate those cases.

Resource pressure also needs another signal. Correlate syscall delays with scheduler events, block I/O, memory pressure, or network retransmissions. One event stream rarely proves causation by itself.

Finally, test the proposed fix under the same reproduction. The original failure should disappear, while successful behavior remains unchanged. Remove the eBPF trace and run the test again so the diagnosis does not depend on the diagnostic tool.

A probe that stays attached after the incident needs an owner, a cost budget, and a defined output consumer. Otherwise, detach it. Yesterday’s useful trace is tomorrow’s unexplained kernel workload.

FAQ

Can eBPF tracing miss events?

Yes. Ring buffers and perf buffers fill when userspace cannot drain records fast enough. Track reservation failures or lost-event counters instead of assuming every record made it across.

If you do not know how much output was lost, you cannot claim a complete event count. The surviving records may still prove that a failure occurred, but they cannot prove how often.

What happens to temporary map state when a process exits?

The kernel does not clean arbitrary map entries for you. State keyed by a thread identifier can remain after an unmatched entry event, especially if the process exits before the return hook fires.

Delete entries on every handled exit path, use bounded maps, and add userspace cleanup where needed. Ignore this on a long-running trace and stale entries eventually become a capacity problem.

Can an eBPF program read passwords or sensitive file data?

Yes, if the hook, helper, and program type allow the read. A buffer that contains an authentication token is still just a buffer to the tracing program, so treat trace output as privileged diagnostic data.

Capture only the fields the investigation requires. Leave request bodies, environment values, full command arguments, and tokens alone unless you have a specific need and controlled storage ready.

Will a kernel update break an existing tracing program?

Yes. Kprobes and hand-copied internal structures are the first places I look because kernel function names and layouts change.

Prefer tracepoints for stable event boundaries. For libbpf programs, use kernel type information and test loading against the kernels you deploy before treating the program as production tooling.

Why does a probe attach successfully but print nothing?

Assume the hook or filter is wrong before blaming eBPF. Confirm the workload reaches that hook with strace, list the tracepoints exposed by the host, and remove filters temporarily in a controlled test.

Then check namespace and process identity assumptions. A filter may use the process ID printed inside the container while the hook reports the host-visible task identifier.