I use eBPF shell detection to catch stealthy programs that land inside the kernel and then hide from normal tools. You will see why watching load-time events matters more than chasing artifacts later.

The kernel grants deep visibility into system activity, but that same access gives attackers a way to hide a reverse shell or backdoor inside programs. Traditional security tools running in user space can miss kernel-resident hooks or be fed forged information after compromise.

In my experience, the most reliable approach is to log high-fidelity data at program load and attachment time, then use cross-layer tools for verification. This guide focuses on practical steps you can run with bpftool, Tracee, Falco and memory inspection when needed.

Key Takeaways

• Prioritize catching load events and tracking suspicious hooks within linux kernel.
• Combine runtime tools and forensics to validate kernel-resident programs.
• Log verifiable artifacts you can check from outside the operating system.
• Use bpftool, Tracee, Falco and memory tools as complementary layers.
• Expect precise commands, fields to collect, and post-incident checklists.
• Plan for cloud and Kubernetes where namespace tricks can mask activity.

What eBPF Is and Why Shell Detection Matters on Linux

I explain how running small bytecode inside the linux kernel changes visibility and risk. You should watch load and attach events, not only artifacts left later.

From classic BPF to ebpf: kernel, system calls, and verifier basics

Classic bpf filtered packets. Modern ebpf adds many hook points and maps for state. Developers load code via the bpf() syscall. The verifier checks bounds and loops. The kernel then JIT compiles safe bytecode so programs run fast in kernel space.

Shell activity signals: exec, socket use, and file access patterns

Watch execve events, unexpected socket families, and access to /etc/shadow or SSH keys. Include PID, UID, container or cgroup tags, and parent lineage to reduce false positives.

• Limit noise by whitelisting known maintenance tools.
• Keep minimal per-event data: timestamp, program, args.
• Test filters against your kernel version and machine to tune alerts.

Threat Landscape: eBPF Backdoors and Kernel-Level Shells

Modern threats place small programs inside the kernel to bypass user-space controls and hide activity.

Two notable cases illustrate the risk. BPFDoor used BPF filters that listened for magic packets. When triggered, it opened a hidden shell while keeping ports invisible to scans.

Pamspy took a different route. It attached a uretprobe to pam_get_authtok. That hook captured credentials during logins without needing a visible process on the host.

Traditional antivirus looks at files and user processes. Programs running inside the linux kernel can bypass those checks. They can also alter outputs from bpftool and debugfs so live listings are unreliable.

Concrete indicators and actions

• Watch for setsockopt calls that install BPF filters and unexpected raw socket use.
• Look for unexplained kprobe/uprobes on authentication paths and deleted executables.
• Exclude known tools like tcpdump and dhclient to reduce alert noise.
• Prefer catching loads and attach events; if missed, use hypervisor or memory dumps for forensics.

Core Challenges in Detecting eBPF-Based Shell Activity

Kernel-resident programs can appear without files and evade common monitoring paths.

You cannot rely on file scans. Malicious code lives in memory and attaches to kernel events without a persistent file on disk. That breaks many endpoint assumptions fast.

Once code runs in kernel context, a rootkit can tamper with local tools and telemetry. Your own process listings, bpftool outputs, and logs may be poisoned after compromise.

• High event volume causes noise. Pick tight attach points and log only essential fields.
• Some techniques hide links or override return paths. Simple enumeration then fails.
• The verifier blocks many unsafe tricks, but attackers reuse allowed helpers and hooks. Technology controls alone are not enough.
• If you miss the load event, you face a window problem and must move to memory or hypervisor-based forensics.

Reduce exposure by shrinking who can load programs and by restricting CONFIG and capability flags. Still, prevention does not replace fast load-time logging and off-host collection for trusted views.

Build a small, stable log schema. Combine quick alerts with scheduled deep snapshots. This balances overhead while keeping enough data to investigate risk and apply practical techniques.

Real-Time Detection: Catching eBPF Program Loads and Attachments

Focus on the bpf() path and attach events to record who loaded a program and where it hooked into the kernel.

You must log minimal, verifiable facts at the instant a program loads. Capture timestamp, PID, UID, and the invoking command. Tie that to the process tree so you can map user intent to kernel activity.

What to watch

• Record the syscall entry (bpf()) and any link create events with link ID.
• Log attach type: kprobe, uprobe, tracepoint, perf_event, XDP, or TC.
• Include program type, map count, and helpers used to spot risky behavior.
• Capture target function names and binary paths for uprobes and kprobes.

Why load-time logs matter

After a program is resident, tools and listings may be forged. Rootkits can hide links and alter outputs. That makes later discovery unreliable.

Alert on CAP_BPF changes and unprivileged toggles. Test your pipeline by loading known benign samples and confirming Tracee bpf_attach and Falco bpf() alerts show correct fields.

eBPF shell detection: Tools, Programs, and Practical Workflows

Focus on reproducible steps you can run now: Tracee alerts, Falco policies, bpftool audits, and offline memory checks.

Tracee rules and quick commands

Enable Tracee rules for bpf_attach and bpf() loads. Tracee logs IDs, types, helper lists, and program names.

Example rule snippet:

- rule: BPF_Attach_Load
  condition: evt.type == bpf_attach or evt.type == bpf
  output: "bpf_attach by %{proc.cmdline} prog_id=%{bpf.id} helpers=%{bpf.helpers}"

Falco policies and scope

Add Falco rules for bpf() and perf_event_open. Limit scope to containers or host as needed. Tune suppressions for tcpdump, wireshark, and dhclient to reduce noise.

bpftool and on-host triage

Use these commands for a quick audit:

• bpftool prog show
• bpftool map show
• bpftool link show

Walk /sys/fs/bpf for pinned objects. Keep a daily export under git for simple diff-based management. A small script can flag deleted executables holding BPF sockets as a hint, not proof.

Memory forensics with Volatility

On compromised machines, use Volatility’s eBPF plugin and a matching kernel profile. It extracts programs via prog_idr and highlights suspicious helpers.

Hardening the Kernel to Reduce Shell Risk Within the Linux Kernel

A small change in kernel build options can remove many attack paths. I focus on settings you can apply now. Each item explains why it matters and how to verify it.

Privilege controls

Enable CONFIG_BPF_UNPRIV_DEFAULT_OFF at build time. This blocks unprivileged users from loading eBPF programs. It removes a common low-privilege vector.

Gate the SYS_bpf syscall to root and CAP_BPF. Add audit rules so every use is logged. Audit records tie actions to accounts for later review.

Reducing attack surface

Disable CONFIG_BPF_KPROBE_OVERRIDE. That helper can force return paths. Turning it off closes an abuse route.

Remove unused kprobe and tracepoint options when compiling the kernel. Fewer attach targets means fewer places for malicious programs to hide.

Keep your kernel version current. Patch verifier and helper bugs quickly. Pair hardening with monitoring and strict module policies. Document approved exceptions for observability so security and operations stay aligned.

Post-Incident Hunting: Finding Stealthy Shell Behavior After the Fact

Post-incident hunting is a methodical sweep of the kernel and user artifacts. Start with a trusted snapshot. Save outputs off-host so you can compare later.

Immediate steps

Run these exact commands and store results with timestamps. Keep copies outside the host for integrity.

1. bpftool prog show > /var/log/bpftool-prog-$(date -I).log
2. bpftool map show > /var/log/bpftool-map-$(date -I).log
3. bpftool link show > /var/log/bpftool-link-$(date -I).log
4. cat /sys/kernel/debug/kprobes/list > /var/log/kprobes-$(date -I).log

Inspect interfaces, pins, and perf owners

• ip link show — note XDP flags per interface.
• tc filter show dev $iface — list BPF TC filters on each device.
• Find pins: ls -l /sys/fs/bpf and record unknown paths.
• bpftool perf show — map PIDs to probes and correlate to ps and pstree output.

Workflows and diffs

Diff today’s files against yesterday’s to surface new program IDs, maps, or pins. For deleted-on-disk binaries, search /proc for open descriptors and sockets. Collect file hashes, command lines, and ancestry for any suspicious PID.

If host trust is low, export memory and run an offline analysis with Volatility and its ebpf plugin. Finish with a network sweep for raw sockets and odd listeners to confirm or rule out a hidden shell.

Case Studies: What Worked Against BPFDoor and Pamspy

This section shows exact hooks and IOCs that stopped two real attacks. I focus on the calls, probe names, and network signs you should log.

BPFDoor used setsockopt to install complex filters and hidden ports. Trend Micro linked that to raw sockets opened by processes whose binaries were deleted.

• Monitor setsockopt inserts for SO_ATTACH_FILTER and unexpected raw socket() calls.
• Flag processes with deleted executables that call network syscalls.
• Treat unusually long or obfuscated filter strings as suspicious compared to tcpdump or libpcap patterns.

Pamspy attached a uretprobe on pam_get_authtok inside libpam.so to harvest credentials. Tracee captured the probe name and offsets at attach time.

• Alert on uretprobe hooks targeting pam_get_authtok. Record offset, target path, and attaching PID.
• Correlate probe attach time with a spike in login events or unexpected child processes.
• Keep an allow list of approved auth function names; any new hook is high priority.

Keep example traces with sanitized args. Include process ancestry and the full command line for any alert. Share these cases with ops so small guardrails can block common attacker behavior.

Cloud and Virtualization: OpenStack and vSphere/NSX Detection Paths

Platform teams must bridge guest kernel visibility with network-level controls. In multi-tenant environments, you can often pull kernel state from many machines without interactive logins.

OpenStack host-driven inspection of guest VMs with bpftool

Use host-side commands to run a small scan inside guests. This helps when user access is blocked.

1. Run a quick script via the cloud CLI:

   openstack server ssh --login root $VM -- 'bpftool prog show; bpftool map show' > /var/log/bpftool-$VM-$(date -I).log

2. Check XDP and TC filters per interface:

   openstack server ssh $VM -- 'ip link; bpftool net; tc filter show dev eth0' > /var/log/bpf-iface-$VM.log

3. Save outputs per machine and version your scripts so you can diff states over time.

vSphere + NSX network signals: DFW, IDS/IPS, flows, and ATP intel

NSX spots odd flows and tunneling patterns such as hidden UDP or ICMP channels. It can block outbound ports at the DFW.

• Use IDS/IPS to flag unusual east-west communication and correlate with ATP feeds.
• Accept limits: NSX sees network events, not guest kernel tampering. Keep a runtime sensor inside guests for kernel-level context.
• Document control split: network teams run DFW/IDS, security teams manage in-guest scans and response handoffs.

Kubernetes Runtime: Detecting Reverse Shells and Privileged Containers Using eBPF

Detecting container escape requires tying process execs to container IDs and watching for host namespace joins. Watch runtime events on nodes and map them back to pods, namespaces, and service accounts. This gives context for quick response.

Signals of container escape

• Alert when a pod uses hostPID or hostNetwork with CAP_SYS_ADMIN or CAP_NET_RAW on nodes that should not allow it.
• Match commands like nsenter -t 1 -m -u -n -i -p to the container ID and service account for fast triage.
• Flag access to /etc/shadow or other host files after a namespace join as a high-priority event.

Runtime monitoring of sockets and RCE patterns

Use a runtime sensor to watch system calls for exec of bash or sh with a TTY over a non-standard socket. That is a strong sign of a reverse connection.

1. Log process, pod, node, namespace, and labels with each event.
2. Flag raw socket opens and outbound connects from unexpected namespaces.
3. Track program loads on nodes to spot attempts to hide at the kernel level.

Action Plan: Layered Strategies and Tools to Lower Risk Now

Action Plan: Focus on layered strategies you can test today to lower risk across kernel and cloud environments.

Start with visibility. Deploy Tracee or Falco to alert on bpf() loads and attachments. Route compact fields into your SIEM so events are usable.

Harden the kernel. Enable CONFIG_BPF_UNPRIV_DEFAULT_OFF, restrict CAP_BPF and disable the kprobe override. Add LKRG for integrity monitoring.

Schedule management tasks. Export bpftool prog/map/link daily, store in git, and diff for new program IDs. Test memory capture and the Volatility plugin in a lab.

Cover cloud and Kubernetes. Use host-run bpftool in OpenStack. Leverage NSX flows and block privileged pods at admission. Train owners with short runbooks and keep a small, consistent event schema. Review monthly and tune tools and processes to keep this plan simple and effective.

Related: Operationalizing Falco: eBPF Driver Setup and Alert Routing