Linux server running shell diagnostics, with monitoring equipment and network cables in a home lab
eBPF Use Cases
William  

Probe Shell Errors: Trace the Real Process

A probe-shell failure is not fixed by toggling random shell settings. First identify the process, account, executable path, working directory, environment, and returned error. Enable shell integration only in the terminal and shell running the probe. Otherwise, you inspect one session while another process fails.

Last updated: 2026-08-11

What does “probe shell” mean in this diagnostic context?

A probe shell is the shell process that a discovery tool, automation agent, terminal, or monitoring service starts to run commands. The shell might be Bash, Dash, Zsh, PowerShell, or a restricted wrapper supplied by the target system.

The shell you use over Secure Shell (SSH) may not be the shell used by the probe. A service can run under another account, inside a container, or with a nearly empty environment. That split causes most “no shell detected” errors.

The phrase also appears in appliance parts catalogs. There it means a physical temperature probe, sometimes with a double shell, threaded connection, or replacement bulb. That hardware has nothing to do with command execution and belongs on a separate product page.

How do you prove which shell the probe is actually running?

Run a small diagnostic payload through the probe itself. Do not run it in your own terminal and assume the result carries over.

id
printf 'argv0=%s\n' "${0}"
printf 'SHELL=%s\n' "${SHELL-<unset>}"
ps -p "$$" -o pid=,ppid=,user=,args=
readlink /proc/$$/exe
pwd
printf 'PATH=%s\n' "$PATH"
env | sort

Each line settles a different part of the argument:

  • id shows the real user and groups.
  • The argument-zero value often reveals login or non-login mode.
  • ps shows the command line used to start the shell.
  • /proc/$$/exe identifies the executable actually running on Linux.
  • pwd exposes a missing or inaccessible working directory.
  • PATH shows where command lookup will occur.
  • env records the environment the probe received.

Do not treat $SHELL as proof. It usually records the account’s preferred login shell, not the current process. A tool can start /bin/sh while $SHELL still says /bin/bash.

Next, ask the active shell for its version. Use the command that matches the executable you found.

bash --version
zsh --version
dash -V
pwsh --version

Check the probe log beside this output. You want the configured path, supplied parameters, exit code, and standard error from the same run. Anything else is a comparison against the wrong process.

Why does a probe report that no shell was detected?

“No shell detected” usually means the probe could not start the configured executable or could not recognize its output. It does not prove that the host lacks a shell.

The usual failures leave evidence you can check:

  • The configured binary does not exist on that host.
  • The file exists but the probe account cannot execute it.
  • The tool supports Bash or PowerShell but receives a custom wrapper.
  • The shell starts non-interactively and skips the expected startup file.
  • The service has a stripped PATH, so helper commands cannot be found.
  • Quoting changes while passing through Java, SSH, sudo, or another shell.
  • A required home or working directory does not exist.
  • The probe connects to the wrong host, container, namespace, or account.
  • Startup output corrupts the probe’s machine-readable response.

That last case wastes time. A .bashrc file that prints a banner can break a tool expecting clean output. Interactive terminals tolerate decoration. Parsers do not.

Run the exact configured path without relying on PATH:

/bin/bash --noprofile --norc -c 'printf "shell-ok\n"; command -v uname; uname'
printf 'exit=%s\n' "$?"

If this works under your account, repeat it under the probe account. If it fails there, shell detection is doing its job. The environment is broken.

Logs and traces expose the real shell failure

Administrator tracing shell processes beside a Linux server using diagnostic tools without visible screens

Start with the application or agent log. Find the attempted executable, arguments, target account, standard error, and exit code. A generic status such as “probe failed” is the wrapper’s opinion, not the cause.

For a systemd service, inspect the unit and its journal:

systemctl show <agent-unit> \
 -p User \
 -p Group \
 -p Environment \
 -p EnvironmentFiles \
 -p WorkingDirectory \
 -p ExecStart

journalctl -u <agent-unit> -b

That tells you which identity and environment systemd supplied. It also catches failures before the shell starts, including a missing working directory or denied executable.

If the log hides the command, trace process creation. Attach strace to the agent, reproduce the failure once, and read the resulting files.

sudo strace -ff \
 -o /tmp/probe-shell.exec \
 -e trace=process,execve \
 -p <agent-pid>

Search the trace for execve. The returned error matters:

  • ENOENT means the path or interpreter is missing.
  • EACCES means execution was denied.
  • ENOEXEC points to an invalid executable format or broken script header.
  • A successful execve followed by a nonzero exit moves the fault inside the shell or command.

Then check operating system security messages:

sudo ausearch -m AVC,USER_AVC -ts recent
dmesg | grep -i denied
journalctl -k -b

SELinux, AppArmor, and the kernel ring buffer often name the denied path. A chmod 777 does not repair a policy denial. It only adds another problem.

Where eBPF fits into shell detection

Kernel-level shell monitoring represented by Linux server instrumentation and connected tracing hardware

Extended Berkeley Packet Filter (eBPF) tools help when you need to observe process execution without changing the target application. They can record execve events, parent lineage, user IDs, container context, socket use, and kernel hook attachment.

For process-level work, start with tracing system calls with eBPF rather than writing a large detector first. Use bpftool, Tracee, Falco, and memory inspection only where the evidence calls for them.

Falco’s official run_shell_in_container rule uses this condition to identify a shell spawned inside a container:

container and proc.name = bash and spawned_process and proc.pname exists and not proc.pname in (bash, docker)

The rule is useful because it includes process ancestry. A process named bash alone tells you almost nothing. Read the Falco rule documentation before changing the condition, because broad exclusions turn a detector into decoration.

For kernel-level threats, process tracing is only one layer. BPFDoor used BPF filters that listened for trigger packets. When triggered, it opened a hidden shell while keeping ports invisible to scans.

Pamspy took a different route. It attached a uretprobe to pam_get_authtok. That hook captured credentials during logins without needing a visible process on the host.

Watch eBPF program load and attachment events before chasing artifacts later. Record the process ID, user ID, command, parent process, hook type, and target. If local listings may be compromised, use off-host logs or memory inspection. The live machine does not get the final vote after kernel tampering.

For a deployed monitor, Falco alert routing and driver setup covers the operational side. Use bpftool for on-host inventory, Tracee for runtime events, and Falco for policy alerts. They answer different questions.

Enable shell integration after startup works

Enable shell integration only after the shell starts cleanly. Integration can improve command boundaries, working-directory tracking, and exit-status detection. It cannot repair a missing executable or a denied execve.

Use this order:

  1. Confirm the terminal and active shell.
    Check the terminal application, the argument-zero value, /proc/$$/exe, and the shell version. Do not configure a Bash hook while the terminal launches Zsh.

  2. Enable the terminal’s supported integration hook.
    Use the terminal’s own setting or documented startup snippet. Do not source an unknown script copied from a forum into every shell session.

  3. Restart the terminal session.
    Reloading one startup file may leave old hooks in memory. Start a new shell and inspect it from scratch.

  4. Verify the injected state.
    Compare the environment before and after integration. Depending on the terminal, you may see an integration variable, prompt hook, precmd function, or control sequence around commands.

  5. Test command boundaries and exit status.
    Run a successful command, a missing command, and a command that returns failure.

printf 'integration-test\n'
command -v sh
command-that-does-not-exist
false
printf 'last-exit=%s\n' "$?"
  1. Repeat the original detection test.
    Check whether the tool now identifies the command, working directory, and exit status correctly.

Shell integration belongs to interactive terminal sessions. A background agent launched by systemd usually has no terminal, prompt, or integration hook. Enabling integration in your desktop terminal will not change that service. Linux remains stubbornly literal about process boundaries.

Unsupported shell configurations have specific causes

An unsupported shell configuration usually has the wrong executable, wrong parameter shape, or output the probe cannot parse. Changing the target’s login shell should be the last move.

Validate the configured executable first:

test -f /path/to/configured-shell
test -x /path/to/configured-shell
file /path/to/configured-shell
sed -n '1p' /path/to/configured-shell

A wrapper script needs a valid interpreter line. If its first line points to a missing interpreter, executing the wrapper returns a path error even though the wrapper itself exists.

Next, inspect how arguments are stored. These are not equivalent:

shell=/bin/bash
arguments=-lc
shell=/bin/bash -lc

Some tools expect an executable field and a separate argument list. Others pass one string to another shell for parsing. If you put flags inside the executable field, the operating system looks for a file whose name includes those flags.

Restricted shells can also block directory changes, command lookup, redirection, or external programs. Custom paths under a user’s home directory may disappear for service accounts or during early boot.

Finally, check version-specific behavior against the tool’s own support list. Do not replace a production shell because one probe parser lags behind. Correct the parser configuration first.

Permissions, sudo, and non-interactive sessions change the result

Linux server access troubleshooting with separated user sessions and secured hardware connections

An interactive SSH test proves what your account can do in a terminal. A service probe may use another user, another home directory, no terminal, and no login startup files.

Reproduce the probe identity directly:

sudo -u <probe-account> -H /bin/bash --noprofile --norc -c '
id
pwd
printf "HOME=%s\nPATH=%s\n" "$HOME" "$PATH"
command -v <required-command>
'

That tells you whether the account can start the shell and find the required command without your profile files. If this fails, adding aliases to your own .bashrc is wasted motion.

Now test privilege escalation without allowing a password prompt:

sudo -u <probe-account> -H sudo -n <required-command>
printf 'exit=%s\n' "$?"

A failure here often means the sudoers policy expects authentication or a terminal. Background probes cannot answer prompts. Fix the narrow command rule or remove the unnecessary privilege step.

Also inspect every parent directory in the executable path:

namei -l /full/path/to/configured-shell

One missing execute bit on a parent directory blocks access. Reading the final file’s mode over and over will not show it.

Startup files differ by mode and shell. Login, interactive, and non-interactive Bash sessions do not read the same files. Zsh and PowerShell have their own rules. Put required service environment in the systemd unit, agent configuration, or deployment environment instead of relying on an interactive profile.

Isolate detection failures from execution failures

Test one boundary at a time. Start with shell discovery and move down the table. If you run the full job after every change, you learn only that something still failed.

StageTestWhat failure means
Shell discoveryResolve and inspect the configured executableWrong path, unsupported shell, or missing interpreter
Shell startupRun a fixed printf commandPermission, startup, or invocation failure
Integration signalingInspect hooks, variables, and command markersTerminal integration was not loaded or is unsupported
Command lookupRun command -v <name>The probe’s PATH cannot find the command
QuotingPrint arguments exactly as receivedAn intermediate parser changed spaces, quotes, or expansion
ExecutionRun the command by absolute pathThe command itself failed, independent of lookup
Output captureWrite known text to standard output and standard errorThe transport or parser dropped a stream
Exit handlingReturn success and failure deliberatelyThe wrapper ignores or rewrites the exit code

For quoting, use a payload that shows each argument separately:

/bin/sh -c '
for arg do
 printf "<%s>\n" "$arg"
done
' probe-shell-test "two words" 'literal $HOME' '*.log'

Check the output against what you supplied. If two words becomes separate arguments or $HOME expands too early, the shell itself may be fine. The quoting layer before it is broken.

Then test output streams:

printf 'stdout-marker\n'
printf 'stderr-marker\n' >&2
exit 1

If the log records the text but reports success, exit-status handling is wrong. If it records neither marker, the command may never have executed.

This ladder keeps detection, invocation, command lookup, and output parsing separate. That makes the failure reproducible, which is the part that matters.

Repair the probe before changing the target shell

Repair the probe configuration when the target shell works under the probe account but the tool invokes it incorrectly. The usual repairs are a corrected executable path, separate argument fields, a valid working directory, or an explicit service environment.

Repair the agent deployment when different nodes have different binaries, users, home directories, or startup files. For example, one node may have /bin/bash while another points the agent at a wrapper under a missing home directory. The shell error is only the first place that drift becomes visible.

Change the target shell only when the logs prove the installed shell is unsupported, damaged, or blocked by policy. Before touching production, capture these facts:

  • The exact execve call and returned error
  • The probe account and groups
  • The executable’s resolved path and interpreter
  • The working directory
  • The effective PATH
  • Standard output, standard error, and exit code
  • The tool’s documented shell support

Do not change /bin/sh system-wide to satisfy one probe. System scripts may depend on the existing implementation. Point the tool at the shell it supports instead.

Likewise, do not add broad permissions because execution was denied. Find the denied file, directory, mount option, or security policy. Fix the one boundary the logs named.

Appliance probe hardware belongs on a separate page

Put appliance probe hardware on a separate page. Someone looking for a double-shell temperature probe, bulb diameter, bulb length, or half-inch female connection is trying to identify a physical replacement part.

That page should cover the appliance model, connector type, probe dimensions, temperature range, and verified Electrolux or Zanussi compatibility. Those details do not help an engineer debug a shell process, and shell commands do not help someone order a commercial replacement probe.

Mixing both meanings leaves each reader with half a page they cannot use. Keep software diagnostics here and hardware purchasing details with the appliance part.

FAQ

Can a probe use /bin/sh even when the account’s login shell is Bash?

Yes. The login shell entry does not control every process the account starts. A probe can call /bin/sh, use a wrapper, or execute a script whose interpreter line selects it. Check /proc/<pid>/exe or the execve trace. The running executable settles it.

What does a command-not-found error mean during a shell probe?

It means the shell probably started and command lookup failed. Read the service’s PATH, then run the command by absolute path. If the absolute path works, replacing the shell is the wrong repair. Put the required directory in the probe’s environment.

Should a probe load .bashrc to get the right environment?

Usually not. A .bashrc may set aliases, draw a prompt, check for a terminal, or print a banner into output the probe expects to parse. Put required variables in the agent configuration or systemd unit. Service startup should not depend on someone’s interactive prompt file.

Can a container hide which shell the host probe is seeing?

Yes. A host probe can watch one process namespace while the command runs in another. Record the namespace, container identifier, parent process, and executable path from the same event. Otherwise, you can prove the right shell exists and still be looking in the wrong container.

Is strace enough for every shell-detection failure?

No. Use it for process creation, file access, signals, and returned errors. However, a compromised kernel can lie to local tracing tools. For suspected kernel tampering, pair load-time eBPF telemetry with off-host logs, bpftool inventory, and memory inspection.

Related on this blog