Kernel debugging is evidence work, not a contest to add more printk calls. Start with dmesg or journalctl, record the exact failure, and build a reproducible trigger. Then classify it as a module load error, oops, panic, boot hang, lockup, or regression before adding tracing, gdb, kdump, or bisecting commits.

Last updated: 2026-07-31

How does linux kernel debugging actually work?

Kernel code runs with full access to memory, devices, interrupts, and scheduler state. A bad pointer can corrupt another subsystem long before the crash appears. As a result, the line that fails is often not the line that caused the damage.

The reliable method is to narrow the failure before changing code:

1. Capture the logs.
2. Record the kernel, module, hardware, and boot parameters.
3. Reproduce the failure with the smallest test.
4. Add targeted instrumentation.
5. Decode the stack trace.
6. Capture a crash dump if the machine dies.
7. Bisect the change if the bug appeared after an update.

A kernel module is only one part of this process. If you need the basics first, read this guide to Linux kernel modules before assuming every kernel failure belongs to your module.

Run this workflow before changing code

Do not reboot first. A reboot destroys useful evidence and can make a timing bug disappear until the next production outage.

1. Capture the current boot

On a systemd machine, start with the kernel journal:

sudo journalctl -k -b --no-pager
sudo journalctl -k -b -p warning..alert --no-pager

The first command shows kernel messages from the current boot. The second filters out routine output and leaves warnings, errors, and emergency messages.

You can also read the kernel ring buffer directly:

sudo dmesg -T | tail -100

The -T flag converts kernel timestamps into readable times. The final lines often contain the actual reason for a module failure, an oops, or a device timeout.

If the system is still running while you reproduce the bug, follow the log live:

sudo journalctl -k -f

That tells you which message appears first. Usually, the first relevant error matters more than the final panic message.

2. Record the system state

Save the details that affect the result:

uname -a
cat /proc/cmdline
cat /proc/sys/kernel/tainted
lsmod

uname identifies the running kernel. /proc/cmdline shows boot parameters that may change debugging or driver behavior. The taint value tells you whether the kernel has loaded an out-of-tree module, forced a module action, or hit another condition that makes the report less trustworthy.

Also record the module build information:

modinfo mymodule

Check the vermagic, source version, signer, parameters, and dependency list. If the module was built for a different kernel, stop debugging its runtime behavior and fix the build mismatch first.

3. Reduce the reproducer

Remove unrelated hardware, services, and workload from the test. Then run one command that triggers the failure and keep the input constant.

For a module, test the smallest sequence:

sudo modprobe mymodule
sudo journalctl -k -n 50 --no-pager
sudo modprobe -r mymodule

If loading fails, the log is your starting point. If unloading fails, look for open file descriptors, active work queues, timers, references, or a device still bound to the module.

Do not test an unstable module on the machine that stores your only copy of the data. A kernel bug does not respect your backup schedule.

Read the kernel logs before adding printk

printk still works, but scattered messages are a poor first tool. On a busy system, they bury the useful line and can change timing enough to hide a race.

Use the modern helpers that match the message:

pr_info("mymodule: loaded\n");
pr_err("mymodule: DMA setup failed: %d\n", ret);
pr_warn("mymodule: unexpected state: %u\n", state);

For device drivers, use device-aware helpers:

dev_err(dev, "DMA setup failed: %d\n", ret);
dev_dbg(dev, "queue depth: %u\n", depth);

The device helpers include the device name in the output. That matters when several identical devices are present.

Log state transitions and failure values, not every loop iteration. A useful message answers three questions:

• Which function ran?
• What state or input did it receive?
• What error caused the path to stop?

Avoid printing raw pointers unless you need them for a specific investigation. Use stable identifiers, return codes, device names, and indices first.

Turn on dynamic debug output

Dynamic debug lets you enable pr_debug() and dev_dbg() messages without rebuilding the module. First, check whether the control file exists:

sudo test -e /sys/kernel/debug/dynamic_debug/control && echo available

Enable messages from one source file:

echo 'file drivers/example/foo.c +p' | \
 sudo tee /sys/kernel/debug/dynamic_debug/control

Enable messages for one module:

echo 'module mymodule +p' | \
 sudo tee /sys/kernel/debug/dynamic_debug/control

Then reproduce the failure and read the output:

sudo journalctl -k -f

The output tells you whether execution reached the expected function. If it did, add the next message after the state change. This is faster than enabling debug output for the entire kernel and trying to find your line in the flood.

Turn it off when finished:

echo 'module mymodule -p' | \
 sudo tee /sys/kernel/debug/dynamic_debug/control

If the control file is missing, the running kernel may lack dynamic debug support or debugfs may not be mounted. Check that before changing source code.

Read an oops instead of staring at the stack

An oops is a kernel fault that may leave the system running. A panic stops the kernel or makes continued operation unsafe. Both need the first error message and the complete stack trace.

Start by finding the fault class:

• NULL pointer dereference points toward an invalid object or missing check.
• general protection fault often means a corrupted pointer or invalid address.
• Unable to handle kernel paging request means the kernel touched an unmapped address.
• BUG: sleeping function called from invalid context means code slept where it could not sleep.
• BUG: unable to handle page fault needs the faulting address, process context, and call trace.

Do not jump straight to the function at the top of the trace. That function may only be where corrupted state became visible.

Save the full report:

sudo journalctl -k -b --no-pager > kernel-current-boot.log
sudo dmesg -T > kernel-ring-buffer.log

If the stack contains addresses or offsets, resolve them against the exact unstripped kernel and module that produced the report. /boot/vmlinuz is often compressed and stripped, so it is not always enough.

For a kernel tree, use the supplied decoder when available:

./scripts/decode_stacktrace.sh \
 /path/to/vmlinux \
 /path/to/kernel-source \
 kernel-current-boot.log

For a module function offset, faddr2line can map an address back to a source line:

./scripts/faddr2line \
 /path/to/mymodule.ko \
 function_name+0xOFFSET/0xSIZE

The source line is useful only if the symbols match the running image. If you rebuilt the module after the crash, keep the old .ko, vmlinux, System.map, and build configuration. Otherwise, you may resolve the trace to a line that was never running.

Build with symbols before you need them

Debug symbols turn an address into a function and source location. Without them, kernel debugging becomes guesswork with better formatting.

For a custom kernel, enable debug information in the kernel configuration and keep the unstripped vmlinux file. This guide to compiling a kernel with debug symbols covers the build side.

Useful options include:

• CONFIG_DEBUG_INFO for symbol information.
• CONFIG_FRAME_POINTER for more reliable stack traces on supported architectures.
• CONFIG_KALLSYMS for readable symbols in kernel messages.
• CONFIG_KGDB when you need a source-level remote debugger.
• CONFIG_KASAN for memory errors.
• CONFIG_KCSAN for data races.
• CONFIG_KMEMLEAK for certain kernel memory leaks.
• CONFIG_LOCKDEP for lock dependency problems.

Do not enable every debugging feature on a production kernel. Sanitizers and lock tracking can change timing, memory use, and scheduling. Build a debug kernel for the failing workload, then confirm the fix on a normal configuration.

Capture crashes with kdump

If the machine reboots or freezes before you can read the screen, configure kdump before the next crash. It reserves memory for a small capture kernel, boots that kernel after the failure, and saves the crashed kernel’s state.

The package and service name depend on the distribution. First, check the service and crash kernel status:

systemctl status kdump
cat /sys/kernel/kexec_crash_loaded

A loaded crash kernel should report a ready state. The exact dump directory also varies, so check the service configuration and the files created after a test crash.

Use the crash utility with the matching debug vmlinux and the captured vmcore:

crash /path/to/vmlinux /path/to/vmcore

Inside crash, start with:

bt
ps
log
sys
mod

bt prints the current backtrace. ps shows tasks and their states. log reads the captured kernel messages. sys reports kernel and system details. mod lists loaded modules.

The matching vmlinux matters. A dump from one kernel build opened with another build can produce a confident-looking lie.

Use ramoops for early or persistent messages

ramoops stores selected kernel output in reserved memory so it survives a reboot. It helps when the failure happens during early boot or when the console disappears before kdump can run.

After reboot, check the persistent storage:

sudo ls -l /sys/fs/pstore
sudo cat /sys/fs/pstore/* 2>/dev/null

If the directory is empty, that does not prove the kernel produced no messages. The reserved memory, device-tree entry, kernel parameters, or pstore support may be missing.

Ramoops captures logs. It does not replace a full crash dump. Use it to preserve the last messages, then use the stack and source to find the fault.

Debug boot hangs before systemd starts

A boot problem can belong to the kernel, an initramfs, a device driver, or systemd. Treating all of them as “the boot is stuck” wastes time.

Add temporary kernel parameters from the bootloader:

ignore_loglevel initcall_debug

ignore_loglevel makes more kernel messages visible. initcall_debug reports which kernel initialization function is running. If the last message names a driver or subsystem, you have a useful boundary.

For early serial output, use the correct earlycon setting for the platform. A wrong console parameter produces silence, not evidence.

If the kernel finishes and userspace hangs, use systemd’s emergency target:

systemd.unit=emergency.target

That separates kernel initialization from service startup. Once you reach the emergency shell, inspect the failed unit:

journalctl -b -p err
systemctl --failed

A module load job can hold up boot while the kernel itself is healthy. If that is the symptom, use this guide to fix failed kernel module load jobs instead of rebuilding the kernel.

Find soft lockups, hard lockups, and deadlocks

A soft lockup means a CPU stopped scheduling other work for too long. A hard lockup means the CPU stopped responding to watchdog activity. Both can point to an infinite loop, an interrupt problem, a spinlock deadlock, or a driver that never returns.

Read the kernel messages from the affected boot:

sudo journalctl -k -b --no-pager | \
 grep -Ei 'lockup|blocked|watchdog|stall|hung|deadlock'

If the machine still responds, ask the kernel for blocked task information:

echo w | sudo tee /proc/sysrq-trigger

The output shows tasks waiting in uninterruptible states. If you need task stacks, use the SysRq task dump:

echo t | sudo tee /proc/sysrq-trigger

Run these only on a system where you understand the impact. They produce large logs and can add pressure to an already sick machine.

For a deadlock, inspect lock dependency reports and the functions holding or waiting on the lock. Do not “fix” it by adding arbitrary delays. A delay can hide the ordering bug while leaving the lock cycle intact.

Use kgdb and gdb when logs stop being enough

Treat gdb as a separate tool, not a drop-in replacement for printk. You cannot attach to a running kernel like you attach to an ordinary user-space process and expect safe breakpoints.

For source-level kernel debugging, use kgdb with a debug-enabled kernel and a separate machine or virtual machine. QEMU is the safer target because a breakpoint can stop the entire guest without taking down your workstation.

The general setup is:

1. Build the kernel with debug symbols and KGDB support.
2. Boot the debug kernel with the required KGDB console or network settings.
3. Connect gdb to the target.
4. Set a breakpoint in the suspect function.
5. Reproduce the failure.
6. Inspect registers, arguments, stack frames, and memory.

A remote session may look like this:

gdb /path/to/vmlinux

Then inside gdb:

target remote SERIAL_OR_NETWORK_TARGET
break suspect_function
continue
bt
info registers

Use this when you need to stop execution at a precise point. Do not start here for a failure that journalctl, dynamic debug, or a decoded oops can already explain. The heavy debugger is useful, but it is not a substitute for understanding the failure.

Use sanitizers for memory and race bugs

Some kernel bugs do not fail where they start. Sanitizers catch them closer to the cause.

Use KASAN for out-of-bounds accesses and use-after-free errors. Use KCSAN for data races. Use lock dependency checking for incorrect lock ordering. Use kmemleak when memory appears to remain allocated without a reachable reference.

The diagnostic output names the access type, allocation or free path, and relevant stack. That is far more useful than printing a pointer after the object has already been freed.

Run these tools in a test kernel. They add overhead and can change scheduling, which matters for race bugs. If enabling a sanitizer makes the bug disappear, keep the original reproducer and compare both configurations.

Bisect a regression instead of guessing

If the bug worked with one kernel and fails with another, use git bisect. Do not read hundreds of commit messages and pick the one whose subject sounds suspicious. Test each candidate with the same build and the same reproducer.

From the kernel source tree:

git bisect start
git bisect bad
git bisect good <known-good-commit>

Build and test the selected commit. Then mark the result:

git bisect good

or:

git bisect bad

Repeat until Git identifies the first bad commit.

The test must be deterministic enough to classify. If the failure is intermittent, improve the reproducer first. A noisy test can send the bisect toward the wrong change and waste the same time more efficiently.

When the bisect ends, inspect the commit and its parent:

git show <suspect-commit>
git diff <suspect-commit>^ <suspect-commit>

Then test a revert or a targeted fix. The first bad commit gives you a lead. You still need to prove that it caused the failure, because it may have exposed an older bug in a different subsystem.

Report kernel issues with useful evidence

A kernel bug report should let another person reproduce the failure without interviewing you for missing details.

Include:

• The exact kernel release and build configuration. Do not make the maintainer reconstruct your build from guesses.
• Hardware and virtualization details. A driver failure without the device context is only half a report.
• The full reproduction command or test case. “It crashes under load” gives nobody a test to run.
• The complete oops, panic, or lockup report. The quiet lines before the crash often carry the real clue.
• The output of /proc/sys/kernel/tainted. This tells readers how much of the result comes from code outside the main kernel tree.
• Relevant module information from modinfo. Include the build identity, parameters, and dependencies.
• The first known good and first known bad kernel. That gives a regression investigation a starting boundary.
• Any recent firmware, driver, or configuration change. Timing matters here, even when the change looks unrelated.
• Whether the bug still occurs with third-party modules removed. Otherwise, the upstream kernel may be blamed for someone else’s code.

Keep the original log formatting. Do not paste only the line that looks dramatic. The lines before the oops often show the first failed allocation, timeout, or device state change.

If the report concerns a module, include the source revision and build command. If it concerns a subsystem in the main kernel tree, identify the suspected maintainer from the source tree and send the smallest reproducer you have.

Common kernel debugging mistakes

The same wrong turns appear in almost every failed investigation.

• Rebooting before saving logs. That throws away the best evidence. Save the current journal and ring buffer first.
• Adding printk everywhere. It changes timing and floods the ring buffer. Instrument the state transitions that separate one path from another.
• Using a mismatched vmlinux. The decoded stack can point to the wrong source line. Keep the exact build artifacts with the report.
• Testing on production hardware. A kernel crash can corrupt more than the test process. Use a spare machine or virtual machine when you can.
• Assuming the last stack frame caused the bug. The corruption may have happened earlier. Trace the bad state backward.
• Ignoring tainted state. An out-of-tree module can invalidate conclusions about an upstream kernel.
• Using chmod, delays, or retries as a fix. These can hide the failure without removing its cause.
• Bisecting with an unreliable test. Git will still produce an answer. It just may be the wrong one.
• Capturing only dmesg | tail. The useful warning may be older than the final panic.
• Stripping symbols from the only debug build. Keep the exact build artifacts beside the crash report.

Good Linux kernel debugging means collecting the smallest set of evidence that separates one cause from the next. More output is not automatically better output.

FAQ

How do I debug a Linux kernel crash after an automatic reboot?

Set up kdump before the crash, then inspect the saved vmcore with the matching vmlinux. If a full dump is not available, check /sys/fs/pstore for ramoops output and compare its final messages with the previous boot’s journal.

Can I debug a kernel module without rebuilding the whole kernel?

Yes. Build the module with symbols, use modinfo to verify it matches the running kernel, and enable targeted dev_dbg() or pr_debug() output through dynamic debug. Rebuild the whole kernel only when you need kernel-wide options such as KASAN, KGDB, or special tracing support.

Why does dmesg show fewer messages than expected?

Fewer messages usually mean the command is restricted, the ring buffer has overwritten older entries, or the output is stored in the systemd journal instead. Compare sudo dmesg -T with sudo journalctl -k -b, then check the previous boot with journalctl -k -b -1.

Should I use gdb or crash for kernel debugging?

Use crash to inspect a saved crash dump. Use KGDB with gdb when you need to stop a live debug kernel, set breakpoints, and inspect execution before the crash. For most first investigations, logs and a decoded stack give you the answer faster.

How can I tell if a kernel bug is caused by a recent update?

Boot the previous kernel and reproduce the same test. If the older kernel remains stable, preserve both logs and use git bisect between the known-good and failing source revisions. That turns “the update broke it” into a commit you can inspect and test.

Related: Kernel Panic Explained: Causes, Clues, and Fixes