Linux Kernel Panic: Find the Real Cause, Not the Reboot
Don't reboot and hope. A Linux kernel panic is a fatal, unrecoverable error where the kernel halts the whole machine on purpose to stop data corruption, and the only real fix is finding the exact cause from the panic message, the kernel logs, and hardware diagnostics. Generic patches and "reinstall the kernel" advice paper over the failure. Read the top line of the panic first, because that one line usually names the subsystem that died.
Last updated: 2026-07-21
What a kernel panic actually means
The kernel hit a state it cannot trust, so it stopped rather than keep running and scribble garbage to your disks. That is the whole idea. A panic is the last-resort safety switch, not a crash the system can shrug off.
This is different from an application crash. When a userspace program dies, the kernel kills that one process, frees its memory, and the machine keeps humming. A panic happens one layer down, in the code that manages every process, so there is nothing left to catch it.
There is also a middle state worth knowing: the soft lockup. One CPU core gets stuck spinning in the kernel while the rest of the box keeps working. You see soft lockup - CPU#2 stuck in the log and sluggish performance, not a dead screen. It is a warning, and it often precedes the real panic.
A "fatal machine check" panic is its own category. That message means the CPU's own error-reporting hardware flagged a fault it could not correct: bad memory, a failing cache line, or an overheating core. When you see machine check in the panic, stop looking at software. The hardware just told you where to look.
What causes a linux kernel panic?
Two buckets cover most of what I see: hardware that lies to the kernel, and a boot path that got broken by an update. Everything else is a variation on those.
Hardware faults sit at the top. Bad RAM flips a bit the kernel was counting on, an overheating CPU throws a machine check, and a dying disk returns corruption instead of data. These trigger panics on bare metal and inside virtual machines alike, because the physical layer underneath the guest is still real.
Drivers are the other frequent offender. After a kernel update, DKMS sometimes fails to rebuild an out-of-tree module, and now you are running a driver compiled against the wrong kernel. Load it and the machine goes down. Firmware and microcode bugs behave the same way, feeding the kernel bad answers from the CPU or BIOS.
The rest cluster around boot:
- A corrupt or missing
initramfs, so the kernel cannot mount the root filesystem. - A wrong root UUID in the GRUB config or
/etc/fstab, pointing at a device that no longer exists. - Filesystem corruption that forces the root device read-only, so the kernel panics rather than write to a device it cannot trust.
- Secure Boot rejecting an unsigned module during early boot.
Name the bucket before you touch anything. That choice decides whether you reach for memtest or update-initramfs.
How do you read a panic message without getting lost?
Read the top line, not the stack trace. Unable to handle kernel NULL pointer dereference and VFS: Unable to mount root fs are completely different failures and want completely different fixes. People stare at the call trace and miss the one line that already named the problem.
Here is what the fields actually tell you:
| Field in the panic | What it means | What to do with it |
|---|---|---|
Top line (not syncing: …) | The kernel's reason for giving up | Start here; it names the class of failure |
RIP: address | The instruction the CPU was on when it died | Maps to a function once you have symbols |
| Call trace | The chain of functions that led there | Read the module names, not every frame |
Tainted: flags | Non-standard or proprietary modules were loaded | A tainted kernel means suspect that module first |
Machine check fields | The CPU's error hardware fired | Treat as hardware until proven otherwise |
The taint flag is the one people skip. If the panic says the kernel was tainted, an out-of-tree module was almost certainly loaded, and that module is your first suspect. A clean, untainted panic pushes you back toward the mainline kernel or hardware instead.
The call trace tells you which module was on the CPU when it died. That is your lead, not your verdict. A driver can appear in the trace because it was the victim of memory another module corrupted. Confirm it, do not convict on the trace alone.
Where the logs go after the screen clears
By default they are gone, which is the first thing to fix. Most distros keep the systemd journal in RAM, so a reboot wipes the evidence you need. Turn on persistent logging before the next panic, not after.
Edit /etc/systemd/journald.conf and set Storage=persistent, then systemctl restart systemd-journald. Now the journal survives reboots. After a crash, read the previous boot with:
journalctl -k -b -1
The -k gives you kernel messages only, and -b -1 reads the boot before this one, which is exactly the boot that died. That alone recovers most panic messages people think they lost.
For a full capture, configure kdump. It boots a tiny second kernel after a panic and writes a complete memory image, the vmcore, to /var/crash/. You reserve memory for it with a crashkernel= boot parameter. Then open the dump in the crash utility with the matching vmlinux symbol file and run bt for the backtrace at the exact moment of death.
If the panic happens too early for any disk to be up, kdump cannot save it. That is where a serial console or netconsole earns its place: the kernel streams its final messages over the serial line or the network to another machine before it dies. On a remote server with no monitor, this is often the only way to see the panic at all.
How do you prove hardware is the cause?
Test the hardware before you blame the software, or you will spend the night chasing a ghost in a driver that was innocent. When a panic recurs at random with no pattern tied to a kernel update, hardware is the first thing I rule out.
Start with memory. Boot memtest86+ from the GRUB menu or a USB stick and let it run for hours, ideally overnight. A single failing bit is enough to panic a kernel, and a five-minute pass proves nothing. If memtest finds errors, you are done: replace the stick.
Check the disks next with SMART data using smartmontools. Run smartctl -a /dev/sda and read the reallocated sector and pending sector counts. Numbers climbing above zero mean a drive that is failing and can hand the kernel corruption. Kick off a self-test with smartctl -t long /dev/sda and check back later.
For machine-check panics, install rasdaemon (or the older mcelog) and read the decoded errors. It translates the raw machine-check event into plain text: which memory bank, which CPU, corrected or fatal. That tells you whether the CPU is quietly correcting errors now and heading for a fatal one soon.
Finally, stress the CPU under load and heat with a tool like stress-ng while you watch temperatures with sensors. If the panic reappears only when the core gets hot, you have found an overheating or marginal chip, not a kernel bug.
Fixing a panic caused by a broken module or driver
Isolate the module first, then decide whether to rebuild it or blacklist it. If a panic started right after a kernel update, the odds are a DKMS module that did not rebuild cleanly against the new kernel.
Boot the previous kernel from the GRUB menu to confirm. If the old kernel is stable and the new one panics, you are hunting a regression or a stale module, not hardware. Then rebuild the module against the running kernel:
dkms status
dkms autoinstall
dkms status shows you which modules are built for which kernels. A module listed as built for the old kernel but missing for the new one is your culprit. A rebuild needs the matching kernel headers and toolchain installed, so check those first; a header mismatch is the usual reason the build failed silently. Our guide on fixing failed kernel module loading walks through the build errors in detail.
Secure Boot adds a wall here. It blocks unsigned modules, and in some hardware and driver setups it causes a boot-time panic before you ever reach a login prompt. Your freshly rebuilt out-of-tree module is unsigned unless you sign it with your own key, so either enroll a Machine Owner Key and sign the module, or turn Secure Boot off while you diagnose.
When you just need the machine up now, blacklist the offending driver. Append module_blacklist=drivername to the kernel command line at the GRUB prompt, boot without it, and confirm the panic is gone. That proves which module was at fault before you commit to a fix. To go deeper, our notes on debugging a kernel module cover the tooling.
Recovering a machine that will not boot
Get a shell before you get creative. Do not reinstall a system that has one wrong line in a config file. At the GRUB menu, pick an older kernel or the recovery entry, and you are usually back in.
If the current kernel is the problem, boot the last known-good one and stay there while you fix the new one. Rolling back cleanly is its own skill; our walkthrough on reverting a kernel update covers doing it without breaking the bootloader.
When GRUB itself cannot hand you a working kernel, boot a live USB and chroot into the broken install:
mount /dev/sda2 /mnt
mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys
chroot /mnt
From inside the chroot you can fix the real cause. Rebuild a broken initramfs with update-initramfs -u (or dracut -f on Fedora-family systems). Correct a wrong root UUID in /etc/fstab and the GRUB config. If a filesystem is corrupt, run fsck on the unmounted device; fsck checks and repairs filesystem inconsistencies, and repairing them is what stops the read-only-then-panic loop from recurring. Our boot error troubleshooting guide has the full recovery sequence.
Prove the fix by rebooting into the repaired system, not by assuming. A panic that vanishes without a known cause is a panic that comes back at the worst time.
Keeping panics from coming back
The durable fix is understanding the failure, but a few settings buy you time and evidence for next time. Configure kdump now, while the machine is healthy, so the next panic writes a vmcore instead of scrolling off a screen nobody was watching.
Tune what the kernel does on panic with the kernel.panic sysctl. Setting kernel.panic=10 in /etc/sysctl.conf reboots the box ten seconds after a panic, which keeps a remote server available while you still capture the dump. On a machine you can walk up to, I leave it at zero so the message stays on screen.
Keep DKMS modules current with each kernel so an update never leaves you running a driver built for a kernel you no longer boot. If your out-of-tree modules keep going missing, the missing module tree error guide covers why.
On hardware where uptime matters, ECC memory is the real safeguard. It corrects the single-bit errors that would otherwise flip a value under the kernel and take the whole box down, and it reports the corrected errors so you replace the stick before it fails hard. Consumer RAM cannot do either.
FAQ
What is the very first command to run after a kernel panic reboot?
Run journalctl -k -b -1 to read the kernel messages from the boot that crashed. If it comes back empty, your journal is not persistent, so set Storage=persistent in journald.conf and you will have the evidence next time. Read the top not syncing: line before anything else.
Does a kernel panic damage my hardware or data?
No, the panic is the kernel protecting your data, not harming it. It halts precisely so it does not write corrupted memory to disk. The underlying fault, like failing RAM or an overheating CPU, can cause damage over time, but the panic itself is the safe response to that fault.
What does "fatal machine check" mean specifically?
The CPU's built-in error hardware detected a fault it could not correct and reported it to the kernel, which then panicked. It almost always points to physical hardware: a bad memory bank, a marginal CPU, or overheating. Decode the raw event with rasdaemon to see which component fired, then test that part.
Can a kernel panic happen inside a virtual machine?
Yes, and the cause is often the physical host underneath the guest. Bad RAM or a failing disk on the hypervisor surfaces as a panic inside the VM. Guest-side settings like panic_on_oom can also halt the machine under memory pressure, so check those before assuming the host is at fault.
How long should I run memtest before trusting the result?
Run it for several hours or overnight, not one quick pass. Marginal memory errors show up only after many passes cover every address under sustained load. A short clean run tells you almost nothing; a full night of clean passes is what lets you rule memory out and move on to the disks and CPU.
