Start with the manual kernel config, then let Buildroot reproduce it. For a BR2_ENABLE_DEBUG debug-symbols workflow, enable BR2_ENABLE_DEBUG for packages, CONFIG_DEBUG_INFO and CONFIG_GDB_SCRIPTS in the kernel, leave CONFIG_DEBUG_INFO_REDUCED off, and keep the matching vmlinux beside the image. BR2_ENABLE_DEBUG alone will not produce kernel symbols.

Last updated: 2026-07-31

You want clear backtraces and fast troubleshooting, so I show how to compile the kernel with debug symbols and turn cryptic traces into actionable clues. I’ve guided many engineers through building a linux kernel that keeps rich DWARF info, frame pointers, and module matches – so GDB and SystemTap find meaningful names instead of hex addresses. We’ll cover exact config flags, two build paths, and a quick QEMU/KVM loop to boot and attach a debugger.

By the end you’ll have a repeatable workflow to produce vmlinux, load module data at runtime, and verify your running Linux image matches its debug info – so troubleshooting feels predictable, not painful.

Key takeaways

• Start with CONFIG_DEBUG_INFO, CONFIG_GDB_SCRIPTS, and frame pointers where the architecture supports them.
• Leave CONFIG_DEBUG_INFO_REDUCED off when you need complete types and useful source-level traces.
• Treat BR2_ENABLE_DEBUG as package-level support. Configure kernel symbols separately.
• Keep the exact vmlinux, modules, kernel configuration, and root filesystem from the same build.
• Use direct QEMU with -s -S when you need a GDB loop you can account for.
• Load module symbols with lx-symbols, inspect loaded modules with lx-lsmod, and read kernel messages with lx-dmesg.

Why kernel debug symbols matter for Linux kernel debugging today

When addresses become names and lines, your troubleshooting shifts from guessing to solving. Enabling full symbol data turns raw addresses into function names, file and line information, and readable variables.

That change makes every backtrace meaningful. I recommend building with CONFIG_GDB_SCRIPTS enabled and leaving CONFIG_DEBUG_INFO_REDUCED off. When supported, enable CONFIG_FRAME_POINTER – it produces far clearer backtraces across architectures.

GDB can load the kernel helpers generated by the Linux kernel GDB debugging guide. Those helpers provide commands such as lx-symbols, lx-lsmod, and lx-dmesg.

Keep a symbolized vmlinux for every debug build as a default practice. It costs disk space, but saves hours during live triage. Tools such as SystemTap, perf, and BPF workflows also rely on precise mappings between addresses, source, and types.

Kernel symbols are not the same as package debug symbols. The kernel needs its own CONFIG_DEBUG_INFO build, while user-space programs such as glibc need matching user-space debuginfo.

How do you compile the Linux kernel with debug symbols manually?

Use a separate build directory for debug work. Do not mix a stripped production tree with a symbolized development build and then wonder which vmlinux GDB opened.

Install a suitable GCC or Clang toolchain, make, GDB, and the normal kernel build dependencies. If you target another architecture, use the matching cross-toolchain and cross-GDB. A host GDB pointed at the wrong architecture produces confusing failures that look like bad symbols.

If you need a refresher on the configuration flow, see how make menuconfig configures the Linux kernel.

Enable the right options

Open the kernel configuration with make menuconfig, then enable the debugging options that match the job.

Confirm the important values before building.

grep -E 'CONFIG_(DEBUG_KERNEL|DEBUG_INFO|DEBUG_INFO_REDUCED|GDB_SCRIPTS|FRAME_POINTER)=' .config

The output should match the debugging job. In particular, check that CONFIG_DEBUG_INFO_REDUCED is not enabled when you need full types.

Build the GDB helpers before you attach

On kernels v5.1 and above, run make scripts_gdb from the kernel source tree. This step is required for the kernel GDB helper workflow.

make scripts_gdb
find . -name vmlinux-gdb.py -print

The build produces vmlinux-gdb.py. Let find show the path in your tree instead of guessing it. Then add that exact path to GDB's auto-load allowlist.

printf '%s\n' \
 'add-auto-load-safe-path /absolute/path/printed/by/find/vmlinux-gdb.py' \
 >> ~/.gdbinit

If GDB reports that auto-loading is disabled, read the warning. It tells you which path GDB rejected. Add that path deliberately. Do not disable all auto-loading just to silence one warning.

Frame pointers and build artifacts

Where the architecture permits, enable CONFIG_FRAME_POINTER. Frame pointers stabilize backtraces and make stepping in GDB far easier during live sessions.

Keep these outputs separate:

Expect larger files and longer links. Those trade-offs cut troubleshooting time when you can see names and lines instead of addresses.

Keep a debug defconfig for iterative hacking and a lean release config for production. That split prevents a debug build from quietly becoming the image you ship.

Set up a debuggable target with QEMU and GDB

I start with a virtual target that exposes a GDB stub. This lets me pause the CPU, set breakpoints, and step through kernel code without risking hardware.

QEMU can boot a kernel directly with -kernel and -append for fast iteration. When you need a full userspace and loaded modules, boot the root filesystem from a guest disk instead.

Start QEMU with the GDB stub

The common mistake is using -s and assuming the guest waits for GDB. It does not. -s opens the GDB stub on TCP port 1234. Add -S to stop the virtual CPU until the debugger attaches.

A direct launch looks like this:

qemu-system-x86_64 \
 -kernel output/images/bzImage \
 -append "root=/dev/vda console=ttyS0" \
 -drive file=output/images/rootfs.ext4,format=raw,if=virtio \
 -net nic,model=virtio \
 -net user,hostfwd=tcp::5555-:22 \
 -s -S

Add -enable-kvm when the host and target architecture support it. KVM makes the guest faster, but it does not change the symbol workflow.

The important split is clear:

• -s starts the GDB stub on port 1234.
• -S pauses execution before the kernel runs.
• -net user,hostfwd=tcp::5555-:22 forwards host port 5555 to guest SSH.

Attach GDB and load kernel symbols

Start GDB with the uncompressed vmlinux, not the compressed boot image.

gdb vmlinux

Then attach to the paused guest.

(gdb) target remote :1234
(gdb) lx-symbols
(gdb) lx-lsmod
(gdb) lx-dmesg

lx-symbols loads kernel and module symbols using the running kernel's module paths. lx-lsmod shows which modules are loaded. lx-dmesg reads the kernel log from the live target.

Useful interactive commands include:

(gdb) continue
(gdb) step
(gdb) finish
(gdb) $lx_current()
(gdb) $lx_per_cpu()

Set pending breakpoints for drivers that load later. If a breakpoint stays unresolved, check the module name and load state before changing the breakpoint. A missing symbol is usually a mismatch, not a GDB failure.

For a module-specific workflow, see how to debug a kernel module.

Where quickqemu fits

quickqemu is usually a mistaken reference to Quickemu. The official project name is quickemu-project/quickemu, and its VM configuration format uses a shell-style .conf file.

Quickemu is a VM launcher. It is not a replacement for understanding QEMU's kernel debugging flags. Direct QEMU remains the reliable choice for this loop because -s -S makes the stub and stopped CPU explicit.

If you use Quickemu, confirm that its configuration can pass the QEMU arguments needed for the GDB stub. If it cannot pass -s -S, use direct QEMU for kernel work and keep Quickemu for ordinary VM launches. The launcher is not the debugger.

What does the buildroot BR2_ENABLE_DEBUG debug symbols manual workflow require?

BR2_ENABLE_DEBUG controls package debug builds. It does not replace the Linux kernel's CONFIG_DEBUG_INFO, CONFIG_GDB_SCRIPTS, or frame pointer settings.

For the manual Buildroot debug-symbols workflow, start with a known board or QEMU configuration. Then change Buildroot and kernel settings in the same output tree so the toolchain, kernel, modules, and root filesystem agree.

The Buildroot manual documents the surrounding configuration system. The practical sequence is:

1. Start with a QEMU configuration.

make qemu_x86_64_defconfig

2. Open Buildroot configuration.

make menuconfig

3. Enable BR2_ENABLE_DEBUG for package-level debug symbols.

4. In the Linux kernel configuration, enable CONFIG_DEBUG_KERNEL, CONFIG_DEBUG_INFO, CONFIG_GDB_SCRIPTS, and CONFIG_FRAME_POINTER where supported.

5. Leave CONFIG_DEBUG_INFO_REDUCED off when complete type data matters.

6. Enable the toolchain debugging support and target-side GDB server option if you need to attach from inside the target.

7. Select OpenSSH if you need an SSH shell.

8. Select an ext4 root filesystem with BR2_TARGET_ROOTFS_EXT2_4 when you want a writable, familiar guest filesystem.

9. Build the complete target.

make

The important separation is host versus target. Use the unstripped files in the Buildroot output tree for host-side GDB. The target root filesystem may contain stripped binaries, and that is fine for a small runtime image. It is not the file you should hand to GDB.

Pick matching kernel and toolchain settings

In the Toolchain and Kernel menus, select one Linux source revision and matching headers. Do not build the kernel from one output tree and use modules from another. The names may look similar while the module version data disagrees.

If you change debug settings after a package already built, rebuild the affected package. A top-level make does not always invalidate old package artifacts.

make <package>-rebuild

Use a clean rebuild when the old package was already stripped or compiled with different compiler flags. Otherwise you can spend an hour debugging a binary that never contained the symbols you needed.

Key outputs and next steps

After the Buildroot build finishes, the Linux source is under output/build/linux-…. The uncompressed kernel image is:

output/build/linux-…/vmlinux

The boot image and root filesystem are under output/images.

Keep these paths tied to the same output tree:

Boot the Buildroot output with the same GDB flags:

qemu-system-x86_64 \
 -kernel output/images/bzImage \
 -append "root=/dev/vda console=ttyS0" \
 -drive file=output/images/rootfs.ext4,format=raw,if=virtio \
 -net nic,model=virtio \
 -net user,hostfwd=tcp::5555-:22 \
 -s -S

Then attach with the matching vmlinux from the same Buildroot output tree. Use SSH on host port 5555 to run commands and reproduce the failure.

QEMU snapshots help during iterative hacking. Save a known state through the monitor, reproduce the bug, then restore that state instead of rebuilding the whole guest after every failed experiment.

Why are module debuginfo and SystemTap warnings still appearing?

A warning such as cannot find module nfs debuginfo usually means one of two things. The module lacks DWARF data, or the tool cannot find the symbols that were built.

Start by checking the module itself.

readelf -wi module.ko

If the command shows no useful DWARF sections, the module was not built with the required debug information. If the sections exist, check the path used by SystemTap, GDB, or your other tracing tool.

Ensure modules are built with DWARF

The kernel's debug settings apply to modules as well as the core image. Confirm that CONFIG_DEBUG_INFO is enabled in the same kernel configuration used to build the .ko file.

If the module came from a different build, stop there. Rebuild it from the exact source tree and configuration that produced the running kernel.

A module-specific trace is easier when you keep the module build output, its .config, and the loaded target image together. Forcing a tool to search every directory on the host is not a symbol strategy.

Match the running kernel, module versions, and symbol paths

Even a small version mismatch can stop symbol resolution. Check the running kernel:

uname -r

Compare that value with the version string in the build output and with the module version data. Then point GDB at the matching vmlinux and run lx-symbols.

If the target loads a module after you attach, run lx-symbols again. The helper cannot load symbols for a module that did not exist at the first lookup.

Do glibc-debuginfo packages fix kernel symbols?

No. glibc-debuginfo is user-space libc debug information. It helps GDB explain calls inside glibc, but it does not add symbols to the Linux kernel or a kernel module.

For a Buildroot target, keep the matching glibc and toolchain artifacts from the same build. For a distribution target, install the debuginfo package that matches the running glibc build. Do not install a random package with the right name and assume the build IDs will match.

What should you check before shipping a debug build?

Do not ship the full debug tree just because it made development easier. Keep the symbol files on the host, strip the runtime image only after you have saved the matching artifacts, and record the exact configuration used to build them.

Run these checks before shipping:

• The production image does not contain unnecessary DWARF data.
• The matching vmlinux is stored with the build record.
• Kernel modules and the core image came from the same build.
• The kernel configuration is saved.
• The target's uname -r matches the module version data.
• The GDB helper script was built when the kernel version requires it.
• The debug image was tested under the same boot arguments used during diagnosis.
• Package-level symbols are retained only where you still need them.

DWARF mainly costs storage and build time. Frame pointers can affect runtime performance on some architectures. BR2_ENABLE_DEBUG can also enlarge package outputs, so use it for development and controlled test images rather than enabling it blindly in production.

The useful compromise is straightforward: keep a symbolized host-side build, ship a lean target image, and make the two easy to match later.

FAQ

Can I use this workflow on physical hardware instead of QEMU?

Yes, but don't carry the QEMU assumptions over to hardware. Replace QEMU's remote target with the hardware's kernel debug transport, then keep the same vmlinux, module, and configuration matching rules. Use serial access and a recovery path first, because a bad breakpoint on real hardware doesn't give you a reset button.

Why do source paths in GDB point into the Buildroot output tree?

Because the compiler recorded those paths, and GDB is reporting what it was given. If the source moved, use GDB's set substitute-path command to map the recorded build path to the current source directory. Do not rebuild first unless the source or configuration actually changed.

Should I keep vmlinux inside the target root filesystem?

No. Keep vmlinux on the host. The target needs the bootable kernel and its modules, while GDB needs the uncompressed host-side vmlinux, which is normally too large and too revealing for a production root filesystem.

Can package debug symbols explain a kernel panic?

No. Package symbols explain user-space code, not a kernel panic. Use the matching kernel vmlinux, module symbols, and the panic message itself. Start with the kernel log and exact build artifacts before installing unrelated user-space debuginfo; a package with the right name is still wrong if it came from another build.

Related on this blog

• Enable Kernel CONFIG Options: What to Know First