Run journalctl -b -p err first. That single command lists every error from the current boot, in order, across every unit on the box, and it's where I start any time I need to check logs on Linux. From there I narrow to the failing service with journalctl -u, then cross-check the kernel's own account with dmesg -T. Grepping around in /var/log before you have a time window is how you end up reading noise for an hour instead of finding the cause.

Last updated: 2026-08-02

Where Linux logs live and which one to open first

Your logs are scattered across more than one store, and those stores don't hold the same things. The systemd journal is a structured binary store fed by the kernel, by services, and by anything writing to syslog. Alongside it sit plain text files. /var/log is the default directory for traditional Linux log files, as specified by the Filesystem Hierarchy Standard.

Plenty of applications ignore both. Anything that predates systemd, or just prefers its own logger, writes wherever its config file says. Nginx, PostgreSQL, and most Java services do exactly that, and people forget it every time.

So pick your source by what failed, not by habit:

That table saves more time than any clever grep. Because the wrong source hands you a plausible-looking line with no connection to your outage, and you'll believe it for twenty minutes.

How do you check logs in Linux with journalctl?

journalctl is the command-line utility for querying the systemd journal, and a handful of flags handle nearly every investigation I run. Learn them together, since they compose.

journalctl -b # this boot only
journalctl --list-boots # every boot the journal still holds
journalctl -b <boot-id> # an earlier boot, which is what you want after a crash
journalctl -u nginx # scope to one unit
journalctl -p warning # warning and everything more severe
journalctl -f # follow live while you reproduce the fault
journalctl --since yesterday --until now

The one that matters most is --since. Reading logs with no time anchor is the mistake I see constantly. You get thousands of lines, all of them true, none of them tied to the moment things went wrong. Start with a loose window like yesterday, then swap in an exact timestamp once you have one.

After a crash, reach for --list-boots. It prints the boot identifiers the journal still holds, and you pass the one you want straight to -b. That beats guessing, because a box that rebooted several times overnight gives you several candidates.

Which output format should you use?

Add -o short-precise when two events land in the same second and you need to know which came first. Use -o json-pretty when you want the structured fields, including _PID, _COMM, and _SYSTEMD_UNIT.

Those fields are the reason the journal beats a text file. You can filter on them directly, like journalctl _PID=<pid>. The systemd project documentation covers the full field list if you want to go further.

One caution. On plenty of distributions the journal is stored in memory only, so it disappears on reboot. Check /etc/systemd/journald.conf for Storage=persistent before you assume last week's crash is still there.

Reading /var/log without missing the rotated files

Never open a log with cat. Use less and you get search, line numbers, and no scrollback flooding your terminal. For live tailing, use tail -F rather than tail -f, because the capital F reopens the file after rotation. Lowercase f keeps holding a deleted file descriptor and quietly shows you nothing while the problem keeps happening.

Rotation is where most searches silently fail. Logrotate renames yesterday's file and gzips the older ones, so grep over syslog skips everything before the last rotation. Use zgrep for the compressed ones:

grep -i "out of memory" /var/log/syslog
zgrep -i "out of memory" /var/log/syslog.*.gz
find /var/log -type f -newermt "-2 days" -printf "%T+ %p\n" | sort

That find command is what I reach for on an unfamiliar box. It tells you which files are actually being written to right now. As a result you skip somebody's stale documentation about what should be there.

Add -B and -A to grep for context lines. A stack trace or a failed DNS lookup sits in the lines around the match, not in the matched line itself.

How much should you trust log levels?

Less than you'd like. Syslog priorities run from emerg at the top through alert, crit, err, warning, notice, and info, down to debug. journalctl -p err shows err and everything above it, which is my default first pass. Both the names and their numeric equivalents work as arguments.

Here's the part nobody says out loud. Severity is set by whoever wrote the code, and a lot of them set it badly. Some services log a fatal condition at warning and a routine retry at err. So a clean -p err sweep tells you nothing about whether the box is healthy.

I use levels to shrink the haystack, then drop to -p info inside the failure window. The story is usually in the info lines: the service starting, binding a socket, failing to resolve a name. The err line only tells you where the story ended.

Turn on debug deliberately and turn it back off. Debug logging on a busy service will fill a disk faster than you expect, and then you have two outages.

When dmesg beats the system log

Reach for dmesg -T when the fault is below userspace. The kernel ring buffer is where you see disk read errors, filesystem remounts to read-only, the out-of-memory killer choosing a victim, USB devices enumerating late, and drivers refusing to load. Always pass -T. Raw kernel timestamps count seconds since boot, which is no use when you're lining events up against a user's report.

dmesg -T --level=err,crit,alert,emerg
dmesg -T | grep -i -E "oom|i/o error|read-only|segfault"

The journal captures kernel messages too, so journalctl -k gets you the same stream with the journal's filtering. Still, the ring buffer itself is finite and wraps. A chatty driver can push your evidence out before you look, which is an argument for reading it early rather than rebooting.

If the kernel messages end mid-sentence and the machine died there, you're in different territory. That's a panic, and it needs the console output, because the disk stopped being written to. My method for finding the real cause of a kernel panic covers that path.

Chasing one service, one app, or one container

Start with systemctl status <unit> before you open any log file. It gives you the current state, the main PID, the last few journal lines, and the exit status of the last run. That header answers the first question I always have: did it fail to start, or did it start and then die?

systemctl status nginx --no-pager -l
journalctl -u nginx -b --since "<timestamp of the failure>"
systemctl cat nginx # the actual unit file plus every drop-in override

systemctl cat matters more than it looks. Distributions ship drop-in overrides under /etc/systemd/system/<unit>.d/ that change ExecStart, environment variables, and output redirection. I've watched people debug a unit file that hadn't been in effect for months.

For applications that log to their own files, don't guess the filename. Ask the app instead. Nginx tells you with nginx -T, which dumps the full parsed configuration including every error_log directive. PostgreSQL keeps its answer in postgresql.conf under log_directory.

Containers are their own world. docker logs -f --since <duration> <name> reads the container's stdout and stderr. Meanwhile the host journal will have none of it, unless the daemon is configured with the journald logging driver.

What does real log analysis look like?

Searching for the word "error" is guessing with extra steps. Analysis means you can say what happened, in what order, and prove it. Here's the sequence I run, and it's reproducible on any box.

1. Fix the window. Get a timestamp from a user report, a monitoring alert, or the process start time in systemctl status. Everything after this step is scoped with --since and --until.
2. Read the whole window, unfiltered. Run journalctl -b --since "..." --until "..." with no priority filter. Read it top to bottom, because the first anomaly is rarely the loudest line.
3. Correlate the layers. Compare the service's journal entries against dmesg -T for the same minutes. A service crash that lines up with a disk I/O error is a storage problem wearing a service's name.
4. Trace one identifier. Pick a PID, a request ID, or a session ID and follow it across sources with journalctl _PID= or grep. One thread of events beats a thousand unrelated lines.
5. Get a known-good baseline. Run the same query against a healthy period or a healthy host. The difference is your signal, and it's usually a line that went missing rather than one that appeared.
6. Test the hypothesis. Reproduce the fault with journalctl -f running. If your theory is right, you'll watch it happen. If nothing appears, your theory is wrong and you just saved yourself a day.

Step five is the one people skip. A log line looks alarming until you check yesterday and find the log_influx_ram crontab entry fires every hour on a perfectly healthy machine.

Why the log you expected isn't there

Missing logs have a short list of causes, and the fix depends on which one you hit. Work through them in this order.

Journal persistence comes first. If Storage=volatile is set, or /var/log/journal doesn't exist, the journal lives in RAM and clears on reboot. Create the directory, set Storage=persistent, and restart systemd-journald.

Rate limiting is next. Journald drops messages when a service floods it, and it says so with a "Suppressed N messages" line. Look for that line before you conclude a service went quiet. The limits live in journald.conf as RateLimitIntervalSec and RateLimitBurst.

Then permissions. A non-root user only sees their own journal entries unless they're in the systemd-journal or adm group. That's the usual reason a command "works for root" and shows nothing otherwise.

Finally, check whether rsyslog or syslog-ng is even installed. Many current distributions ship the journal alone, so /var/log/syslog and /var/log/messages never get created. Nothing is broken; the file you're grepping simply doesn't exist on that system.

What to run when the log points deeper

A log line is evidence, not a diagnosis. Once the journal tells you a service is dying, stop reading and start inspecting the running system.

systemctl show <unit> -p ExecMainStatus -p Restart -p NRestarts
ps -eo pid,ppid,stat,wchan:20,cmd --sort=-%mem | head
ss -tulpn | grep <port>
strace -f -e trace=file -p <pid>

A non-zero exit code that matches a documented failure mode is a real lead. A clean exit code paired with an unexpected restart means something else told the process to stop. In that case, look at signals and at whatever supervises it. Reading the Linux process life cycle makes those state letters in ps worth something.

strace covers the gap between what the log says and what the process is doing. Scope it with -e trace=file or -e trace=network so you get a readable trace rather than a wall of futex calls. When a service reports "permission denied" and names no path, strace hands you the exact path it tried.

If the trail leads to timeouts and refused connections, you're out of log territory. Work the stack from the bottom, the way I do when troubleshooting a Linux network, because link and DNS problems produce very convincing application errors.

FAQ

Can I read the journal from a machine that won't boot?
Yes, and this is the flag worth memorizing. Boot a live image, mount the broken root filesystem, then run journalctl -D /mnt/var/log/journal --list-boots to see what survived. Pass the identifier you want back to the same command with -b to read it. If you only have loose journal files copied off the disk, journalctl --file <path> reads them directly.

How do I stop the journal from eating the disk?
Set SystemMaxUse in /etc/systemd/journald.conf to a cap you choose, then restart journald. For immediate relief, journalctl --vacuum-size= and journalctl --vacuum-time= delete old entries down to a size or age you specify. Check what it's currently consuming with journalctl --disk-usage before you pick a number.

Do I still need rsyslog if I have journald?
Only for a reason you can name. Central forwarding to a remote collector, a compliance requirement for plain text files, or a legacy tool that parses /var/log/messages all justify it. Otherwise you're storing the same messages in more than one place and paying for the extra disk writes to no purpose.

Why does a service log at info to the journal but nothing lands in its own file?
Because the unit is capturing stdout and stderr while the application's internal logger is configured separately. StandardOutput=journal in the unit sends console output to the journal. The app's own config decides what goes to its file, so changing one won't affect the other.

How do I search the journal for a message when I don't know the unit?
Use journalctl -g <pattern> for a case-insensitive regex match across all entries, combined with --since to keep it bounded. Once you get a hit, read the _SYSTEMD_UNIT field on that entry with -o verbose and you've found your unit.