You wake up to find MySQL has stopped, a site is down, and there is nothing useful in the application logs. The service just vanished. On a Linux server this is the classic fingerprint of the Linux OOM killer (out-of-memory killer), a kernel mechanism that terminates processes when the system runs out of memory. It protects the server from a total freeze, but it often kills the most important process on the box. This guide shows how to confirm an OOM event, find what caused it and make sure it does not happen again.

What the OOM Killer Does

Linux lets processes allocate more memory than physically exists (overcommit), betting that not all of it will be used at once. When actual demand outpaces RAM plus swap and the kernel cannot reclaim cache, it picks a victim based on an oom_score, which favours processes using a lot of memory that are not essential, and sends it SIGKILL. Because a database often holds the largest memory footprint, it is a frequent victim.

Because SIGKILL cannot be caught, the application writes nothing to its own log. That is why the evidence is in the kernel log.

Step 1: Confirm It Was the Linux OOM Killer

# Kernel ring buffer with readable timestamps
dmesg -T | grep -iE "out of memory|killed process|oom-kill"

# systemd journal (persists across reboots if configured)
journalctl -k --since "yesterday" | grep -iE "oom|killed process"

# Classic log files
grep -iE "out of memory|killed process" /var/log/messages /var/log/syslog 2>/dev/null

A typical entry looks like this:

Out of memory: Killed process 2314 (mysqld) total-vm:4864120kB, anon-rss:2951000kB, file-rss:0kB, shmem-rss:0kB, UID:27

Read it as: the process name, how much memory it used (anon-rss is the real resident usage) and which user it ran as. Just above the kill line the kernel prints a table of every process and its memory, which shows who was competing.

On systemd servers you may also see systemd-oomd or earlyoom acting before the kernel does.

Step 2: Find Out Who Was Using the Memory

Check the current state:

free -h
vmstat 1 5
# Top memory consumers
ps -eo pid,user,%mem,rss,cmd --sort=-rss | head -15

For shared-memory-aware numbers, smem -tk gives proportional set size (PSS), which is more honest than RSS when many PHP-FPM or Apache workers share pages. For history, install sar and review past memory use; see Install and Use SAR on Linux.

Typical causes on hosting servers

Cause	Why it exhausts memory
Too many PHP-FPM or Apache workers	pm.max_children multiplied by memory per worker exceeds RAM
Oversized MySQL buffers	innodb_buffer_pool_size plus per-connection buffers times connections
Traffic spike or bot crawl	Hundreds of simultaneous dynamic requests
Memory leak	A long-running script or a misbehaving plugin
No swap	Nothing absorbs a short burst
Backup, malware scan or cron job running at peak	Temporary but heavy
Containers/VMs without limits	One tenant starves the rest

Step 3: Do the Memory Maths

For web servers, the most common fix is simply sizing worker counts to the hardware. Measure the memory of one worker:

ps --no-headers -o rss -C php-fpm | awk '{s+=$1;n++} END {print s/n/1024 " MB average over " n " workers"}'

If workers average 80 MB and you have 4 GB of RAM with about 1.5 GB reserved for MySQL and the OS, you can safely run about (4096 - 1536) / 80 = 32 workers, so set pm.max_children = 30, not 200. The same logic applies to Apache’s MaxRequestWorkers.

For the database side, see our MySQL and MariaDB performance tuning guide: the worst-case MySQL memory is the buffer pool plus max_connections times the per-thread buffers.

Step 4: Add and Tune Swap

Swap is not a substitute for RAM, but a small amount lets the kernel evict idle pages instead of killing a process during a spike. Check it:

swapon --show
free -h

Create a 2 GB swap file if there is none:

fallocate -l 2G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab

Tune how eagerly the kernel swaps. The default vm.swappiness=60 is fine for general servers; many database servers prefer a lower value such as 10:

sysctl vm.swappiness=10
echo 'vm.swappiness=10' > /etc/sysctl.d/99-swappiness.conf

If vmstat shows constant si/so activity (swap in and out), the server is thrashing: it needs more RAM or fewer workers, not more swap. Note that KVM and dedicated servers allow swap freely, while some container-based VPS platforms do not; compare them in KVM vs OpenVZ vs LXC vs Docker.

Step 5: Protect Critical Services

You can tell the kernel which processes to avoid killing by adjusting oom_score_adj (from -1000, never kill, to +1000, kill first). With systemd:

# systemctl edit mariadb
[Service]
OOMScoreAdjust=-500

Then systemctl daemon-reload && systemctl restart mariadb. Use this sparingly. If everything is protected, the kernel will kill something worse. A better approach is to make the sacrificial process more killable, such as backup jobs or PHP workers:

# systemctl edit php-fpm
[Service]
OOMScoreAdjust=300

Also set a memory limit on a service so only it is affected rather than the whole system:

[Service]
MemoryMax=2G
MemoryHigh=1800M

Step 6: Stop Misbehaving Sites and Requests

• Set memory_limit in PHP to a realistic value such as 256M, not -1
• Limit crawlers and bad bots: block bad bots with .htaccess and Cloudflare
• Use full-page and object caching so fewer requests reach PHP
• Move heavy cron jobs to quiet hours (Cron Jobs Explained)
• On cPanel servers, CloudLinux LVE limits keep one account from consuming everything; see 508 Resource Limit

Step 7: Monitor So You See It Coming

Set an alert at 85% memory use and alert on any oom-kill line. A tiny script from cron is enough:

#!/bin/bash
if dmesg -T | grep -q "Out of memory" ; then
  dmesg -T | grep -i "killed process" | tail -3 | mail -s "OOM on $(hostname)" admin@example.com
fi

For continuous visibility use Netdata, Prometheus with node_exporter, or Zabbix. See also How to Diagnose and Fix High CPU or Memory Usage on a Linux VPS.

When You Simply Need More RAM

If you have right-sized workers, tuned MySQL and the server still reaches its limits at normal traffic levels, it is outgrown. Upgrading RAM is cheaper than debugging recurring outages.

OOM Quick Reference

Goal	Command
Find OOM events	`dmesg -T \	grep -i “killed process”`
Memory overview	free -h and vmstat 1 5
Top consumers	`ps -eo pid,user,%mem,rss,cmd –sort=-rss \	head`
Check swap	swapon --show
Protect a service	OOMScoreAdjust=-500 in systemd unit
Cap a service	MemoryMax=2G

Frequently Asked Questions

How do I know if the OOM killer killed my process?

Search the kernel log with dmesg -T | grep -i "killed process" or journalctl -k. The application’s own log usually shows nothing, because SIGKILL cannot be handled.

Is swap a good idea on a web server?

A modest amount of swap is useful as a safety buffer. Constant swapping signals too little RAM or too many workers, not a need for more swap.

Can I disable the OOM killer?

You can set vm.panic_on_oom or disable overcommit, but this usually makes things worse, as the server may panic or refuse allocations. Fix the memory demand instead.

Why did the OOM killer choose MySQL?

The kernel scores processes mainly by memory use. A database with a large buffer pool is typically the biggest process, so it scores highest even though it is the most important.