Cron job killed with exit 137 or signal 9: was it the OOM killer?

The nightly report didn’t arrive. Its log file stops halfway through a line, with no error and no stack trace. If cron mails you, the mail says one word: Killed. If something recorded the exit code, it’s 137. The script has a try/except around everything and an error handler that posts to Slack, and neither ran.

Something sent the job SIGKILL. Usually it’s the kernel’s out-of-memory killer, but not always, and the job itself can’t tell you which. We OOM-killed the same Python job in a cgroup, a shell script, a systemd unit configured like Ubuntu’s cron.service and a Podman container, and noted where each one left a trace.

Short answer

Exit 137 and “signal 9” mean SIGKILL. To find out who sent it, read the kernel log around the time the job died:

journalctl -k --since "03:00" --until "03:30" | grep -iE 'killed process|oom-kill:'
# or, as root: dmesg -T | grep -iE 'killed process|oom-kill:'

Out of memory: Killed process 4127 (python3) is the OOM killer, and the oom-kill: line above it names the cgroup the process lived in. No line there? Check journalctl -u systemd-oomd, then your crontab for timeout -k, then whoever has a shell on the box. In Docker, docker inspect -f '{{.State.OOMKilled}}' <container> gives you the same answer for a container.

The job can’t log its own death, since SIGKILL can’t be caught, so the evidence has to come from outside. To stop one job from taking down the rest of the server, give it its own memory limit: systemd-run --scope -p MemoryMax=1G -p MemorySwapMax=0 in front of the command.

If you’d rather have every run’s exit code, output and peak memory recorded, and the run log say “killed by SIGKILL, check the OOM killer” for you, RunWisp, an open-source cron replacement that ships as a single binary, does that. Details below.

Where each piece of evidence lives:

137, signal 9, -9 and “Killed” are the same death

A process killed by a signal has no exit code of its own. What you see depends on who’s watching:

$ python3 hog.py; echo "exit $?"
exit 137
$ sh -c 'python3 hog.py; echo after'
Killed
after
$ python3 -c 'import subprocess; print(subprocess.run(["python3", "hog.py"]).returncode)'
-9

A parent that waits on the process directly gets “terminated by signal 9”, which Python shows as -9. A shell turns that into 128 + 9 = 137 and prints Killed to stderr. (dash, which cron uses as /bin/sh on Debian and Ubuntu, does so even when the job is the last command. bash skips both when it execs the last command, and then its caller sees signal 9 instead.) That Killed is what lands in cron’s mail. Same kill, four spellings. (All three runs above happened under a 100 MB memory limit, which is how we made the kernel pick hog.py every time.)

The second run shows the first trap. The shell printed Killed, carried on, ran echo after and exited 0. A wrapper script without set -e reports success for a job the kernel killed, and anything that only checks the exit code never hears about it.

SIGKILL is also sent by more than the OOM killer:

• timeout -k 30s 20m sends it when the job ignores SIGTERM, and then exits 137 itself. See how to put a timeout on a cron job.
• systemctl stop sends it when a unit outlives TimeoutStopSec= (90 seconds by default), and docker stop after 10 seconds. A deploy or a reboot in the middle of the job does this.
• A person with kill -9, or pkill -9 -f python aimed at a different Python.
• Hosting panels and PaaS agents that enforce their own memory or CPU limits.

So 137 is a lead, not an answer.

Step 1: the kernel log

When the kernel runs out of memory, it picks a process to kill and logs it. This is what it wrote when our job hit its limit:

$ journalctl -k --since "-10min" | grep -iE 'killed process|oom-kill:'
kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/user.slice/user-1000.slice/[email protected]/app.slice/nightly-report.scope,task_memcg=/user.slice/user-1000.slice/[email protected]/app.slice/nightly-report.scope,task=python3,pid=1829633,uid=1000
kernel: Memory cgroup out of memory: Killed process 1829633 (python3) total-vm:123736kB, anon-rss:101884kB, file-rss:7036kB, shmem-rss:0kB, UID:1000 pgtables:280kB oom_score_adj:100

Three things to read in there:

• Out of memory: means the whole machine ran out. Memory cgroup out of memory: means a limit was hit: a container’s --memory, a unit’s MemoryMax=, a Kubernetes limit.
• task_memcg= is the cgroup the killed process was in. A cron job on Debian or Ubuntu shows up under /system.slice/cron.service, a systemd timer’s job under its own .service, a container under its scope. This is how you tie a python3 or php (the name is cut to 15 characters) to a job.
• anon-rss: is how much memory the process held when it died.

Just above those lines the kernel prints a table of every process in the affected cgroup or, on a machine-wide OOM, every process on the box, with its RSS. Read it. The OOM killer kills the biggest process, which isn’t always the one that asked for memory. A cron job that grows to 3 GB on a 4 GB VPS can get MySQL killed instead, and then the first thing you see is “database went away” in your web app, not a dead cron job.

Some things that hide this log:

• dmesg needs root on Ubuntu. kernel.dmesg_restrict is 1, so a normal user gets read kernel buffer failed: Operation not permitted. journalctl -k reads the same messages without root if you’re in the adm group.
• The ring buffer is a fixed size. On a chatty box, last night’s OOM may have been overwritten. journalctl -k keeps it if the journal is persistent (/var/log/journal exists).
• dmesg -T times can be wrong. It converts uptime to wall-clock time and doesn’t account for time spent suspended. journalctl -k stamps each line when it’s received.

Step 2: systemd-oomd and earlyoom

Fedora and Ubuntu Desktop run systemd-oomd, and some people install earlyoom. Both kill from user space before the kernel has to, and the kernel logs nothing when they do. systemd-oomd also kills whole cgroups, so the job and every process it started go at once.

systemctl is-active systemd-oomd && journalctl -u systemd-oomd --since "03:00"
journalctl -u earlyoom --since "03:00"

A systemd-oomd kill reads Killed /system.slice/... due to memory pressure ... or ... due to memory used ... swap used. If your job died there, the fix is the same as for the kernel: give it less to do or more room.

Step 3: what cron’s systemd unit saw

On Debian and Ubuntu, cron jobs stay inside cron.service’s cgroup. systemd watches that cgroup for OOM kills, and its default OOMPolicy=stop treats one as the service failing. Both distros ship a cron.service with KillMode=process and Restart=on-failure, so in practice systemd stops the cron daemon, leaves the other running jobs alone, and starts cron again. The journal keeps a record of it. We reproduced it with a stand-in unit that has the same KillMode and Restart settings and a 100 MB limit:

$ journalctl --user -u fakecron -o cat
fakecron.service: The kernel OOM killer killed some processes in this unit.
fakecron.service: Failed with result 'oom-kill'.
fakecron.service: Unit process 1838479 (sleep) remains running after unit stopped.
fakecron.service: Consumed 24ms CPU time over 1.120s wall clock time, 100M memory peak.
fakecron.service: Scheduled restart job, restart counter is at 1.
fakecron.service: Found left-over process 1838479 (sleep) in control group while starting unit. Ignoring.

The sleep is the other job, still running.

On a real server that’s journalctl -u cron | grep -i oom. It’s a handy check because it names cron, so you know the kill was inside a cron job without matching PIDs. The cron daemon itself logs only CMD lines when a job starts, and nothing when one dies (more in where are cron logs).

Not every scheduler gets off this lightly. With KillMode=control-group, the default for a unit, systemd’s stop kills every process in the unit, so one OOM-killed job takes every other running job down with it. Our stand-in did exactly that when we switched it to the default. Any daemon that runs jobs as children under a systemd unit has this problem unless it sets OOMPolicy=continue.

If the job runs in its own cgroup (a systemd scope or service, a container), the cgroup also keeps a counter. This is our test scope’s, read from inside the scope after the kill:

$ cat /sys/fs/cgroup/user.slice/user-1000.slice/[email protected]/app.slice/nightly-report.scope/memory.events
low 0
high 0
max 37
oom 1
oom_kill 1
oom_group_kill 0

oom_kill counts processes in that cgroup killed by the kernel’s OOM killer. It goes away with the cgroup, so read it while the unit or container still exists. max is how many times the cgroup hit its MemoryMax= and had to reclaim, which is a warning you get before the kill.

Step 4: Docker and Podman

The container runtime watches the container’s cgroup and sets a flag. Here with Podman, where the container’s main process was the one that ran out:

$ podman run --name oom-pid1 -m 64m --memory-swap 64m alpine tail /dev/zero; echo "exit $?"
exit 137
$ podman inspect -f '{{.State.OOMKilled}} {{.State.ExitCode}}' oom-pid1
true 137

docker inspect takes the same -f template.

Two caveats. The container has to still exist, so docker run --rm loses it, and so does anything that cleans up stopped containers. And the flag is about the whole container, not its main process. In Podman 5.7 we OOM-killed a child of the container’s shell, and the shell went on and exited normally:

$ podman run --name oom-child -m 64m --memory-swap 64m alpine sh -c 'tail /dev/zero; echo "job died: $?"'
Killed
job died: 137
$ podman inspect -f '{{.State.OOMKilled}} {{.State.ExitCode}}' oom-child
true 0

Exit code 0, OOMKilled true. A cron container that runs jobs under a shell or supercronic looks exactly like this: the container stays up, the job is gone. Docker sets the flag from the same kind of cgroup event, but if OOMKilled says false and you still see 137, the kernel log from step 1 is the source of truth. It’s in the host’s journal, since containers share the kernel.

Step 5: when there’s no OOM anywhere

No kernel line, no systemd-oomd line, no flag. Then something else sent the SIGKILL:

• Your own crontab. grep -n 'timeout' /etc/crontab /etc/cron.d/* /var/spool/cron/crontabs/*. A timeout -k that fired exits 137 and logs nothing.
• A stop or a deploy. journalctl --since "03:00" --until "03:30" for Stopping, Killing process or a reboot (last -x reboot).
• A person or another script. Shell history, auth.log for who was logged in, and any cleanup script that runs pkill -9.

Why the job can’t tell you itself

SIGKILL is the one signal a process can’t catch, block or ignore. The kernel removes the process without running any of its code. No trap in bash, no finally or atexit in Python, no register_shutdown_function in PHP, no process.on('exit') in Node. Any “send an alert if the job fails” logic inside the job is dead before it starts.

It also loses output you thought it had written. Under cron, stdout is a pipe, not a terminal, and Python, like most runtimes, buffers output to a pipe and writes it out a few kilobytes at a time. Our job printed two progress lines and then started eating memory. Through a pipe, neither line appeared. With buffering off, both did:

$ python3 hog.py | cat

$ PYTHONUNBUFFERED=1 python3 hog.py | cat
step 1: loaded 120k rows
step 2: building report

So the last line in the job’s log is not where it died; it’s where the buffer was last flushed. For jobs that might be killed:

• Set PYTHONUNBUFFERED=1 in the crontab, or print(..., flush=True). Write progress to stderr, which isn’t block-buffered.
• End wrapper scripts with the job’s exit code, or start them with set -e, so a 137 doesn’t turn into a 0.
• Record how much memory normal runs use, so you know how close you’re running. GNU time, which is /usr/bin/time and not the shell’s time, prints the peak even when the job is killed:

$ /usr/bin/time -v python3 hog.py 2>&1 | grep -E 'signal|Maximum resident'
Command terminated by signal 9
	Maximum resident set size (kbytes): 108696

(Ignore the Exit status: 0 line it prints for a killed command.)

Cap the job, not the machine

The OOM killer exists to save the machine, not your job. Without a limit, a job that grows takes memory from everything else until the kernel kills whatever is biggest. Give the job a limit of its own, and the kernel kills inside that limit and leaves the rest alone.

From a root crontab, systemd-run --scope runs the command in the foreground, in a new cgroup with a limit:

0 3 * * * systemd-run --scope -q -p MemoryMax=1G -p MemorySwapMax=0 -p OOMPolicy=continue /usr/local/bin/nightly-report

Output still goes to cron, and the kill shows up in the kernel log as Memory cgroup out of memory, with the scope’s name in task_memcg=. Without MemorySwapMax=0, the job pushes the rest into swap first and gets slow before it gets killed. OOMPolicy=continue matters: a scope defaults to OOMPolicy=stop, so after the kernel kills one process systemd stops the whole scope and SIGTERMs the job’s other processes. A wrapper script around the job then exits 143 instead of carrying on to report the 137. With continue, only the process the kernel picked dies, and the exit code is the job’s. MemoryHigh= is a softer limit below MemoryMax=: past it, the kernel slows the job down and reclaims its memory instead of killing it.

From a user crontab, it’s systemd-run --user, and cron doesn’t set the variable that finds your user’s systemd. Without it, systemd-run --user fails with $DBUS_SESSION_BUS_ADDRESS and $XDG_RUNTIME_DIR not defined. Set it at the top of the crontab, and run loginctl enable-linger once so your user’s systemd runs when you aren’t logged in:

XDG_RUNTIME_DIR=/run/user/1000
0 3 * * * systemd-run --user --scope -q -p MemoryMax=1G -p MemorySwapMax=0 -p OOMPolicy=continue /home/me/bin/report

A systemd timer gets the same keys on its service unit:

[Service]
Type=oneshot
ExecStart=/usr/local/bin/nightly-report
MemoryMax=1G
MemorySwapMax=0

In Docker, docker run --memory 1g --memory-swap 1g, or mem_limit: 1g in Compose. Setting --memory-swap to the same value turns swap off for the container.

ulimit -v looks like the simple option, and works for some jobs. It limits virtual address space, so the allocation fails and the job gets an error instead of SIGKILL: our Python job raised MemoryError and could log it. But runtimes that reserve address space up front break. Under ulimit -v 1000000 (about 1 GB), Node.js crashed before printing 1, with exit 133. Java reserves large ranges too, and fails at 1 GB. Use it for plain C tools and Python scripts, not for anything with a JIT or a garbage collector of its own.

Pick the limit from the job’s real peak, plus headroom. A limit below the job’s normal peak turns a working job into one that dies every night.

RunWisp: the run log names the signal

RunWisp, our project, is an open-source cron replacement that ships as a single binary. It reads your crontab with runwisp import cron and runs each line as a task, and it keeps every run’s exit code, output and duration. Three things in it are about this problem.

When a run dies from a signal RunWisp didn’t send, the run log says so, also when a shell turned it into 137, and for SIGKILL it points at the OOM killer. This is our test job under RunWisp:

$ runwisp run nightly-report
[INFO] Task triggered name=nightly-report run=01M5WFQ47AB063X9HC9WBH63B5
Killed
Exit code 137: a process the run started was killed by SIGKILL from outside RunWisp. The kernel's out-of-memory killer is the usual cause: check `journalctl -k | grep -i oom` or `dmesg`.
[ERROR] Task failed name=nightly-report reason=failed duration=121ms exit=137

The run counts as failed and alerts like any other failure. Runs are started as sh -e -c, so a multi-line run stops at the killed command instead of carrying on and exiting 0.

Every run records its peak memory and CPU time. The run lists in the TUI and the Web UI color them against the task’s usual run, amber from 1.5× and red from 2×, so the run that got killed, and the runs that were getting close before it, stand out.

And its systemd unit sets OOMPolicy=continue, so a task the kernel kills doesn’t stop the daemon and every other task with it. (If you installed the service before 1.8.0, run sudo runwisp service install again to update the unit.)

[tasks.nightly-report]
cron   = "0 3 * * *"
run    = "systemd-run --scope -q -p MemoryMax=1G -p MemorySwapMax=0 -p OOMPolicy=continue /usr/local/bin/nightly-report"
notify = ["slack-ops"]

The limits. RunWisp reads the signal, not the kernel log, so “usually the OOM killer” is an inference: a kill -9 gets the same message, and journalctl -k is still the proof. It has no memory limit of its own per task, so you wrap the command in systemd-run --scope as above, the same as in a crontab. It can’t catch the signal for your job either; it records it because it’s the parent watching. And it runs on Linux and macOS, not Windows outside WSL.

How we tested

Ubuntu 26.04.1 with kernel 7.0 and systemd 259, cgroup v2. The job was a Python 3 script that prints two lines and then allocates 10 MB every 10 ms. Each run had a 100 MB limit from systemd-run --user --scope -p MemoryMax=100M -p MemorySwapMax=0, so the kernel’s cgroup OOM killer picked it every time. We didn’t run a machine-wide OOM; it logs Out of memory: instead of Memory cgroup out of memory: and is otherwise the same. The cron.service behaviour is from a transient user unit with KillMode=process, Restart=on-failure and the default OOMPolicy, the settings of the cron.service that Ubuntu 26.04 and Debian 13 ship. Containers: Podman 5.7.0 with crun and alpine:3. Node.js 24.22 for ulimit -v. RunWisp 1.8.0.