What happens to a running job when you kill -9 cron, supervisord or systemd

The scheduler is the one process every job depends on, and the one nobody tests killing. The OOM killer, a kill -9 on the wrong PID, a pkill -9 -f python aimed at something else: sooner or later the daemon dies with a job half done. What happens to that job?

We tried it. On Debian 13, with each tool’s own packaged systemd unit, we started a job that ticks once a second for a minute or two, sent the scheduler’s main process kill -9 in the middle, and watched the job, the restart, and what each tool remembered afterwards.

Short answer

It depends less on the scheduler than on its systemd unit’s KillMode=. With KillMode=process, the job outlives the scheduler. With mixed or control-group, systemd kills it along with the scheduler.

• cron handles it best. Its per-job child keeps the job and its output, and systemd has cron back in 0.1 seconds. Nothing records the crash except a few lines in the journal.
• supervisord leaves its programs running as orphans and starts every program again 50 seconds later. You end up with two copies, and supervisorctl only shows you one.
• systemd itself ignores kill -9 1. A per-user manager (systemd --user) dies, takes its running timer job with it, and stays down.
• RunWisp, an open-source cron replacement that ships as a single binary, has its job killed by systemd, comes back in 5 seconds and records the run as crashed, exit -2.

To see what your box does, read the unit:

systemctl show -p KillMode,Restart,RestartUSec cron supervisor

What we killed, exactly

kill -9 PID sends SIGKILL to that one process. Its children aren’t signalled. A child whose parent dies is handed to PID 1 (or the nearest subreaper) and keeps running. Whether the job survives the scheduler is decided by who else acts on it afterwards: systemd, cleaning up the unit’s cgroup, or the kernel, raising SIGPIPE when the job writes to a pipe its dead parent was reading.

So this test is not the same as:

• systemctl stop, which runs the unit’s stop logic and then kills what’s left according to KillMode=. That’s a shutdown, not a crash.
• docker kill on a container whose PID 1 is the scheduler. When a PID namespace’s init dies, the kernel kills every process in the namespace (pid_namespaces(7)), so the job always dies with it, whichever scheduler it is. The restart policy brings the container back.
• Killing the job’s whole process group (kill -9 -PGID). Cron, supervisord and RunWisp all start jobs in a process group of their own, so a group kill aimed at the scheduler misses the job too.

What we did is what an out-of-memory kill or a stray kill -9 does: one SIGKILL to the scheduler’s main PID, taken from systemctl show -p MainPID.

cron: the job never notices

Debian’s cron forks twice per job. The main cron -f process forks a CRON child, and that child runs /bin/sh -c with your command and reads its output, so it can mail it when the job ends:

$ ps -eo pid,ppid,cmd --forest
     35       1 /usr/sbin/cron -f
     84      35  \_ /usr/sbin/CRON -f
     86      84      \_ /bin/sh -c /usr/local/bin/job cron
     87      86          \_ /bin/bash /usr/local/bin/job cron

Killing 35 leaves 84 alone. The job ran its remaining 34 seconds, wrote done, and the CRON child handled the output as usual (it logged No MTA installed, discarding output in our container; with a mail server it sends the mail). Meanwhile systemd restarted cron because cron.service has Restart=on-failure, and left the job alone because the unit has KillMode=process:

$ journalctl -u cron -o cat
cron.service: Main process exited, code=killed, status=9/KILL
cron.service: Failed with result 'signal'.
cron.service: Unit process 2530 (cron) remains running after unit stopped.
cron.service: Unit process 2533 (job) remains running after unit stopped.
cron.service: Scheduled restart job, restart counter is at 1.
cron.service: Found left-over process 2533 (job) in control group while starting unit. Ignoring.
Started cron.service - Regular background program processing daemon.
(CRON) INFO (Skipping @reboot jobs -- not system startup)

The 2530 (cron) that “remains running” is the CRON child, still holding the job’s output.

The new cron ran the next minute’s job on time. It also didn’t re-run @reboot jobs: Debian’s cron only runs them once per boot (more in when does cron’s @reboot run, which covers the exceptions).

Cron’s weak spot is the case without systemd. In a container where cron isn’t PID 1, or on a box where someone started it by hand, nothing brings it back. The job that was running finishes, and then nothing runs at all, with no error anywhere, until somebody notices that a nightly file is missing.

supervisord: two of everything

supervisord starts each program as its direct child and reads its stdout and stderr into log files. Debian’s supervisor.service has KillMode=process, Restart=on-failure and RestartSec=50s. We gave it two 2-minute programs, one that prints to stdout and one that only writes a file, and killed supervisord four seconds in:

$ ps -eo pid,ppid,cmd | grep job          # 4 s after kill -9
     55       1 /bin/bash /usr/local/bin/job sup-loud x 120
     56       1 /bin/bash /usr/local/bin/job sup-quiet quiet 120

$ ps -eo pid,ppid,cmd | grep job          # 59 s after
     55       1 /bin/bash /usr/local/bin/job sup-loud x 120
     56       1 /bin/bash /usr/local/bin/job sup-quiet quiet 120
    292     287 /bin/bash /usr/local/bin/job sup-loud x 120
    293     287 /bin/bash /usr/local/bin/job sup-quiet quiet 120

$ supervisorctl status
loud                             RUNNING   pid 292, uptime 0:00:08
quiet                            RUNNING   pid 293, uptime 0:00:08

Both originals kept going, and their output went nowhere. supervisord came back with no memory of them and started both programs again, as autostart says it should. For 50 seconds nothing supervised them, and after that each one ran twice. For a queue worker that means two consumers where you configured one.

supervisorctl only knows the new copies, and stopping the service doesn’t find the old ones either: after systemctl stop supervisor, both orphans were still running, a minute old, with the unit inactive. They stay until they exit on their own, someone kills them, or the box reboots. stopasgroup and killasgroup don’t help either. They change how a running supervisord stops a program, and after a SIGKILL supervisord doesn’t stop anything.

The fix is on the systemd side. With KillMode=mixed, when supervisord dies, systemd SIGKILLs everything left in the unit before it restarts it:

sudo systemctl edit supervisor
# [Service]
# KillMode=mixed

With the drop-in, the same kill -9 ended like this, and 50 seconds later supervisord started one copy of each program:

supervisor.service: Main process exited, code=killed, status=9/KILL
supervisor.service: Killing process 2425 (job) with signal SIGKILL.
supervisor.service: Killing process 2426 (job) with signal SIGKILL.
supervisor.service: Failed with result 'signal'.

A normal systemctl stop still goes through supervisord, which stopped both programs with SIGTERM the way it does without the drop-in. You trade the orphans for the work they were doing, the same trade RunWisp’s unit makes below.

systemd: PID 1 shrugs, a user manager doesn’t

You can’t SIGKILL PID 1. The kernel only delivers signals to init that init has installed a handler for, and SIGKILL can’t be handled (kill(2) says so, “to assure the system is not brought down accidentally”). kill -9 1 returned success and did nothing: systemd stayed up and the timer’s job kept ticking. System timers are the one setup here a stray kill -9 can’t touch.

User timers run under a per-user systemd --user, which is an ordinary process. With lingering on (loginctl enable-linger), it runs a user’s timers without anyone logged in. We killed the manager three seconds into a timer job:

$ journalctl -u [email protected] -o cat
[email protected]: Main process exited, code=killed, status=9/KILL
[email protected]: Killing process 1073 (job) with signal SIGKILL.
[email protected]: Killing process 1084 (sleep) with signal SIGKILL.
[email protected]: Failed with result 'signal'.

[email protected] has KillMode=mixed and no Restart=, so the job died with the manager, and the manager stayed failed. Every timer that user had stopped firing. Logging in as the user, a reboot or systemctl start [email protected] brings it back. A missed run of a timer with Persistent=true then runs on the next start.

Why writing to a dead pipe didn’t kill the jobs

The job in each test printed a line every second. When the process reading that pipe is dead, the write raises SIGPIPE, and SIGPIPE’s default action is to kill the writer. None of ours died that way, and the reason is who started them:

• systemd starts services with SIGPIPE ignored (IgnoreSIGPIPE=yes is the default), and children inherit an ignored signal. So under any unit that doesn’t turn that off, a job keeps running after its reader dies and gets an EPIPE error instead. (Debian’s cron.service sets IgnoreSIGPIPE=false, but cron’s CRON child keeps the pipe open, so it doesn’t matter there.)
• Python ignores SIGPIPE at startup, and supervisord starts programs with a plain fork and exec, so its programs ignore it too. That’s why supervisord’s printing program survived the kill even without systemd.

Under a scheduler that leaves SIGPIPE at its default, outside systemd, “did the job survive?” depends on whether it happens to print something. A job that’s quiet for twenty minutes and then prints its summary dies at minute twenty.

RunWisp: killed, restarted, recorded

RunWisp, our project, is an open-source cron replacement that ships as a single binary. It reads your crontab with runwisp import cron and runs each line as a task, and it keeps every run’s exit code and output in its own history. runwisp service install writes a unit with Restart=on-failure, RestartSec=5s and KillMode=mixed. Here’s the same test:

$ journalctl -u runwisp -o cat
[INFO] run started task=job run=01M6R7G351WR2B4XWHZ0D25F2A
runwisp.service: Main process exited, code=killed, status=9/KILL
runwisp.service: Killing process 1230 (sh) with signal SIGKILL.
runwisp.service: Killing process 1231 (job) with signal SIGKILL.
runwisp.service: Failed with result 'signal'.
runwisp.service: Scheduled restart job, restart counter is at 1.
[WARN] marked crashed runs from previous session count=1

$ runwisp logs 01M6R7G351WR2B4XWHZ0D25F2A
tick 1
...
tick 6
[WARN] run failed task=job run=01M6R7G351WR2B4XWHZ0D25F2A exit=-2 reason=crashed dur=11.235s

The job died with the daemon, which is the price of that unit setting: unlike cron, a crash costs you the run in progress. In return nothing is left running that the daemon doesn’t know about, the interrupted run is in the history as crashed with its output up to the kill, and crashed counts as a failure, so it alerts like one. The next run started on schedule. If the daemon had been down across a scheduled time, catch_up (default 1) would have run the latest missed one and recorded a missed run.

The limits. A crashed run isn’t resumed or re-run; a retry fires only if you set retry_attempts, and then the job has to be safe to run twice. It also doesn’t make a job’s side effects exactly-once: a job killed halfway through a transfer has transferred half.

Check your own box

The unit file tells you most of it:

systemctl show -p KillMode,Restart,RestartUSec,IgnoreSIGPIPE cron supervisor runwisp

• KillMode=process: running jobs outlive the scheduler. Fine for cron. For anything that restarts its programs on startup, like supervisord, it means duplicates.
• KillMode=mixed or control-group: running jobs die with the scheduler.
• Restart=no: one crash and the scheduler stays down. Check [email protected] too if you use user timers.

To see it happen, on a test VM or a container with systemd, not on a server you care about:

sudo kill -9 "$(systemctl show -p MainPID --value supervisor)"
systemd-cgls -u supervisor.service   # leftovers: processes with no supervisord above them
sleep 60; pgrep -af 'your-program'   # two PIDs per program means duplicates

And on a server that’s already had a crash, look for orphans. A job whose parent is PID 1 and that isn’t a daemon in its own right deserves a second look:

ps -eo pid,ppid,etime,cmd | awk '$2 == 1'
journalctl -b | grep -E 'Found left-over process|code=killed, status=9/KILL'

How we tested

Debian 13 (trixie) in a rootless Podman 5.7.0 container booted with systemd 257.13 as PID 1, on an Ubuntu host with kernel 7.0. Debian’s packages and their stock units: cron 3.0pl1-197 and supervisor 4.2.5-3. RunWisp 1.9.0 with the unit from runwisp service install. The job was a bash script that wrote a line per second to a file and, unless told to be quiet, to stdout, for 40 to 120 seconds. cron ran it every minute from /etc/cron.d, supervisord as a program with autorestart=false, systemd as a Type=oneshot service on an OnCalendar=*:*:00 timer (system and user), and RunWisp as a task with cron = "* * * * *". Each time we sent kill -9 to the unit’s MainPID a few seconds into the run. The “no systemd” rows ran in a plain Debian 13 container with Podman’s --init as PID 1 and the scheduler started from a shell. We didn’t test cronie, busybox crond or Supercronic, and other distributions’ units may set KillMode= and Restart= differently, which is the point of the systemctl show line above.