systemd timer email on failure: an OnFailure= unit that mails the failed run's log

A systemd timer fails the same way cron does: quietly. systemctl --failed lists the unit, the journal has the error, and nobody looks at either until the backup is needed. systemd has the hook, OnFailure=, and leaves the email part to you. Most examples mail the last 20 journal lines, which for a short job can be the previous run’s.

We built the setup below on systemd 259 and broke it on purpose: a non-zero exit, a timeout, SIGKILL, a skipped condition, a failed assert, Restart=on-failure, and two units failing at once into one handler.

Short answer

Put OnFailure=unit-fail-mail@%n.service in the [Unit] section of each service your timers start. Make [email protected] a oneshot that runs a short script: it pipes the failed run’s journal (journalctl -u "$unit" --invocation="$MONITOR_INVOCATION_ID") into sendmail -t. Install msmtp-mta so sendmail exists, and set TimeoutStartSec on every job, or a hung run never fails. To cover every job without editing each unit, give them a common prefix and put the OnFailure= line in a prefix drop-in, /etc/systemd/system/job-.service.d/.

If you’d rather keep every run’s output and exit code in a web UI and get one mail per stretch of failures instead of one per run, RunWisp (an open-source cron replacement that ships as a single binary) imports .timer units with runwisp import systemd and sends failures through the same sendmail. Setup below.

What made OnFailure= run in our tests:

1. Something that sends mail: msmtp as sendmail

A fresh VPS can’t send mail. You don’t need a mail server for alerts, only a sendmail command that hands the message to an SMTP account you already have. msmtp does that, and msmtp-mta installs it as /usr/sbin/sendmail:

apt install msmtp msmtp-mta      # Arch: pacman -S msmtp msmtp-mta

# /etc/msmtprc
defaults
auth           on
tls            on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
syslog         on

account        default
host           smtp.example.com
port           587
from           [email protected]
user           [email protected]
passwordeval   "cat /etc/msmtp-password"

chmod 600 /etc/msmtprc /etc/msmtp-password, since the handler runs as root. Gmail wants an app password here, not your normal one. Check it works before wiring anything to it:

printf 'Subject: test from %s\n\nIt works.\n' "$(hostname)" | sendmail [email protected]

One thing to know: msmtp has no queue. If the SMTP server is down at the moment the job fails, msmtp exits non-zero and the mail is gone. msmtpq, which ships with msmtp, queues and retries if that matters to you.

2. The handler: a template unit and a script

The script gets the failed unit’s name as its argument. systemd sets the rest in its environment:

#!/bin/sh
# /usr/local/bin/unit-fail-mail: called by [email protected]
unit="$1"
{
    printf 'To: %s\n' "${MAILTO:-root}"
    printf 'Subject: [%s] %s failed (%s)\n' "$(hostname)" "$unit" "$MONITOR_SERVICE_RESULT"
    printf 'Auto-Submitted: auto-generated\n\n'
    printf 'Result: %s, %s %s\n\n' "$MONITOR_SERVICE_RESULT" "$MONITOR_EXIT_CODE" "$MONITOR_EXIT_STATUS"
    journalctl -u "$unit" --invocation="$MONITOR_INVOCATION_ID" -o short-iso --no-pager
} | sendmail -t

# /etc/systemd/system/[email protected]
[Unit]
Description=Mail the failure of %i

[Service]
Type=oneshot
Environment=MAILTO=[email protected]
ExecStart=/usr/local/bin/unit-fail-mail %i

And on the job:

# /etc/systemd/system/job-backup.service
[Unit]
Description=Nightly backup
OnFailure=unit-fail-mail@%n.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
TimeoutStartSec=2h

# /etc/systemd/system/job-backup.timer
[Timer]
OnCalendar=*-*-* 03:30
Persistent=true

[Install]
WantedBy=timers.target

chmod +x the script, systemctl daemon-reload, then systemctl enable --now job-backup.timer. This is the mail a failing run produced in our test, with the unit and host renamed:

To: [email protected]
Subject: [web1] job-backup.service failed (exit-code)
Auto-Submitted: auto-generated

Result: exit-code, exited 2

2026-10-27T11:13:57+01:00 web1 systemd[1]: Starting job-backup.service...
2026-10-27T11:13:57+01:00 web1 backup.sh[72275]: pg_dump: error: connection to server failed
2026-10-27T11:13:57+01:00 web1 systemd[1]: job-backup.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
2026-10-27T11:13:57+01:00 web1 systemd[1]: job-backup.service: Failed with result 'exit-code'.
2026-10-27T11:13:57+01:00 web1 systemd[1]: Failed to start job-backup.service.
2026-10-27T11:13:57+01:00 web1 systemd[1]: job-backup.service: Triggering OnFailure= dependencies.

systemd names exit codes after the old LSB init-script meanings, so ignore INVALIDARGUMENT. The parts that matter:

• The @ template and %n. One handler file serves every job, and each failure starts its own instance, [email protected]. We also tried a plain shared OnFailure=mail.service with two units failing at the same moment: it ran once, and the $MONITOR_* variables were empty, so the mail couldn’t say which unit failed or how. The systemd.exec man page says as much and recommends the template.
• --invocation= limits the journal to that one run, including systemd’s own lines about it. journalctl -u unit -n 20, the usual answer, can mix in the previous run.
• Older systemd. --invocation= arrived in systemd 257 (Debian 13, Ubuntu 25.04 and newer). On 251 to 256, which covers Debian 12, Ubuntu 24.04 and RHEL 9, use journalctl _SYSTEMD_INVOCATION_ID="$MONITOR_INVOCATION_ID" --no-pager: it prints the run’s own output, without systemd’s status lines. Below 251 (Ubuntu 22.04) there are no $MONITOR_* variables, and journalctl -u "$unit" -n 30 is the best you can do.

3. Every job at once: a prefix drop-in

Adding OnFailure= to each unit by hand is how one gets forgotten. A drop-in adds it for you. A drop-in directory named after a prefix of a unit name, ending in a dash, applies to every unit whose name starts with it:

# /etc/systemd/system/job-.service.d/10-onfailure.conf
[Unit]
OnFailure=unit-fail-mail@%n.service

After systemctl daemon-reload, every job-*.service has the hook, and %n still expands to each unit’s own name. Check one:

$ systemctl show -p OnFailure job-backup.service
[email protected]

You can go wider with /etc/systemd/system/service.d/, which applies to every service on the machine. That includes the ones your distribution ships, which then mail you when they fail at boot, and the handler itself, which would then call itself when it fails. systemd tries to detect that loop, and systemd.unit(5) shows how to break it by masking the drop-in for the handler. The prefix is less to think about: the handler’s name doesn’t start with job-.

4. What doesn’t send a mail, and what sends too many

• A hung run. A Type=oneshot service has no start timeout, so a backup stuck on a dead NFS mount stays “activating” forever, never fails, and the timer’s next run does nothing because the unit is still running. TimeoutStartSec sends SIGTERM when it runs out, the unit fails with timeout, and you get the mail. How to time out a cron job covers picking the number.
• A skipped run. A Condition...= that isn’t met skips the run, and the unit stays inactive. We expected Assert...= to count as a failure, since systemctl start printed Assertion failed and exited 1. The unit still ended up inactive, and the handler didn’t run. If “the NAS isn’t mounted” should wake you up, check it in the service: ExecStartPre=/usr/bin/mountpoint -q /mnt/nas fails the unit and fires OnFailure=.
• Restart=on-failure mails every attempt. The handler runs after each failed attempt, and once more when the start limit ends the restarts: three runs, four mails. RestartMode=direct (systemd 254 and newer) skips the handler on attempts systemd is going to restart, and we got one mail when it gave up. Combine it with a start limit the job can actually hit. With RestartSec=3 and the default limit of 5 starts in 10 seconds, our test service restarted 8 times in 25 seconds, never gave up, and never sent anything.
• A job that fails every run mails every run. A timer every 5 minutes on a broken job is 288 mails a day, and systemd has nothing to group them. Fix the job fast, or have the script skip sending if it mailed about the same unit in the last hour.
• The handler can fail too. If msmtp can’t reach the server, unit-fail-mail@... ends up failed and nothing tells you. systemctl --failed shows it. Glance at it now and then.
• A dead box mails nothing. Everything above runs on the machine that failed. If it’s off, or someone ran systemctl disable on the timer, there’s no failure to report. A heartbeat service that alerts when a ping doesn’t arrive covers that; cron job failure alerts shows the setup, and it works the same from a systemd service.

Testing it

Break a job once and read the mail you get:

systemctl edit job-backup.service      # add: [Service] Environment=PGHOST=nowhere.invalid
systemctl start job-backup.service
systemctl revert job-backup.service

Or keep a unit that always fails, job-mailtest.service with ExecStart=/bin/false, and start it when you change the mail setup. Running systemctl start [email protected] yourself only tests the transport: there’s no failed run behind it, so the $MONITOR_* variables are empty.

The restic and borg guide uses the same handler pattern to send to ntfy instead of email. RunWisp vs systemd timers compares the two schedulers in general.

RunWisp: import the timers, keep the sendmail

RunWisp (our project, so weigh accordingly) is an open-source cron replacement that ships as a single binary, with a web UI that keeps every run’s output, exit code and duration. Since 1.8 it can read your timer units:

runwisp import systemd /etc/systemd/system/job-backup.timer -o runwisp.toml

That turns the timer and the service it starts into one task, with OnCalendar= as a cron line. The import doesn’t carry over OnFailure= or TimeoutStartSec, so add timeout and notify yourself. The sendmail notifier runs the same sendmail -t -i that msmtp-mta installed:

# runwisp.toml
[tasks.job-backup]
run     = "/usr/local/bin/backup.sh"
cron    = "30 3 * * *"
timeout = "2h"
notify  = ["mail"]

[notifiers.mail]
type = "sendmail"
from = "[email protected]"
to   = ["[email protected]"]

Then systemctl disable --now job-backup.timer, so the job doesn’t run twice. Compared with the handler above:

• Repeat failures are grouped. A job that fails every 5 minutes sends the first failure, a check-in every few repeats and a summary, about one mail every ten minutes with the defaults. A recovered mail says when it works again.
• More things count as a failure. A timeout, a run killed by a signal, and a run missed while the daemon was down all alert, without per-unit setup. A missed run also runs once when the daemon starts, like Persistent=true.
• Each run stays browsable with its output and exit code, for as long as you keep it, instead of until the journal rotates.
• A failed delivery shows up. If sendmail fails, the Web UI’s bell gets a notify.delivery_failed warning with msmtp’s error, instead of a failed unit nobody checks.
• The daemon pings out. [daemon] heartbeat_url pings a service like Healthchecks every minute, so a dead box or a stopped daemon gets reported from somewhere else.

The limits, honestly:

• The mail is short. It holds the exit code, the duration, the last three lines of output and a link to the full run in the Web UI (when external_url is set). The script above mails the whole run’s log, which is better if you read alerts on a phone with no VPN.
• The import needs a human. Sandboxing like ProtectSystem= is dropped, extra ExecStart lines are flagged, and each note is listed in the migration docs.
• It’s a daemon, where the setup in this post is two unit files and a script, using only what systemd already does.
• Keep OnFailure= for services you don’t move. RunWisp only alerts on what it runs, not on the distribution’s own units.

Which one to use

• A few timers, and you want the whole log in your inbox: the template handler, a prefix drop-in, msmtp, and TimeoutStartSec on every job.
• Restart= on a job: add RestartMode=direct and a start limit it can reach, or expect one mail per attempt.
• Jobs that should stop when a mount is missing: ExecStartPre=mountpoint -q, not Condition...= or Assert...=.
• Whatever you pick: a heartbeat from outside the box for the jobs that matter, and one test failure after every change to the mail setup.
• Many jobs, and you want their history, timeouts and grouped alerts in one place: move the timers to RunWisp with runwisp import systemd. The quick start covers the daemon, and the code is on GitHub.