Scheduling restic or borg backups with alerts when they fail: systemd timer, cron or borgmatic

The backup “ran every night”. Then a disk died, and the newest snapshot was from March, because the repository password changed in April and the job has exited 12 every night since, into a crontab with no MAILTO. Most guides to scheduling restic or borg end at “add this to cron”. The part that matters is the next one: you hear about it when a backup fails, when it doesn’t run at all, and when it runs but quietly saves less than you think.

We tested restic 0.18.1 and 0.19.1 on systemd 259 and checked four setups against that: a systemd timer with OnFailure=, cron with a Healthchecks ping, resticprofile and borgmatic.

Short answer

On a box with systemd, run the backup from a systemd timer with Persistent=true, set OnFailure= on the service to a small unit that sends you the journal, and set TimeoutStartSec, because a oneshot service never times out on its own. Add a Healthchecks ping as well if you want to hear about it when the whole box is gone. On borg, use borgmatic: it runs create, prune, compact and check, and has hooks for Healthchecks, ntfy, Uptime Kuma and others built in. On restic, resticprofile gives you the same in one config file.

If you also want every run’s output and exit code in a web UI, with backup, prune and check as separate jobs and alerts on Slack, ntfy or email, RunWisp (an open-source cron replacement that ships as a single binary) runs restic or borg as plain tasks and catches up on a backup missed while it was down. Setup below.

“Nothing else to run” means no service besides the backup tool and the scheduler the OS already has. Healthchecks is a hosted service you can also self-host, and RunWisp is a daemon.

What counts as a failed backup

Before picking a scheduler, know what restic tells it. We made each case happen:

$ restic backup /srv           # one file chmod 000
error: open /srv/b: permission denied
Warning: at least one source file could not be read
exit 3

$ restic snapshots             # wrong RESTIC_PASSWORD
Fatal: wrong password or no key found
exit 12

$ restic snapshots             # repository path typo, or the NAS isn't mounted
Fatal: repository does not exist: unable to open config file
exit 10

$ restic forget --prune        # while a backup is running
unable to create lock in backend: repository is already locked by PID 1825129
exit 11

Exit 3 is the tricky one. The snapshot was saved and is usable, minus the files restic couldn’t read (on restic 0.19 and later also a source path that doesn’t exist). Every scheduler below treats it as a failure, which is what you want the first time: either fix the permissions or exclude the files on purpose. Don’t make 3 a success to silence it, or the night a whole directory turns unreadable looks the same as a good one. Codes 10 to 12 came with the restic 0.17 releases; older versions exit 1 for all three.

Borg has three: 0 for success, 1 for a warning (a file changed while it was read, a file it couldn’t open), 2 for an error, plus 128+N if a signal killed it. Borg 1.4 can return finer codes with BORG_EXIT_CODES=modern. A warning is non-zero, so cron wrappers and systemd count it as a failure.

1. systemd timer with OnFailure=

Three units: the backup, its timer, and a template that sends the alert.

# /etc/systemd/system/restic-backup.service
[Unit]
Description=restic backup
Wants=network-online.target
After=network-online.target
OnFailure=notify-failure@%n.service

[Service]
Type=oneshot
EnvironmentFile=/etc/restic/env
ExecStart=/usr/bin/restic backup --one-file-system --exclude-caches --retry-lock 30m /etc /home /srv
TimeoutStartSec=6h
Nice=10
IOSchedulingClass=idle

# /etc/systemd/system/restic-backup.timer
[Timer]
OnCalendar=*-*-* 02:30
Persistent=true
RandomizedDelaySec=10m

[Install]
WantedBy=timers.target

# /etc/systemd/system/[email protected]
[Unit]
Description=Failure alert for %i

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'journalctl -u %i -n 20 --no-pager -o cat | curl -fsS -m 10 -H "Title: %i failed on %H" --data-binary @- https://ntfy.sh/backups-7f3kq9'

/etc/restic/env holds RESTIC_REPOSITORY, RESTIC_PASSWORD_FILE and any cloud credentials, readable by root only. Then systemctl enable --now restic-backup.timer.

We ran a service that exited 3 through this. systemd logged status=3/NOTIMPLEMENTED (it names exit codes after the LSB init-script ones, so ignore the name), skipped any later ExecStart= lines, and started the template with %i set to restic-backup.service. The alert carried the last 20 journal lines, which can include the tail of the previous run. Swap the curl for mail or a Slack webhook as you like; the Arch wiki’s status_email_user@ unit is the same idea with email.

What this setup gets right:

• Persistent=true catches up. The timer stores when it last fired. If the box was off at 02:30, the backup runs once when the timer starts at the next boot. Without it, a laptop that sleeps every night never backs up.
• The output is in the journal. journalctl -u restic-backup shows every run, with timestamps.

What it doesn’t do on its own:

• A hung backup is never a failure. A Type=oneshot service has no start timeout: we checked, and TimeoutStartUSec is infinity. An SFTP backend that stops answering keeps the unit in “activating” forever, nothing fires, and the next night’s timer does nothing because the unit is still running. Set TimeoutStartSec to a few times your longest backup. systemd sends SIGTERM when it runs out, and restic removes its lock on SIGTERM.
• A dead box sends nothing. OnFailure= runs on the box that failed. If the box is off, the disk is gone or someone ran systemctl disable by accident, you hear nothing. Add a heartbeat (option 2) for that.
• A skipped run is silent. A unit with a Condition...= that isn’t met, such as borgmatic’s sample ConditionACPower=true on a laptop that’s on battery, is skipped and doesn’t count as failed.

RunWisp vs systemd timers compares timers with RunWisp in general.

2. cron with a heartbeat ping

Cron runs the job and does nothing else: no catch-up, no timeout, and the output goes to mail you probably don’t read. (Cron job failure alerts covers the general case.) What makes it workable is a heartbeat service like Healthchecks, which alerts you when a ping says the job failed and when no ping arrives on time.

# /etc/cron.d/restic
30 2 * * * root set -a; . /etc/restic/env; set +a; curl -fsS -m 10 --retry 5 -o /dev/null https://hc-ping.com/<uuid>/start; restic backup --one-file-system --exclude-caches /etc /home /srv > /var/log/restic.log 2>&1; curl -fsS -m 10 --retry 5 -o /dev/null --data-binary @/var/log/restic.log https://hc-ping.com/<uuid>/$?

• /start tells Healthchecks the backup started, so it can alert on a run that started and never finished, and shows how long each run took.
• /$? sends restic’s exit code: 0 is a success, anything else a failure. The body is the run’s output, and Healthchecks keeps the first 100 kB of it.
• set -a; . /etc/restic/env is there because cron starts the job with an almost empty environment. set -a exports every variable the file sets, so the same plain KEY=value file works for systemd’s EnvironmentFile= too.

Two common versions of this line are subtly wrong. restic backup ... && curl hc-ping.com/<uuid> only pings on success, so a failure shows up as a late ping, an hour or a day later depending on the grace time. restic backup ... | logger -t restic; curl .../$? sends the exit code of logger, which is 0.

A ping to a service outside the box is what tells you when the box itself is dead, because the alert comes from somewhere else. It’s worth adding to any of the others.

3. resticprofile

resticprofile wraps restic in a config file and writes the schedule for you, as systemd timers on Linux (with Persistent=true in its template), launchd on macOS or Task Scheduler on Windows:

# /etc/resticprofile/profiles.yaml
default:
    repository: "sftp:backup@nas:/srv/restic"
    password-file: "/etc/restic/password"
    run-after-fail: 'curl -fsS -m 10 -H "Title: restic $PROFILE_COMMAND failed" -d "$ERROR_MESSAGE" https://ntfy.sh/backups-7f3kq9'

    backup:
        source: ["/etc", "/home", "/srv"]
        one-file-system: true
        exclude-caches: true
        schedule: "*-*-* 02:30"
        schedule-permission: system

    forget:
        keep-daily: 7
        keep-weekly: 5
        keep-monthly: 12
        prune: true
        schedule: "Sun 04:00"
        schedule-permission: system
        schedule-lock-wait: 2h

run-after-fail gets $ERROR_MESSAGE, $ERROR_EXIT_CODE and $ERROR_STDERR, and send-after-fail does the same as an HTTP request without a shell. resticprofile schedule installs the timers. It fixes most of what’s missing from a hand-written timer: schedule-lock-wait makes a scheduled run wait for another one instead of failing, and it unlocks stale locks older than an hour by default. It is one more tool and config format between you and restic, and since the alert runs on the same box, the dead-box problem stays.

4. borgmatic

borgmatic does for borg what resticprofile does for restic, and goes further on monitoring. One run of borgmatic creates the archive, prunes, compacts and checks, and it has built-in hooks for Healthchecks, ntfy, Uptime Kuma, Cronitor, PagerDuty, Pushover, Apprise and others:

# /etc/borgmatic/config.yaml
source_directories:
    - /etc
    - /home
    - /srv
repositories:
    - path: ssh://backup@nas/./borg
encryption_passcommand: cat /etc/borgmatic/passphrase

keep_daily: 7
keep_weekly: 5
keep_monthly: 12

checks:
    - name: repository
      frequency: 2 weeks
    - name: archives
      frequency: 1 month

healthchecks:
    ping_url: https://hc-ping.com/<uuid>
    send_logs: true

commands:
    - after: error
      run:
          - curl -fsS -m 10 -d {error} https://ntfy.sh/backups-7f3kq9

The frequency on checks means you can run borgmatic daily and the slow checks happen only every two weeks or month; borgmatic remembers when each last ran. Its sample systemd timer uses Persistent=true, so missed nights catch up. The commands: hook syntax is borgmatic 2.0; older versions used on_error:.

Two things to know. By default borgmatic treats borg’s exit code 1 as a warning, not an error, so a file borg couldn’t read doesn’t fire the error hook; borg_exit_codes changes that per code. And its sample service has ConditionACPower=true, so on a laptop running on battery the backup is skipped without a word. Remove the line if that’s not what you want.

If you’re on borg, borgmatic is the answer, and nothing below beats its hooks.

Pruning, checking and the lock between them

A backup is three jobs on different schedules:

• backup: daily, or hourly for data that changes a lot.
• forget + prune: weekly. Prune rewrites pack files, takes an exclusive lock, and is the slowest and most I/O-heavy part. On borg, borg prune then borg compact (1.2 and newer free no space without it).
• check: weekly or monthly. restic check alone only checks the structure. Add --read-data-subset=$(date +%-m)/12 to read and decrypt a different twelfth of the data each month, so the whole repository is read once a year. On borg, borg check --verify-data does the full read.

They share one repository, so they collide. Every restic command takes a lock, and prune’s is exclusive. We ran forget --prune while a backup was writing, and it failed with exit 11 at once. --retry-lock 2h made it wait for the backup and succeed. Add it to prune and check, and a shorter one to the backup.

The trap is a backup that dies hard. We SIGKILLed a running backup, the way the OOM killer or a power cut would, and its lock stayed in the repository:

$ restic prune
unable to create lock in backend: repository is already locked by PID 1829344 on 137PC
lock was created at 2026-10-26 17:10:25 (32.677494256s ago)
the `unlock` command can be used to remove stale locks
exit 11

The PID no longer existed, and prune still refused, again 30 seconds later. Backups kept working, since their locks aren’t exclusive, so nothing looks wrong until the disk fills up from snapshots nobody prunes. restic unlock removed the three stale locks and prune ran. A SIGTERM is fine: we sent one to a running backup, restic printed signal terminated received, cleaning up, and no lock was left. This is why prune needs its own alert, not only the backup.

Testing the alert path

An alert you’ve never seen arrive is a guess. Break each setup on purpose once:

• systemd: point RESTIC_REPOSITORY at a path that doesn’t exist and run systemctl start restic-backup. You should get exit 10 in the alert.
• Healthchecks: curl https://hc-ping.com/<uuid>/fail, and pause the cron line for a night to see the “down” alert.
• borgmatic: borgmatic --verbosity 1 against a repository path that doesn’t exist.
• Restore: once a month, restic restore latest --target /tmp/restore-test --include /etc/hostname or a real file you’d miss. A backup you can’t restore from is the failure none of the above catch.

RunWisp: backup, prune and check as tasks

RunWisp (our project, so weigh accordingly) is an open-source cron replacement that ships as a single binary, with a web UI that records every run’s output, exit code and duration. It has no restic or borg integration. It runs the commands above as tasks, each with its own schedule, timeout and alerts:

# runwisp.toml
[defaults]
secrets_file = "/etc/restic/env"   # RESTIC_REPOSITORY, RESTIC_PASSWORD_FILE, keys

[tasks.restic-backup]
group      = "Backups"
cron       = "30 2 * * *"
run        = "restic backup --one-file-system --exclude-caches --retry-lock 30m /etc /home /srv"
on_overlap = "skip"
timeout    = "6h"
keep_for   = "90d"
notify     = ["phone"]

[tasks.restic-forget]
group      = "Backups"
cron       = "0 4 * * 0"
run        = "restic forget --keep-daily 7 --keep-weekly 5 --keep-monthly 12 --prune --retry-lock 2h"
on_overlap = "skip"
notify     = ["phone"]

[tasks.restic-check]
group      = "Backups"
cron       = "0 6 1 * *"
run        = "restic check --read-data-subset=$(date +%-m)/12 --retry-lock 2h"
on_overlap = "skip"
notify     = ["phone"]

[notifiers.phone]
type  = "ntfy"
topic = "backups-7f3kq9"

What that gives you, compared with the options above:

• Each run is kept with its output and exit code for 90 days, so “when did the backup last skip files” is a scroll through the history instead of a journalctl session.
• A missed backup runs when the daemon starts. catch_up = 1 is the default, so a 02:30 backup the box was off for runs once at startup, like Persistent=true. RunWisp also sends a missed-run alert for the gap, so you know it happened.
• A hung run times out. timeout = "6h" stops it with SIGTERM, which lets restic remove its lock, and the alert says it timed out.
• Secrets stay out of the logs. Values from secrets_file are masked in the output and the UI.
• Exit 3 can be a recorded non-failure, if skipped files are expected on that box: failures = ["-failed", "+1-2", "+4-255"] alerts on every other non-zero exit. We’d still rather exclude the files.

To test the alert path, use runwisp run --daemon restic-backup with a broken repository path; the notifications docs have a task that always fails for testing a channel. The nightly backup recipe does the same for pg_dump, with a daily restore test.

The limits, honestly:

• It knows nothing about restic or borg. Retention flags, check frequencies and locks are yours to get right, which resticprofile and borgmatic do for you. On borg, borgmatic’s hooks are better than anything here; you can still run borgmatic as a RunWisp task to get its history in the UI.
• The alert comes from the same box. If the machine is dead, RunWisp is too. Set [daemon] heartbeat_url to a Healthchecks ping URL (1.8 and later) and you hear about it when the daemon or the machine goes silent. It watches the daemon, not each backup, so keep the missed-run and failure alerts too.
• The boot catch-up may beat the network. A catch-up run at startup can start before a network mount or VPN is up, fail and alert you. That’s noise, but it’s not silent.
• It’s a daemon, where a timer is three unit files and no extra process.

Which one to use

• borg: borgmatic, with its Healthchecks or ntfy hook, and its systemd timer.
• restic, and you want one config file: resticprofile with run-after-fail.
• restic, and you’d rather write it yourself: a systemd timer with Persistent=true, a TimeoutStartSec, OnFailure= and --retry-lock on everything.
• cron only (no systemd, or a NAS): Healthchecks with /start and /$?, and accept that a missed night stays missed.
• Whatever you pick: a heartbeat from outside the box, prune and check on their own schedule with their own alert, and a test restore now and then.
• You want backup, prune and check side by side with their history and alerts, next to your other jobs: run them as RunWisp tasks. The quick start covers the daemon, and the code is on GitHub.