systemd service that will not stay up: restart rules, start limits and start types

The defaults restart nothing

A systemd service written without a restart setting is not restarted when it crashes. The Restart option defaults to no, and the delay before a restart, RestartSec, defaults to 100 milliseconds when you do use it. The manual lists the conditions: on-success, on-failure, on-abnormal, on-watchdog, on-abort and always, and recommends on-failure for long-running services. on-failure restarts after a non-zero exit, a signal that is not clean or a timeout; always restarts whatever the cause. Pick on purpose and write the reason in a comment, because the next person will need to know whether a clean exit is supposed to stay stopped.

• on-failure: restart after a crash, stay stopped after a clean exit.
• always: restart even after a clean exit, except an explicit stop.
• Add a delay of a few seconds so a crashing process does not spin.

Start limits turn a loop into a failed unit

Restarting forever is not safe. systemd rate-limits starts with a burst count within an interval, both set in the Unit section. When the limit is hit, the unit that uses Restart stops being restarted automatically and sits in a failed state until you start it by hand or clear it. The limit counts every kind of start, including manual ones, not only automatic restarts. That behaviour is good: a service that cannot start should become visible instead of burning CPU and filling the journal. Check the numbers, because the defaults are five starts within ten seconds: with a restart delay of about three seconds or more, a crash loop never reaches that limit and restarts for ever. The example below sets a longer interval for that reason. Test it with a deliberately failing copy of the unit, and learn the command that clears the failed state.

• Start-limit settings live in the [Unit] section.
• The reset-failed command clears the counter when you start a unit by hand.
• A manual systemctl stop is not restarted by the Restart rule.

Start type decides when a start counts as success

With the default simple type systemd counts the unit as started immediately after forking, before the binary has been executed, so a missing binary or user does not fail the start command. The exec type waits until the binary has run, which reports those mistakes; it needs systemd 240 or later. The notify type waits until the program tells systemd it is ready. Forking is for programs that daemonise and is discouraged. Choosing the type that matches how the program behaves matters because other units ordered after this one wait for it to be considered started. If your program takes a while to be ready, simple will report success too early.

• exec is a safer default than simple when the program does not signal readiness.
• notify needs support in the program.
• oneshot suits a task that runs and exits.

Run as a user, in a directory, with sensible confinement

Set the user and working directory explicitly so the process does not run as root in whatever directory systemd chooses. Optional confinement settings, such as preventing the process from gaining new privileges or making most of the filesystem read-only, can reduce the damage a bug does, but they can also stop the program writing a file it needs. Add them one at a time, test the program after each, and keep only those that it tolerates. Do not put secret values in the unit file, since unit files are readable by other users; the secrets guide covers the alternatives. Order the unit after the network only if the process needs the network up when it starts: use Wants=network-online.target together with After=network-online.target, because After= alone only orders the unit if the target happens to be started, and a service that merely listens should not pull the target in at all.

• User and working directory are explicit.
• Confinement settings are tested one by one.
• No secrets in unit files.
• Network ordering needs Wants= and After= together, and only when the process needs the network at start.

If the matrix is wider than the box, scroll horizontally to read every column. Keyboard: focus the matrix and use Left/Right.

[Unit]
Description=Worker for the app
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
Type=exec
User=appworker
WorkingDirectory=/srv/appworker
ExecStart=/srv/appworker/run
Restart=on-failure
RestartSec=5
NoNewPrivileges=yes

[Install]
WantedBy=multi-user.target

Prove it by breaking it

Reading the file proves little. Kill the process with a hard signal through systemctl, so only the unit's main process is signalled, and watch a new one appear within the restart delay. Stop the unit and confirm it stays stopped. Reboot and confirm it starts without a login. Run a copy that exits at once and confirm the start limit stops it and the failed state can be cleared. The fixed systemd job performs and records all four, sets the user, directory and journal logging, and does not fix whatever makes your program crash.