A disk filling with logs: three sources, three different limits

Find what is actually growing

Before touching any setting, measure. Look at the largest directories on the one filesystem, not the whole machine, and note the sizes of the container data directory, the system log directory and the journal. Logs are only the cause if they are the largest growing consumers. If the large entries are uploads, a database or backups, rotating logs will not help and deleting those files can lose data. Send numbers, not file contents, when you ask for help; log lines can include personal data.

• du limited to one filesystem avoids mounted volumes and virtual filesystems.
• Compare two measurements a day apart to see the growth rate.
• Treat unknown large files as unknown; do not delete to find out.

Container logs: the default does not rotate, and the safe way to change it

Docker's default json-file driver writes each container's output to a file and, unless configured, never rotates it, because the size limit defaults to unlimited. Set max-size and max-file, and note that max-file only has an effect when max-size is also set. The lower-risk route is per service in the Compose file, as in the example: recreating that one service applies the options and leaves the others running. The alternative is the daemon default in daemon.json, where every option value must be a string, and it carries two costs. A change to daemon.json needs a Docker restart, and restarting the daemon stops every running container unless live-restore is enabled; the live-restore setting itself can be applied with a reload instead of a restart, so enable and check it first, and run the restart in a window you control. And the changed default still affects only containers created afterwards, so existing containers keep their old settings until they are recreated. Recreating a container discards whatever it wrote to its own writable layer, so check that each one keeps its data in a named volume or a bind mount first; Compose keeps those mounted volumes when it recreates a service. The page for the local driver recommends it for most cases because it rotates by default and stores logs more efficiently.

• Prefer the per-service logging block in the Compose file; recreate only that service.
• Before touching daemon.json, check live-restore, keep a copy of the old file and validate the new one with dockerd --validate. Roll back by putting the old file back and restarting again.
• Check each container's effective setting with the inspect command, and confirm every container that was running is running afterwards.
• Before recreating a container, check that its data is in a named volume or bind mount, not only inside the container.

If the matrix is wider than the box, scroll horizontally to read every column. Keyboard: focus the matrix and use Left/Right.

services:
  app:
    image: registry.yourbusiness.co.uk/app:1.4.2
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

The journal has its own budget

The systemd journal is bounded by size options of its own. In the version of the manual used for this guide, the journal may use up to 10% of the filesystem and journald aims to leave 15% free, each figure capped at four gigabytes, and a maximum retention time is disabled unless you set it. Distributions ship different versions and settings, so read the manual and the effective values on your host rather than assuming. Put your limits in a drop-in file under the journald configuration directory so package updates do not overwrite them, and check usage with the journal's own disk usage command.

• Choose a maximum size you can afford, not the default percentage of a large disk.
• Use a drop-in file; do not edit the packaged main file.
• Check whether the journal is stored on disk at all on your system.

Application files: rotate, and signal the application

Files an application writes itself are rotated by logrotate rules. A rule names the files, how many rotated copies to keep, whether to compress, and what to do afterwards. The safe pattern is to rename the file and tell the application to reopen it in a post-rotation script. The alternative, copytruncate, copies the file and empties the original in place for programs that cannot reopen a log; the manual warns that some logging data might be lost between the copy and the truncation. Use explicit file paths, not wide patterns, so a rule cannot reach uploads or a database directory.

• Prefer a reopen signal over copytruncate.
• Use the debug option to see what a rule would do before running it for real.
• Keep the number of rotated files to what your retention target allows.

Test with a synthetic log, then check tomorrow

Prove each limit with a made-up file that grows past its threshold, and watch it rotate or cap. Then check the real disk-use figure the next day. A rule that has never fired is untested. The fixed log-rotation job does exactly this for up to five application files plus the container and journal sources, recreates the containers whose settings changed, deletes only what you approve one item at a time, and records the budget it sized the limits against. It does not decide how long you must keep logs for legal reasons; you do.