Job logging-rotation-stops-disk-filling-one-server · revised 11 October 2026
Stop logs filling the disk: rotation and size limits on one server
Container, journal and application logs on one Linux server get size limits and rotation, so the disk stops filling. A growth test shows the limits working, and nothing outside the logs is deleted.
You might be seeing
- The disk fills again a few weeks after someone deleted old log files
- The directory that holds container data is the largest thing on the server
- Nobody can say how long logs are kept or how large they may grow
No passwords, keys, card details or admin invites needed to start.
What usually happened
Logs reach the disk through three different routes, each with its own limit or none. Docker's default json-file driver does no rotation unless it is configured. A per-service logging option in a Compose file takes effect when that one service is recreated. Changing the daemon default in daemon.json needs a Docker restart, which stops every running container unless live-restore is enabled, and even then reaches only containers created afterwards. The systemd journal has its own size limits. Files written by applications need logrotate rules, and a rule that truncates a file in place can lose lines. Deleting old files only resets the clock.
Who it’s for: A founder or developer running their own Linux server whose disk keeps filling up and who suspects logs, but has not set limits on any of them.
Usually starts when: A disk-usage warning, a service that failed to write, or a clean-up that worked for a few weeks and then the disk filled again.
The result: Each log source on one server that the agreed inventory names has a size limit and a retention rule, a growth test shows each rule rotating or capping a synthetic log, and the disk-use figure is checked again a day later. No file outside the named log locations is changed or deleted.
Check whether this job fits
Four checks that decide whether this fixed job fits. Your answers stay on this page unless you choose to email them.
Checks you can run yourself
Find the largest consumers
Run on the server. Both commands only read. The second is limited to one filesystem and may take a minute.
df -h; sudo du -xh /var --max-depth=2 2>/dev/null | sort -h | tail -15Look for: Large entries in the Docker data directory (container data and logs), the system log directory and the journal directory. Send the numbers, not the contents.
See the Docker log setting
Run on the server. It prints the default driver and whether live-restore is on, and then the driver of one named container. It changes nothing.
docker info | grep -iE "logging driver|live restore"; docker inspect CONTAINER_NAME | grep -A6 LogConfigLook for: json-file means size limits are not set unless someone configured them. A max-size line under LogConfig means a limit exists for that container. Live Restore Enabled: false means restarting the Docker daemon stops every container.
What you get
- The inventory of log sources with current and new limits, and for each container where its data lives
- The configuration files, with the dry-run and growth-test output
- A next-day disk-use check and the steps to revert each change
Included
- One Linux server and an inventory of what writes logs: container logs, the journal and up to five application log files
- Docker log options set per service in the Compose file, which recreates only that service, as the default route; changing the daemon default in daemon.json only where you ask for it, with a daemon restart in an approved window
- The journal size limits, and logrotate rules for the named files
- A growth test for each source using a synthetic log, run with a dry run first
- A short note of what is kept, for how long, and how to read older logs
Not included
- Recovering a server that is already full: that is the disk-full repair job
- Sending logs to an outside service, structured logging or alert rules
- Deleting data, uploads, database files or backups
- Resizing disks or changing the hosting plan
- Any promise that logs are kept for legal or audit purposes
How we know it’s done
Agreed with you before work starts. Each check produces evidence you keep.
A synthetic log in each named source grows past its limit in the test and is rotated or capped, and the rotated file count never exceeds the agreed number.
Evidence: File listings before and after the test for each source.
Every container named in the inventory shows the new log driver options in its effective settings after it was recreated.
Evidence: The inspect output for each container.
docker inspect CONTAINER_NAME | grep -A6 LogConfigAfter the window, every container that was running before is running, each recreated container still has its named volumes and bind mounts, and if the daemon default changed the daemon configuration passed validation and shows the new default.
Evidence: The container list and the mounts of each recreated container before and after, and the daemon validation and settings output.
docker info | grep -iE "logging driver|live restore"The journal size limit is in force and the journal disk usage is below it.
Evidence: The journal disk-usage figure and the effective configuration.
systemd-analyze cat-config systemd/journald.conf; journalctl --disk-usageThe dry run of the log rotation tool lists only the named files, and the real run on the synthetic file changes only that file.
Evidence: Dry-run output and the file list after the real run.
sudo logrotate -d YOUR_ROTATION_FILEDisk use of the log locations measured the next day is within the budget derived from the retention you named.
Evidence: Usage figures at hand-over and 24 hours later, with the budget calculation.
Sign-off. You sign off when every named source has a tested limit, the containers are recreated with their data intact and the next-day figure is inside the budget. Payment follows sign-off.
If it fails. If a limit cannot be made to work on a source, we name the source and the reason and you do not pay for that part of the fixed scope. If logs turn out not to be the cause of the growth, we say so with the figures and stop.
When it fits, and when we stop
It fits when
- The server has free space to work with, or the logs have already been cleared safely by your administrator
- You can say how long logs must be kept, even roughly
- You can recreate the containers whose log settings change, in an agreed window, and each of them keeps its data in a named volume or a bind mount, not only inside the container
- If the Docker daemon default must change, you can accept a daemon restart in that window, with live-restore enabled first where your setup allows it
- The log files are plain files that rotate safely, or the application can be told to reopen its log
We stop and tell you if
- The disk is still full and nothing disposable has been identified, so deletion would be guesswork
- The largest consumer is customer data, uploads, a database or a backup, not logs
- The logs are the only copy of records you are required to keep
- A container holds data that matters only in its own writable layer, and nobody can move it to a volume first
- The daemon default would have to change but no window for a daemon restart can be agreed: only the per-service route is then offered
- Signs of intrusion appear in the logs
What could go wrong
Each setting is a separate file or option, so removing it restores the earlier behaviour: take the logging block out of the Compose file and recreate that service, or put the saved daemon.json back and restart the Docker daemon again in a window. A container recreated by Compose keeps its named volumes and bind mounts, but anything it kept only in its own writable layer is gone, which is why that is checked first. Log files already rotated and compressed remain readable. Deleted old logs are not restored, which is why deletions are itemised and approved first.
Scroll the table sideways to read it all.
| Risk | How we handle it |
|---|---|
| Restarting the Docker daemon to apply a changed daemon default stops every running container unless live-restore is on. | Use the per-service Compose options as the default route. If the daemon default must change, enable live-restore first (it reloads without a restart), validate the new file, restart in the approved window and check that every container that was running is running afterwards. |
| Recreating a container discards its writable layer, so data kept only there is lost. | Record where each container keeps its data before the window, stop if anything important lives only in the container, and recreate through Compose so named volumes and bind mounts are kept. |
| A rotation rule that truncates a file in place loses lines written during the copy. | Prefer telling the application to reopen its log. Use truncation only where it cannot, and state the small loss window in the handover. |
| New container log limits do not apply to containers that already exist. | List every existing container, recreate those that matter in the window, and verify each one's effective setting. |
| A broad path pattern in a rule matches files that are not logs. | Use explicit paths, run each rule in debug mode first, and have the reviewer read the list of files matched. |
An independent reviewer checks that every rule matches only the named log files, that no rule can reach uploads, database files or backups, and that any in-place truncation is justified.
How we deliver
We arrange the work and independent review, then show you the result against the agreed checks. You keep authority over your systems.
- Agree the log sources, retention targets, where each container keeps its data and the restart window in writing
- Measure growth per source and write the limits that fit the retention you named
- Test each rule with a synthetic log on a copy, starting with a dry run, and check any daemon.json change with the daemon's own validation before it is used
- Independent review of every rule that deletes, truncates or compresses a file, and of what a daemon restart or a container recreation would affect
- You approve the window. Add the per-service logging options and recreate only those services; change the daemon default only if agreed, enabling live-restore first and keeping the old daemon.json beside the new one
- Check the disk figure and the container list straight after (every container that was running is running) and again the next day
This is a one-off job, not emergency cover or a subscription. We confirm eligibility, the total price, a start window and a delivery date before you accept. Work starts only after agreed inputs, secure access, any licences and necessary permissions are in place. Hosting, platform and supplier charges are excluded unless the written quote includes them. No charge or booking is created by an enquiry.
Need to keep it working?
To have the disk and log health reviewed every month, ask about the monthly server patch and review service. This job sets the limits once.
Ongoing work is separately scoped and quoted: no monitoring, response-time guarantee or automatic subscription is included in this job.
Explore an ongoing engineering lane, or mention the responsibility you need in your enquiry.
What you can check
This is a new service. We have not delivered this job for a client yet.
Other ways to get this done
- Docker documents the log-opts for size and file count on the json-file driver, which a technical team can apply itself. docs.docker.com
- Docker recommends the local driver for most cases because it rotates by default; its configuration page also explains setting the daemon default. docs.docker.com
- If the disk is already full and services are failing, ask your host for a snapshot and temporary capacity before deleting anything.
Questions
Can you just delete the old logs?
Not as the fix. Deleting resets the clock and the disk fills again. Any deletion is listed and approved item by item, and the limit is what stops the growth.
Will my existing containers be covered?
Only once they are recreated. Docker applies new log options to newly created containers, so the job lists each existing one and recreates those you approve, one service at a time through Compose. Changing the daemon default needs a Docker restart, which stops containers unless live-restore is on, so it is done only if you ask for it and only in your window.
How long will logs be kept?
For the period you choose. We size the limits to that and show you the figures. We do not advise on legal retention requirements.
Do you ship logs to a service?
No. Shipping and alerting are separate work.
Send an enquiry
Send us
- The output of the disk-usage commands in the self-check, with names that identify customers removed
- Which services write logs: containers, the journal, files
- How long you need to keep logs
- Do not send log contents, server logins or customer data in the first enquiry
Later, once you agree
- A read-only usage report, or a time-limited account that you create and revoke
- The names and paths of the application logs and which user owns them
- For each container, whether its data is in a named volume, a bind mount or only inside the container
- A restart window for affected containers (and for the Docker daemon, if its default changes) and a person who can approve each deletion, if any
- A company-controlled secure handoff agreed before access: no live passwords, keys, private code or customer records by ordinary email.
You own the server and its logs. We read usage reports and configuration, write and test the rules on a copy of the log layout with synthetic files, and hand you the files. You apply them, or you create a time-limited account that you revoke afterwards, agreed in writing. We delete nothing from the live server except what you approve one item at a time.
Email fallback: open your mail app
If website submission is unavailable, review and send the fallback email yourself. An email fallback is not a website receipt. Or write to hello@syntheticindustry.ai with “logging-rotation-stops-disk-filling-one-server” as the subject.