Job sftp-scheduled-supplier-fetch-complete-files · revised 11 October 2026
Make a scheduled supplier file fetch import only complete, new files
A scheduled job fetches the latest supplier file over SFTP, skips files still being uploaded, never imports the same file twice, and fails loudly on a changed host key, tested on a synthetic server.
You might be seeing
- A supplier file imported half-way because it was still uploading when the job ran
- The scheduled fetch has not run for days and nothing told anyone
No passwords, keys, card details or admin invites needed to start.
What usually happened
A scheduled job picks up whatever file is on the supplier's server at run time. It cannot tell a finished file from one still uploading, does not remember what it has already imported, trusts or silently ignores a changed server identity, and reports no failure when the file is absent or the connection fails.
Who it’s for: Operations manager or developer at a distributor whose nightly supplier fetch sometimes imports half a file, nothing, or yesterday's file again.
Usually starts when: Some mornings the stock import has a truncated file, runs on yesterday's data, imports twice, or stopped altogether after the supplier's server changed.
The result: Against a synthetic SFTP server you control, the scheduled fetch downloads only a complete file that matches the agreed completeness rule, imports each file once, refuses a server whose host key has changed, and raises a visible failure when no new file appears by the agreed time.
Check whether this job fits
Answer these without sharing keys, passwords or supplier details. Nothing is submitted unless you choose to contact us.
Checks you can run yourself
Check the last three runs
Look at the job's last three log entries: the file name, size and time of each import.
Look for: Two runs with the same file name and size, a file much smaller than the others, or a long gap between runs.
What you get
- A change to the fetch job and its tests against the synthetic server
- A run log format that shows fetched, skipped, repeated and failed files, and runs that exited because another run was in progress
- A note on the completeness rule, schedule, time zone and failure alerts agreed
- Steps to revert the change
Included
- One existing scheduled fetch from one supplier's SFTP location, up to one file pattern
- Reproduce the faults on a synthetic SFTP server with partial, repeated, missing and renamed files
- Add an agreed completeness rule, such as a file that stops changing size for a set time, a companion marker or manifest, or a final rename by the supplier
- Record each imported file's name, size and checksum so a repeat is skipped, take a lock so a second run that starts before the first has finished exits without importing, and fail visibly when no new file arrives or the host key changes
- Add tests against the synthetic server, including a second run that starts before the first has finished, and write down the schedule, time zone and overlap rule
Not included
- Holding or storing the supplier's SFTP credentials; you keep the secret and the key
- Asking the supplier to change how it uploads
- Connections to Google Merchant Center, which has its own fetch rules
- Moving the job to a new server or scheduler
- Processing the file's contents, which is the importer's job
How we know it’s done
Agreed with you before work starts. Each check produces evidence you keep.
On the synthetic server, a file still being written is not fetched, and the complete file is fetched and imported exactly once on the next run.
Evidence: Run logs showing the partial file skipped and the complete file imported, with sizes and checksums.
Running the job again with the same complete file present imports nothing and logs the file as already imported.
Evidence: Run log and the record of imported files.
With a second run started while the first is still fetching the same complete file, only one run fetches and imports it: the other exits without importing and logs that a run is already in progress, and the file appears once in the record of imported files.
Evidence: Both runs' logs with start times, and the record of imported files.
When the synthetic server's host key is changed, the job refuses to connect and raises the agreed failure alert, and no file is fetched.
Evidence: Run log and the alert.
When no new file appears by the agreed time, the job raises the agreed alert instead of finishing silently.
Evidence: Run log and the alert.
Sign-off. You inspect the test logs and sign off in writing. Payment follows sign-off; applying the change to the live job with your own credentials stays with your maintainer.
If it fails. If the agreed tests do not pass, you do not pay for this fixed scope. We hand over what we found and agree whether to stop or re-quote; no surprise work.
When it fits, and when we stop
It fits when
- The fetch job is a script or scheduled task you can share, and runs on a synthetic server without real credentials
- You know how the supplier names and uploads its files, or can ask
- A person on your side holds the supplier credentials and applies the change
We stop and tell you if
- The only way to test is with the supplier's real credentials or a live supplier server
- The supplier overwrites the file in place, with no marker, rename or manifest, and uploads pause for longer than any settle time you would accept, or files arrive more often than a settle time allows, so no completeness rule can be trusted: we show the evidence and stop
- The fetch runs inside a hosted tool that cannot be changed
What could go wrong
Before you merge, closing the change leaves the job as it was. After merge, your maintainer can revert the named commit. The record of imported files is a new file or table; removing it does not touch products already imported.
Scroll the table sideways to read it all.
| Risk | How we handle it |
|---|---|
| A size-stability rule imports a file that paused mid-upload. | The rule is agreed with the supplier, a rename or marker is preferred, and the tests include a paused upload. |
| A script is changed to accept any server key to make a failure go away. | A changed host key must fail visibly and be accepted by a person who verifies it with the supplier. |
| Credentials leak into a repository or log. | We never receive credentials, tests use a synthetic server and the review checks for secrets. |
An independent reviewer checks that no credentials appear in code or logs, that a changed host key stops the job rather than being accepted automatically, and that the completeness rule is the one agreed. Your maintainer reviews and applies the change.
How we deliver
We arrange the work and independent review, then show you the result against the agreed checks. You keep authority over your systems.
- Agree the job, the file pattern, the completeness rule, the schedule and the failure alerts in writing
- Stand up a synthetic SFTP server and reproduce partial, repeated, missing and renamed file cases
- Add the completeness check, the imported-file record and the host key and missing-file failures
- Test overlap and repeat handling on the schedule, including a second run starting before the first ends
- Independent review of the change, the logs and the credential handling
- Hand over the change, the log format and revert steps; your maintainer applies it with your own credentials
This is a one-off job, not emergency cover or a subscription. We confirm eligibility, the total price, a start window and a delivery date before you accept. Work starts only after agreed inputs, secure access and any needed permissions are in place. Hosting, supplier and platform charges are excluded unless the written quote includes them. An enquiry creates no charge or booking.
Need to keep it working?
Discuss a standing check on every scheduled fetch, with a monthly summary.
Ongoing work is separately scoped and quoted: no monitoring, response-time guarantee or automatic subscription is included in this job.
Explore an ongoing engineering lane, or mention the responsibility you need in your enquiry.
What you can check
This is a new service. We have not delivered this job for a client yet.
Other ways to get this done
- Ask the supplier to upload under a temporary name and rename the file when finished, then fetch only finished names. OpenSSH's sftp tool has a rename command, so the supplier can do this on a standard server. man.openbsd.org
- If the supplier can publish a small manifest with each file's size and checksum, comparing it is more reliable than waiting for the size to settle.
Questions
Do you need my SFTP password?
No. We test against a synthetic server with invented files. You apply the change with your own credentials.
What if the supplier cannot give a finished-file signal?
A size that stops changing for a set time can work but is weaker. We say so in the written rule and test a paused upload.
Does it cover the file's contents?
No. It makes sure only a complete, new file reaches the importer. Checks on the content are separate.
Send an enquiry
Send us
- How the supplier names its files and how often they appear, as text
- What went wrong and when, in one or two sentences, without credentials or hostnames you consider private
- How the job is scheduled today and where it runs. No keys, passwords, supplier addresses or code in the first enquiry
Later, once you agree
- The fetch script or task definition through an agreed company-controlled route, with no secrets in it
- The completeness rule agreed with the supplier, in writing
- A named person who holds the real credentials and applies the change
You keep the supplier relationship, the credentials and the host key records. We build and test against a synthetic SFTP server with invented files and return a change for your maintainer to review and apply with your own credentials. We never receive or store supplier credentials.
Email fallback: open your mail app
If website submission is unavailable, review and send the fallback email yourself. An email fallback is not a website receipt. Or write to hello@syntheticindustry.ai with “sftp-scheduled-supplier-fetch-complete-files” as the subject.