Job workflow-api-polling-no-gaps-no-repeats · revised 11 October 2026
Poll an API on a schedule without missing a record or acting on one twice
A scheduled poll of one documented API processes every new or changed record once in effect, including a late arrival and a kill at every step, proved on a synthetic copy.
You might be seeing
- A record created during a poll never appears in your system, although it exists at the source
- A customer is emailed twice, or a task is created twice, for one source record
- After a crash or a timeout, the next run either starts from the beginning or skips ahead
No passwords, keys, card details or admin invites needed to start.
What usually happened
A polling job that asks for everything since its last run time can lose records that arrive late, or that sit beyond the first page of results. It can also repeat records when it re-reads an overlap or when it crashes between doing the work and saving its position. Search endpoints may take a moment to show new records and cap the total results per query. Recording a record as handled before acting on it removes the repeat but creates a gap: a crash between the two leaves the record marked handled and never acted on. Across two systems nothing makes acting and recording a single step, so the job acts first, in a way that is safe to repeat for the same record key, then records the key, then moves the checkpoint. A repeated action then leaves one effect.
Who it’s for: A founder, operations lead or technical contact who runs a scheduled script or function that pulls records from another system and acts on them.
Usually starts when: Some records are never processed, others are processed twice, or both happen after a run was interrupted or the other system was slow.
The result: Against a synthetic copy of the API, the job processes each of 120 synthetic records over three pages once in effect, picks up a late-arriving record whose timestamp is older than the previous run but inside the overlap window, and loses none and duplicates none when it is killed before any one of its steps (reading a page, acting, recording the key, saving the checkpoint) and then restarted.
Check whether this job fits
Answer these without sending keys or customer data. Nothing is submitted unless you choose to contact us.
Checks you can run yourself
Note how the job finds new records
Read the part of the job that decides what counts as new. Write down whether it uses the last run time, the highest key seen or something else, and whether it reads every page. Do not change it.
Look for: Whether the position is saved before or after the work is done, whether the key is recorded before or after the action, and whether pages beyond the first are read.
What you get
- The job's source code in a repository you control, with a readme and a configuration list that names no secret values
- The synthetic API stand-in and the automated test suite, including the kill-at-every-step test
- A run log from the kill, late-arrival and repeat tests
- A short design note on the checkpoint, the overlap size chosen and why, the key and retention rule, and what the job cannot promise
Included
- One scheduled job against one HTTPS API whose documentation describes a way to list records by modification time or stable order, with paging
- A persisted checkpoint, an overlap window and complete paging, with the order act (safe to repeat for the same key), then record the key, then advance the checkpoint
- The action made idempotent by record key (for example create-or-update on the key, or a destination that accepts an idempotency key), or the action's result and the handled key written to one store in one transaction
- A durable record of handled record keys, with the agreed rule for a changed record and a stated retention that covers the overlap window and the longest gap between runs
- A synthetic stand-in for the API and automated tests for paging, late arrival, repeat, kill-at-every-step and error cases
Not included
- Deploying or scheduling the job in production, which stays with your team
- Subscribing to webhooks or building a receiver, which is a different outcome
- More than one API or one action, a historical backfill beyond one agreed window, or buying higher rate limits
- Monitoring or alerting after hand-over, unless you buy a separate service
- An action that is one-way and cannot be made safe to repeat by key: after a crash it forces a choice between a possible repeat and a possible loss, and this fixed job does not make that choice for you
- Real customer data in tests, or calls to the real API with real credentials by us
How we know it’s done
Agreed with you before work starts. Each check produces evidence you keep.
Against the synthetic stand-in holding 120 records over three pages, one run reads every page and leaves exactly one effect at the destination for each of the 120 keys.
Evidence: Test output with the page count, the effect count per key at the destination (120 keys, one effect each) and no key with more than one.
A record created after the poll started, with a timestamp older than the previous run but inside the overlap window, is processed once on the next run.
Evidence: Test output for the late-arrival case showing one effect for that key.
When the job is killed before each of its steps in turn (reading a page, acting on a record, recording its key, saving the checkpoint) and then restarted, every one of the 120 keys ends with exactly one effect: none missing, none doubled.
Evidence: The kill-test log for each stop point with the effect count per key before and after the restart.
A record returned in two consecutive polls is acted on once, and an error partway through a run leaves the checkpoint behind the first unrecorded record.
Evidence: Test output for the repeat and error cases.
The design note states the key rule, the retention of handled keys, the overlap chosen and the limits: a record appearing later than the overlap, or an action that cannot be made safe to repeat, is not covered.
Evidence: The design note.
Sign-off. You run the automated tests and review the evidence, then sign off before payment. Your team deploys and schedules the job and checks one live run under your own keys.
If it fails. If the agreed tests do not pass, you do not pay for this fixed scope. We hand over what we found and agree whether to stop or re-quote; no surprise work.
When it fits, and when we stop
It fits when
- The API documents a modified-since filter or stable ordering, a unique record key and paging, and its terms allow polling at the interval you need
- You run the job on your own hosting and have somewhere to store a checkpoint and the handled keys
- The action taken for each record can be made safe to repeat for the same record key, or its result and the handled key can be written to one store in one transaction
We stop and tell you if
- The API offers no usable ordering, filter or unique key, so complete polling cannot be shown
- The action is one-way and cannot be made safe to repeat by key or written in one transaction with the handled key. A crash between acting and recording then forces a choice between a possible repeat and a possible loss; this fixed job does not cover it, and we quote a different scope with that choice written down
- The only test route would call the real API with live customer data
What could go wrong
Nothing is deployed by us. Your team deploys the job; to reverse it, schedule the previous job again and remove the new configuration. The checkpoint and handled-key store are separate from your source system, so removing them does not change source records.
Scroll the table sideways to read it all.
| Risk | How we handle it |
|---|---|
| The overlap window is too short and a slow-to-appear record is still missed. | Choose the overlap from the API's documented indexing delay and your agreed tolerance, state it in the note, and test a late arrival inside the overlap. A record that appears later than the overlap is not caught, and the note says so. |
| The key is recorded before the action, so a crash between the two leaves a record marked handled that was never acted on. | The job acts first, in a way that is safe to repeat for the same key, then records the key. The kill test stops the job before each step and checks that no key is missing at the destination. |
| The action runs twice after a crash between acting and recording. | The action leaves one effect for one key (create-or-update on the key, an idempotency key, or one transaction with the handled key). The kill test counts effects per key, not calls. |
| Handled keys are pruned too early and an old record read again is acted on a second time. | The note states the retention, which covers the overlap window and the longest gap between runs, and the test replays a record from inside that window. |
| The poll exceeds the API's rate limit or result cap. | Read the documented limits first, page within them, and stop and log rather than silently truncating. |
An independent reviewer checks the redacted before and after evidence, that every test used synthetic data, that nothing outside the agreed scope changed, and whether your authorised owner can safely reverse the change.
How we deliver
We arrange the work and independent review, then show you the result against the agreed checks. You keep authority over your systems.
- Agree the API, the record key, the modified-since or ordering field, the action and how it is made safe to repeat, the overlap window, the key retention and the checks that define done in writing
- Build a synthetic API stand-in from the documented response shape, with three pages of 120 records and the ability to inject a late arrival, a repeat and an error
- Record the starting behaviour of the existing job, or of a naive poll, against the stand-in
- Implement the checkpoint, overlap and paging in the order act (safe to repeat for the same key), record the key, then advance the checkpoint only after the whole run is recorded
- Run the automated tests and the kill test: stop the job before each step in turn (reading a page, acting, recording the key, saving the checkpoint), restart it, and count the effects per key at the destination; capture the logs
- Independent review of the checkpoint logic and of how the action is made safe to repeat, then hand over the repository and notes; your team deploys and schedules the job
This is a one-off job, not emergency cover or a subscription. We confirm eligibility, the total price, a start window and a delivery date before you accept. Work starts only after agreed inputs, secure access, any licences and necessary permissions are in place. Platform, hosting and supplier charges are excluded unless the written quote includes them. No charge or booking is created by an enquiry.
Need to keep it working?
If the job is business-critical, discuss alerting on a missed run as a separate recurring service.
Ongoing work is separately scoped and quoted: no monitoring, response-time guarantee or automatic subscription is included in this job.
Explore an ongoing engineering lane, or mention the responsibility you need in your enquiry.
What you can check
This is a new service. We have not delivered this job for a client yet.
Other ways to get this done
- If the other system can send webhooks, that may be simpler than polling; see the webhook receiver outcome. It brings its own repeat and loss risks, which that outcome addresses.
- Zapier documents how polling triggers use unique IDs and newest-first ordering. If the job can live in a Zap, a built-in trigger may suffice. docs.zapier.com
Questions
Why not just ask for records since the last run time?
A record can appear late, sit on a later page, or be read twice around an overlap. A single timestamp loses or repeats such records. The job keeps an overlap and a record of handled keys.
Can you guarantee nothing is ever missed or done twice?
Not in general. We show the behaviour against a synthetic stand-in built from the API's documentation, at every stop point, and state the overlap and the assumptions. The result is one effect per record key when the action is safe to repeat for that key; where it is not, no polling design avoids a choice between a possible repeat and a possible loss. The live API's behaviour is yours to confirm.
Is this for Zapier or Make?
No. Those tools manage their own polling triggers. This job is for a script or function that you run yourself.
Send an enquiry
Send us
- A link to the API's public documentation for listing records, including its paging and rate-limit sections
- What the job does with each record, in a sentence, and how often it should run
- Where the job runs today, and what has gone wrong: missing records, repeated records or both
- Do not send API keys, customer data or source code in the first enquiry
Later, once you agree
- A repository or code-export route you control, with no production credentials
- Sample response shapes from the documentation or redacted by you, from which we build synthetic records
- A description of the storage available for the checkpoint and handled keys
You keep the live accounts, production keys and customer records. We work only on a copy or a test route with synthetic data, through a company-controlled handoff agreed before any access; a person owns Synthetic Industry and remains accountable. Your authorised account holder approves and carries out the live change.
Email fallback: open your mail app
If website submission is unavailable, review and send the fallback email yourself. An email fallback is not a website receipt. Or write to hello@syntheticindustry.ai with “workflow-api-polling-no-gaps-no-repeats” as the subject.