A timed-out create may have worked: making retried writes safe

Unknown outcome is the real problem

When a create call times out, three things may have happened: the provider never received it, it received and applied it, or it applied it and the response was lost. The client sees the same timeout in all three. AWS's engineering article states the consequence plainly: an API with side effects is not safe to retry unless it provides idempotency, meaning a repeat has the same effect as one call. Reads are generally safe; creations may not be.

• A timeout is not a failure; it is an unknown.
• A second attempt in the unknown case is a gamble with duplicates.
• HTTP's own rules allow automatic retry only for certain methods and conditions.

Technique one: let the provider recognise the repeat

The best case is that the provider supports a client-supplied identifier. Google Calendar lets you set the event ID at creation, and its guide says this prevents duplicates if a call fails partway: the retry names the same event. Some services accept an idempotency key on create calls; where documented, send the same key on every attempt of the same operation. Generate the identifier from the operation, once, before the first attempt, and store it.

• The identifier must be stable across attempts and unique per intended record.
• Read the provider's documentation for how long the key is remembered.
• Test: the fake or sandbox must show one record after two sends.

Technique two: look before you create, with care

If there is no identifier, you can search for the record before retrying: Pipedrive's person search, for example, supports an exact match on chosen fields such as email. This reduces duplicates but does not remove them. Two retries running at once can both find nothing and both create, and the page read does not describe duplicate handling on creation. Use lookup together with a single worker per operation, or accept that a flagged conflict is possible and design a review step.

• Search on a field the operation controls.
• If more than one match appears, flag it; do not guess.
• Search calls may cost more of a rate budget than a plain read.

Technique three: your own ledger, and the case where nothing works

Record each operation in your database before the call, with a status of pending, then set it to done when you see a success. After an unknown outcome, the ledger tells you what you attempted and when, so a person or a reconciliation job can check the provider. If you cannot make a write safe, do not retry it automatically: mark it unknown, tell a person and show them what to check. That is slower than a blind retry and avoids the second invoice.

• Never delete a ledger row; mark it.
• Reconcile unknowns against the provider on a schedule you choose.
• Say in your agreement which writes may be retried and which may not.

How the paid outcome is accepted

The retry outcome for one client includes a test that a write timing out after the fake provider recorded it results in one record, not two. The CRM push and the calendar sync outcomes carry the same check for their own records. Prices are untested, from a fixed £295 for the retry job, and payment follows your sign-off.