Synthetic IndustryGet a price

Case study · Data and spreadsheets

A supplier import held together by scripts was rebuilt, so that a bad file is stopped and explained.

A supplier's daily file had corrupted the catalogue more than once. We built a pipeline that checks every file and holds a bad one with reasons.

Who
A distributor with a small operations team and one technical lead, importing a daily product file from its main supplier.
The problem
The import was a tangle of scripts and spreadsheet steps. It had corrupted data more than once, and only one person understood it.
What we did
Wrote down every stage, rebuilt each as a tested step, and joined them into a pipeline that releases a good file or holds a bad one with reasons.
The result
Every daily file is checked, compared with the previous one, and either released for import or held with a report whose counts add up.
Time
Six weeks.
Get a price for something like thisThe service behind it

What was going on.

Each morning the supplier's product file arrived, and a chain of scripts and manual steps turned it into updates on the company's store. It had grown by accretion: downloading, decoding, reading, category rules, picture checks and publishing lived in different scripts and in one person's head.

It had failed or corrupted data more than once. Each new fault was fixed in one place and broke another. Adding a second supplier meant copying the same fragile steps again.

What we found.

The trouble was not any one script. It was that nowhere in the chain was a bad file stopped and explained. A file with a changed column, a broken character or a thousand missing prices went straight through to the store.

So we started by writing down each stage: what goes in, what comes out, what is checked and what happens to a file or a row that fails. Then we rebuilt each stage as a small tested step, with a set of made-up test files for every kind of fault the company had seen.

The technical detail

The stages are fetch, decode, parse, category mapping, picture checks where the feed has pictures, a comparison with the previous file, and a decision to hold the file with reasons or release it. Each stage has a written specification and tests against made-up files for its kind of fault. A run produces a hold-or-release report whose counts reconcile: rows read equal rows released plus rows held plus rows rejected.

What we did.

How the work ranSteps in order: 1 Agree the stages: Scripts, a sample, the stages and the price; 2 Write it down: Each stage's input, output and checks; 3 Build and test: Each stage on made-up faulty files; 4 Join and run: A real sample, start to finish; 5 Review each stage: And the whole pipeline, independently; 6 You approve it: Each stage, then handover with a run book. You decide at step 1 and step 6.1 Agree the stagesScripts, a sample, thestages and the price2 Write it downEach stage's input,output and checks3 Build and testEach stage on made-upfaulty files4 Join and runA real sample, start tofinish5 Review each stageAnd the whole pipeline,independently6 You approve itEach stage, thenhandover with a runbookYou decide here How the work ranSteps in order: 1 Agree the stages: Scripts, a sample, the stages and the price; 2 Write it down: Each stage's input, output and checks; 3 Build and test: Each stage on made-up faulty files; 4 Join and run: A real sample, start to finish; 5 Review each stage: And the whole pipeline, independently; 6 You approve it: Each stage, then handover with a run book. You decide at step 1 and step 6.1 Agree the stagesScripts, a sample, the stages and the price2 Write it downEach stage's input, output and checks3 Build and testEach stage on made-up faulty files4 Join and runA real sample, start to finish5 Review each stageAnd the whole pipeline, independently6 You approve itEach stage, then handover with a run bookYou decide here
The steps in order. A box with a coloured border is where the client decided.

How we checked it.

These had to pass before we asked for sign-off.

  • For each kind of fault in the agreed test set, the pipeline holds or rejects the file or row, with a reason, and releases the clean files.
  • An end-to-end run on the agreed sample produces a report whose counts reconcile.
  • The company approved every stage, and its technical lead ran the pipeline from the run book without our help.

What they have now.

The company received the specification, the pipeline code and its tests in a repository it owns, the test files, the report from the sample run, and a run book covering scheduling, alerts and what to do when a file is held. Its technical lead ran the pipeline from the run book alone before the company approved it.

The first real bad file the pipeline held was a delivery with a column moved. It was stopped with a reason before it reached the store.

What we did not do.

  • We did not switch the company's live import to the pipeline. That stayed with its team.
  • We never held the supplier's credentials or any live access.
  • We did not clean or correct product data already in the store.
  • This covers one supplier feed and one target import. A second supplier is quoted separately.

The service behind this.

A supplier import you can trust is one that can say no, and say why.

Project

Rebuild a supplier feed pipeline

Replace one fragile supplier-to-store import with a tested pipeline: fetch, check, compare with the last file, then hold or release, with category and image stages if your feed needs them.

From $5,850. The payment schedule is set in the written proposal. Nothing is charged until you have agreed it.

Get a priceSee what is included

More case studies.

All case studies

Get a price for something like this

You are asking about: Rebuild a supplier feed pipeline.

Add detail (optional)

A web address anyone can open, with no login in it. No code or logs.

We reply by email. Sending this emails your request to our team; nothing is kept in a website database. Please do not send passwords, keys or customer records. Privacy notice.

Sending this is not an order. We check we can help, then agree what is included, safe access and the price in writing before any work. Prefer email? hello@syntheticindustry.ai.