Job core-metrics-defined-and-tested-for-ai-queries · revised 12 October 2026
Define our key numbers once so everyone, and any AI tool, gets the same answer
Up to ten metrics are defined as code on one schema, with a 30-question set that reconciles to figures you sign off. Five planted questions outside the definitions must be refused.
You might be seeing
- Two teams quote different numbers for the same metric
- A pilot that lets staff ask questions of the data gave answers nobody trusted
No passwords, keys, card details or admin invites needed to start.
What usually happened
When 'revenue' means three things in three places, a person or an AI tool picks one at random. In one vendor test of 11 questions, a tool writing queries straight from the schema gave more wrong answers than one answering through defined metrics. Defining them is a business decision, and it has to be written down and tested.
Who it’s for: Analytics lead or head of operations at a tech company with conflicting numbers, or a failed 'ask the data' pilot.
Usually starts when: Two dashboards show different revenue, or a pilot that let staff ask the warehouse questions gave wrong answers.
The result: Up to ten metrics are defined as code (semantic-layer files or tested views) on one schema, and a 30-question set returns figures that reconcile to control totals you sign off. In 5 planted tests, questions outside the definitions return an error or refusal, not a figure. A plain query-writing baseline is reported beside them, not promised.
Check whether this job fits
Answer these without sending files or logins. Nothing is submitted unless you choose to contact us.
What you get
- The pull request with definitions and tests
- The glossary
- The 30-question set with expected values
- The benchmark script
- The results table with the baseline
Included
- Up to ten metrics on one schema, as semantic-layer files or tested views; an entry version covers three
- A pull request with definitions and tests, and a glossary
- A 30-question set with expected values: 20 for tuning and 10 that your named person writes and keeps from us until the test run
- A benchmark script and a results table including a plain text-to-query baseline
- Work on a read-only clone with personal columns masked
Not included
- Deciding what a metric should mean: stakeholders sign each definition
- Access to your live warehouse or production data
- Dashboards or reports
- Questions across many hops of joins, which a semantic layer may not answer
- Personal data
How we know it’s done
Agreed with you before work starts. Each check produces evidence you keep.
Each defined metric reconciles to the control total you supply, within the tolerance you set in writing.
Evidence: The reconciliation table.
All 10 questions your named person kept back until the test run return the expected values through the definitions. A wrong figure on any of them fails the whole test. This is a small test (10 of 10 shows the true rate is at least 74% with 95% confidence) and the report says so.
Evidence: The question set and the results beside expected values.
5 out-of-scope questions, which your named person adds and we do not see beforehand, each return an error or refusal, not a figure. One figure returned for any of them fails the whole test.
Evidence: The five questions and the replies.
A plain baseline (the same questions answered straight from raw tables) is run on the same set; both results are reported, and neither is promised for future questions.
Evidence: The results table.
On a second snapshot with changed inputs, each metric moves by an independently computed amount.
Evidence: The snapshot changes and the independent amounts.
Sign-off. Your named person signs the control totals and the results, and your team reviews and merges the pull request. Payment follows sign-off.
If it fails. If the held-back test is not passed, you do not pay for this fixed scope. You keep the results, the failures and the test set, and we agree whether to stop or re-quote. No surprise work.
When it fits, and when we stop
It fits when
- A dbt project or SQL views on one schema
- A read-only clone with personal columns masked, reached by a secure route after agreement
- Stakeholders have agreed each metric definition in writing before the build
- A named person signs the control totals
- Your named person writes 10 of the 30 questions, with expected values, and keeps them from us until the test run
We stop and tell you if
- Stakeholders dispute the definitions: that is a decision, not engineering work
- Only a live production warehouse is available
- Personal columns cannot be masked
- You want the tool to answer anything
What could go wrong
The job reads copies you send and writes files. It changes nothing in your systems, so there is nothing to revert. Our copies of your files are deleted when the job ends.
Scroll the table sideways to read it all.
| Risk | How we handle it |
|---|---|
| A definition encodes the wrong business meaning. | Stakeholders sign each definition before the build. We do not decide meaning. |
| A question outside the definitions gets a guessed answer. | 5 out-of-scope questions you add, unseen by us, must all return an error or refusal. |
| Warehouse access exposes personal data. | Work is on a read-only clone with personal columns masked, reached by a secure route after agreement. |
A separate automated reviewer checks the definitions, the tests and the reconciliation before hand-over and does not mark the held-back test. A named person on your side labels the test set and signs off. No person on our side is included unless your agreement names one, and at launch the work is largely automated.
How we deliver
We arrange the work and independent review, then show you the result against the agreed checks. You keep authority over your systems.
- Agree the metrics, the control totals, the question set and the severe errors
- Receive the masked clone and the written definitions
- Your named person writes 10 questions and keeps them from us
- Write the definitions as code with tests and a glossary
- Run the 20 tuning questions, fix gaps, then the 10 held-back questions
- Independent review of the definitions, tests and reconciliation
- Hand over the pull request and results; delete our copies
This is a one-off job, not emergency cover or a subscription. We confirm eligibility, the price, a start window and a delivery date before you accept, and work starts only after inputs, a secure handoff route and permissions are in place. Model usage up to a cap agreed in writing is included, and an enquiry creates no charge or booking.
Need to keep it working?
Want the definitions kept tested as your schema changes? Ask about a quoted monthly check.
Ongoing work is separately scoped and quoted: no monitoring, response-time guarantee or automatic subscription is included in this job.
Explore an ongoing managed lane, or mention the responsibility you need in your enquiry.
What you can check
This is a new service. We have not delivered this job for a client yet.
Other ways to get this done
- If you already use dbt, dbt Copilot can draft definitions and tests. This job adds the signed-off question set, the reconciliation and the baseline comparison. docs.getdbt.com
- For three metrics in a small team, writing them yourself and agreeing the totals in a meeting may be enough.
Questions
Does this make 'ask the data' work?
It makes answers through defined metrics checkable. In our test, five questions outside the definitions are refused, not guessed; we cannot promise the same for every future question.
Who decides what a metric means?
Your stakeholders. We write down and test what they agree.
Do you need my live warehouse?
No. A read-only clone with personal columns masked is enough.
Send an enquiry
Send us
- The metrics you want defined, in plain words
- Which tool holds the data (dbt, plain SQL, a warehouse) and the table count
- No real records, personal data or credentials in the first enquiry
Later, once you agree
- A read-only clone with personal columns masked, by an agreed secure route
- Written definitions the stakeholders agreed
- Control totals you sign
- A written cap on model usage
You keep your warehouse and every definition. We work on a read-only clone with personal columns masked, return a pull request you review and merge, and delete our copies when the job ends. We never write to your production data.
Email fallback: open your mail app
If website submission is unavailable, review and send the fallback email yourself. An email fallback is not a website receipt. Or write to hello@syntheticindustry.ai with “core-metrics-defined-and-tested-for-ai-queries” as the subject.