Standing service codebase-assistant-citation-and-refusal-tests-rerun-every-month · revised 12 October 2026
Standing service
Have your codebase assistant's citations and refusals re-tested every month
Each month the assistant's test set is re-run on a copy, a sample of its real answers is checked against the code, new files are scanned for secrets and you get a report. Nothing is changed for you.
This starts a conversation by email. Nothing is charged, and nothing is monitored, until we have agreed scope, access and terms with you in writing.
The responsibility you hand over
An assistant tested at launch drifts as the code changes, the index goes stale and the model behind it is updated. Nobody reads the question log. A refusal that nobody re-tests may stop working.
Who it’s for: Engineering lead whose staff use a codebase assistant and who wants proof it still cites correctly and refuses what it should.
Usually starts when: The code moved on, the model vendor changed a version, or someone asked it a question it should have refused.
The result: Every month the 120-question set is re-run on a copy against the current index, a random sample of 30 logged answers is checked against the code by script and by your engineer, the repository is re-scanned for new secrets, and you receive a report listing anything that changed.
What stays true, and what we do about it
No response-time guarantee is published for this new service. A date for each monthly report is agreed in writing before it starts.
Hours are agreed in writing before the service starts. It runs once a month and is not staffed round the clock.
What must remain true
- The report checks every re-run and sampled citation against the indexed commit and flags missing lines or mismatched quotes
- The report lists unanswerable, secret-seeking and hostile-case results and flags any missed refusal or shown canary
- The report states index age against the agreed limit and flags any indexed secret
What we watch
- The month's question log export
- Read-only code at the month-end commit
- The test set, the canary file and the exclusion list
When something happens
Scroll the table sideways to read it all.
| When | What we do |
|---|---|
| A month closes and the exports are available. | We re-run the set, check the sample, scan for secrets and send the report. |
| You change the model, the index settings or the exclusion list. | We re-run the full set on a copy before you rely on the change and list every answer that changed. |
| A large refactor lands. | We re-check the index age and re-run the cited-line checks on the affected files. |
We do on our own
- Re-run the test set on a copy
- Check the sampled answers by script and by reading
- Scan the branch for secrets
- Write the report
We ask you first
- Any request to read staff names or the full log
- Any change to a rule, setting or page in your own systems
- Any wider access than the exports named here
- Contacting your customers or anyone outside your team
We escalate to you when
- An answer showed a secret or a credential
- A shown answer has a citation that does not exist
- A refusal test fails
- A secret is found in the indexed branch
How you know it held. Your engineer selects 10 sampled answers, holds their marks until our report is frozen and agrees on at least 9; any unflagged false citation, secret leak or missed refusal fails the check.
How we keep it true
This service is never finished. Each month's report shows whether the things it watches stayed true, and it continues until you end it.
Give your team a codebase question assistant that cites file and line and refuses the rest Job Optional
An assistant in your own account answers staff questions about one repository. A script checks every file and line cited, and secret-seeking and off-topic questions are refused. You hold the switch.
Bought separately, each time it is needed; not part of the monthly fee
The one-off job; buy it first if you do not yet have one.
What is included, and what is not
- The monthly report
- The test re-run results
- The sample table with the seed
- The secret scan and index-age result
Included
- One assistant on one repository, built under the one-off job on this site or one you can show us
- A re-run of the test set, the trick questions, the secret-seeking questions and the hostile prompts on a copy
- A random sample of 30 logged answers from the month, with the seed recorded, checked for citation and correctness
- A new secret scan of the indexed branch and an index-age check
- A monthly report with counts and anything that needs a decision
Not included
- Changing the assistant, the index or the exclusion list
- Holding your keys or reading answers beyond the sample
- Writing or changing code
- Round-the-clock cover or a response-time guarantee
- Any promise about time saved
How we know it’s done
Agreed with you before work starts. Each check produces evidence you keep.
Your engineer selects 10 of the 30 sampled answers and keeps their marks from us until our report is frozen; at least 9 marks agree. One failed citation, leaked secret or missed refusal that the report fails to flag fails the whole test; this is a report smoke test, not a future-accuracy rate.
Evidence: The seed, the 10 answers and the engineer's marks beside ours.
The report states the sample size, the count of shown answers with a failed citation, the refusal test results and what it cannot show.
Evidence: The monthly report.
The test set is re-run on a copy each month and any answer that changed is listed first.
Evidence: The re-run results.
Nothing in your assistant, index or repository has been changed by us.
Evidence: Your cloud and repository audit logs for the month; we hold no key.
Sign-off. You read each report and re-run the acceptance checks with the files and steps supplied. A month counts as delivered only when those checks pass.
If it fails. If we cannot do a month because an export is missing or has changed, we say so and do not guess. If the service is not working for you, you can end it at the end of any month.
When it fits, and when we stop
It fits when
- An assistant with a question log you can export and at least 30 answers in the month
- A named engineer who rates the sample
- Read-only access to the indexed branch
We stop and tell you if
- The log is missing or has fewer than 30 answers: we stop and agree a reduced test-only scope and price in writing
- A secret is found and not removed within the month
- Nobody can change the assistant or its exclusions
What could go wrong
The service reads exports and reports, so there is nothing to undo. Ending it leaves your systems exactly as they are; you keep every report.
Scroll the table sideways to read it all.
| Risk | How we handle it |
|---|---|
| A sample of 30 misses a rare error. | The report states the sample size and that a sample is a tripwire. The script still checks every citation in the re-run. |
| The question log carries sensitive questions. | Staff names are replaced by codes, and we read the sample only. |
| The report is read as proof the assistant is safe. | The report says what was checked this month and that it describes this month only. |
Each report is checked by a separate automated reviewer before hand-over. Your named engineer re-checks a sample of answers and decides every change to the assistant. No human supervisor is included unless your agreement names one, and at launch the work is largely automated.
Stays with a person
- You or your named person change the assistant, index and exclusions
- You hold the off switch and the keys
Access we would need
- A monthly export of the question log with staff names replaced by codes
- Read-only access to the indexed branch
- A copy of the assistant for the re-runs
Questions
Do you change the assistant?
No. We name the answer or setting behind each issue. You or your person make every change.
Do you tell me how much time it saves?
No. We never state time or cost savings.
Send an enquiry
Send us
- Which assistant and repository
- Roughly how many questions staff ask a month
- No logins, keys, customer data or payment details in the first enquiry
Later, once you agree
- A monthly export of the question log, with staff names replaced by codes
- Read-only code at the month-end commit
- Your list of structural code changes during the month
You keep the repository, assistant, cloud account and every key; we receive the agreed code and log exports and work on a copy, never changing the assistant or code. Before sharing, the terms name any model provider, country, retention and training status, and the deletion date for our copies. You run any key-dependent replay in your own account and send us its results.
Email fallback: open your mail app
If website submission is unavailable, review and send the fallback email yourself. An email fallback is not a website receipt. Or write to hello@syntheticindustry.ai with “codebase-assistant-citation-and-refusal-tests-rerun-every-month” as the subject.