Pass-mark arithmetic: what 38 out of 40 does and does not show about an AI-drafted answer set

What this example is

Every number below is computed, not measured on a client. The method is the exact binomial: it finds the lowest true pass rate for which seeing k passes in n items would still happen at least 5 times in 100, with each bound rounded down. The script on this page prints the table below, and was run on Node.js v24 on 12 October 2026.

The table

A pass mark is only meaningful with the number of test items beside it. The same share of passes supports a lower guaranteed rate when the test is small.

• 20 of 20: the true rate is at least 86.0 percent.
• 30 of 30: at least 90.4 percent.
• 28 of 30: at least 80.4 percent.
• 27 of 30: at least 76.1 percent.
• 38 of 40: at least 85.0 percent.
• 36 of 40: at least 78.5 percent.
• 45 of 50: at least 80.1 percent.
• 56 of 60: at least 85.3 percent.
• 96 of 100: at least 91.0 percent.
• 90 of 100: at least 83.6 percent.
• 144 of 150: at least 92.2 percent.

How to read it

A pass at 38 of 40 on questions you picked, with your answers held back from us until the test run, shows with 95 percent confidence that the true rate for questions like them is at least 85 percent. It does not say any one future answer is right, and it says nothing about questions unlike the sample. That is why every drafted answer also needs a source a person can open, and why a named person signs.

When a pass mark means nothing

These are our rules of thumb: treat the result as meaningless if the builder saw the test items, if two people on your side would not label them the same way, if one class dominates so a model that always says it would score well, or if the pass mark was set after seeing the results. A single run of a system that gives different answers each time is also not enough. If we planted the test cases ourselves, the test is a smoke test: it can show a failure, but it supports no rate on your real data.

The script

Save it as bound.mjs and run it with Node.js. It needs no packages.

If the matrix is wider than the box, scroll horizontally to read every column. Keyboard: focus the matrix and use Left/Right.

// Lower 95% bound on a true pass rate, given k passes in n test items (exact binomial, one-sided), rounded down to one decimal.
const logFact = n => { let s = 0; for (let i = 2; i <= n; i++) s += Math.log(i); return s; };
const tail = (n, k, p) => { // chance of k or more passes if the true rate is p
  let s = 0;
  for (let i = k; i <= n; i++) s += Math.exp(logFact(n) - logFact(i) - logFact(n - i) + i * Math.log(p) + (n - i) * Math.log(1 - p));
  return s;
};
const lowerBound = (n, k) => {
  let lo = 0, hi = 1;
  for (let i = 0; i < 60; i++) { const mid = (lo + hi) / 2; if (tail(n, k, mid) > 0.05) hi = mid; else lo = mid; }
  return lo;
};
const down = x => (Math.floor(x * 1000) / 10).toFixed(1);
for (const [k, n] of [[20, 20], [30, 30], [28, 30], [27, 30], [38, 40], [36, 40], [45, 50], [56, 60], [96, 100], [90, 100], [144, 150]])
  console.log(`${k} of ${n}: at least ${down(lowerBound(n, k))}%`);