PX-bench measures product experience: what the product an agent ships is actually like to use, form and function together. It drops an agent inside a complete, opinionated application, asks it to add a feature the way a product team would, and scores the result across eight categories. Every run produces a number, a composite across the eight.
A number invites comparison. If one agent scores 0.72 and another scores 0.69, the natural read is that the first did better. But with one run each, a narrow gap may say more about the particular build each agent happened to produce than about how the agents perform on average.
Why repeated runs differ
We ran one agent on one task ten times under the same conditions, giving it a fresh attempt each time. The resulting builds scored anywhere from 0.66 to 0.79. That spread is the run-to-run variation of the eval cell.
Two things move it. The scoring layer leans on language-model judges, and even handed identical finished work they wobble a little. The agent is the bigger source: asked to build the same feature twice, it builds genuinely different things — an empty state here, a keyboard shortcut skipped there — and all of it moves the composite.
- Instrument noise is the scorers' share, how much the number moves when the work under it is held fixed.
- Total variation is how much the score moves across complete reruns under the same conditions, including differences in what the agent builds.
Total variation includes instrument noise, and whatever it carries beyond that traces back to the agent.
The reason we measure it
Measuring the noise floor proves useful in more than one way.
First, it makes comparisons defensible. If a score drifts by some amount with nothing changed, then a gap between two agents smaller than that drift is noise dressed up as signal.
Second, it helps determine how many runs each result requires. The results are published as a grid, where a cell is one agent on one task, and total noise decides how many times each cell runs. Averaging N runs shrinks the spread of the average by √N. Say the noise is 0.05: two single runs cannot resolve a gap under 0.1, while four runs of each agent halve that floor to 0.05. We pick the smallest N that puts the floor below the gaps we need to read, and that averaged floor, not the single-run one, is what we print beside the numbers.
Measuring the floor
We measure the two sources separately, because a single spread cannot say which one moved it.
- For instrument noise, we freeze one agent's finished build and re-score only that, several times. Anything that moves is the instrument, since the work beneath it is byte-identical.
- For total variation, we run the whole eval end to end several times under the same conditions. That captures everything the instrument does plus everything the agent did differently from one build to the next.
Both are the ordinary sample standard deviation across the repeats.
In the scenario mentioned above, the split was lopsided. Instrument noise was negligible: most checks score identically on every re-score, and the residual lives in a couple of genuinely subjective judge calls. Total variation came out roughly seven times larger, so the agent rebuilding the feature differently, not the scorers disagreeing, is what actually moves a PX-bench number between runs.
Measuring the floor carefully also surfaces things a single run would miss. The first was the number of runs. A spread estimated from too few of them is unreliable, so the count is fixed up front rather than stopped once the figure looks settled, and the floor is published as a round number rather than a precise-looking one.
The second was a bug in disguise. One accessibility check kept wobbling across re-scores of identical files, which should be impossible for a deterministic check. It counted an element that repeats once per row of a list, and the scoring pass adds rows as it exercises the app, so the check was tracking the row count, not the build. That is a determinism defect, not noise, and it had been hiding inside what we would otherwise have filed as scorer disagreement. Re-scoring a frozen build is the cleanest test we have that a "deterministic" scorer really is deterministic.
The payoff, and the limits
The floor does one concrete thing: it stops us reading meaning into gaps the eval cannot resolve. A reader can take any two published numbers, measure the distance between them against the floor, and throw out the comparisons that are just drift. That is the point of publishing the number rather than keeping it as an internal sanity check.
The floor has limits worth naming.
- It is partly a property of the subject: a weaker agent's messier output may be noisier to score, so a floor measured on one agent does not automatically transfer to another.
- It is partly a property of the task, since a tightly scoped feature leaves the agent less room to build differently.
- It covers the composite only; single categories rest on fewer checks, swing harder, and need wider floors.
- It is tied to the version of the scoring layer that measured it, so we re-measure it as the scorers evolve.
The floor reduces overconfidence, it does not abolish it. It tells you when a gap is too small to trust, not that every gap above it means what you think it does, and a grid of many pairwise comparisons will produce the occasional above-floor gap by chance alone. What it removes is the easiest way to be wrong, which is to read a leaderboard's third decimal as though the eval could see that far.
The floor is a guardrail, not the goal. It guards the comparison coming next — a grid of agents scored on the same scenarios — where a gap has to clear the floor before it counts as a finding.
Version 1.0 — June 2026. Reach us at hello@chordio.com.