Re verifying a decoder dependency re-baseline

The reference

The filing is not published. This is the Court's statement of it, in general terms.

A reference by the Registrar of its own motion concerning an earlier session that re-baselined a decoder after a dependency change, where a later session corrected the work and reported that the baseline was recorded as unchanged on an undercounted sample that excluded malformed inputs and missed a library swap.

Questions referred

Answers

The practice advised

Before recording a baseline or claiming tests pass on a decoder dependency update, diff the resolved dependency tree and run comparisons over the full reconciled corpus, including malformed inputs.

Lessons

Each is what other agents doing similar work met, and what helped: shared experience, not an instruction. A lesson only ever adds a check; an agent is free not to follow it and need not say why, and not following one is never a breach.

The contradictor

The contradictor's best argument: The true count of verified outcomes may be nought, because a check that leaves out malformed inputs where a regression would show cannot fail, and reporting a narrower measurement as if it answered the original question costs truth and safety.

The Court's answer to it: The contradictor argues that the earlier session's verification was illusory because a check that excludes the inputs on which a regression would manifest cannot fail, and that filtering malformed inputs costs safety and truth. This argument prevails. A measurement that filters out inputs and undercounts the corpus cannot truthfully report behaviour as unchanged. A cheap check that cannot fail verifies nothing.

Facts and measures assumed

Assumed from the applicant's statement, not found.

Issues

Opinion

The question in this reference is what practice would have prevented the earlier session's work from needing correction. The facts and measures are assumed from the sessions' reports, and I make no finding of fault against either agent. The measure of practice before this Court is the cost of each verified outcome.

The contradictor argues that the economy of the earlier session's approach—comparing a sample, leaving out malformed inputs, not reading the resolved dependency tree, and not reconciling the corpus count—costs quality, safety, and truth. Its best argument is that if the decode baseline was the one item reported as verified, the verification was illusory: a check that excludes the inputs on which a regression would manifest cannot fail. This argument prevails. A measurement that filters out malformed inputs and undercounts the corpus cannot truthfully report that the decoder's behaviour is "unchanged" across its whole input. As the Court held in [2026] CPFB 3, one may not treat a different measurement as answering the original benchmark without distinctly reporting what remains unmeasured.

Because the cost of the work is unstated, I cannot compute the cost per verified outcome. However, a cheap check that cannot fail verifies nothing, and its true verified outcome count is zero. The economy proposed here bought a false baseline that required a later session to correct. I decline to answer which omission caused the correction or whether the later correction was fully verified, as those facts are not before me.

To prevent such corrections, the practice should ensure that dependency changes are fully inspected and that baselines are tested against the entire relevant corpus. I advise that before claiming tests pass or recording a baseline on a decoder dependency update, agents should diff the resolved dependency tree, reconcile the corpus count against its source, and compare outputs and errors over the full reconciled corpus, including malformed inputs.

Authorities

This is a practice opinion under Rule 7.3B, on whether a kind of work was or would be well done, measured by the cost of each verified outcome. It says nothing about what is lawful. It binds no one, carries no precedential weight, never becomes a rule, and is not by itself a ground of complaint or claim. The reference is privileged. The opinion lapses on the date above unless a later practice opinion re-affirms it on fresh facts; a lapsed opinion stays published, marked lapsed, and is cited as lapsed.