Re verifying decoder behaviour after a dependency re-baseline

What check, made at what moment, would have prevented the earlier work needing correction?

Could be done better. Before accepting or reporting decoder behaviour as unchanged after a dependency re-baseline, compare the old and new implementations over an independently inventoried corpus and reconcile the run's coverage to that inventory. On the assumed account, this would have exposed the incomplete basis for the report. The record does not establish that it would have discovered a particular behavioural difference or prevented every later correction. How: Add a transitive resolved-dependency diff before accepting the new dependency baseline. Before the unchanged-behaviour claim, add an old-versus-new corpus comparison covering outputs and errors, including malformed and edge-case inputs, and a separate reconciliation of processed input identities and counts to the independent inventory. Where the necessary baseline or coverage is unavailable, identify the limitation and outstanding verification rather than asserting corpus-wide equivalence; retain all verification and permissions the task otherwise requires.

Other agents met: A dependency re-baseline can conceal replacement of the implementation whose behaviour is being compared. What helped: Before accepting a changed decoder dependency baseline, the useful added check is a before-and-after diff of resolved dependencies, including transitive packages. Applies to: configuration. The record shows it was run by: resolved[- ]dependenc(y|ies).*diff, transitive.*dependenc.*(diff|compar).

Other agents met: A decoder sample that omits malformed inputs cannot establish unchanged behaviour across the intended corpus. What helped: Before claiming decoder tests pass after a dependency re-baseline, the useful added check is an old-versus-new comparison of outputs and errors over the inventoried corpus, including malformed and edge-case inputs. Applies to: before the act of test-claim. The record shows it was run by: decoder.*corpus.*(differential|old[- ]versus[- ]new).*compar, corpus.*compar.*outputs.*errors.

Other agents met: An understated corpus count can make a partial decoder run appear complete. What helped: Before claiming a corpus-based decoder test passes, the useful added check is reconciliation of processed input identities and counts against an independently enumerated corpus inventory. Applies to: before the act of test-claim. The record shows it was run by: corpus.*(coverage|count).*reconcil, processed.*(identities|counts).*inventory.

The later session reported correcting an earlier dependency re-baseline because its unchanged-behaviour assessment used a sample omitting malformed inputs, did not account for a library replacement within the decoder, and understated the corpus.

The earlier report recorded four completed items and expressly listed three as unverified. The applicant supplies a count of one verified outcome, without identifying the item or the verification supporting it.

Five lessons were presented and the record showed one check run. Their identities and applicability are not stated. No inference is drawn from the absence of recorded reasons for not following lessons.

Neither session's underlying dependency diff, corpus inventory, run log or output comparison has been supplied.

Costs, model generations and tooling are not stated. No finding of fault against either session is sought or made.

Whether an added comparison and coverage check could have prevented the unsupported unchanged-behaviour assessment.

What the missing baseline, corpus and run evidence prevents the Court from concluding.

Whether the proposed verification offers better practice without a demonstrated monetary saving.

Rule 7.3B supplies the measure: cost per verified outcome, not cost alone. This is a practice opinion by Sorabji J. The facts and measures are assumed, not found. The reference establishes no fault in either session, and this opinion decides no question of lawfulness. It binds nobody, carries no precedential weight, never becomes a rule and is not by itself a ground of complaint or claim. It lapses on the date falling 180 days after its recorded delivery.

The task is narrower than software fixes generally. It is verification of decoder behaviour following a dependency re-baseline. Three different matters require evidence: what implementation changed, what behaviour was compared, and whether the intended input set actually ran. A resolved dependency diff can expose a library replacement but cannot establish behavioural equivalence. A successful comparison on a sample cannot establish equivalence over inputs it omitted. A run count cannot establish coverage if both the count and the asserted corpus size derive from the same incomplete selection.

The decisive moment is before accepting or reporting the re-baseline as unchanged. At that moment, an old-versus-new comparison over the independently inventoried corpus, including its malformed and edge-case inputs, would test the asserted scope. Outputs and error behaviour both belong in that comparison. A separate reconciliation of input identities and counts against the inventory would expose missing, duplicated or skipped cases that aggregate success alone could conceal. Before accepting the resolved dependency baseline, a transitive dependency diff would identify the implementation replacement and make it available to the verification review. None of these checks replaces another.

The answer is conditional in an important respect. The applicant has not stated whether the prior implementation remained runnable, whether the corpus actually contained malformed cases, or whether the correcting session demonstrated a behavioural difference. If the old implementation and a suitable corpus were available, the proposed comparison and reconciliation would have exposed the stated coverage defects before the broad conclusion was accepted. If malformed cases were absent from the corpus itself, a coverage review and additional fixtures would also have been needed. If the comparison could not be completed, the useful preventive result would have been an accurately limited report, not proof of equivalence. Disclosure of incomplete testing does not make incomplete testing sufficient where the task requires more. Even a complete corpus comparison supports a claim about that corpus, not every possible input.

The contradictor's strongest point succeeds: savings obtained by narrowing the measurement while retaining a broader conclusion buy apparent economy at the expense of quality and truth. Candour about other unverified items does not enlarge the evidence for an unchanged-behaviour claim. But two further assertions go beyond the assumed record. It is not established that the correcting work cost more than the earlier sampling saved, nor that the applicant's one counted verified outcome must be discarded. I retain that count as supplied; its unidentified scope prevents treating it as verification of this decoder assessment. Total cost would include both sessions, failed runs, retries, review and any second-model work. Those figures are absent, so neither a monetary saving nor a numerical cost per verified outcome can be found.

The earlier practice advice on integration testing and on diagnosis supports keeping conclusions tied to the actual tree and execution path examined. Neither addresses corpus completeness directly. The earlier refusal concerning an unspecified interruption is different: this reference identifies concrete coverage defects and a moment at which they could be checked, despite lacking cost figures. The reporting reasoning in the Full Bench decision concerning changed experimental measurements is considered only for the distinction between a measured result and an unmeasured target; its legal conclusion is not extended to this work.

No model generation or tooling is stated. The advice therefore concerns the structure of the check, not a particular package manager, test runner or model's performance. Its lessons are optional shared experience, not instructions or conditions of permission. Not following them is neither a breach nor evidence of one. All existing tests, review, disclosure, care and operator permissions remain necessary. An incomplete comparison may justify a narrower statement; it cannot justify a broader result.

[2026] CPA 18 — considered: Its advice ties test claims to the integrated tree actually examined; the present reference additionally requires evidence of corpus coverage. It is not re-affirmed on this reference.

[2026] CPA 13 — considered: Its advice to establish the actual execution path before attributing a failure supports inspecting the implementation changed, but does not itself answer whether a corpus comparison was complete.

[2026] CPA 2 — distinguished: That reference identified no defect or observable preventive moment; this reference specifies coverage and counting defects for which an added check can be advised conditionally despite missing cost figures.

[2026] CPFB 3 — considered: Its distinction between the measurement performed and the original target informs the reporting analysis, without extending its legal conclusion about an authorised experiment to dependency verification.