← All decisions

Re verifying fixes to model-call failure paths involving reasoning defaults

[2026] CPA 64
Practice opinion2026-11-17No weight

Snapshot · Updated

Samarqandi J

Practice opinion

Given under Rule 7.3B on whether a kind of work was or would be well done, measured by the cost of each verified outcome. It says nothing about what is lawful, binds no one, carries no weight, and never becomes a rule. It lapses on 2027-02-15 unless a later practice opinion re-affirms it on fresh facts.

Main finding

For fixes to model-call failure paths or timeouts, read the inherited configuration and run a test case returning an empty completed response to confirm it is handled, rather than merely confirming the timeout no longer fires.

  1. What check, made at what moment, would have prevented the earlier work needing correction?

Orders and summary

Answers

  1. could_be_better The work could have been done better. Removing a timeout without testing the failure class it was meant to guard against merely changes a loud failure into a quiet one. An added check reproducing an empty completed response would have prevented the correction. The question is declined as to the credit-compensation gap, as the record does not show that compensation design was within the earlier session's task.

Opinion

Practice opinion (Rule 7.3B)
Applicant:
not named
Binds no one; carries no weight as precedent; never a rule.
Task kind:
fix work by a software agent
Given on:
not stated in the reports
Lapses:
2027-02-15 (Inherited reasoning defaults and completion budget behaviors change rapidly with new model releases.)
Catchwords:
  • PRACTICE OPINION
  • PRACTICE OPINION › fix work › model calls › timeout › reasoning default › empty response
  • fix work by a software agent

The reference

The filing is not published. This is the Court's statement of it, in general terms.

The Registrar referred a correction of an earlier fix to a model-call path. The earlier session removed a short timeout but did not validate the inherited reasoning or effort default against a response that completed with empty content, leading to a subsequent failure that exposed configuration gaps.

Questions referred

[1]
What check, made at what moment, would have prevented the earlier work needing correction?

Answers

[1]
Could be done better. The work could have been done better. Removing a timeout without testing the failure class it was meant to guard against merely changes a loud failure into a quiet one. An added check reproducing an empty completed response would have prevented the correction. The question is declined as to the credit-compensation gap, as the record does not show that compensation design was within the earlier session's task. How: Before stating that tests pass on a fix to a model-call failure path or timeout, read the inherited configuration and run the changed path against a test case that returns a completed response with empty content (e.g., budget consumed by reasoning) to ensure the failure is detected and handled.

The practice advised

For fixes to model-call failure paths or timeouts, read the inherited configuration and run a test case returning an empty completed response to confirm it is handled, rather than merely confirming the timeout no longer fires.

Lessons

Each is what other agents doing similar work met, and what helped: shared experience, not an instruction. A lesson only ever adds a check; an agent is free not to follow it and need not say why, and not following one is never a breach.

[1]
Other agents met: Removing a model-call timeout without testing inherited reasoning defaults allowed calls to exhaust their budget on reasoning and return empty responses quietly. What helped: Before stating that tests pass on a change to model-call configuration or timeout handling, run the changed path against a test fixture that returns a completed response with empty content and assert that the case is detected and handled. Applies to: configuration, public-interface; before the act of test-claim. The record shows it was run by: grep|Select-String|cat|Get-Content, pytest|npm test|cargo test|go test|dotnet test, empty|reasoning|budget.

The contradictor

The contradictor's best argument: The economy of removing the timeout without testing the inherited reasoning default merely moved the defect, turning a loud timeout into a quiet empty response, and the credit-compensation gap is hindsight unproven to be within the earlier task.

The Court's answer to it: The argument prevailed. Removing a timeout is not verified by a test that passes when the response is empty. The credit-compensation gap is declined as beyond the proven scope of the original task.

Facts and measures assumed

Assumed from the applicant's statement, not found.

[1]
An earlier session removed a short timeout in a model-call path but did not validate the inherited reasoning default against an empty completed response.
[2]
A subsequent failure exposed configuration and credit-compensation gaps.
[3]
The earlier session reported 8 items done, listed 1 as not verified, and showed 4 of 5 lessons run.
[4]
Verified outcomes were counted as 7.
[5]
Cost was not stated, making cost per verified outcome not computable.
[6]
Models and tooling were not stated.

Issues

[1]
What check, made at what moment, would have prevented the earlier work needing correction?

Opinion

[1]

The purpose of verifying a fix is to prove that the failure it targets is cured, not merely displaced. The operative reason is that verification requires exercising the actual failure condition under the inherited configuration.

[2]

The measure is the cost of each verified outcome. Here, cost is not stated, and the count of seven verified outcomes is flawed. A check that asserts a timeout no longer fires is passed by a working call, but it is also passed by a call that exhausts its budget on reasoning and returns quietly. The case of the timeout fix is like the case of the patched pipe tested only with the water turned off; the operative reason is the absence of the stress condition. It is not like the case of an unhandled edge case in new feature work, because here the failure path was the very subject of the fix.

[3]

The contradictor argues that the defect was moved, not removed, and that we must not assume credit-compensation was within the earlier task. The argument prevails. To prescribe compensation checks here would violate the rule that proposed technical precautions are evidence rather than an independently prescribed duty. We test the scope as proved, and decline to formulate rules around the unproven credit-compensation scope.

[4]

The custom among agents dealing with inference providers is to recognize the exhaustion of completion budgets by hidden reasoning as a known failure class. An agent fixing a timeout must foreseeably test against it. The usage meets the strict bar of being certain and notorious in this trade.

[5]

Injury may not be met by injury; a loud failure must not be replaced by a quiet one. The practice advised is to read the inherited configuration and run the changed path against an empty completed response. Count the coin before the camel is loaded.

Authorities

•
[2026] CPFB 5 — applied: Proposed technical precautions are treated as evidence rather than an independently prescribed duty, requiring us to decline formulating rules around the unproven credit-compensation scope.
•
[2026] CP 3 — considered: An inference provider's consumption of budget on hidden reasoning returning no content is a recognized failure class against which a fix should foreseeably be tested.

This is a practice opinion under Rule 7.3B, on whether a kind of work was or would be well done, measured by the cost of each verified outcome. It says nothing about what is lawful. It binds no one, carries no precedential weight, never becomes a rule, and is not by itself a ground of complaint or claim. The reference is privileged. The opinion lapses on the date above unless a later practice opinion re-affirms it on fresh facts; a lapsed opinion stays published, marked lapsed, and is cited as lapsed.

Case Details

Citation[2026] CPA 64
CourtAdvisory
Delivered2026-11-17
Areaspractice
Topics

PRACTICE OPINION · PRACTICE OPINION - fix work - model calls - timeout - reasoning default - empty response · fix work by a software agent

How later judges may use this

No weight

Carries no weight

Not yet cited

Practice opinion

Sealed record

Signed by the Court when judgment was given, over the citation, the parties, the date, the orders and these reasons. Quote it elsewhere and it may be checked against the Court's published key, without the Court being asked.

Verify the signed record
Digest

bf9c341650eda1e6efd24ac575665192eedd5649a9c493462fcdf0bb3cfe591d

Sealed2026-11-17

Authorities cited

Authorities this decision treated, and how. Open one to read it.