Replace a model answer with an answer contract
A single reference response can illustrate a task without defining every acceptable answer. When scoring requires semantic correctness rather than exact reproduction, write an admissible-answer contract before reviewing candidate outputs.
Identify required facts, permitted variation, contradictions that cause failure and evidence gaps that prevent a definitive judgment. This makes the scoring boundary inspectable without pretending every possible answer can be listed. It gives reviewers something more concrete than close enough to the reference.
Work through a bounded example
Consider an invented document-extraction fixture. Its source says: The workshop is on 14 November. Registration closes on 7 November. The question asks for the registration closing date, and the reference is 7 November.
If the contract requires the day and month, November 7 and Registration closes on 7 November preserve the required fact. One week before the workshop supplies a relationship rather than the requested date. Whether that is acceptable depends on the task contract, not whether a reviewer finds it understandable.
Adding an unsupported year is a different issue. A response can contain the correct day and month while introducing a claim the source does not support. Decide whether unsupported additions fail the task, reduce a separate groundedness score or require review. This fixture is synthetic guidance, not a delivered project or a claim about actual model performance.
Separate dimensions before combining scores
A practical contract for this fixture could assess four distinct properties.
- Required content identifies 7 November as the registration deadline.
- Grounding excludes unsupported factual details.
- Format follows any explicitly required representation.
- Answerability establishes whether the supplied source supports the requested information.
Do not quietly substitute one dimension for another. Exact-match checks can find wording differences, but those differences are not automatically factual errors. Conversely, a response containing the reference phrase can still include contradictory material.
If using an automated scorer, challenge it with a correct paraphrase, wrong month, correct date plus unsupported year, and an explanation that an actually missing date cannot be established. These boundary tests inspect the scoring rule, not just the candidate model.
Handle newly discovered valid alternatives
A reviewer may find an answer that satisfies the intended task but falls outside the written contract. Avoid accepting it only for a favoured candidate. Record the proposed alternative and supporting reasoning; have an appropriate owner decide whether it reveals an incomplete contract, mistaken reference or unacceptable response.
Version any changed contract and identify affected cases. Re-score comparable outputs under the revised rule, or state clearly that earlier and later results use different definitions. Preserve original judgments rather than rewriting their history.
Keep uncertainty visible
Broader contracts accept legitimate variation but can require specialist review. Narrower contracts are easier to automate but may reject useful answers. Neither choice is neutral. When sources cannot establish an answer, report an unresolved disposition separately: removing uncertainty from the denominator does not remove it from the scope.
The Manchester reference-uncertainty guide covers commissioning. See UK AI Data & Evaluation services for wider workstream selection. This original exercise is not an established scoring standard or evidence that a particular scorer implements these rules.