Replace a model answer with an answer contract
A single reference response can illustrate a task without defining every acceptable answer. When scoring requires semantic correctness rather than exact reproduction, write an admissible-answer contract before reviewing candidate outputs.
Identify required facts, permitted variation, contradictions that cause failure and evidence gaps that prevent a definitive judgment. This makes the scoring boundary inspectable without pretending every possible answer can be listed. It gives reviewers something more concrete than close enough to the reference.
Work through a bounded example
Consider an invented document-extraction fixture. Its source says: The workshop is on 14 November. Registration closes on 7 November. The question asks for the registration closing date, and the reference is 7 November.
If the contract requires the day and month, November 7 and Registration closes on 7 November preserve the required fact. One week before the workshop supplies a relationship rather than the requested date. Whether that is acceptable depends on the task contract, not whether a reviewer finds it understandable.
Adding an unsupported year is a different issue. A response can contain the correct day and month while introducing a claim the source does not support. Decide whether unsupported additions fail the task, reduce a separate groundedness score or require review. This fixture is synthetic guidance, not a delivered project or a claim about actual model performance.
Separate dimensions before combining scores
A practical contract for this fixture could assess four distinct properties.
- Required content identifies 7 November as the registration deadline.
- Grounding excludes unsupported factual details.
- Format follows any explicitly required representation.
- Answerability establishes whether the supplied source supports the requested information.