An acceptance dataset should answer an engineering decision, not merely produce a convenient score. State which change is being assessed, what tasks matter and who can approve the result.
Define the decision before choosing the metric
Retrieving the right document and producing a supported answer are different behaviours. An agent may also need checks for tool selection and unavailable dependencies. Name the acceptance owner and the conditions that require investigation. Do not copy an arbitrary threshold from another application.
Keep development and acceptance examples separate
Examples used to tune a prompt have influenced the system. Record that history and retain held-out cases for acceptance. Document provenance, permitted uses, sampling and exclusions. Removing difficult cases simply to improve a score defeats the comparison. Avoid unnecessary personal information in test fixtures.
Inspect failure slices
Group cases by meaningful conditions such as missing context, ambiguous instructions and failed dependencies. A higher average can hide a consequential regression. Small samples and uncertain labels limit what the result establishes; describe those limits alongside the score.
Version the evidence
Retain dataset and rubric versions, application configuration, scorer identity and run settings. Inspect automated judgments against qualified human review. If the judge or sample changes, the score may no longer be comparable even when the application is unchanged.
Hand over an acceptance record
Include representative failures, coverage gaps, the decision and receiving owner. State what input or task changes trigger re-evaluation. These practices support a bounded decision; they do not guarantee behaviour on every future input.
Compare dataset, reviewer and infrastructure responsibilities in XIVTech's AI data and evaluation scope. Agree permitted inputs, delivery arrangements and handoff before implementation.