Designing Evaluation Dataset Acceptance Criteria
Define representative cases, rubric versions and failure slices before using an evaluation score to accept an AI system change.
AI data and evaluation practice / Toronto
For Toronto teams commissioning AI evaluation, a collection of examples is not yet an acceptance set. Specify which cases may guide development, which remain held out and what the acceptance owner needs when a candidate system differs from the baseline.
Illustrative technical example, not a client result: a support-answer prompt is revised after failing a question with an ambiguous product name. That question now belongs in the development history; passing it demonstrates a targeted repair.
Run baseline and candidate on the same separately held-out cases, rubric and source snapshot. Include answerable, ambiguous and unsupported questions where they matter.
A candidate may improve the combined score while giving an unsupported answer to a previously correct case. Preserve that regression for review.
The owner decides using the failure policy set before the run. A better average must not silently override an acceptance-critical failure.
Case admission is a practical way to test dataset fitness before investing in more automated runs.
| Condition or decision | Evidence and boundary to agree |
|---|---|
| Permitted and traceable? | Record the source, permitted use and dataset version. Exclude material that cannot be used for the agreed purpose. |
| Exposed during development? | Track whether the case or a close duplicate shaped a prompt, model or retrieval change. Do not present exposed cases as independent evidence for that repair. |
| Relevant and judgeable? | Connect the case to a task or failure condition. Supply enough evidence to judge the expected behavior. |
| Unresolved expectation? | Clarify the rubric or retain the case as diagnostic. Do not turn a disputed expectation into a definitive pass or fail. |
Agree these record fields for the work package. Sensitive source evidence can remain access-controlled rather than copied into every test fixture.
Bring baseline and candidate identifiers, authorised examples, development history and a named acceptance owner. Holding cases out reduces the material available for development; repeated exposure can weaken independence.
XIVTech's published Canadian data-quality and evaluation scope includes Toronto, Vancouver, Montreal and Calgary as service areas. Confirm engagement availability, communication hours, authorised data access and delivery arrangements. These pages do not imply local offices, on-site staff or previous local clients.
Engagement models
Bring baseline and candidate identifiers, authorised examples, development history and a named acceptance owner. Holding cases out reduces the material available for development; repeated exposure can weaken independence.
Opinions, reviews, and focused direction.
ExploreOngoing capacity in your engineering team.
ExploreIncidents, rotations, and production response.
ExploreRoadmaps with clear delivery ownership.
ExplorePlan and deliver a defined technical outcome.
ExploreOngoing engineering care and improvement.
ExploreQuestions answered
A short set of practical questions to clarify the first conversation.
Keep it as a regression case, but record that it influenced development. It is not independent held-out evidence for that repair.
Clarify the expected behavior or keep the case as diagnostic. An unresolved judgment should not silently become a definitive pass or fail.
Define that policy before reviewing the candidate. Some failures may block acceptance while others prompt investigation. An aggregate score should not override an agreed critical-failure rule.
Next step
Start with cloud platforms that support enterprise and product teams together and the technical or organizational boundary that makes it difficult today.
Practical guidance on datasets, reviewer decisions and evaluation evidence.
Define representative cases, rubric versions and failure slices before using an evaluation score to accept an AI system change.
Build a review workflow that preserves disagreement, clarifies rubrics and records how adjudication changes AI data quality decisions.
An independent reading of Anthropic's documented production Research system, focusing on rubric evidence, human review and recoverable execution.
Define permitted uses, schema, review states and receiving responsibilities that make an AI dataset handoff inspectable rather than merely transferable.
Separate a changed criterion, a changed candidate and a rerun so an evaluation decision retains its original evidence and approval history.
Report timeouts, missing outputs and invalid judgments alongside completed runs so an evaluation average does not quietly select its own evidence.
An independent analysis of Stanford CRFM's documented HELM benchmark: why human judgments about reference summaries complicate automatic evaluation scores.
Test label renames, splits and merges with explicit fixtures and reconciliation checks before moving an annotation dataset to a new taxonomy.