Designing Evaluation Dataset Acceptance Criteria
Define representative cases, rubric versions and failure slices before using an evaluation score to accept an AI system change.
AI data and evaluation practice / Vancouver
For Vancouver teams planning annotation or human feedback, more labels do not resolve an unclear judgment. Test whether reviewers apply the same rule. Keep their independent reasoning, then decide which disputes need clearer instructions, missing evidence or a policy choice.
A final answer alone can hide the reason reviewers split. Preserve the difference between a first judgment, an agreement measure and an authorised resolution.
Before expanding a batch, diagnose why the same task produced different judgments. One response will not fit every cause.
| Condition or decision | Evidence and boundary to agree |
|---|---|
| Different readings of a rule | Clarify the definition and add contrasting examples. Keep the old and revised rules for comparison. |
| Missing context | Supply permitted evidence or allow an insufficient-information outcome. Do not force a confident label. |
| A genuine policy choice | Escalate to the authorised decision owner. Reviewers should not have to guess the policy. |
| Several defensible readings | Preserve uncertainty. A forced single answer can conceal a real limit in the task. |
Illustrative rubric repair, not a delivered project: reviewers label an answer 'helpful'. One rewards a plausible recommendation; another rejects it because the supplied source does not support it.
Assess groundedness and usefulness separately. State which evidence may support each choice.
Apply the revised rubric to shared cases without first agreeing the answers. Inspect the reasons as well as labels.
Identify which criteria changed and which records may be affected. Do not rewrite history as if the first labels never existed.
Bring a draft rubric, difficult examples, authorised reviewers and an adjudicator. Calibration takes review capacity and does not promise consensus. Ongoing reviewer provision and production annotation must be scoped separately.
XIVTech's published Canadian data-quality and evaluation scope includes Toronto, Vancouver, Montreal and Calgary as service areas. Confirm engagement availability, communication hours, authorised data access and delivery arrangements. These pages do not imply local offices, on-site staff or previous local clients.
Engagement models
Bring a draft rubric, difficult examples, authorised reviewers and an adjudicator. Calibration takes review capacity and does not promise consensus. Ongoing reviewer provision and production annotation must be scoped separately.
Opinions, reviews, and focused direction.
ExploreOngoing capacity in your engineering team.
ExploreIncidents, rotations, and production response.
ExploreRoadmaps with clear delivery ownership.
ExplorePlan and deliver a defined technical outcome.
ExploreOngoing engineering care and improvement.
ExploreQuestions answered
A short set of practical questions to clarify the first conversation.
Collect independent judgments first when the goal is to expose different readings. Discussion can follow, but early alignment can hide the disagreement the exercise needs to find.
Not automatically. Identify which criteria changed and which records depend on them. Document the review boundary and any uncertainty about cases left unchanged.
Keep the first judgments alongside the resolved result. Otherwise, the dataset loses evidence about ambiguity and the reason for the final choice.
Next step
Start with data platforms for research, content and operational workloads and the technical or organizational boundary that makes it difficult today.
Practical guidance on datasets, reviewer decisions and evaluation evidence.
Define representative cases, rubric versions and failure slices before using an evaluation score to accept an AI system change.
Build a review workflow that preserves disagreement, clarifies rubrics and records how adjudication changes AI data quality decisions.
An independent reading of Anthropic's documented production Research system, focusing on rubric evidence, human review and recoverable execution.
Define permitted uses, schema, review states and receiving responsibilities that make an AI dataset handoff inspectable rather than merely transferable.
Separate a changed criterion, a changed candidate and a rerun so an evaluation decision retains its original evidence and approval history.
Report timeouts, missing outputs and invalid judgments alongside completed runs so an evaluation average does not quietly select its own evidence.
An independent analysis of Stanford CRFM's documented HELM benchmark: why human judgments about reference summaries complicate automatic evaluation scores.
Test label renames, splits and merges with explicit fixtures and reconciliation checks before moving an annotation dataset to a new taxonomy.