Designing Evaluation Dataset Acceptance Criteria
Define representative cases, rubric versions and failure slices before using an evaluation score to accept an AI system change.
AI data and evaluation practice / Calgary
For Calgary teams reviewing an AI data refresh, establish what changed before interpreting a new score. Source versions, transformations, missing records and timing conditions can change the task presented to the system, even when the model is unchanged.
A score without its source conditions cannot explain a refresh. Identify the artifacts that connect input records to evaluation findings.
Illustrative experiment, not an industrial deployment: a refreshed dataset excludes records missing a required field. Its score rises, but the evaluated population has become easier.
| Condition or decision | Evidence and boundary to agree |
|---|---|
| Same system, different snapshots | Run the unchanged system on old and refreshed inputs to inspect the data effect. Keep exclusions visible. |
| Different systems, fixed snapshot | Compare baseline and candidate on the same inputs to inspect the system change under that condition. |
| Limits of the comparison | These checks help isolate changes; they do not prove causality alone. Explain whether excluded records still occur in intended use. |
Agree triggers before the next refresh. A higher score is not a reason to assume an earlier acceptance decision transfers to new input conditions.
If inputs and judgments are unaffected, record the change and why a rerun is not required. Retain the decision owner.
Inspect affected tasks. Exercise changed, partial, delayed or unavailable inputs against agreed behavior instead of silently dropping difficult cases.
Do not present earlier evidence as transferable. Identify missing snapshots, tests or owners needed for review.
Bring source versions, transformation rules, freshness expectations, authorised samples and data owners. Traceable origins do not establish representative sampling or correct behavior by themselves.
Supply approved snapshots, record transformations and explain exclusions. Separate source remediation from evaluation work.
Maintain input-condition cases and rerun instructions with the relevant dataset and rule versions.
Review findings and remaining gaps. Assign continued monitoring and future refresh reviews explicitly, or leave them outside the engagement.
XIVTech's published Canadian data-quality and evaluation scope includes Toronto, Vancouver, Montreal and Calgary as service areas. Confirm engagement availability, communication hours, authorised data access and delivery arrangements. These pages do not imply local offices, on-site staff or previous local clients.
Engagement models
Bring source versions, transformation rules, freshness expectations, authorised samples and data owners. Traceable origins do not establish representative sampling or correct behavior by themselves.
Opinions, reviews, and focused direction.
ExploreOngoing capacity in your engineering team.
ExploreIncidents, rotations, and production response.
ExploreRoadmaps with clear delivery ownership.
ExplorePlan and deliver a defined technical outcome.
ExploreOngoing engineering care and improvement.
ExploreQuestions answered
A short set of practical questions to clarify the first conversation.
No. Sampling, exclusions or source conditions may have changed. Compare versions and individual cases before attributing a result to the system.
Only under an explicit task and sampling rule. Removing missing records that occur in intended use can hide a relevant failure condition. Document treatment and fallback behavior.
Configuration alone is insufficient. Record the missing snapshot and the resulting reproducibility limit. A different source cannot support a claim of exact reproduction.
Next step
Start with trusted data pipelines from operational and enterprise sources and the technical or organizational boundary that makes it difficult today.
Practical guidance on datasets, reviewer decisions and evaluation evidence.
Define representative cases, rubric versions and failure slices before using an evaluation score to accept an AI system change.
Build a review workflow that preserves disagreement, clarifies rubrics and records how adjudication changes AI data quality decisions.
An independent reading of Anthropic's documented production Research system, focusing on rubric evidence, human review and recoverable execution.
Define permitted uses, schema, review states and receiving responsibilities that make an AI dataset handoff inspectable rather than merely transferable.
Separate a changed criterion, a changed candidate and a rerun so an evaluation decision retains its original evidence and approval history.
Report timeouts, missing outputs and invalid judgments alongside completed runs so an evaluation average does not quietly select its own evidence.
An independent analysis of Stanford CRFM's documented HELM benchmark: why human judgments about reference summaries complicate automatic evaluation scores.
Test label renames, splits and merges with explicit fixtures and reconciliation checks before moving an annotation dataset to a new taxonomy.