Designing Evaluation Dataset Acceptance Criteria
Define representative cases, rubric versions and failure slices before using an evaluation score to accept an AI system change.
AI data and evaluation practice / Montreal
For Montreal teams planning AI evaluation, list the languages and tasks the decision concerns. Establish which combinations have representative examples and qualified review. This English-language page does not establish French-language delivery or performance.
Build the evaluation around language-task combinations. A populated English cell cannot stand in for an empty cell in another language.
| Condition or decision | Evidence and boundary to agree |
|---|---|
| Language and usage | Identify the required language, domain terms and mixed-language inputs where they matter. Do not assume a translated interface establishes coverage. |
| Task being judged | Separate retrieval relevance, answer correctness and translation review. These questions need different evidence. |
| Review basis | Record example sources, reference material and required reviewer qualifications. Confirm expertise and delivery availability before commissioning work. |
| Coverage status | Mark each cell reviewed, limited or not evaluated. Explain the reason rather than treating an untested condition as a passing result. |
Illustrative example, not a client result: a French question receives a fluent answer, but the retrieved document concerns a different version of the requested policy.
Judge whether the selected reference is relevant to the question and intended source version.
Check whether the answer is supported by that reference. Translation quality alone cannot establish task correctness.
If required reference material or qualified review is unavailable, mark the finding unresolved. Do not infer correctness from fluent wording.
Sourcing changes the meaning of a language comparison. Record origins and transformations before attributing a difference to language.
| Condition or decision | Evidence and boundary to agree |
|---|---|
| Translated parallel cases | Aligned questions can support a defined comparison. They may carry unnatural wording or translation artifacts and miss native usage. |
| Naturally occurring cases | Real phrasing can expose different vocabulary and tasks. Difficulty and source coverage may differ from the parallel set. |
| Mixed-language cases | Use an explicit slice with an expected behavior and review basis when these inputs matter. Do not assign them arbitrarily to a single language. |
Bring required user journeys, permitted examples, language requirements and references. Translation alone does not establish representative coverage; further tasks and vocabulary changes need an agreed extension.
XIVTech's published Canadian data-quality and evaluation scope includes Toronto, Vancouver, Montreal and Calgary as service areas. Confirm engagement availability, communication hours, authorised data access and delivery arrangements. These pages do not imply local offices, on-site staff or previous local clients.
Engagement models
Bring required user journeys, permitted examples, language requirements and references. Translation alone does not establish representative coverage; further tasks and vocabulary changes need an agreed extension.
Opinions, reviews, and focused direction.
ExploreOngoing capacity in your engineering team.
ExploreIncidents, rotations, and production response.
ExploreRoadmaps with clear delivery ownership.
ExplorePlan and deliver a defined technical outcome.
ExploreOngoing engineering care and improvement.
ExploreQuestions answered
A short set of practical questions to clarify the first conversation.
It can test the defined translated cases. It does not by itself establish coverage of naturally occurring French questions, vocabulary or source material.
Define an explicit slice if these inputs matter to the task. Record expected behavior and review requirements instead of assigning the examples arbitrarily to one language.
No. Confirm language expertise, reviewer qualifications and actual delivery availability during scoping before commissioning language-specific work.
Next step
Start with evaluation and data lineage for AI-enabled systems and the technical or organizational boundary that makes it difficult today.
Practical guidance on datasets, reviewer decisions and evaluation evidence.
Define representative cases, rubric versions and failure slices before using an evaluation score to accept an AI system change.
Build a review workflow that preserves disagreement, clarifies rubrics and records how adjudication changes AI data quality decisions.
An independent reading of Anthropic's documented production Research system, focusing on rubric evidence, human review and recoverable execution.
Define permitted uses, schema, review states and receiving responsibilities that make an AI dataset handoff inspectable rather than merely transferable.
Separate a changed criterion, a changed candidate and a rerun so an evaluation decision retains its original evidence and approval history.
Report timeouts, missing outputs and invalid judgments alongside completed runs so an evaluation average does not quietly select its own evidence.
An independent analysis of Stanford CRFM's documented HELM benchmark: why human judgments about reference summaries complicate automatic evaluation scores.
Test label renames, splits and merges with explicit fixtures and reconciliation checks before moving an annotation dataset to a new taxonomy.