Begin with the decision, not the average
A language-slice scorecard explains which examples support a quality decision. It is not a league table of languages. Start by naming the task and the action a result could change: accept a candidate answer workflow, revise instructions or gather missing review evidence. A single aggregate can obscure a smaller group even when its failures matter to the application.
Record the slice contract
Give each case a stable identifier, intended language, task category, reference source and criterion version. Separate original-language cases from translations and machine-generated candidates. If a set contains mixed-language inputs, define how they are assigned rather than letting an automatic detector silently decide the population. Keep an explicit unknown group where the evidence is insufficient.
Keep three questions separate
Ask whether the answer is factually supported, whether it follows the task and whether its language or style fits the audience. These questions can need different reviewers. A technically correct fact does not establish suitable formality; fluent wording does not establish correct evidence. Agree observable examples for each requirement and record who is qualified to make the judgement.
Display coverage beside findings
A useful row includes eligible case count, inspected count, criterion findings, excluded cases and unresolved review. Do not express an unreviewed slice as zero failures. When reference answers are missing, preserve the distinction between a failed answer and an answer that cannot yet be assessed. Document any combined weighting and show the component results so the receiving owner can inspect its consequences.
Use a small rehearsal to expose gaps
Illustrative example: a support assistant is evaluated on English and Japanese answers. Most available cases are English and a Japanese style requirement has no approved reviewer. The appropriate output is a coverage gap and a review dependency, not a passing overall result. Rehearse the record format on a permitted sample, resolve ambiguous criteria and only then decide whether more cases or additional reviewer expertise are needed.