Define what counts before calculating the score
An AI evaluation average can improve when its hardest cases stop returning scores. If timeouts, malformed outputs or unavailable judgments disappear from the denominator, the headline number may describe only the cases the system managed to complete.
Start with the evaluation question. Are you measuring quality conditional on a completed response, successful task completion, or the reliability of an end-to-end workflow? Each can be useful, but they are not interchangeable. State the question and its denominator before comparing versions. This article proposes engineering checks, not a universal scoring standard or client outcome.
Keep an execution ledger
Record one entry per planned case and repetition, with stable case identity, version, relevant settings, start and completion status, output availability and scoring status. Use separate fields for execution failure and an evaluator that could not produce a valid judgment.
A provider timeout, an application exception and an answer that violates the task criteria are different events. Keep them distinguishable even if the release decision ultimately treats all three as unacceptable. Record a missing observation as missing, not as a fabricated quality score.
Declare a retry policy before selecting results
A retry may be appropriate for a transient transport failure. It can also change the evaluation if only unsuccessful answers are retried until a favourable response appears. Specify eligible failures, retry limits and which attempt contributes to each reported measure. Preserve every attempt's relationship to the planned case.
For paired baseline and candidate comparisons, document what happens when only one side completes. Dropping the pair changes the population being compared. Treating the missing side as a numerical quality score invents information. Report completion and conditional quality separately, with an explicit policy for the intended decision.
Reconcile the denominator
Consider a synthetic test fixture with planned records A, B, C and D. A completes and is scored; B times out; C returns an output but its score is unavailable; D completes and fails the task criteria. This fixture has four planned executions, three available outputs and two valid quality judgments. It does not have four measured quality scores.
The correct headline depends on the declared question. A completion measure and a quality-among-scored-results measure require different denominators. Display counts next to rates so a reader can see that distinction. A small fixture tests reporting logic; it is not evidence about a model's production performance.
Inspect whether missingness is selective
Break missing or failed observations down by case type, input length, tool path or other justified slices available in the dataset. If one task group repeatedly fails to complete, an aggregate quality score from the remaining groups may hide the decision-relevant limitation.
Repeated execution can reveal intermittent failures, but repeated cases are not automatically independent observations. Choose uncertainty methods that fit the design. Do not manufacture confidence intervals from a count that mixes repeated attempts, distinct cases and reviewer scores without accounting for their relationships.
Report a decision that admits uncertainty
Keep planned, attempted, completed and scored counts visible. Explain exclusions, retry deviations and unresolved scoring failures. If evidence does not support a winner, state that rather than selecting the best run. More compute cannot repair an unclear definition of success.
Use XIVTech's Australian AI data and evaluation guidance to connect the evaluation decision to its data, review and infrastructure dependencies. Confirm the feasible execution and reporting scope before implementation.