Two questions require different samples
An AI evaluation can ask whether a system handles important situations, or how often it succeeds across a defined stream of use. Those are different questions. A dataset designed for the first does not automatically answer the second.
Diagnostic coverage deliberately includes situations worth investigating: ambiguous requests, missing information, uncommon formats or consequential mistakes. A prevalence-oriented sample instead aims to represent a specified population during a specified period.
Neither approach is inherently better. The error is treating the composition of a diagnostic collection as evidence of how frequently those situations occur outside it.
Define the population before counting coverage
Begin with the unit being sampled. Is it a message, a complete conversation, a document or a user task? Counting messages can give long conversations more influence than short ones. Counting conversations answers a different question.
Next, define the sampling frame: the eligible records from which selection occurs. Record the time window, channels, languages, exclusions and handling of incomplete records. A frame that omits abandoned sessions cannot establish their frequency or outcomes.
Coverage also needs an explicit definition. All languages covered might mean one item per language or substantial testing across different tasks within each language. A coverage matrix should show which combinations were examined, how many eligible items were included and which combinations remain empty.
Keep diagnostic and representative results separate
Maintain a representative sample for population-oriented estimates and a diagnostic set for targeted investigation. They can share evaluation criteria without sharing the same interpretation.
If difficult situations are deliberately oversampled, an unweighted overall score describes that selected mixture. It does not describe the original traffic distribution. Weighting may support a population estimate when selection probabilities or defensible population proportions are available, but it does not repair an unknown sampling frame.