RAG evaluation metrics are useful when they distinguish what failed. A retrieval-augmented generation workflow can return a poor answer because the source does not exist, retrieval missed it, the context was incomplete or generation used it incorrectly. One overall score cannot identify all four causes.
Start with an answerable task
For each authorized test question, record whether the approved source collection contains the evidence needed to answer it. If that evidence is absent, a confident answer should not be treated as a retrieval success. Include tasks where the appropriate behavior is to ask for clarification or explain that the available material is insufficient.
Inspect retrieval before scoring the response
Retain the selected document or chunk identifiers and their versions. Where relevant documents are labeled, retrieval measures such as precision or recall can help compare the chosen set with the expected set. The labels and relevance definition are part of the measurement: a document about the same subject is not necessarily sufficient evidence for the requested answer.
A useful illustrative test asks for a policy's effective date. The correct document may be retrieved, but an obsolete version or missing heading can still make the context inadequate. This is a proposed diagnostic scenario, not an XIVTech project result.
Separate answer support from task completion
Check whether each material claim is supported by the permitted context, then check whether the response actually answers the question. A response can quote the source faithfully while omitting the requested decision. It can also sound helpful while introducing an unsupported detail. Keep those findings separate so a prompt change is not used to conceal a source-quality problem.
Record the evaluator's limits
Human and model-based judgments need an explicit rubric and examples. A model judge is another system to validate, not independent ground truth. Preserve uncertain results for review, and do not infer that a score transfers unchanged between domains, languages or document collections.