An LLM evaluation rubric states what a reviewer should inspect and how that observation affects a decision. Words such as helpful, accurate or safe are not sufficient instructions on their own. A usable rubric connects the intended task to observable evidence and explains what to do when that evidence is incomplete.
Define the decision and permitted evidence
Start with a particular workflow rather than a universal quality score. State what the user asked, which sources the reviewer may consult and whether the system must answer, clarify, refuse or escalate. Keep the criterion within the reviewer's authority; a factual review is not automatically legal, medical or security approval.
Split criteria that imply different fixes
Task completion, source support and presentation can fail independently. If a single score combines them, an engineer may not know whether to change retrieval, generation or the interface. Keep separate judgments where their remedies differ, and explain which failures block acceptance rather than averaging them away.
Write anchors at the boundary
Provide examples of acceptance, failure and a meaningful borderline case. Explain the reason, not merely the label. An illustrative source-support criterion might reject a response that adds a deadline absent from the approved document, even when the remainder is accurate. Another criterion can assess whether the requested action is explained. These are proposed examples, not measured client results.
Make uncertainty a valid result
A reviewer who lacks evidence should not be forced to guess. Define an uncertain or not-applicable outcome and who resolves it. Record the missing evidence separately from an observed error. If reviewers consistently choose uncertainty, inspect the task or instructions before interpreting the pattern as poor reviewer performance.
Test the rubric before broad use
Ask reviewers to apply the criteria independently, then examine disagreement and their reasons. The goal is to find ambiguous boundaries, not to produce agreement at any cost. If a model judge is used, test it against reviewed examples and inspect its mistakes. Do not treat fluent explanations from an automated judge as proof that its decision is correct.