An exception is a question with an owner
An annotation exception queue records why a normal decision cannot be made under the current rules. Treat it as a controlled review workflow, not a bin for difficult examples. Its output may be a clarified instruction, a retained unresolved state, an exclusion or a justified label. Each outcome has a different downstream meaning.
Distinguish the causes before routing
Separate missing source context, inconsistent taxonomy, conflicting instructions, inaccessible records and genuine ambiguity. A reviewer may resolve a simple instruction lookup but cannot grant missing data rights or invent client context. Reason codes should direct the case to someone authorised to answer the actual question. Avoid a broad other category becoming the default route for every problem.
Preserve the minimum decision record
Keep a case identifier, instruction version, proposed interpretations, exception reason and current owner. Attach only the context needed to understand the decision, respecting source permissions. Retain the original question after resolution. Replacing it with a final label would erase the evidence explaining why that record differs from an ordinary annotation.
Separate decision rights
The annotator proposes or flags a decision; an adjudicator applies an agreed interpretation; the taxonomy owner can approve changes to label meaning. These roles can be held by the same person in a small project, but their responsibilities still need definition. An acceptance owner decides whether unresolved cases prevent the output being used. Queue throughput is not proof that these decisions were correct.
Close the loop on previously labelled records
Illustrative example: a rule about quoted sentiment changes after reviewers disagree. The change can affect more than the escalated example. Record the old and new instruction versions, identify potentially affected records and agree whether they require re-review. Do not imply that a new rule retrospectively validates work produced under the previous one.