An annotator who lacks enough information should not have to invent a label. Define an abstention path that preserves why an item is unresolved and what the downstream system should do with it. Unknown, negative, not applicable and disputed are different claims; collapsing them can change both training data and evaluation results.
Define the states before adding a button
Start with the task's evidence requirements. A negative label means the relevant condition is absent according to the agreed criterion. Unknown means the available evidence cannot support that decision. Not applicable means the task does not apply to this item. Disputed means reviewers reached incompatible judgements that have not been resolved.
Use only the states the workflow can explain and preserve. Too many overlapping reasons can create another source of inconsistent annotation. Write a short decision rule and a permitted next action for each state, with examples reviewed against the actual instructions. Do not assume a low confidence score is equivalent to missing evidence.
Keep the reason and the judgement separate
Store the assigned class, disposition and reason as distinct concepts where the existing data contract allows them. A missing class value alone cannot explain whether a reviewer abstained, a task was never completed or an export failed. Retain reviewer and instruction-version references appropriate to the project's privacy rules.
Route a missing-evidence item to an authorised evidence owner, not automatically to a second annotator who sees the same incomplete record. Route a disputed interpretation to criterion review or adjudication. Additional review is useful only when it can address the cause.
Test the export, not just the annotation screen
Select a small accepted example for every supported state and follow it through export and ingestion. Check that unknown is not converted to false, a zero-valued class or an empty string that the consumer interprets differently. Confirm whether omitted fields, null values and explicit disposition codes survive the actual loader.