AI preference data records a judgment about alternatives under stated conditions. A reviewer choosing response A over response B does not establish that A is correct, safe or suitable for every user. It establishes a relative decision that needs its prompt, criteria and context to remain interpretable.
Choose the feedback format for the decision
Pairwise comparisons, rankings and individual ratings answer different questions. A rating asks how one response meets a scale; a comparison asks which alternative is preferred. Decide how ties, unacceptable alternatives and insufficient evidence are represented before collecting judgments. Do not force a winner when the task needs a valid no-preference outcome.
Preserve what the reviewer actually saw
Record the authorized prompt context, candidate identifiers, presentation order and rubric version. If the reviewer saw a shortened response or hidden source panel, that interface condition can matter to the judgment. Keep original content behind approved access controls rather than copying sensitive conversations into an unrestricted feedback export.
Separate preference dimensions
A response can be clearer but less well supported. Another can be concise while omitting an important qualification. Where those differences affect training or evaluation decisions, capture them explicitly rather than allowing one overall preference to conceal the tradeoff. The appropriate dimensions depend on the application, not a universal list.
Check for presentation effects
Review whether response order, length or formatting systematically influences the decision. An illustrative calibration exercise can show the same alternatives in a changed order and inspect the reasons for any different judgment. That exercise diagnoses a possible bias; it does not prove that every reviewer or dataset has the same effect.