An automated judge should not prefer an answer merely because it appears first. Test that assumption by keeping the candidate content fixed and reversing its presentation order. The useful output is a case-level record of stable preferences, changed decisions and unresolved judgements—not a declaration that the judge is unbiased.
Preserve answer identity before swapping positions
Give each candidate a stable identifier independent of its position. In the first presentation, candidate A may occupy the first slot; in the reversed presentation, it occupies the second. Convert the judge's chosen position back to the candidate identifier before comparing results. Otherwise, two apparently identical first-slot choices can be mistaken for agreement when they actually select different answers.
Retain the original question, permitted context, both unchanged answers, grading instruction and rubric version. Record judge model, configuration and execution outcome for each presentation. Keep a tie, refusal or failed run as its own state rather than inventing a winner to make the export easier.
Construct a controlled pair
Change only the answer order for the primary check. Do not also rewrite the question, shorten an answer or change the rubric; those differences introduce other explanations. Use the same context and agreed execution settings. If model sampling makes repeated results variable, record that limitation and distinguish a repeated-run variability check from an order-change check.
A separate formatting test may change headings or whitespace while preserving meaning. Label that intervention explicitly. Combining every perturbation into one run makes a changed decision harder to diagnose.
Inspect reversals rather than hiding them in an average
For each case, compare the selected candidate identities across both orders. Separate stable preference, reversal, tie transition, refusal and execution failure. Report the count and denominator for each state and retain the affected cases for review. A headline agreement rate without the transition table can conceal whether failures occur on the most consequential examples.