OpenAI's HealthBench is a real, publicly released evaluation project, not an XIVTech client engagement. This independent analysis examines its published methodology and implementation to explain how conversation-specific criteria connect expert judgement to automated grading. It does not reproduce the benchmark's test conversations or claim that XIVTech built, validated or clinically deployed the system.
The documented project and its evaluation problem
In Introducing HealthBench, published 12 May 2025, OpenAI describes a benchmark with 5,000 realistic health conversations created with 262 physicians who had practised in 60 countries. These are OpenAI's reported project figures, not independently measured outcomes or evidence of XIVTech delivery. The conversations include synthetic generation and human adversarial testing; realistic does not mean they are a disclosed set of actual patient records.
The project addresses a mismatch between exam-style questions and the contextual exchanges people may have with a model. Its unit of assessment is a response within a conversation, evaluated against criteria specific to that conversation. The engineering lesson is to define the behaviour and context an evaluation represents before comparing headline model scores.
Criteria make the judgement inspectable
OpenAI reports physician-written rubric criteria with point values reflecting their intended importance. Criteria describe behaviours to include or avoid. This offers a more inspectable record than a single unexplained preference, but the quality of that record still depends on whether the criteria represent the task and whether a grader applies them appropriately.
The public HealthBench implementation in simple-evals associates rubric items with criterion text, points and tags. Its scoring function sums points for criteria judged met and divides by the available positive points. Negative-point criteria therefore need careful interpretation: meeting an undesirable criterion can reduce a result rather than reward it. This is a description of the inspected public code, not a recommendation to reuse its scoring contract unchanged.