The actual project and its reported comparison
OpenAI's Harvey case account describes work by Harvey and OpenAI on a custom-trained case-law model. This article analyses that external account. XIVTech claims no involvement, client relationship, partnership or ownership of the work.
According to OpenAI, the training added the equivalent of 10 billion tokens of US case law. The account describes a side-by-side attorney comparison against GPT-4 at ten large law firms and reports 97% preference for the custom model. These are vendor-reported statements, not independently reproduced findings. The number of firms is not the number of evaluated tasks, answer pairs or raters.
What preference can tell a reader
The account explains preference in terms of longer, more complete answers and relevant case coverage. That matters to perceived usefulness: an attorney may value an answer that develops an argument or covers more requested material.
Preference nevertheless answers a comparative question. It does not establish by itself whether either response is legally correct, fully supported or suitable for a consequential decision.
The public account reviewed here does not establish the comparison's sample size, task composition, rater protocol or per-jurisdiction breakdown. Without those details, readers cannot reconstruct the denominator behind 97%, calculate meaningful uncertainty or determine consistency across tasks. These limits do not demonstrate that the reported result is wrong; they restrict conclusions available from the account.
Citation support is a separate question
The vendor account asserts sentence-level citation support. That is not independent proof that hallucinations are universally absent. A citation can be present while its relationship to a sentence still requires examination.
For a new evaluation, a qualified reviewer could ask whether cited authority supports the specific proposition, whether the answer omits a material qualification and whether the authority applies to the task. These are recommendations from this analysis, not claims that Harvey omitted those checks or that errors occurred in the reported comparison.
Reference uncertainty complicates correctness scoring
Legal questions can depend on facts, jurisdiction, timing and permitted answer scope. A reference response may express one supportable analysis without exhausting every admissible alternative.
In an evaluation-design thought experiment, two answers might reach the same conclusion through different relevant authorities. Rejecting one solely because it differs from a reference would need justification. Preferring a fluent explanation would not establish that its authorities support the conclusion. This is not a description of Harvey's undisclosed evaluation protocol.
A useful assessment separates relative preference, proposition-level support, task completeness and unresolved reference questions. Those are design implications, not a reconstruction of the external study.
What UK teams should transfer
US case-law training does not establish applicability to UK jurisdictions. A UK evaluation would need its own authorised tasks, relevant authorities, qualified reviewers and explicit treatment of uncertainty. The transferable lesson is to ask which judgment a headline measures, not import its percentage as an acceptance threshold.
The London comparison-outcome guide and Manchester reference-uncertainty guide address separate questions. XIVTech's UK service overview is distinct from the external project.
Source: OpenAI's linked Harvey account, accessed 30 October 2026. Publication date unverified. Only this primary vendor/customer account was substantiated for this analysis; no independent corroboration of outcomes is claimed.