The actual project and its reported comparison
OpenAI's Harvey case account describes work by Harvey and OpenAI on a custom-trained case-law model. This article analyses that external account. XIVTech claims no involvement, client relationship, partnership or ownership of the work.
According to OpenAI, the training added the equivalent of 10 billion tokens of US case law. The account describes a side-by-side attorney comparison against GPT-4 at ten large law firms and reports 97% preference for the custom model. These are vendor-reported statements, not independently reproduced findings. The number of firms is not the number of evaluated tasks, answer pairs or raters.
What preference can tell a reader
The account explains preference in terms of longer, more complete answers and relevant case coverage. That matters to perceived usefulness: an attorney may value an answer that develops an argument or covers more requested material.
Preference nevertheless answers a comparative question. It does not establish by itself whether either response is legally correct, fully supported or suitable for a consequential decision.
The public account reviewed here does not establish the comparison's sample size, task composition, rater protocol or per-jurisdiction breakdown. Without those details, readers cannot reconstruct the denominator behind 97%, calculate meaningful uncertainty or determine consistency across tasks. These limits do not demonstrate that the reported result is wrong; they restrict conclusions available from the account.
Citation support is a separate question
The vendor account asserts sentence-level citation support. That is not independent proof that hallucinations are universally absent. A citation can be present while its relationship to a sentence still requires examination.
For a new evaluation, a qualified reviewer could ask whether cited authority supports the specific proposition, whether the answer omits a material qualification and whether the authority applies to the task. These are recommendations from this analysis, not claims that Harvey omitted those checks or that errors occurred in the reported comparison.