The actual project and its authors
Stanford's Center for Research on Foundation Models developed Holistic Evaluation of Language Models, or HELM. Its November 2022 launch account describes an implemented benchmarking effort, not a proposed example. This article independently analyses that publicly documented external research. XIVTech did not conduct the project and claims no client, delivery or partnership relationship with Stanford or the evaluated providers.
The HELM paper, first submitted on 16 November 2022 and revised on 1 October 2023, reports evaluation of 30 language models across 42 scenarios. Those numbers describe the historical study, not the current leaderboard or a recommended set of models. The official account and paper are related primary sources; independent outcome replication is not established here.
Why the reference became part of the investigation
In its findings section, Stanford CRFM reports that human evaluation judged generated summaries better than reference summaries in the CNN/DailyMail and XSum summarisation datasets under the study's evaluation conditions. The researchers also describe a mismatch between automatic ROUGE-L results and human evaluation when comparing fine-tuned models with few-shot prompting.
This is a specific, attributed finding. It does not establish that all reference summaries are poor, that ROUGE-L is always inappropriate, or that current systems reproduce the historical result. It shows why a stored reference should be an object of scrutiny rather than an unquestioned source of truth.
Similarity and usefulness answer different questions
An automatic comparison can reward agreement with a reference while a human review examines properties that the reference or metric does not fully represent. The analytical lesson is to state what each measure captures and inspect disagreements, not simply replace every automatic measure with an unexamined human preference.