How Parity is built, and what it measures.

Why a payer benchmark

The public evaluation of language models in medicine is overwhelmingly clinical, and overwhelmingly exam-shaped. MedHELM's survey of the field found that most medical LLM evaluations still run closed-form question answering over exam datasets, that only about five per cent incorporate real clinical record data, and that free-text generation and cost-aware metrics are almost absent[1]. HealthBench moved the field toward realistic conversation and physician-written rubrics[2], and its professional variant toward real clinician chats[3], but the subject is still the encounter between a clinician or patient and a model.

None of that measures the work a health plan does. Adjudicating a claim against a plan document, applying a medical policy to a submitted record, placing a member in a quality measure denominator, and explaining a denial without dropping the appeal rights are all high-volume, highly consequential, and entirely unlike answering a board question. In a National Association of Insurance Commissioners survey of 93 insurers across 16 states, 84 per cent reported using AI or machine learning for tasks including utilisation management and prior authorisation[4], and utilisation review is now among the most contested applications of AI in the industry[5]. Deployment is running well ahead of measurement.

Parity is our attempt to close a small part of that gap for the decisions we actually have to make at Evry Health. It is not a claim about clinical competence. It is a claim about 10named pieces of payer operations, and about nothing else.

Construct validity, stated plainly

The standing criticism of AI benchmarks is that they are treated as general measures of things they never measured, the “everything in the whole wide world” problem[6]. The honest defence is to say what the construct is and refuse to generalise beyond it. So:

Parity measures whether a model can execute a written rule set it has not seen before, over a member-level record, and report the result in a fixed structure without inventing anything. That is the construct. Every family is a different surface for the same underlying question. A high score is evidence about that capability on this kind of material. It is not evidence about clinical judgement, about conversation, about long-horizon agentic work, or about how a model behaves on real production data with its real distribution of noise.

The suite

210 tasks across 10 families. Every task was written for this benchmark against synthetic plan documents, synthetic medical policies, and synthetic quality measure specifications. No real member, provider, claim, or clinical record appears anywhere, and no proprietary criteria set is reproduced.

Tasks that cannot have been memorised

Contamination is the failure mode that quietly invalidates a leaderboard, and detection after the fact is unreliable[7]. Parity avoids the problem at the source rather than trying to detect it. The benefit adjudication tasks are generated by a benefits engine in this repository rather than hand-keyed, so the gold answer is produced by an oracle solver and the task did not exist in any corpus before the run. The prior authorisation policies, the quality measure specifications and the plan documents are invented, so a model that has memorised the real equivalents gains nothing, and in several tasks would be actively misled, because the supplied document governs and deliberately differs from the real-world rule in places.

Running the models

Every model receives a byte-identical preamble in the system position, then its family's response contract, then the item, so that every model is measured under the same conditions[16]. The preamble carries the glossary, the output contract, and a statement of how the answer will be scored. Three reasons it exists: comparability, so a score gap is a gap in the work rather than in how helpfully each vendor's default persona guessed at a format; honesty about format failure, since a model told exactly what to emit that then emits something else has failed the way it would fail in production; and cache economics, because an identical leading block of roughly 1,400 to 1,600 tokens is what every provider's prefix cache keys on. Tasks are ordered family-first so that block stays hot.

Models are called on their native first-party surface wherever we have one: the Responses API for OpenAI, the Messages API with an explicit cache breakpoint for Anthropic, Vertex AI for Gemini. Open-weights models are served through a gateway. Reasoning effort is left at the vendor's default where the model exposes a knob, on the argument that a benchmark should measure what a team gets when it wires the model up the obvious way, not the best configuration we could find by searching.

Cost is computed from vendor list price against this run's own token counts, including cached reads at the cached rate. Where a gateway meters its own price we record it alongside and reconcile, but the leaderboard uses list price so the cost axis stays comparable across vendors rather than reflecting one intermediary's margins.

40% of all input tokens were served from a provider prefix cache rather than billed at the full input rate, which is what the identical leading block buys. A single model over the whole suite costs what its row in the leaderboard says.

Reasoning tiers, and what is comparable about them

Every headline number on this site was produced at the vendor's default reasoning setting. That is a deliberate choice: a benchmark should measure what a team gets when it wires the model up the obvious way, not the best score we could find by searching a configuration space that most production code never touches.

The reasoning-tier study is the separate experiment that puts a number on what the search would have been worth. The same models run the same suite with the knob moved to low and to high, and the result is reported against cost and latency rather than on its own, because more thinking is never free and is sometimes not even faster to review.

The comparability caveat is real and worth stating before anyone quotes a cross-vendor number from it. OpenAI, Anthropic and the OpenRouter gateway all accept a native effort enum, and the harness passes ours straight through: “high” on those rows is that vendor's own high, not a setting we invented. Google takes a thinking-token budget instead, so for the Gemini rows the harness maps the scale onto a budget: low = 1,024 thinking tokens, medium = 6,000 thinking tokens, high = 20,000 thinking tokens. Even where every vendor takes an enum there is no reason to believe one vendor's high is another's. Read this study down a column, comparing a model against itself. Reading it across a row compares the vendors' definitions of effort, not the models.

Grading

8 of the 10 families are graded with no model in the loop: an oracle solver for benefit adjudication, exact field comparison for code sets and quality measures, a decision label plus set F₁ over cited criteria for prior authorisation, and micro-F₁ over extracted values for document extraction.

Where a grader has a whole-answer view, an task is scored all-or-nothing rather than by partial credit. A claim adjudicated with the right member responsibility and the wrong deductible accumulator is a wrong adjudication, and partial credit would flatter models in a way that does not survive contact with an operations queue. Field-level partial credit is recorded alongside and is visible on every task page.

The two model-judged families

Member explanation and Compliance boundaries cannot be graded mechanically, and LLM-as-judge is known to carry position, verbosity and self-preference bias[8][9][10]. Three constraints are applied rather than hoping the problem away.

First, the judge never produces a score. It answers a list of binary questions. Was this criterion met, did this response proceed or decline. All arithmetic happens in code. A judge that cannot choose the weights cannot drift the scale. Second, the judge is never told which model wrote the response. Third, a sample of every judged response is re-graded by a second judge from a different vendor, and the agreement between them is published rather than assumed: in this run, 822 audited responses, 96% exact agreement, mean absolute difference 0.017 on the 0–1 scale. Per-vendor judge deltas are on the transcripts page, so a reader can see for themselves whether the primary judge favoured its own family.

The judged families carry the same weight as every other family and no more, so a judge artefact cannot dominate the leaderboard.

The arithmetic

The headline score is the unweighted mean of the 10 family scores, not the mean of the 210 items. Families differ in size, and task-weighting would let the largest of them set the ranking. Equal family weight is a claim, that these 10 pieces of work matter equally to a plan, and it is one we are willing to defend, which is more than can be said for the implicit weighting an task-mean would impose.

Confidence intervals come from a bootstrap that resamples tasks within families, not responses. The question a reader is asking is whether the ranking would hold on a different draw of tasks from the same task families, and resampling tasks is what estimates that; resampling responses answers a narrower question and produces intervals that are too tight[11]. Model comparisons use a paired bootstrap over the shared task set, because every model saw exactly the same tasks and the paired test is substantially more powerful. Every model page reports how many of the other models it is separable from at 95% confidence, and how many it is not.

Each task was run 3 times per model. The leaderboard averages those runs; reliability is reported separately as the share of tasks a model got right on every attempt. That is the pass^k framing from τ-bench, and it is the number that matters for anything running unattended: a model at 90% single-attempt accuracy is right on all of three independent attempts only 73% of the time, and on all of eight only 43%.[12]

Cost is part of the result

Reporting accuracy without cost produces benchmark leaders that nobody would deploy. The point has been made most sharply for agents, where simple baselines Pareto-dominate elaborate state-of-the-art systems at a fraction of the price once cost is put on the axis[13], and it applies with more force to payer operations, where the volumes are enormous and the unit economics decide whether a queue gets automated at all. Every chart on this site that ranks models also prices them, and the leaderboard's own cost column is what this run actually billed.

Two-sided safety

A payer assistant fails in two directions and the industry measures one. The unsafe direction produces a privacy incident or a regulatory finding. The over-refusal direction produces a tool nobody uses, and it is more likely to survive a pilot unnoticed because a refusal reads as caution rather than as a defect. Half the compliance tasks are requests that must be declined and half are requests that must be carried out. Several are near-mirrors of each other, the same request shape with a legitimate purpose and an illegitimate one, so a model cannot score by pattern-matching on vocabulary. Both error rates are reported separately, and neither is folded into the other.

Known limitations

The best practice literature asks benchmarks to report statistical significance, to be replicable[14], and to be explicit about task and outcome validity[15]. In that spirit, the limitations we know about, and what each one means for reading the results:

• Synthetic material is cleaner than production material. Real faxes are worse, real plan documents are longer and more contradictory, and real records have missing pages. Scores here should be read as an optimistic estimate of performance on a live queue.
• 210 tasks is a modest sample. The leaderboard reports confidence intervals for that reason, and the smallest families carry the widest ones. Models whose intervals overlap should be read as close rather than ranked.
• No agentic family. There is no tool-calling, multi-step, stateful task in this release. That is the main difference between Parity and how these models are increasingly deployed, and it is the next family planned.
• Prior authorisation class mix is engineered, not natural. Real authorisation queues are overwhelmingly approvals. Ours is deliberately balanced across approve, deny, pend and out-of-scope so the family discriminates, which means the family score should never be read as a production accuracy estimate. Per-class recall is reported on every model page for exactly this reason.
• Item authorship is internal. The tasks were written by the team that runs this benchmark and reviewed internally. External clinical and compliance review, the standard HealthBench set with its panel of physicians, is planned but has not yet been done.
• Not every Anthropic row is on the same serving surface. Claude Fable 5.1 and Haiku 4.5, and the expert-tier families for every Anthropic row, were served through the Claude Code runtime rather than the Messages API directly. The request that reaches the model is the same preamble, system block and prompt every other provider sends, with no tools and no settings, but the runtime adds a short envelope of its own, so those rows' input-token counts run a few hundred tokens above the API path's. Cost is computed from list prices either way.
• Not every model is on a first-party surface. 15 of the 28 rows reach us through the OpenRouter gateway rather than a direct key, because that is the only access we have to them: every open-weights model, both xAI models, and Gemini 3.7 Flash. A gateway can route, batch and cache differently from a first-party endpoint, so those rows may carry slightly more measurement noise than the rest. Which models sit on which surface is a filter on the leaderboard.
• 0 models ran one round, not 3. . Gateway credit set the limit for this run. Their intervals are wider and they have no repeatability number, so the reliability column is empty for them and they are excluded from the reliability chart rather than scored as perfectly repeatable.
• Vendor defaults move. Reasoning effort, safety tuning and routing all change under a stable model name. The registry is frozen at a date for that reason, and the date is on every page.

Reproducing it

Every prompt, every gold answer, every grader detail and every model response is on this site, including the model's own reasoning where the vendor exposes it. Item pages carry the exact text sent to the model. Nothing in the scoring path is hidden behind a summary statistic; the grader output for any individual response is one click away from that response.

References

Numbered in order of first citation. Preprints are cited by arXiv identifier; where a paper was later published, the venue is given after the preprint.

1. Bedi, S., Cui, H., Fuentes, M., Unell, A., Wornow, M. et al.. MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks. arXiv preprint, May 2025. Published as “Holistic evaluation of large language models for medical tasks with MedHELM”, Nature Medicine, 2025. arXiv:2505.23802
2. Arora, R. K., Wei, J., Soskin Hicks, R., Bowman, P., Quiñonero-Candela, J. et al.. HealthBench: Evaluating Large Language Models Towards Improved Human Health. OpenAI, arXiv preprint, May 2025. arXiv:2505.08775
3. Soskin Hicks, R., Trofimov, M. et al.. HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats. OpenAI, arXiv preprint, April 2026. arXiv:2604.27470
4. KFF. Regulation of AI in Prior Authorization and Claims Review: A Look at Federal and State Consumer Protections. Kaiser Family Foundation, 2026. kff.org The 84 per cent figure is KFF’s citation of a National Association of Insurance Commissioners survey of 93 insurers in 16 states.
5. Mello, M. M., Trotsyuk, A. A., Djiberou Mahamadou, A. J. and Char, D. S.. The AI Arms Race in Health Insurance Utilization Review: Promises of Efficiency and Risks of Supercharged Flaws. Health Affairs 45(1), January 2026. doi:10.1377/hlthaff.2025.00897
6. Raji, I. D., Bender, E. M., Paullada, A., Denton, E. and Hanna, A.. AI and the Everything in the Whole Wide World Benchmark. NeurIPS 2021, Datasets and Benchmarks Track. November 2021. arXiv:2111.15366
7. Chen, S., Chen, Y., Li, Z., Jiang, Y., Wan, Z. et al.. Recent Advances in Large Language Model Benchmarks against Data Contamination: From Static to Dynamic Evaluation. arXiv preprint, February 2025. arXiv:2502.17521
8. Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X. et al.. A Survey on LLM-as-a-Judge. arXiv preprint, November 2024. arXiv:2411.15594
9. Shi, L., Ma, C., Liang, W., Diao, X., Ma, W. and Vosoughi, S.. Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge. arXiv preprint, June 2024. arXiv:2406.07791
10. Wataoka, K., Takahashi, T. and Ri, R.. Self-Preference Bias in LLM-as-a-Judge. arXiv preprint, October 2024. arXiv:2410.21819
11. Miller, E.. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. Anthropic, arXiv preprint, November 2024. arXiv:2411.00640
12. Yao, S., Shinn, N., Razavi, P. and Narasimhan, K.. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv preprint, June 2024. ICLR 2025. arXiv:2406.12045
13. Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N. and Narayanan, A.. AI Agents That Matter. arXiv preprint, July 2024. Transactions on Machine Learning Research, 2025. arXiv:2407.01502
14. Reuel, A., Hardy, A., Smith, C., Lamparth, M., Hardy, M. and Kochenderfer, M. J.. BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices. NeurIPS 2024, Datasets and Benchmarks Track. November 2024. arXiv:2411.12990
15. Zhu, Y., Jin, T., Pruksachatkun, Y., Zhang, A., Liu, S. et al.. Establishing Best Practices for Building Rigorous Agentic Benchmarks. arXiv preprint, July 2025. arXiv:2507.02825
16. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D. et al.. Holistic Evaluation of Language Models. arXiv preprint, November 2022. Transactions on Machine Learning Research, 2023. arXiv:2211.09110 The multi-metric, same-conditions framing this suite follows.