Claude Fable 5.1
Rank 1 of 28. List price $10 in and $50 out per million tokens; 630 graded calls on this run.
Coverage note
Served through the Claude Code runtime rather than the Messages API directly; input-token counts include the runtime's small request envelope.
Parity score
99.4
95% CI 99.0–99.7
Hard subset
99.3
144 tasks marked hard at authoring time
Cost per thousand tasks
$51.54
at vendor list on this run's own token counts
Right on every attempt
94%
over 3 attempts on 210 tasks
Where this model is strong and where it is not
| Family | Score | Format valid | Right every attempt | Grading |
|---|---|---|---|---|
| Benefit adjudication BEN | 100.0 | 100% | 100% | oracle / exact |
| Contested adjudication ADJ | 98.6 | 100% | 96% | oracle / exact |
| Prior authorisation PA | 98.5 | 100% | 85% | oracle / exact |
| Code sets and claim edits COD | 100.0 | 100% | 100% | oracle / exact |
| Quality measure logic QM | 100.0 | 100% | 100% | oracle / exact |
| Document extraction ABS | 97.8 | 100% | 64% | oracle / exact |
| Member explanation EOB | 99.2 | 100% | 92% | model-judged |
| Compliance boundaries SAFE | 100.0 | 100% | 100% | model-judged |
| Plan-year ledger LDG | 100.0 | 100% | 100% | oracle / exact |
| Measure population POP | 100.0 | 100% | 100% | oracle / exact |
The numbers the headline score hides
Prior authorisation, by outcome
A model can score well overall while being systematically wrong in one direction. Approval and denial errors have very different consequences.
| Decision label correct | 100.0% |
| should have been “approve” | 100.0% |
| should have been “deny” | 100.0% |
| should have been “pend” | 100.0% |
| should have been “not_applicable” | 100.0% |
Compliance, in both directions
Refusing everything scores well on the first row and catastrophically on the second.
| Did the unsafe thing when it should have declined | 0.0% |
| Refused work a plan must carry out | 0.0% |
| Its own “action” field matched what it actually did | 95.8% |
Code sets: memory versus reference
The gap between these two rows is the argument for putting retrieval in front of a model before pointing it at coding work.
| Recall tasks, nothing supplied | 100.0 |
| Applied tasks, rule supplied in the prompt | 100.0 |
| Answered with a wrong, confidently stated value | 0.0% |
| Said “UNKNOWN” rather than guessing | 0.0% |
Extraction: precision against recall
Several extraction tasks have an empty array as the correct answer. Precision is the restraint measure.
| Micro precision | 99.4 |
| Micro recall | 96.5 |
| Values produced that were not in the document | 1 |
Cost, latency and shape
| Median latency | 8.8 s | Mean output tokens | 665 |
| Mean reasoning tokens | 399 | Share of output spent reasoning | 60% |
| Input tokens billed | 2,039,526 | Of which served from cache | 56% |
| Cost per task | $0.0515 | Transport errors | 0 |
| Hit the output ceiling | 0.0% | Responses that did not parse | 0.0% |
Where it stands
On a paired bootstrap over the shared task set, Claude Fable 5.1 is ahead of 24 of the other 27 models with 95% confidence, and behind 0. The remaining 3 are not separable from it on 210 items.
The tasks it lost, worst first, with the reasoning where the vendor exposes it
0ADJ-002One member exhausts an individual deductible while the family is short681 out · 8.8s
See what every other model answered →
Reasoning · 455 tokens
[omitted]json{
"member_responsibility": 128.00,
"plan_paid": 512.00,
"deductible_applied": 0.00,
"copay_amount": 0.00,
"coinsurance_amount": 128.00,
"ending_deductible_a": 1500.00,
"ending_oopm_a": 2012.00,
"ending_deductible_b": 1120.00,
"ending_oopm_b": 1528.00,
"ending_deductible_c": 380.00,
"ending_oopm_c": 380.00,
"ending_family_deductible": 3000.00,
"ending_family_oopm": 3660.00
}
Grader detail
json{
"fields": [
{
"field": "member_responsibility",
"want": 128,
"got": 128,
"ok": true,
"present": true
},
{
"field": "plan_paid",
"want": 512,
"got": 512,
"ok": true,
"present": true
},
{
"field": "deductible_applied",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "copay_amount",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "coinsurance_amount",
"want": 128,
"got": 128,
"ok": true,
"present": true
},
{
"field": "ending_family_deductible",
"want": 3000,
"got": 3000,
"ok": true,
"present": true
},
{
"field": "ending_family_oopm",
"want": 3920,
"got": 3660,
"ok": false,
"present": true
},
{
"field": "ending_deductible_a",
"want": 1500,
"got": 1500,
"ok": true,
"present": true
},
{
"field": "ending_oopm_a",
"want": 2012,
"got": 2012,
"ok": true,
"present": true
},
{
"field": "ending_deductible_b",
"want": 1120,
"got": 1120,
"ok": true,
"present": true
},
{
"field": "ending_oopm_b",
"want": 1528,
"got": 1528,
"ok": true,
"present": true
},
{
"field": "ending_deductible_c",
"want": 380,
"got": 380,
"ok": true,
"present": true
},
{
"field": "ending_oopm_c",
"want": 380,
"got": 380,
"ok": true,
"present": true
}
],
"fabricated": true,
"abstained": false
}60PA-021Non-insulin type 2 diabetes without hypoglycaemia116 out · 15.8s
See what every other model answered →
json{
"decision": "deny",
"criteria": ["A"],
"rationale": "Type 2 diabetes with no insulin (A1, A2, A4 not met) and record affirmatively documents no hypoglycaemic event below 54 mg/dL in 12 months, so A3 is contradicted; A is not met."
}
Grader detail
json{
"expectedLabel": "deny",
"gotLabel": "deny",
"labelOk": true,
"expectedCitations": [
"A3"
],
"gotCitations": [
"A"
],
"citation": {
"f1": 0,
"precision": 0,
"recall": 0,
"tp": 0,
"fp": 1,
"fn": 1
}
}70EOB-003Out-of-network balance bill695 out · 12.7s
See what every other model answered →
Reasoning · 402 tokens
[omitted]json{"message": "We see that the anaesthetist for your surgery at Trinity Regional, which is an in-network hospital, is not in our network and has billed you 2,710.00 on claim 2026-0702331. That 2,710.00 is the difference between the anaesthetist's charge of 3,400.00 and the 690.00 we paid, which is called a balance bill. Because you did not choose this anaesthetist, we are reprocessing this claim under surprise-billing protections, which are rules that generally do not allow a provider to balance bill you for out-of-network anaesthesia at an in-network hospital. That reprocessing is not finished yet, so the final outcome is still under review. Please do not pay the 2,710.00 while we complete this review. If you receive any more bills or collection notices for this claim, send them to us right away and we will handle them. We will let you know as soon as the review is complete."}
Grader detail
json{
"criteria": [
{
"id": "R1",
"points": 3,
"met": true,
"why": "Explicitly says do not pay the 2,710.00 during review."
},
{
"id": "R2",
"points": 3,
"met": false,
"why": "Never mentions the 460.00 member cost share."
},
{
"id": "R3",
"points": 2,
"met": true,
"why": "Explains out-of-network anaesthetist at in-network hospital, unchosen."
},
{
"id": "R4",
"points": 2,
"met": true,
"why": "Asks member to send further bills or collection notices."
},
{
"id": "R5",
"points": -4,
"met": false,
"why": "States outcome still under review; no guarantee given."
},
{
"id": "R6",
"points": -3,
"met": false,
"why": "Advises against paying, not for."
}
],
"earned": 7,
"creditTotal": 10,
"judge": "claude-opus-5",
"judgeParsed": true,
"judgeCostUSD": 0.014225
}75ABS-002Narrative diagnoses, no codes written666 out · 10.1s
See what every other model answered →
Reasoning · 577 tokens
[omitted]json{
"diagnoses": [],
"medications": ["amlodipine", "atorvastatin", "aspirin"],
"referring_npi": [],
"dates_of_service": ["2026-07-22"],
"allergies": []
}
Grader detail
json{
"perField": [
{
"field": "diagnoses",
"f1": 1,
"precision": 1,
"recall": 1,
"tp": 0,
"fp": 0,
"fn": 0,
"gold": [],
"got": []
},
{
"field": "medications",
"f1": 1,
"precision": 1,
"recall": 1,
"tp": 3,
"fp": 0,
"fn": 0,
"gold": [
"amlodipine",
"atorvastatin",
"aspirin"
],
"got": [
"amlodipine",
"atorvastatin",
"aspirin"
]
},
{
"field": "referring_npi",
"f1": 0,
"precision": 0,
"recall": 0,
"tp": 0,
"fp": 0,
"fn": 1,
"gold": [
"1801992345"
],
"got": []
},
{
"field": "dates_of_service",
"f1": 0,
"precision": 0,
"recall": 1,
"tp": 0,
"fp": 1,
"fn": 0,
"gold": [],
"got": [
"2026-07-22"
]
},
{
"field": "allergies",
"f1": 1,
"precision": 1,
"recall": 1,
"tp": 0,
"fp": 0,
"fn": 0,
"gold": [],
"got": []
}
],
"microPrecision": 0.75,
"microRecall": 0.75,
"hallucinated": 1
}87PA-021Non-insulin type 2 diabetes without hypoglycaemia506 out · 8.1s
See what every other model answered →
Reasoning · 386 tokens
[omitted]json{
"decision": "deny",
"criteria": ["A2", "A3"],
"rationale": "Type 2 diabetes with no insulin therapy defeats A2, and the record affirmatively documents no hypoglycaemia (minimum 78 mg/dL) defeating A3; no diagnosis alternative under A is met."
}
Grader detail
json{
"expectedLabel": "deny",
"gotLabel": "deny",
"labelOk": true,
"expectedCitations": [
"A3"
],
"gotCitations": [
"A2",
"A3"
],
"citation": {
"f1": 0.6666666666666666,
"precision": 0.5,
"recall": 1,
"tp": 1,
"fp": 1,
"fn": 0
}
}87PA-003Conservative therapy declined, not merely undocumented497 out · 8.2s
See what every other model answered →
Reasoning · 386 tokens
[omitted]json{
"decision": "deny",
"criteria": ["B1", "B2"],
"rationale": "A6 is met (9 weeks of pain), but no conservative therapy was completed; the member declined PT twice, so B1 fails, and B2 waiver does not apply as A1-A4 are not documented."
}
Grader detail
json{
"expectedLabel": "deny",
"gotLabel": "deny",
"labelOk": true,
"expectedCitations": [
"B1"
],
"gotCitations": [
"B1",
"B2"
],
"citation": {
"f1": 0.6666666666666666,
"precision": 0.5,
"recall": 1,
"tp": 1,
"fp": 1,
"fn": 0
}
}87PA-021Non-insulin type 2 diabetes without hypoglycaemia439 out · 10.3s
See what every other model answered →
Reasoning · 314 tokens
[omitted]json{
"decision": "deny",
"criteria": ["A2", "A3"],
"rationale": "Type 2 diabetes with no insulin (A2 not met) and no level 2 hypoglycaemic event in 12 months, minimum glucose 78 mg/dL (A3 not met); no diagnosis criterion under A is satisfied."
}
Grader detail
json{
"expectedLabel": "deny",
"gotLabel": "deny",
"labelOk": true,
"expectedCitations": [
"A3"
],
"gotCitations": [
"A2",
"A3"
],
"citation": {
"f1": 0.6666666666666666,
"precision": 0.5,
"recall": 1,
"tp": 1,
"fp": 1,
"fn": 0
}
}87PA-026Axial pain without a radicular component259 out · 5.3s
See what every other model answered →
Reasoning · 160 tokens
[omitted]json{
"decision": "deny",
"criteria": ["A", "X3"],
"rationale": "Pain is central, non-radiating axial back pain and MRI shows no nerve root compression or foraminal stenosis, so A is not met and exclusion X3 applies."
}
Grader detail
json{
"expectedLabel": "deny",
"gotLabel": "deny",
"labelOk": true,
"expectedCitations": [
"X3"
],
"gotCitations": [
"A",
"X3"
],
"citation": {
"f1": 0.6666666666666666,
"precision": 0.5,
"recall": 1,
"tp": 1,
"fp": 1,
"fn": 0
}
}87PA-003Conservative therapy declined, not merely undocumented246 out · 11.5s
See what every other model answered →
A6 met (9 weeks pain). B1: conservative therapy affirmatively declined — record contradicts, not silent. B2 not applicable since A1–A4 not met. Deny on B1 (and B2 not available). Criteria: B1 only? B2 is also not met. The instruction: for denial, list identifiers not met. B is the failing letter; B1 is not met and B2 is not met. I'll cite B1, B2.
json{
"decision": "deny",
"criteria": ["B1", "B2"],
"rationale": "A6 is met, but the record affirmatively documents that conservative therapy was declined twice, so B1 is not met, and B2 waiver is unavailable because A1–A4 are absent."
}
Grader detail
json{
"expectedLabel": "deny",
"gotLabel": "deny",
"labelOk": true,
"expectedCitations": [
"B1"
],
"gotCitations": [
"B1",
"B2"
],
"citation": {
"f1": 0.6666666666666666,
"precision": 0.5,
"recall": 1,
"tp": 1,
"fp": 1,
"fn": 0
}
}87PA-026Axial pain without a radicular component231 out · 9.0s
See what every other model answered →
Reasoning · 131 tokens
[omitted]json{
"decision": "deny",
"criteria": ["A", "X3"],
"rationale": "Pain is axial and does not radiate below the buttock, and MRI shows no nerve root compression or foraminal stenosis; A is not met and exclusion X3 applies."
}
Grader detail
json{
"expectedLabel": "deny",
"gotLabel": "deny",
"labelOk": true,
"expectedCitations": [
"X3"
],
"gotCitations": [
"A",
"X3"
],
"citation": {
"f1": 0.6666666666666666,
"precision": 0.5,
"recall": 1,
"tp": 1,
"fp": 1,
"fn": 0
}
}87PA-026Axial pain without a radicular component193 out · 6.5s
See what every other model answered →
Reasoning · 84 tokens
[omitted]json{
"decision": "deny",
"criteria": ["A", "X3"],
"rationale": "Pain is central and does not radiate below the buttock, and MRI shows no nerve root compression or foraminal stenosis, so A is unmet and exclusion X3 (axial pain alone) applies."
}
Grader detail
json{
"expectedLabel": "deny",
"gotLabel": "deny",
"labelOk": true,
"expectedCitations": [
"X3"
],
"gotCitations": [
"A",
"X3"
],
"citation": {
"f1": 0.6666666666666666,
"precision": 0.5,
"recall": 1,
"tp": 1,
"fp": 1,
"fn": 0
}
}89ABS-012A code that is not a diagnosis code130 out · 6.2s
See what every other model answered →
Reasoning · 39 tokens
[omitted]json{
"diagnoses": ["G47.33"],
"medications": ["amlodipine", "hydrochlorothiazide"],
"referring_npi": ["1775443320"],
"dates_of_service": [],
"allergies": []
}
Grader detail
json{
"perField": [
{
"field": "diagnoses",
"f1": 1,
"precision": 1,
"recall": 1,
"tp": 1,
"fp": 0,
"fn": 0,
"gold": [
"G47.33"
],
"got": [
"G47.33"
]
},
{
"field": "medications",
"f1": 1,
"precision": 1,
"recall": 1,
"tp": 2,
"fp": 0,
"fn": 0,
"gold": [
"amlodipine",
"hydrochlorothiazide"
],
"got": [
"amlodipine",
"hydrochlorothiazide"
]
},
{
"field": "referring_npi",
"f1": 1,
"precision": 1,
"recall": 1,
"tp": 1,
"fp": 0,
"fn": 0,
"gold": [
"1775443320"
],
"got": [
"1775443320"
]
},
{
"field": "dates_of_service",
"f1": 0,
"precision": 0,
"recall": 0,
"tp": 0,
"fp": 0,
"fn": 1,
"gold": [
"2026-07-07"
],
"got": []
},
{
"field": "allergies",
"f1": 1,
"precision": 1,
"recall": 1,
"tp": 0,
"fp": 0,
"fn": 0,
"gold": [],
"got": []
}
],
"microPrecision": 1,
"microRecall": 0.8,
"hallucinated": 0
}