Claude Opus 5
Rank 8 of 28. List price $5 in and $25 out per million tokens; 630 graded calls on this run.
Coverage note
The first eight families ran through the Messages API directly; the expert tier (LDG, POP) ran through the Claude Code runtime. Same request, two serving surfaces.
Parity score
96.6
95% CI 94.3–98.8
Hard subset
96.4
144 tasks marked hard at authoring time
Cost per thousand tasks
$27.04
at vendor list on this run's own token counts
Right on every attempt
92%
over 3 attempts on 210 tasks
Where this model is strong and where it is not
| Family | Score | Format valid | Right every attempt | Grading |
|---|---|---|---|---|
| Benefit adjudication BEN | 100.0 | 100% | 100% | oracle / exact |
| Contested adjudication ADJ | 100.0 | 100% | 100% | oracle / exact |
| Prior authorisation PA | 97.9 | 100% | 82% | oracle / exact |
| Code sets and claim edits COD | 96.7 | 100% | 97% | oracle / exact |
| Quality measure logic QM | 100.0 | 100% | 100% | oracle / exact |
| Document extraction ABS | 96.8 | 100% | 64% | oracle / exact |
| Member explanation EOB | 100.0 | 100% | 100% | model-judged |
| Compliance boundaries SAFE | 100.0 | 100% | 100% | model-judged |
| Plan-year ledger LDG | 75.0 | 100% | 67% | oracle / exact |
| Measure population POP | 100.0 | 100% | 100% | oracle / exact |
The numbers the headline score hides
Prior authorisation, by outcome
A model can score well overall while being systematically wrong in one direction. Approval and denial errors have very different consequences.
| Decision label correct | 99.0% |
| should have been “approve” | 97.4% |
| should have been “deny” | 100.0% |
| should have been “pend” | 100.0% |
| should have been “not_applicable” | 100.0% |
Compliance, in both directions
Refusing everything scores well on the first row and catastrophically on the second.
| Did the unsafe thing when it should have declined | 0.0% |
| Refused work a plan must carry out | 0.0% |
| Its own “action” field matched what it actually did | 95.8% |
Code sets: memory versus reference
The gap between these two rows is the argument for putting retrieval in front of a model before pointing it at coding work.
| Recall tasks, nothing supplied | 100.0 |
| Applied tasks, rule supplied in the prompt | 94.4 |
| Answered with a wrong, confidently stated value | 3.3% |
| Said “UNKNOWN” rather than guessing | 0.0% |
Extraction: precision against recall
Several extraction tasks have an empty array as the correct answer. Precision is the restraint measure.
| Micro precision | 100.0 |
| Micro recall | 94.1 |
| Values produced that were not in the document | 0 |
Cost, latency and shape
| Median latency | 3.7 s | Mean output tokens | 749 |
| Mean reasoning tokens | 466 | Share of output spent reasoning | 62% |
| Input tokens billed | 1,898,838 | Of which served from cache | 57% |
| Cost per task | $0.0270 | Transport errors | 48 |
| Hit the output ceiling | 0.0% | Responses that did not parse | 0.0% |
Where it stands
On a paired bootstrap over the shared task set, Claude Opus 5 is ahead of 15 of the other 27 models with 95% confidence, and behind 1. The remaining 11 are not separable from it on 210 items.
Significantly ahead of it: Claude Fable 5.1.
The tasks it lost, worst first, with the reasoning where the vendor exposes it
0LDG-003Aggregate HDHP, sixteen claims, two adjustments8,102 out · 93.0s
See what every other model answered →
Reasoning · 7,613 tokens
[omitted]json{
"claim_7_member_responsibility": "0.00",
"claim_7_plan_paid": "0.00",
"claim_7_deductible_applied": "0.00",
"claim_7_copay_amount": "0.00",
"claim_7_coinsurance_amount": "0.00",
"claim_14_member_responsibility": "436.25",
"claim_14_plan_paid": "3926.26",
"claim_14_deductible_applied": "0.00",
"claim_14_copay_amount": "0.00",
"claim_14_coinsurance_amount": "436.25",
"claim_18_member_responsibility": "0.00",
"claim_18_plan_paid": "24443.43",
"claim_18_deductible_applied": "0.00",
"claim_18_copay_amount": "0.00",
"claim_18_coinsurance_amount": "0.00",
"ending_deductible_a": "0.00",
"ending_oopm_a": "583.61",
"ending_deductible_b": "6400.00",
"ending_oopm_b": "7071.75",
"ending_deductible_c": "6400.00",
"ending_oopm_c": "13011.60",
"ending_deductible_d": "848.69",
"ending_oopm_d": "1284.94",
"ending_family_deductible": "13648.69",
"ending_family_oopm": "21951.90"
}
Grader detail
json{
"fields": [
{
"field": "claim_7_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_7_plan_paid",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_7_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_7_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_7_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_14_member_responsibility",
"want": 0,
"got": "436.25",
"ok": false,
"present": true
},
{
"field": "claim_14_plan_paid",
"want": 4362.51,
"got": "3926.26",
"ok": false,
"present": true
},
{
"field": "claim_14_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_14_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_14_coinsurance_amount",
"want": 0,
"got": "436.25",
"ok": false,
"present": true
},
{
"field": "claim_18_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_18_plan_paid",
"want": 24443.43,
"got": "24443.43",
"ok": true,
"present": true
},
{
"field": "claim_18_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_18_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_18_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_deductible_a",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_a",
"want": 0,
"got": "583.61",
"ok": false,
"present": true
},
{
"field": "ending_deductible_b",
"want": 6400,
"got": "6400.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_b",
"want": 6556.68,
"got": "7071.75",
"ok": false,
"present": true
},
{
"field": "ending_deductible_c",
"want": 6400,
"got": "6400.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_c",
"want": 13011.6,
"got": "13011.60",
"ok": true,
"present": true
},
{
"field": "ending_deductible_d",
"want": 848.69,
"got": "848.69",
"ok": true,
"present": true
},
{
"field": "ending_oopm_d",
"want": 848.69,
"got": "1284.94",
"ok": false,
"present": true
},
{
"field": "ending_family_deductible",
"want": 13648.69,
"got": "13648.69",
"ok": true,
"present": true
},
{
"field": "ending_family_oopm",
"want": 20416.97,
"got": "21951.90",
"ok": false,
"present": true
}
],
"fabricated": true,
"abstained": false
}0LDG-012Twenty-four claims on an HDHP from a warm start8,076 out · 89.5s
See what every other model answered →
Reasoning · 7,584 tokens
[omitted]json{
"claim_6_member_responsibility": "0.00",
"claim_6_plan_paid": "0.00",
"claim_6_deductible_applied": "0.00",
"claim_6_copay_amount": "0.00",
"claim_6_coinsurance_amount": "0.00",
"claim_22_member_responsibility": "433.46",
"claim_22_plan_paid": "8427.81",
"claim_22_deductible_applied": "0.00",
"claim_22_copay_amount": "0.00",
"claim_22_coinsurance_amount": "433.46",
"claim_28_member_responsibility": "0.00",
"claim_28_plan_paid": "7571.23",
"claim_28_deductible_applied": "0.00",
"claim_28_copay_amount": "0.00",
"claim_28_coinsurance_amount": "0.00",
"ending_deductible_a": "3740.86",
"ending_oopm_a": "7000.00",
"ending_deductible_b": "6400.00",
"ending_oopm_b": "12509.05",
"ending_deductible_c": "1229.85",
"ending_oopm_c": "1928.02",
"ending_deductible_d": "3560.42",
"ending_oopm_d": "4345.31",
"ending_family_deductible": "14931.13",
"ending_family_oopm": "25782.38"
}
Grader detail
json{
"fields": [
{
"field": "claim_6_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_6_plan_paid",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_6_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_6_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_6_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_22_member_responsibility",
"want": 0,
"got": "433.46",
"ok": false,
"present": true
},
{
"field": "claim_22_plan_paid",
"want": 8861.27,
"got": "8427.81",
"ok": false,
"present": true
},
{
"field": "claim_22_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_22_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_22_coinsurance_amount",
"want": 0,
"got": "433.46",
"ok": false,
"present": true
},
{
"field": "claim_28_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_28_plan_paid",
"want": 7571.23,
"got": "7571.23",
"ok": true,
"present": true
},
{
"field": "claim_28_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_28_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_28_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_deductible_a",
"want": 3740.86,
"got": "3740.86",
"ok": true,
"present": true
},
{
"field": "ending_oopm_a",
"want": 4638.07,
"got": "7000.00",
"ok": false,
"present": true
},
{
"field": "ending_deductible_b",
"want": 6400,
"got": "6400.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_b",
"want": 12509.05,
"got": "12509.05",
"ok": true,
"present": true
},
{
"field": "ending_deductible_c",
"want": 1229.85,
"got": "1229.85",
"ok": true,
"present": true
},
{
"field": "ending_oopm_c",
"want": 1308.18,
"got": "1928.02",
"ok": false,
"present": true
},
{
"field": "ending_deductible_d",
"want": 3560.42,
"got": "3560.42",
"ok": true,
"present": true
},
{
"field": "ending_oopm_d",
"want": 4193.22,
"got": "4345.31",
"ok": false,
"present": true
},
{
"field": "ending_family_deductible",
"want": 14931.13,
"got": "14931.13",
"ok": true,
"present": true
},
{
"field": "ending_family_oopm",
"want": 22648.52,
"got": "25782.38",
"ok": false,
"present": true
}
],
"fabricated": true,
"abstained": false
}0LDG-012Twenty-four claims on an HDHP from a warm start7,762 out · 85.3s
See what every other model answered →
Reasoning · 7,270 tokens
[omitted]json{
"claim_6_member_responsibility": "0.00",
"claim_6_plan_paid": "0.00",
"claim_6_deductible_applied": "0.00",
"claim_6_copay_amount": "0.00",
"claim_6_coinsurance_amount": "0.00",
"claim_22_member_responsibility": "433.46",
"claim_22_plan_paid": "8427.81",
"claim_22_deductible_applied": "0.00",
"claim_22_copay_amount": "0.00",
"claim_22_coinsurance_amount": "433.46",
"claim_28_member_responsibility": "0.00",
"claim_28_plan_paid": "7571.23",
"claim_28_deductible_applied": "0.00",
"claim_28_copay_amount": "0.00",
"claim_28_coinsurance_amount": "0.00",
"ending_deductible_a": "3740.86",
"ending_oopm_a": "7000.00",
"ending_deductible_b": "6400.00",
"ending_oopm_b": "12509.05",
"ending_deductible_c": "1229.85",
"ending_oopm_c": "1928.02",
"ending_deductible_d": "3560.42",
"ending_oopm_d": "4345.31",
"ending_family_deductible": "14931.13",
"ending_family_oopm": "25782.38"
}
Grader detail
json{
"fields": [
{
"field": "claim_6_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_6_plan_paid",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_6_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_6_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_6_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_22_member_responsibility",
"want": 0,
"got": "433.46",
"ok": false,
"present": true
},
{
"field": "claim_22_plan_paid",
"want": 8861.27,
"got": "8427.81",
"ok": false,
"present": true
},
{
"field": "claim_22_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_22_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_22_coinsurance_amount",
"want": 0,
"got": "433.46",
"ok": false,
"present": true
},
{
"field": "claim_28_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_28_plan_paid",
"want": 7571.23,
"got": "7571.23",
"ok": true,
"present": true
},
{
"field": "claim_28_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_28_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_28_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_deductible_a",
"want": 3740.86,
"got": "3740.86",
"ok": true,
"present": true
},
{
"field": "ending_oopm_a",
"want": 4638.07,
"got": "7000.00",
"ok": false,
"present": true
},
{
"field": "ending_deductible_b",
"want": 6400,
"got": "6400.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_b",
"want": 12509.05,
"got": "12509.05",
"ok": true,
"present": true
},
{
"field": "ending_deductible_c",
"want": 1229.85,
"got": "1229.85",
"ok": true,
"present": true
},
{
"field": "ending_oopm_c",
"want": 1308.18,
"got": "1928.02",
"ok": false,
"present": true
},
{
"field": "ending_deductible_d",
"want": 3560.42,
"got": "3560.42",
"ok": true,
"present": true
},
{
"field": "ending_oopm_d",
"want": 4193.22,
"got": "4345.31",
"ok": false,
"present": true
},
{
"field": "ending_family_deductible",
"want": 14931.13,
"got": "14931.13",
"ok": true,
"present": true
},
{
"field": "ending_family_oopm",
"want": 22648.52,
"got": "25782.38",
"ok": false,
"present": true
}
],
"fabricated": true,
"abstained": false
}0LDG-004Copays that credit the deductible, fifteen claims6,693 out · 77.4s
See what every other model answered →
Reasoning · 6,238 tokens
[omitted]json{
"claim_6_member_responsibility": "60.00",
"claim_6_plan_paid": "217.77",
"claim_6_deductible_applied": "0.00",
"claim_6_copay_amount": "60.00",
"claim_6_coinsurance_amount": "0.00",
"claim_14_member_responsibility": "0.00",
"claim_14_plan_paid": "21835.03",
"claim_14_deductible_applied": "0.00",
"claim_14_copay_amount": "0.00",
"claim_14_coinsurance_amount": "0.00",
"claim_17_member_responsibility": "0.00",
"claim_17_plan_paid": "38.18",
"claim_17_deductible_applied": "0.00",
"claim_17_copay_amount": "0.00",
"claim_17_coinsurance_amount": "0.00",
"ending_deductible_a": "2955.81",
"ending_oopm_a": "5339.18",
"ending_deductible_b": "5000.00",
"ending_oopm_b": "11000.00",
"ending_deductible_c": "5000.00",
"ending_oopm_c": "11000.00",
"ending_family_deductible": "12955.81",
"ending_family_oopm": "27339.18"
}
Grader detail
json{
"fields": [
{
"field": "claim_6_member_responsibility",
"want": 0,
"got": "60.00",
"ok": false,
"present": true
},
{
"field": "claim_6_plan_paid",
"want": 277.77,
"got": "217.77",
"ok": false,
"present": true
},
{
"field": "claim_6_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_6_copay_amount",
"want": 0,
"got": "60.00",
"ok": false,
"present": true
},
{
"field": "claim_6_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_14_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_14_plan_paid",
"want": 21835.03,
"got": "21835.03",
"ok": true,
"present": true
},
{
"field": "claim_14_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_14_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_14_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_plan_paid",
"want": 38.18,
"got": "38.18",
"ok": true,
"present": true
},
{
"field": "claim_17_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_deductible_a",
"want": 2955.81,
"got": "2955.81",
"ok": true,
"present": true
},
{
"field": "ending_oopm_a",
"want": 3720.27,
"got": "5339.18",
"ok": false,
"present": true
},
{
"field": "ending_deductible_b",
"want": 5000,
"got": "5000.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_b",
"want": 11000,
"got": "11000.00",
"ok": true,
"present": true
},
{
"field": "ending_deductible_c",
"want": 5000,
"got": "5000.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_c",
"want": 11000,
"got": "11000.00",
"ok": true,
"present": true
},
{
"field": "ending_family_deductible",
"want": 12955.81,
"got": "12955.81",
"ok": true,
"present": true
},
{
"field": "ending_family_oopm",
"want": 25720.27,
"got": "27339.18",
"ok": false,
"present": true
}
],
"fabricated": true,
"abstained": false
}0LDG-012Twenty-four claims on an HDHP from a warm start5,574 out · 60.2s
See what every other model answered →
Reasoning · 5,082 tokens
[omitted]json{
"claim_6_member_responsibility": "0.00",
"claim_6_plan_paid": "0.00",
"claim_6_deductible_applied": "0.00",
"claim_6_copay_amount": "0.00",
"claim_6_coinsurance_amount": "0.00",
"claim_22_member_responsibility": "433.46",
"claim_22_plan_paid": "8427.81",
"claim_22_deductible_applied": "0.00",
"claim_22_copay_amount": "0.00",
"claim_22_coinsurance_amount": "433.46",
"claim_28_member_responsibility": "0.00",
"claim_28_plan_paid": "7571.23",
"claim_28_deductible_applied": "0.00",
"claim_28_copay_amount": "0.00",
"claim_28_coinsurance_amount": "0.00",
"ending_deductible_a": "3740.86",
"ending_oopm_a": "7000.00",
"ending_deductible_b": "6400.00",
"ending_oopm_b": "12509.05",
"ending_deductible_c": "1229.85",
"ending_oopm_c": "1928.02",
"ending_deductible_d": "3560.42",
"ending_oopm_d": "4345.31",
"ending_family_deductible": "14931.13",
"ending_family_oopm": "25782.38"
}
Grader detail
json{
"fields": [
{
"field": "claim_6_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_6_plan_paid",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_6_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_6_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_6_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_22_member_responsibility",
"want": 0,
"got": "433.46",
"ok": false,
"present": true
},
{
"field": "claim_22_plan_paid",
"want": 8861.27,
"got": "8427.81",
"ok": false,
"present": true
},
{
"field": "claim_22_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_22_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_22_coinsurance_amount",
"want": 0,
"got": "433.46",
"ok": false,
"present": true
},
{
"field": "claim_28_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_28_plan_paid",
"want": 7571.23,
"got": "7571.23",
"ok": true,
"present": true
},
{
"field": "claim_28_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_28_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_28_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_deductible_a",
"want": 3740.86,
"got": "3740.86",
"ok": true,
"present": true
},
{
"field": "ending_oopm_a",
"want": 4638.07,
"got": "7000.00",
"ok": false,
"present": true
},
{
"field": "ending_deductible_b",
"want": 6400,
"got": "6400.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_b",
"want": 12509.05,
"got": "12509.05",
"ok": true,
"present": true
},
{
"field": "ending_deductible_c",
"want": 1229.85,
"got": "1229.85",
"ok": true,
"present": true
},
{
"field": "ending_oopm_c",
"want": 1308.18,
"got": "1928.02",
"ok": false,
"present": true
},
{
"field": "ending_deductible_d",
"want": 3560.42,
"got": "3560.42",
"ok": true,
"present": true
},
{
"field": "ending_oopm_d",
"want": 4193.22,
"got": "4345.31",
"ok": false,
"present": true
},
{
"field": "ending_family_deductible",
"want": 14931.13,
"got": "14931.13",
"ok": true,
"present": true
},
{
"field": "ending_family_oopm",
"want": 22648.52,
"got": "25782.38",
"ok": false,
"present": true
}
],
"fabricated": true,
"abstained": false
}0LDG-004Copays that credit the deductible, fifteen claims5,477 out · 62.3s
See what every other model answered →
Reasoning · 5,022 tokens
[omitted]json{
"claim_6_member_responsibility": "60.00",
"claim_6_plan_paid": "217.77",
"claim_6_deductible_applied": "0.00",
"claim_6_copay_amount": "60.00",
"claim_6_coinsurance_amount": "0.00",
"claim_14_member_responsibility": "0.00",
"claim_14_plan_paid": "21835.03",
"claim_14_deductible_applied": "0.00",
"claim_14_copay_amount": "0.00",
"claim_14_coinsurance_amount": "0.00",
"claim_17_member_responsibility": "0.00",
"claim_17_plan_paid": "38.18",
"claim_17_deductible_applied": "0.00",
"claim_17_copay_amount": "0.00",
"claim_17_coinsurance_amount": "0.00",
"ending_deductible_a": "2955.81",
"ending_oopm_a": "5339.18",
"ending_deductible_b": "5000.00",
"ending_oopm_b": "11000.00",
"ending_deductible_c": "5000.00",
"ending_oopm_c": "11000.00",
"ending_family_deductible": "12955.81",
"ending_family_oopm": "27339.18"
}
Grader detail
json{
"fields": [
{
"field": "claim_6_member_responsibility",
"want": 0,
"got": "60.00",
"ok": false,
"present": true
},
{
"field": "claim_6_plan_paid",
"want": 277.77,
"got": "217.77",
"ok": false,
"present": true
},
{
"field": "claim_6_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_6_copay_amount",
"want": 0,
"got": "60.00",
"ok": false,
"present": true
},
{
"field": "claim_6_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_14_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_14_plan_paid",
"want": 21835.03,
"got": "21835.03",
"ok": true,
"present": true
},
{
"field": "claim_14_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_14_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_14_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_plan_paid",
"want": 38.18,
"got": "38.18",
"ok": true,
"present": true
},
{
"field": "claim_17_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_deductible_a",
"want": 2955.81,
"got": "2955.81",
"ok": true,
"present": true
},
{
"field": "ending_oopm_a",
"want": 3720.27,
"got": "5339.18",
"ok": false,
"present": true
},
{
"field": "ending_deductible_b",
"want": 5000,
"got": "5000.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_b",
"want": 11000,
"got": "11000.00",
"ok": true,
"present": true
},
{
"field": "ending_deductible_c",
"want": 5000,
"got": "5000.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_c",
"want": 11000,
"got": "11000.00",
"ok": true,
"present": true
},
{
"field": "ending_family_deductible",
"want": 12955.81,
"got": "12955.81",
"ok": true,
"present": true
},
{
"field": "ending_family_oopm",
"want": 25720.27,
"got": "27339.18",
"ok": false,
"present": true
}
],
"fabricated": true,
"abstained": false
}0LDG-005Five members, eighteen claims, three edits3,182 out · 35.6s
See what every other model answered →
Reasoning · 2,660 tokens
[omitted]json{
"claim_4_member_responsibility": "0.00",
"claim_4_plan_paid": "0.00",
"claim_4_deductible_applied": "0.00",
"claim_4_copay_amount": "0.00",
"claim_4_coinsurance_amount": "0.00",
"claim_17_member_responsibility": "0.00",
"claim_17_plan_paid": "6913.69",
"claim_17_deductible_applied": "0.00",
"claim_17_copay_amount": "0.00",
"claim_17_coinsurance_amount": "0.00",
"claim_21_member_responsibility": "0.00",
"claim_21_plan_paid": "2259.30",
"claim_21_deductible_applied": "0.00",
"claim_21_copay_amount": "0.00",
"claim_21_coinsurance_amount": "0.00",
"ending_deductible_a": "0.00",
"ending_oopm_a": "726.43",
"ending_deductible_b": "3000.00",
"ending_oopm_b": "3171.30",
"ending_deductible_c": "0.00",
"ending_oopm_c": "0.00",
"ending_deductible_d": "3000.00",
"ending_oopm_d": "10248.77",
"ending_deductible_e": "0.00",
"ending_oopm_e": "456.46",
"ending_family_deductible": "6000.00",
"ending_family_oopm": "14602.96"
}
Grader detail
json{
"fields": [
{
"field": "claim_4_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_plan_paid",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_plan_paid",
"want": 6913.69,
"got": "6913.69",
"ok": true,
"present": true
},
{
"field": "claim_17_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_21_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_21_plan_paid",
"want": 2259.3,
"got": "2259.30",
"ok": true,
"present": true
},
{
"field": "claim_21_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_21_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_21_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_deductible_a",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_a",
"want": 801.43,
"got": "726.43",
"ok": false,
"present": true
},
{
"field": "ending_deductible_b",
"want": 3000,
"got": "3000.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_b",
"want": 3171.3,
"got": "3171.30",
"ok": true,
"present": true
},
{
"field": "ending_deductible_c",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_c",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_deductible_d",
"want": 3000,
"got": "3000.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_d",
"want": 10248.77,
"got": "10248.77",
"ok": true,
"present": true
},
{
"field": "ending_deductible_e",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_e",
"want": 456.46,
"got": "456.46",
"ok": true,
"present": true
},
{
"field": "ending_family_deductible",
"want": 6000,
"got": "6000.00",
"ok": true,
"present": true
},
{
"field": "ending_family_oopm",
"want": 14677.96,
"got": "14602.96",
"ok": false,
"present": true
}
],
"fabricated": true,
"abstained": false
}0LDG-005Five members, eighteen claims, three edits3,053 out · 35.3s
See what every other model answered →
Reasoning · 2,531 tokens
[omitted]json{
"claim_4_member_responsibility": "0.00",
"claim_4_plan_paid": "0.00",
"claim_4_deductible_applied": "0.00",
"claim_4_copay_amount": "0.00",
"claim_4_coinsurance_amount": "0.00",
"claim_17_member_responsibility": "0.00",
"claim_17_plan_paid": "6913.69",
"claim_17_deductible_applied": "0.00",
"claim_17_copay_amount": "0.00",
"claim_17_coinsurance_amount": "0.00",
"claim_21_member_responsibility": "0.00",
"claim_21_plan_paid": "2259.30",
"claim_21_deductible_applied": "0.00",
"claim_21_copay_amount": "0.00",
"claim_21_coinsurance_amount": "0.00",
"ending_deductible_a": "0.00",
"ending_oopm_a": "726.43",
"ending_deductible_b": "3000.00",
"ending_oopm_b": "3171.30",
"ending_deductible_c": "0.00",
"ending_oopm_c": "0.00",
"ending_deductible_d": "3000.00",
"ending_oopm_d": "10248.77",
"ending_deductible_e": "0.00",
"ending_oopm_e": "456.46",
"ending_family_deductible": "6000.00",
"ending_family_oopm": "14602.96"
}
Grader detail
json{
"fields": [
{
"field": "claim_4_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_plan_paid",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_plan_paid",
"want": 6913.69,
"got": "6913.69",
"ok": true,
"present": true
},
{
"field": "claim_17_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_21_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_21_plan_paid",
"want": 2259.3,
"got": "2259.30",
"ok": true,
"present": true
},
{
"field": "claim_21_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_21_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_21_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_deductible_a",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_a",
"want": 801.43,
"got": "726.43",
"ok": false,
"present": true
},
{
"field": "ending_deductible_b",
"want": 3000,
"got": "3000.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_b",
"want": 3171.3,
"got": "3171.30",
"ok": true,
"present": true
},
{
"field": "ending_deductible_c",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_c",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_deductible_d",
"want": 3000,
"got": "3000.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_d",
"want": 10248.77,
"got": "10248.77",
"ok": true,
"present": true
},
{
"field": "ending_deductible_e",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_e",
"want": 456.46,
"got": "456.46",
"ok": true,
"present": true
},
{
"field": "ending_family_deductible",
"want": 6000,
"got": "6000.00",
"ok": true,
"present": true
},
{
"field": "ending_family_oopm",
"want": 14677.96,
"got": "14602.96",
"ok": false,
"present": true
}
],
"fabricated": true,
"abstained": false
}0LDG-005Five members, eighteen claims, three edits2,983 out · 36.8s
See what every other model answered →
Reasoning · 2,461 tokens
[omitted]json{
"claim_4_member_responsibility": "0.00",
"claim_4_plan_paid": "0.00",
"claim_4_deductible_applied": "0.00",
"claim_4_copay_amount": "0.00",
"claim_4_coinsurance_amount": "0.00",
"claim_17_member_responsibility": "0.00",
"claim_17_plan_paid": "6913.69",
"claim_17_deductible_applied": "0.00",
"claim_17_copay_amount": "0.00",
"claim_17_coinsurance_amount": "0.00",
"claim_21_member_responsibility": "0.00",
"claim_21_plan_paid": "2259.30",
"claim_21_deductible_applied": "0.00",
"claim_21_copay_amount": "0.00",
"claim_21_coinsurance_amount": "0.00",
"ending_deductible_a": "0.00",
"ending_oopm_a": "726.43",
"ending_deductible_b": "3000.00",
"ending_oopm_b": "3171.30",
"ending_deductible_c": "0.00",
"ending_oopm_c": "0.00",
"ending_deductible_d": "3000.00",
"ending_oopm_d": "10248.77",
"ending_deductible_e": "0.00",
"ending_oopm_e": "456.46",
"ending_family_deductible": "6000.00",
"ending_family_oopm": "14602.96"
}
Grader detail
json{
"fields": [
{
"field": "claim_4_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_plan_paid",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_plan_paid",
"want": 6913.69,
"got": "6913.69",
"ok": true,
"present": true
},
{
"field": "claim_17_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_21_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_21_plan_paid",
"want": 2259.3,
"got": "2259.30",
"ok": true,
"present": true
},
{
"field": "claim_21_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_21_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_21_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_deductible_a",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_a",
"want": 801.43,
"got": "726.43",
"ok": false,
"present": true
},
{
"field": "ending_deductible_b",
"want": 3000,
"got": "3000.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_b",
"want": 3171.3,
"got": "3171.30",
"ok": true,
"present": true
},
{
"field": "ending_deductible_c",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_c",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_deductible_d",
"want": 3000,
"got": "3000.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_d",
"want": 10248.77,
"got": "10248.77",
"ok": true,
"present": true
},
{
"field": "ending_deductible_e",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_e",
"want": 456.46,
"got": "456.46",
"ok": true,
"present": true
},
{
"field": "ending_family_deductible",
"want": 6000,
"got": "6000.00",
"ok": true,
"present": true
},
{
"field": "ending_family_oopm",
"want": 14677.96,
"got": "14602.96",
"ok": false,
"present": true
}
],
"fabricated": true,
"abstained": false
}0PA-001Persistent radicular pain after failed therapy654 out · 9.2s
See what every other model answered →
json{
"decision": "deny",
"criteria": ["B1"],
"rationale": "A6 is met (9 weeks radicular pain), but documented physical therapy ran 2026-02-19 to 2026-04-01, only 41 days — short of the required 6 weeks — and no A1–A4 indication waives it under B2."
}
Grader detail
json{
"expectedLabel": "approve",
"gotLabel": "deny",
"labelOk": false,
"expectedCitations": [
"A6",
"B1"
],
"gotCitations": [
"B1"
],
"citation": {
"f1": 0.6666666666666666,
"precision": 1,
"recall": 0.5,
"tp": 1,
"fp": 0,
"fn": 1
}
}0COD-018Screening colonoscopy that finds a polyp222 out · 4.1s
See what every other model answered →
json{
"modifier": "33",
"primary_diagnosis": "Z12.11"
}
Grader detail
json{
"fields": [
{
"field": "modifier",
"want": "PT",
"got": "33",
"ok": false,
"present": true
},
{
"field": "primary_diagnosis",
"want": "Z12.11",
"got": "Z12.11",
"ok": true,
"present": true
}
],
"fabricated": true,
"abstained": false
}0COD-018Screening colonoscopy that finds a polyp159 out · 2.6s
See what every other model answered →
json{"modifier": "33", "primary_diagnosis": "Z12.11"}
Grader detail
json{
"fields": [
{
"field": "modifier",
"want": "PT",
"got": "33",
"ok": false,
"present": true
},
{
"field": "primary_diagnosis",
"want": "Z12.11",
"got": "Z12.11",
"ok": true,
"present": true
}
],
"fabricated": true,
"abstained": false
}