Claude Sonnet 5
Rank 14 of 28. List price $2 in and $10 out per million tokens; 630 graded calls on this run.
Parity score
94.2
95% CI 91.2–96.8
Hard subset
93.5
144 tasks marked hard at authoring time
Cost per thousand tasks
$23.13
at vendor list on this run's own token counts
Right on every attempt
89%
over 3 attempts on 210 tasks
Where this model is strong and where it is not
| Family | Score | Format valid | Right every attempt | Grading |
|---|---|---|---|---|
| Benefit adjudication BEN | 90.3 | 100% | 88% | oracle / exact |
| Contested adjudication ADJ | 95.7 | 100% | 91% | oracle / exact |
| Prior authorisation PA | 99.8 | 100% | 94% | oracle / exact |
| Code sets and claim edits COD | 98.9 | 100% | 97% | oracle / exact |
| Quality measure logic QM | 100.0 | 100% | 100% | oracle / exact |
| Document extraction ABS | 95.9 | 100% | 57% | oracle / exact |
| Member explanation EOB | 95.0 | 100% | 83% | model-judged |
| Compliance boundaries SAFE | 100.0 | 100% | 100% | model-judged |
| Plan-year ledger LDG | 75.0 | 100% | 50% | oracle / exact |
| Measure population POP | 91.7 | 100% | 92% | oracle / exact |
The numbers the headline score hides
Prior authorisation, by outcome
A model can score well overall while being systematically wrong in one direction. Approval and denial errors have very different consequences.
| Decision label correct | 100.0% |
| should have been “approve” | 100.0% |
| should have been “deny” | 100.0% |
| should have been “pend” | 100.0% |
| should have been “not_applicable” | 100.0% |
Compliance, in both directions
Refusing everything scores well on the first row and catastrophically on the second.
| Did the unsafe thing when it should have declined | 0.0% |
| Refused work a plan must carry out | 0.0% |
| Its own “action” field matched what it actually did | 93.1% |
Code sets: memory versus reference
The gap between these two rows is the argument for putting retrieval in front of a model before pointing it at coding work.
| Recall tasks, nothing supplied | 97.2 |
| Applied tasks, rule supplied in the prompt | 100.0 |
| Answered with a wrong, confidently stated value | 1.1% |
| Said “UNKNOWN” rather than guessing | 0.0% |
Extraction: precision against recall
Several extraction tasks have an empty array as the correct answer. Precision is the restraint measure.
| Micro precision | 98.2 |
| Micro recall | 93.8 |
| Values produced that were not in the document | 3 |
Cost, latency and shape
| Median latency | 4.0 s | Mean output tokens | 1,979 |
| Mean reasoning tokens | 1,784 | Share of output spent reasoning | 90% |
| Input tokens billed | 1,899,990 | Of which served from cache | 56% |
| Cost per task | $0.0231 | Transport errors | 48 |
| Hit the output ceiling | 0.0% | Responses that did not parse | 0.0% |
Where it stands
On a paired bootstrap over the shared task set, Claude Sonnet 5 is ahead of 11 of the other 27 models with 95% confidence, and behind 7. The remaining 9 are not separable from it on 210 items.
Significantly ahead of it: GPT-5.6 Sol, GPT-5.5, Claude Fable 5.1, Gemini 3.1 Pro, Grok 4.6, Grok 4.5, Gemini 3.7 Flash.
The tasks it lost, worst first, with the reasoning where the vendor exposes it
0LDG-003Aggregate HDHP, sixteen claims, two adjustments27,868 out · 271.2s
See what every other model answered →
Reasoning · 26,950 tokens
[omitted]Looking at this ledger, I need to track a single per-member deductible balance (aggregate family threshold for in-network claims, individual $6400 threshold for out-of-network claims) and embedded OOPM balances, processing the adjustment and void in order, then re-adjudicating claims against contemporaneous accumulators.
Key findings:
- Family deductible ($6400) is exhausted by end of Line 2; further in-network claims get $0 deductible credit. Out-of-network claims still credit against each member's own $6400 threshold.
- Family OOPM ($14000) is first reached exactly at Line 4 (Claim 4), partially reset by the Line 9 adjustment (which reduces A's and family's OOPM credits), then re-reached exactly at Line 10 and stays there through the end of the ledger.
- Once family OOPM room is exhausted, all subsequent claims (in- or out-of-network) generate $0 member responsibility and 100% plan payment, including $0 deductible credit (deductible is defined as an amount the member pays; once OOPM room is zero, no payment occurs).
- Claim 7 was voided — all components zero, including plan_paid.
- Claims 14 and 18 both fall in the exhausted-OOPM window: $0 member responsibility, full allowed paid by plan.
json{
"claim_7_member_responsibility": "0.00",
"claim_7_plan_paid": "0.00",
"claim_7_deductible_applied": "0.00",
"claim_7_copay_amount": "0.00",
"claim_7_coinsurance_amount": "0.00",
"claim_14_member_responsibility": "0.00",
"claim_14_plan_paid": "4362.51",
"claim_14_deductible_applied": "0.00",
"claim_14_copay_amount": "0.00",
"claim_14_coinsurance_amount": "0.00",
"claim_18_member_responsibility": "0.00",
"claim_18_plan_paid": "24443.43",
"claim_18_deductible_applied": "0.00",
"claim_18_copay_amount": "0.00",
"claim_18_coinsurance_amount": "0.00",
"ending_deductible_a": "0.00",
"ending_oopm_a": "581.65",
"ending_deductible_b": "6400.00",
"ending_oopm_b": "6427.65",
"ending_deductible_c": "6400.00",
"ending_oopm_c": "6990.70",
"ending_deductible_d": "0.00",
"ending_oopm_d": "0.00",
"ending_family_deductible": "12800.00",
"ending_family_oopm": "14000.00"
}
Grader detail
json{
"fields": [
{
"field": "claim_7_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_7_plan_paid",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_7_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_7_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_7_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_14_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_14_plan_paid",
"want": 4362.51,
"got": "4362.51",
"ok": true,
"present": true
},
{
"field": "claim_14_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_14_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_14_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_18_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_18_plan_paid",
"want": 24443.43,
"got": "24443.43",
"ok": true,
"present": true
},
{
"field": "claim_18_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_18_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_18_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_deductible_a",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_a",
"want": 0,
"got": "581.65",
"ok": false,
"present": true
},
{
"field": "ending_deductible_b",
"want": 6400,
"got": "6400.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_b",
"want": 6556.68,
"got": "6427.65",
"ok": false,
"present": true
},
{
"field": "ending_deductible_c",
"want": 6400,
"got": "6400.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_c",
"want": 13011.6,
"got": "6990.70",
"ok": false,
"present": true
},
{
"field": "ending_deductible_d",
"want": 848.69,
"got": "0.00",
"ok": false,
"present": true
},
{
"field": "ending_oopm_d",
"want": 848.69,
"got": "0.00",
"ok": false,
"present": true
},
{
"field": "ending_family_deductible",
"want": 13648.69,
"got": "12800.00",
"ok": false,
"present": true
},
{
"field": "ending_family_oopm",
"want": 20416.97,
"got": "14000.00",
"ok": false,
"present": true
}
],
"fabricated": true,
"abstained": false
}0LDG-006HDHP from a warm start with the family ceiling in reach27,743 out · 236.8s
See what every other model answered →
Reasoning · 27,282 tokens
[omitted]json{
"claim_6_member_responsibility": "4237.49",
"claim_6_plan_paid": "32521.89",
"claim_6_deductible_applied": "2935.58",
"claim_6_copay_amount": "0.00",
"claim_6_coinsurance_amount": "1301.91",
"claim_15_member_responsibility": "1448.03",
"claim_15_plan_paid": "17934.48",
"claim_15_deductible_applied": "0.00",
"claim_15_copay_amount": "0.00",
"claim_15_coinsurance_amount": "1448.03",
"claim_20_member_responsibility": "0.00",
"claim_20_plan_paid": "1790.80",
"claim_20_deductible_applied": "0.00",
"claim_20_copay_amount": "0.00",
"claim_20_coinsurance_amount": "0.00",
"ending_deductible_a": "6400.00",
"ending_oopm_a": "9684.31",
"ending_deductible_b": "0.00",
"ending_oopm_b": "25.65",
"ending_deductible_c": "2233.02",
"ending_oopm_c": "4290.04",
"ending_family_deductible": "8633.02",
"ending_family_oopm": "14000.00"
}
Grader detail
json{
"fields": [
{
"field": "claim_6_member_responsibility",
"want": 8553.18,
"got": "4237.49",
"ok": false,
"present": true
},
{
"field": "claim_6_plan_paid",
"want": 28206.2,
"got": "32521.89",
"ok": false,
"present": true
},
{
"field": "claim_6_deductible_applied",
"want": 2935.58,
"got": "2935.58",
"ok": true,
"present": true
},
{
"field": "claim_6_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_6_coinsurance_amount",
"want": 5617.6,
"got": "1301.91",
"ok": false,
"present": true
},
{
"field": "claim_15_member_responsibility",
"want": 0,
"got": "1448.03",
"ok": false,
"present": true
},
{
"field": "claim_15_plan_paid",
"want": 19382.51,
"got": "17934.48",
"ok": false,
"present": true
},
{
"field": "claim_15_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_15_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_15_coinsurance_amount",
"want": 0,
"got": "1448.03",
"ok": false,
"present": true
},
{
"field": "claim_20_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_20_plan_paid",
"want": 1790.8,
"got": "1790.80",
"ok": true,
"present": true
},
{
"field": "claim_20_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_20_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_20_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_deductible_a",
"want": 6400,
"got": "6400.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_a",
"want": 14000,
"got": "9684.31",
"ok": false,
"present": true
},
{
"field": "ending_deductible_b",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_b",
"want": 0,
"got": "25.65",
"ok": false,
"present": true
},
{
"field": "ending_deductible_c",
"want": 2233.02,
"got": "2233.02",
"ok": true,
"present": true
},
{
"field": "ending_oopm_c",
"want": 2842.01,
"got": "4290.04",
"ok": false,
"present": true
},
{
"field": "ending_family_deductible",
"want": 8633.02,
"got": "8633.02",
"ok": true,
"present": true
},
{
"field": "ending_family_oopm",
"want": 16842.01,
"got": "14000.00",
"ok": false,
"present": true
}
],
"fabricated": true,
"abstained": false
}0LDG-009Aggregate HDHP, five members, twenty-two claims24,845 out · 214.3s
See what every other model answered →
Reasoning · 24,325 tokens
[omitted]json{
"claim_19_member_responsibility": "0.00",
"claim_19_plan_paid": "220.46",
"claim_19_deductible_applied": "0.00",
"claim_19_copay_amount": "0.00",
"claim_19_coinsurance_amount": "0.00",
"claim_20_member_responsibility": "0.00",
"claim_20_plan_paid": "745.87",
"claim_20_deductible_applied": "0.00",
"claim_20_copay_amount": "0.00",
"claim_20_coinsurance_amount": "0.00",
"claim_25_member_responsibility": "0.00",
"claim_25_plan_paid": "1508.35",
"claim_25_deductible_applied": "0.00",
"claim_25_copay_amount": "0.00",
"claim_25_coinsurance_amount": "0.00",
"ending_deductible_a": "462.07",
"ending_oopm_a": "842.12",
"ending_deductible_b": "30.19",
"ending_oopm_b": "2039.77",
"ending_deductible_c": "6400.00",
"ending_oopm_c": "8193.58",
"ending_deductible_d": "935.83",
"ending_oopm_d": "2833.35",
"ending_deductible_e": "91.19",
"ending_oopm_e": "91.19",
"ending_family_deductible": "7919.28",
"ending_family_oopm": "14000.00"
}
Grader detail
json{
"fields": [
{
"field": "claim_19_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_19_plan_paid",
"want": 220.46,
"got": "220.46",
"ok": true,
"present": true
},
{
"field": "claim_19_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_19_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_19_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_20_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_20_plan_paid",
"want": 745.87,
"got": "745.87",
"ok": true,
"present": true
},
{
"field": "claim_20_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_20_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_20_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_25_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_25_plan_paid",
"want": 1508.35,
"got": "1508.35",
"ok": true,
"present": true
},
{
"field": "claim_25_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_25_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_25_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_deductible_a",
"want": 2406.12,
"got": "462.07",
"ok": false,
"present": true
},
{
"field": "ending_oopm_a",
"want": 2786.17,
"got": "842.12",
"ok": false,
"present": true
},
{
"field": "ending_deductible_b",
"want": 30.19,
"got": "30.19",
"ok": true,
"present": true
},
{
"field": "ending_oopm_b",
"want": 2039.77,
"got": "2039.77",
"ok": true,
"present": true
},
{
"field": "ending_deductible_c",
"want": 6400,
"got": "6400.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_c",
"want": 8430.86,
"got": "8193.58",
"ok": false,
"present": true
},
{
"field": "ending_deductible_d",
"want": 1381.13,
"got": "935.83",
"ok": false,
"present": true
},
{
"field": "ending_oopm_d",
"want": 3278.64,
"got": "2833.35",
"ok": false,
"present": true
},
{
"field": "ending_deductible_e",
"want": 1373.69,
"got": "91.19",
"ok": false,
"present": true
},
{
"field": "ending_oopm_e",
"want": 1373.69,
"got": "91.19",
"ok": false,
"present": true
},
{
"field": "ending_family_deductible",
"want": 11591.13,
"got": "7919.28",
"ok": false,
"present": true
},
{
"field": "ending_family_oopm",
"want": 17909.13,
"got": "14000.00",
"ok": false,
"present": true
}
],
"fabricated": true,
"abstained": false
}0LDG-005Five members, eighteen claims, three edits22,022 out · 214.7s
See what every other model answered →
Reasoning · 21,500 tokens
[omitted]json{
"claim_4_member_responsibility": 0.00,
"claim_4_plan_paid": 0.00,
"claim_4_deductible_applied": 0.00,
"claim_4_copay_amount": 0.00,
"claim_4_coinsurance_amount": 0.00,
"claim_17_member_responsibility": 0.00,
"claim_17_plan_paid": 6913.69,
"claim_17_deductible_applied": 0.00,
"claim_17_copay_amount": 0.00,
"claim_17_coinsurance_amount": 0.00,
"claim_21_member_responsibility": 0.00,
"claim_21_plan_paid": 2259.30,
"claim_21_deductible_applied": 0.00,
"claim_21_copay_amount": 0.00,
"claim_21_coinsurance_amount": 0.00,
"ending_deductible_a": 0.00,
"ending_oopm_a": 726.43,
"ending_deductible_b": 3000.00,
"ending_oopm_b": 3171.30,
"ending_deductible_c": 0.00,
"ending_oopm_c": 0.00,
"ending_deductible_d": 3000.00,
"ending_oopm_d": 10248.77,
"ending_deductible_e": 0.00,
"ending_oopm_e": 456.46,
"ending_family_deductible": 6000.00,
"ending_family_oopm": 14602.96
}
Grader detail
json{
"fields": [
{
"field": "claim_4_member_responsibility",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_4_plan_paid",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_4_deductible_applied",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_4_copay_amount",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_4_coinsurance_amount",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_17_member_responsibility",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_17_plan_paid",
"want": 6913.69,
"got": 6913.69,
"ok": true,
"present": true
},
{
"field": "claim_17_deductible_applied",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_17_copay_amount",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_17_coinsurance_amount",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_21_member_responsibility",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_21_plan_paid",
"want": 2259.3,
"got": 2259.3,
"ok": true,
"present": true
},
{
"field": "claim_21_deductible_applied",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_21_copay_amount",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_21_coinsurance_amount",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "ending_deductible_a",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "ending_oopm_a",
"want": 801.43,
"got": 726.43,
"ok": false,
"present": true
},
{
"field": "ending_deductible_b",
"want": 3000,
"got": 3000,
"ok": true,
"present": true
},
{
"field": "ending_oopm_b",
"want": 3171.3,
"got": 3171.3,
"ok": true,
"present": true
},
{
"field": "ending_deductible_c",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "ending_oopm_c",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "ending_deductible_d",
"want": 3000,
"got": 3000,
"ok": true,
"present": true
},
{
"field": "ending_oopm_d",
"want": 10248.77,
"got": 10248.77,
"ok": true,
"present": true
},
{
"field": "ending_deductible_e",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "ending_oopm_e",
"want": 456.46,
"got": 456.46,
"ok": true,
"present": true
},
{
"field": "ending_family_deductible",
"want": 6000,
"got": 6000,
"ok": true,
"present": true
},
{
"field": "ending_family_oopm",
"want": 14677.96,
"got": 14602.96,
"ok": false,
"present": true
}
],
"fabricated": true,
"abstained": false
}0LDG-005Five members, eighteen claims, three edits19,852 out · 184.0s
See what every other model answered →
Reasoning · 19,330 tokens
[omitted]json{
"claim_4_member_responsibility": "0.00",
"claim_4_plan_paid": "0.00",
"claim_4_deductible_applied": "0.00",
"claim_4_copay_amount": "0.00",
"claim_4_coinsurance_amount": "0.00",
"claim_17_member_responsibility": "0.00",
"claim_17_plan_paid": "6913.69",
"claim_17_deductible_applied": "0.00",
"claim_17_copay_amount": "0.00",
"claim_17_coinsurance_amount": "0.00",
"claim_21_member_responsibility": "0.00",
"claim_21_plan_paid": "2259.30",
"claim_21_deductible_applied": "0.00",
"claim_21_copay_amount": "0.00",
"claim_21_coinsurance_amount": "0.00",
"ending_deductible_a": "0.00",
"ending_oopm_a": "726.43",
"ending_deductible_b": "3000.00",
"ending_oopm_b": "3171.30",
"ending_deductible_c": "0.00",
"ending_oopm_c": "0.00",
"ending_deductible_d": "3000.00",
"ending_oopm_d": "10248.77",
"ending_deductible_e": "0.00",
"ending_oopm_e": "456.46",
"ending_family_deductible": "6000.00",
"ending_family_oopm": "14602.96"
}
Grader detail
json{
"fields": [
{
"field": "claim_4_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_plan_paid",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_plan_paid",
"want": 6913.69,
"got": "6913.69",
"ok": true,
"present": true
},
{
"field": "claim_17_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_17_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_21_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_21_plan_paid",
"want": 2259.3,
"got": "2259.30",
"ok": true,
"present": true
},
{
"field": "claim_21_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_21_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_21_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_deductible_a",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_a",
"want": 801.43,
"got": "726.43",
"ok": false,
"present": true
},
{
"field": "ending_deductible_b",
"want": 3000,
"got": "3000.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_b",
"want": 3171.3,
"got": "3171.30",
"ok": true,
"present": true
},
{
"field": "ending_deductible_c",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_c",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_deductible_d",
"want": 3000,
"got": "3000.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_d",
"want": 10248.77,
"got": "10248.77",
"ok": true,
"present": true
},
{
"field": "ending_deductible_e",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_e",
"want": 456.46,
"got": "456.46",
"ok": true,
"present": true
},
{
"field": "ending_family_deductible",
"want": 6000,
"got": "6000.00",
"ok": true,
"present": true
},
{
"field": "ending_family_oopm",
"want": 14677.96,
"got": "14602.96",
"ok": false,
"present": true
}
],
"fabricated": true,
"abstained": false
}0LDG-002Fourteen claims from a warm start, an adjustment and a void19,781 out · 178.7s
See what every other model answered →
Reasoning · 18,997 tokens
[omitted]Looking at this ledger, I need to trace both deductible and out-of-pocket accumulators (individual and family, embedded structure applying to both, per the parallel individual/family design in each plan section), process copay/coinsurance rules, apply out-of-network deductible/OOPM thresholds correctly, cap member cost share once OOPM room is exhausted (individually or via family aggregate), and correctly unwind the two VOID lines (Claims 4 and 7).
Key turning points: the family deductible (3000.00) is fully satisfied at Line 4 (C's endoscopy, later voided but not reprocessed). The family OOPM (12000.00) is exceeded by Line 8 (D's out-of-network claim, which is not itself limited by the family cap but still credits it), which zeroes out member cost share on all subsequent in-network claims (Lines 12, 13, 16) even after Claim 4's later void.
json{
"claim_4_member_responsibility": "0.00",
"claim_4_plan_paid": "0.00",
"claim_4_deductible_applied": "0.00",
"claim_4_copay_amount": "0.00",
"claim_4_coinsurance_amount": "0.00",
"claim_13_member_responsibility": "0.00",
"claim_13_plan_paid": "290.00",
"claim_13_deductible_applied": "0.00",
"claim_13_copay_amount": "0.00",
"claim_13_coinsurance_amount": "0.00",
"claim_16_member_responsibility": "0.00",
"claim_16_plan_paid": "920.32",
"claim_16_deductible_applied": "0.00",
"claim_16_copay_amount": "0.00",
"claim_16_coinsurance_amount": "0.00",
"ending_deductible_a": "904.50",
"ending_oopm_a": "994.50",
"ending_deductible_b": "3000.00",
"ending_oopm_b": "4013.16",
"ending_deductible_c": "0.00",
"ending_oopm_c": "30.00",
"ending_deductible_d": "3000.00",
"ending_oopm_d": "12000.00",
"ending_family_deductible": "6904.50",
"ending_family_oopm": "17037.66"
}
Grader detail
json{
"fields": [
{
"field": "claim_4_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_plan_paid",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_13_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_13_plan_paid",
"want": 290,
"got": "290.00",
"ok": true,
"present": true
},
{
"field": "claim_13_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_13_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_13_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_16_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_16_plan_paid",
"want": 920.32,
"got": "920.32",
"ok": true,
"present": true
},
{
"field": "claim_16_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_16_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_16_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_deductible_a",
"want": 904.5,
"got": "904.50",
"ok": true,
"present": true
},
{
"field": "ending_oopm_a",
"want": 964.5,
"got": "994.50",
"ok": false,
"present": true
},
{
"field": "ending_deductible_b",
"want": 3000,
"got": "3000.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_b",
"want": 3953.16,
"got": "4013.16",
"ok": false,
"present": true
},
{
"field": "ending_deductible_c",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_c",
"want": 30,
"got": "30.00",
"ok": true,
"present": true
},
{
"field": "ending_deductible_d",
"want": 3000,
"got": "3000.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_d",
"want": 12000,
"got": "12000.00",
"ok": true,
"present": true
},
{
"field": "ending_family_deductible",
"want": 6904.5,
"got": "6904.50",
"ok": true,
"present": true
},
{
"field": "ending_family_oopm",
"want": 16947.66,
"got": "17037.66",
"ok": false,
"present": true
}
],
"fabricated": true,
"abstained": false
}0LDG-008Twenty claims, mixed network, three edits18,900 out · 171.3s
See what every other model answered →
Reasoning · 18,414 tokens
[omitted]json{
"claim_7_member_responsibility": "391.98",
"claim_7_plan_paid": "2155.88",
"claim_7_deductible_applied": "0.00",
"claim_7_copay_amount": "0.00",
"claim_7_coinsurance_amount": "391.98",
"claim_16_member_responsibility": "0.00",
"claim_16_plan_paid": "2069.76",
"claim_16_deductible_applied": "0.00",
"claim_16_copay_amount": "0.00",
"claim_16_coinsurance_amount": "0.00",
"claim_23_member_responsibility": "0.00",
"claim_23_plan_paid": "189.56",
"claim_23_deductible_applied": "0.00",
"claim_23_copay_amount": "0.00",
"claim_23_coinsurance_amount": "0.00",
"ending_deductible_a": "1500.00",
"ending_oopm_a": "4190.16",
"ending_deductible_b": "710.81",
"ending_oopm_b": "6000.00",
"ending_deductible_c": "789.19",
"ending_oopm_c": "828.55",
"ending_deductible_d": "0.00",
"ending_oopm_d": "981.29",
"ending_family_deductible": "3000.00",
"ending_family_oopm": "12000.00"
}
Grader detail
json{
"fields": [
{
"field": "claim_7_member_responsibility",
"want": 0,
"got": "391.98",
"ok": false,
"present": true
},
{
"field": "claim_7_plan_paid",
"want": 2547.86,
"got": "2155.88",
"ok": false,
"present": true
},
{
"field": "claim_7_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_7_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_7_coinsurance_amount",
"want": 0,
"got": "391.98",
"ok": false,
"present": true
},
{
"field": "claim_16_member_responsibility",
"want": 2069.76,
"got": "0.00",
"ok": false,
"present": true
},
{
"field": "claim_16_plan_paid",
"want": 0,
"got": "2069.76",
"ok": false,
"present": true
},
{
"field": "claim_16_deductible_applied",
"want": 2069.76,
"got": "0.00",
"ok": false,
"present": true
},
{
"field": "claim_16_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_16_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_23_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_23_plan_paid",
"want": 189.56,
"got": "189.56",
"ok": true,
"present": true
},
{
"field": "claim_23_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_23_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_23_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_deductible_a",
"want": 0,
"got": "1500.00",
"ok": false,
"present": true
},
{
"field": "ending_oopm_a",
"want": 2204.04,
"got": "4190.16",
"ok": false,
"present": true
},
{
"field": "ending_deductible_b",
"want": 3000,
"got": "710.81",
"ok": false,
"present": true
},
{
"field": "ending_oopm_b",
"want": 12000,
"got": "6000.00",
"ok": false,
"present": true
},
{
"field": "ending_deductible_c",
"want": 789.19,
"got": "789.19",
"ok": true,
"present": true
},
{
"field": "ending_oopm_c",
"want": 828.55,
"got": "828.55",
"ok": true,
"present": true
},
{
"field": "ending_deductible_d",
"want": 3000,
"got": "0.00",
"ok": false,
"present": true
},
{
"field": "ending_oopm_d",
"want": 4153.38,
"got": "981.29",
"ok": false,
"present": true
},
{
"field": "ending_family_deductible",
"want": 6789.19,
"got": "3000.00",
"ok": false,
"present": true
},
{
"field": "ending_family_oopm",
"want": 19185.97,
"got": "12000.00",
"ok": false,
"present": true
}
],
"fabricated": true,
"abstained": false
}0LDG-005Five members, eighteen claims, three edits15,862 out · 144.8s
See what every other model answered →
Reasoning · 15,341 tokens
[omitted]json{
"claim_4_member_responsibility": 0.00,
"claim_4_plan_paid": 0.00,
"claim_4_deductible_applied": 0.00,
"claim_4_copay_amount": 0.00,
"claim_4_coinsurance_amount": 0.00,
"claim_17_member_responsibility": 0.00,
"claim_17_plan_paid": 6913.69,
"claim_17_deductible_applied": 0.00,
"claim_17_copay_amount": 0.00,
"claim_17_coinsurance_amount": 0.00,
"claim_21_member_responsibility": 0.00,
"claim_21_plan_paid": 2259.30,
"claim_21_deductible_applied": 0.00,
"claim_21_copay_amount": 0.00,
"claim_21_coinsurance_amount": 0.00,
"ending_deductible_a": 0.00,
"ending_oopm_a": 726.42,
"ending_deductible_b": 3000.00,
"ending_oopm_b": 3171.30,
"ending_deductible_c": 0.00,
"ending_oopm_c": 0.00,
"ending_deductible_d": 3000.00,
"ending_oopm_d": 10248.77,
"ending_deductible_e": 0.00,
"ending_oopm_e": 456.46,
"ending_family_deductible": 6000.00,
"ending_family_oopm": 14602.96
}
Grader detail
json{
"fields": [
{
"field": "claim_4_member_responsibility",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_4_plan_paid",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_4_deductible_applied",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_4_copay_amount",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_4_coinsurance_amount",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_17_member_responsibility",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_17_plan_paid",
"want": 6913.69,
"got": 6913.69,
"ok": true,
"present": true
},
{
"field": "claim_17_deductible_applied",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_17_copay_amount",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_17_coinsurance_amount",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_21_member_responsibility",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_21_plan_paid",
"want": 2259.3,
"got": 2259.3,
"ok": true,
"present": true
},
{
"field": "claim_21_deductible_applied",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_21_copay_amount",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "claim_21_coinsurance_amount",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "ending_deductible_a",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "ending_oopm_a",
"want": 801.43,
"got": 726.42,
"ok": false,
"present": true
},
{
"field": "ending_deductible_b",
"want": 3000,
"got": 3000,
"ok": true,
"present": true
},
{
"field": "ending_oopm_b",
"want": 3171.3,
"got": 3171.3,
"ok": true,
"present": true
},
{
"field": "ending_deductible_c",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "ending_oopm_c",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "ending_deductible_d",
"want": 3000,
"got": 3000,
"ok": true,
"present": true
},
{
"field": "ending_oopm_d",
"want": 10248.77,
"got": 10248.77,
"ok": true,
"present": true
},
{
"field": "ending_deductible_e",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "ending_oopm_e",
"want": 456.46,
"got": 456.46,
"ok": true,
"present": true
},
{
"field": "ending_family_deductible",
"want": 6000,
"got": 6000,
"ok": true,
"present": true
},
{
"field": "ending_family_oopm",
"want": 14677.96,
"got": 14602.96,
"ok": false,
"present": true
}
],
"fabricated": true,
"abstained": false
}0LDG-002Fourteen claims from a warm start, an adjustment and a void13,880 out · 119.0s
See what every other model answered →
Reasoning · 13,392 tokens
[omitted]json{
"claim_4_member_responsibility": "0.00",
"claim_4_plan_paid": "0.00",
"claim_4_deductible_applied": "0.00",
"claim_4_copay_amount": "0.00",
"claim_4_coinsurance_amount": "0.00",
"claim_13_member_responsibility": "75.00",
"claim_13_plan_paid": "215.00",
"claim_13_deductible_applied": "0.00",
"claim_13_copay_amount": "75.00",
"claim_13_coinsurance_amount": "0.00",
"claim_16_member_responsibility": "184.06",
"claim_16_plan_paid": "736.26",
"claim_16_deductible_applied": "0.00",
"claim_16_copay_amount": "0.00",
"claim_16_coinsurance_amount": "184.06",
"ending_deductible_a": "904.50",
"ending_oopm_a": "1178.56",
"ending_deductible_b": "3000.00",
"ending_oopm_b": "4088.16",
"ending_deductible_c": "0.00",
"ending_oopm_c": "2943.49",
"ending_deductible_d": "3000.00",
"ending_oopm_d": "12000.00",
"ending_family_deductible": "6904.50",
"ending_family_oopm": "20210.21"
}
Grader detail
json{
"fields": [
{
"field": "claim_4_member_responsibility",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_plan_paid",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_4_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_13_member_responsibility",
"want": 0,
"got": "75.00",
"ok": false,
"present": true
},
{
"field": "claim_13_plan_paid",
"want": 290,
"got": "215.00",
"ok": false,
"present": true
},
{
"field": "claim_13_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_13_copay_amount",
"want": 0,
"got": "75.00",
"ok": false,
"present": true
},
{
"field": "claim_13_coinsurance_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_16_member_responsibility",
"want": 0,
"got": "184.06",
"ok": false,
"present": true
},
{
"field": "claim_16_plan_paid",
"want": 920.32,
"got": "736.26",
"ok": false,
"present": true
},
{
"field": "claim_16_deductible_applied",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_16_copay_amount",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "claim_16_coinsurance_amount",
"want": 0,
"got": "184.06",
"ok": false,
"present": true
},
{
"field": "ending_deductible_a",
"want": 904.5,
"got": "904.50",
"ok": true,
"present": true
},
{
"field": "ending_oopm_a",
"want": 964.5,
"got": "1178.56",
"ok": false,
"present": true
},
{
"field": "ending_deductible_b",
"want": 3000,
"got": "3000.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_b",
"want": 3953.16,
"got": "4088.16",
"ok": false,
"present": true
},
{
"field": "ending_deductible_c",
"want": 0,
"got": "0.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_c",
"want": 30,
"got": "2943.49",
"ok": false,
"present": true
},
{
"field": "ending_deductible_d",
"want": 3000,
"got": "3000.00",
"ok": true,
"present": true
},
{
"field": "ending_oopm_d",
"want": 12000,
"got": "12000.00",
"ok": true,
"present": true
},
{
"field": "ending_family_deductible",
"want": 6904.5,
"got": "6904.50",
"ok": true,
"present": true
},
{
"field": "ending_family_oopm",
"want": 16947.66,
"got": "20210.21",
"ok": false,
"present": true
}
],
"fabricated": true,
"abstained": false
}0POP-001Blood pressure control, twelve members4,471 out · 40.9s
See what every other model answered →
Reasoning · 4,312 tokens
[omitted]json{
"denominator": 7,
"numerator": 3,
"excluded": 0,
"not_eligible": 5,
"rate": 42.9,
"compliant_members": ["71011", "71012", "71017"],
"non_compliant_members": ["71014", "71019", "71020", "71021"],
"excluded_members": [],
"not_eligible_members": ["71013", "71015", "71016", "71018", "71022"]
}
Grader detail
json{
"fields": [
{
"field": "denominator",
"want": 6,
"got": 7,
"ok": false,
"present": true
},
{
"field": "numerator",
"want": 3,
"got": 3,
"ok": true,
"present": true
},
{
"field": "excluded",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "not_eligible",
"want": 6,
"got": 5,
"ok": false,
"present": true
},
{
"field": "rate",
"want": 50,
"got": 42.9,
"ok": false,
"present": true
},
{
"field": "compliant_members",
"want": [
"71011",
"71012",
"71017"
],
"got": [
"71011",
"71012",
"71017"
],
"ok": true,
"present": true
},
{
"field": "non_compliant_members",
"want": [
"71014",
"71019",
"71020"
],
"got": [
"71014",
"71019",
"71020",
"71021"
],
"ok": false,
"present": true
},
{
"field": "excluded_members",
"want": [],
"got": [],
"ok": true,
"present": true
},
{
"field": "not_eligible_members",
"want": [
"71013",
"71015",
"71016",
"71018",
"71021",
"71022"
],
"got": [
"71013",
"71015",
"71016",
"71018",
"71022"
],
"ok": false,
"present": true
}
],
"fabricated": true,
"abstained": false
}0POP-001Blood pressure control, twelve members4,217 out · 39.3s
See what every other model answered →
Reasoning · 4,058 tokens
[omitted]json{
"denominator": 7,
"numerator": 3,
"excluded": 0,
"not_eligible": 5,
"rate": 42.9,
"compliant_members": ["71011", "71012", "71017"],
"non_compliant_members": ["71014", "71019", "71020", "71021"],
"excluded_members": [],
"not_eligible_members": ["71013", "71015", "71016", "71018", "71022"]
}
Grader detail
json{
"fields": [
{
"field": "denominator",
"want": 6,
"got": 7,
"ok": false,
"present": true
},
{
"field": "numerator",
"want": 3,
"got": 3,
"ok": true,
"present": true
},
{
"field": "excluded",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "not_eligible",
"want": 6,
"got": 5,
"ok": false,
"present": true
},
{
"field": "rate",
"want": 50,
"got": 42.9,
"ok": false,
"present": true
},
{
"field": "compliant_members",
"want": [
"71011",
"71012",
"71017"
],
"got": [
"71011",
"71012",
"71017"
],
"ok": true,
"present": true
},
{
"field": "non_compliant_members",
"want": [
"71014",
"71019",
"71020"
],
"got": [
"71014",
"71019",
"71020",
"71021"
],
"ok": false,
"present": true
},
{
"field": "excluded_members",
"want": [],
"got": [],
"ok": true,
"present": true
},
{
"field": "not_eligible_members",
"want": [
"71013",
"71015",
"71016",
"71018",
"71021",
"71022"
],
"got": [
"71013",
"71015",
"71016",
"71018",
"71022"
],
"ok": false,
"present": true
}
],
"fabricated": true,
"abstained": false
}0POP-001Blood pressure control, twelve members3,267 out · 31.1s
See what every other model answered →
Reasoning · 3,108 tokens
[omitted]json{
"denominator": 7,
"numerator": 3,
"excluded": 0,
"not_eligible": 5,
"rate": 42.9,
"compliant_members": ["71011", "71012", "71017"],
"non_compliant_members": ["71014", "71019", "71020", "71021"],
"excluded_members": [],
"not_eligible_members": ["71013", "71015", "71016", "71018", "71022"]
}
Grader detail
json{
"fields": [
{
"field": "denominator",
"want": 6,
"got": 7,
"ok": false,
"present": true
},
{
"field": "numerator",
"want": 3,
"got": 3,
"ok": true,
"present": true
},
{
"field": "excluded",
"want": 0,
"got": 0,
"ok": true,
"present": true
},
{
"field": "not_eligible",
"want": 6,
"got": 5,
"ok": false,
"present": true
},
{
"field": "rate",
"want": 50,
"got": 42.9,
"ok": false,
"present": true
},
{
"field": "compliant_members",
"want": [
"71011",
"71012",
"71017"
],
"got": [
"71011",
"71012",
"71017"
],
"ok": true,
"present": true
},
{
"field": "non_compliant_members",
"want": [
"71014",
"71019",
"71020"
],
"got": [
"71014",
"71019",
"71020",
"71021"
],
"ok": false,
"present": true
},
{
"field": "excluded_members",
"want": [],
"got": [],
"ok": true,
"present": true
},
{
"field": "not_eligible_members",
"want": [
"71013",
"71015",
"71016",
"71018",
"71021",
"71022"
],
"got": [
"71013",
"71015",
"71016",
"71018",
"71022"
],
"ok": false,
"present": true
}
],
"fabricated": true,
"abstained": false
}