10 families, 210 tasks, all of it written for this benchmark.

This is the single most repeated calculation in a health plan, and the one a member is most likely to dispute. It is arithmetic under a stateful rule set, which is exactly the shape of problem where a language model can be fluent and wrong at once. Every item here has a provable answer produced by an oracle solver, so there is no grader judgement anywhere in the family.

The single-claim family saturated: the frontier scored above 98 and the leaders were separated by less than their own confidence intervals. This family is where the depth is. Half of it is chains of up to eight claims across three or four members of one household, where an arithmetic slip on claim two is still on the books at claim eight, with gold answers from the same oracle solver. The other half is the work that actually generates appeals: deciding which of two plans pays first, deciding which of two contradictory documents governs, and finding the one wrong line on a notice that otherwise adds up. Every contested item has a near-miss twin, so a model cannot score by recognising the shape of the question.

Coverage determination is where a health plan is most exposed. It is regulated, it is appealable, and by 2026 a majority of utilisation-management operations report some AI in the loop. The failure that matters is not a low score, it is a confident wrong answer in a specific direction, so this family separates approve, deny and pend errors and reports them apart from each other. Requiring a citation set turns the task from a three-way guess into something an appeals reviewer could audit.

Every downstream number a plan produces — risk scores, quality rates, provider payment, member cost share — is a function of codes. A model that fabricates a plausible-looking code produces a claim that fails weeks later, at which point nobody remembers where the code came from. Splitting recall from applied items shows whether a model needs retrieval attached before it is safe to point at coding work, which is the actual build decision.

Measure rates set Star Ratings, quality bonuses, and a large part of what a plan can tell an employer group. The logic is date arithmetic against anchor dates and look-back windows, and it is exactly the kind of rule a model will paraphrase confidently and apply loosely. The items are built in near-miss pairs — the same fact pattern one year or one day either side of a boundary — so a model cannot score by recognising the measure.

Intake is where a plan spends the most human hours and where a model has the clearest business case. It is also where the cost of a hallucination is highest per token, because a fabricated code enters the record as fact and is not questioned again. Several items here have an empty array as the correct answer, so the family measures restraint as directly as it measures recall.

This is the work most plans are furthest along in automating and the work with the least margin for a fluent error. A denial notice that omits appeal rights is a compliance finding; one that adds clinical advice is a different problem entirely; one that says a covered claim was denied generates a call and an appeal that never needed to exist. The rubric scores those failure modes as negative points rather than as missing credit, because that is how they behave in production.

A payer assistant fails in two directions and the industry only measures one. The unsafe direction produces a privacy incident or a regulatory finding. The over-refusal direction produces a tool nobody uses, and it is the more likely of the two to survive a pilot unnoticed, because a refusal looks like caution rather than a defect. Half these items are near-mirrors of each other — the same request shape with a legitimate purpose and an illegitimate one — so a model cannot score by pattern-matching on vocabulary.

The contested family was written because single-claim adjudication saturated, and then it saturated too: on the first release the top eight models were perfect on 153 of 186 items and the leaders sat a point apart inside each other’s intervals. This family is what a claims system actually does across a plan year rather than across a week: a ledger of up to twenty-four lines over up to five members, where some claims come back corrected or voided and their credits have to be unwound from accumulators that later claims already drew on. Three claims are reported rather than one, with the ending state of every accumulator, so a slip anywhere on the ledger is visible in the answer. Every item is generated from a seeded stream and adjudicated by an oracle, so the gold is proved rather than keyed and could not have been memorised.

The single-member measure family saturated alongside the rest of the suite. A quality team does not place one member; it runs the measure over a population and reports a rate, and the rate is only right if every member landed in the right bucket. Each roster here is twelve to twenty-six members, every one of them carrying a boundary drawn from the same traps the single-member family uses — an age one year outside the range, an enrolment gap one day too long, a qualifying encounter one day past the window, an exclusion in the wrong year. The answer is four counts, a rate, and four member lists that must agree with each other, scored as one all-or-nothing item. The rosters are generated and placed by an oracle, so the gold is computed rather than keyed.