Evaluation

Failure cases

Method

Data. Eight published Medicare coverage policies: six NCDs parsed from the full text returned by the CMS Medicare Coverage Database API, and two LCD-derived trees that are paraphrases — CMS returns 401 for LCD detail text without a licence token, so those thresholds are unverified and are labelled as such in the UI. Twenty synthetic notes, written for this project, distributed 8 approve / 8 deny / 4 indeterminate, with three cases carrying two or more independent failures.

Both systems. The criteria graph evaluates hand-labelled gold facts against the policy tree with three-valued logic. The baseline chunks the same policy text at roughly 400 tokens with 50 of overlap, embeds with all-MiniLM-L6-v2, retrieves the top 5 chunks by cosine similarity, and asks the same model at the same temperature for a decision and a citation.

Baseline tuning is disclosed. The baseline prompt was written and revised to be competent — it is given the retrieved text, the note, an explicit three-class output contract and an instruction to abstain when documentation is missing. Its free-text citation is mapped onto a node by token overlap against node labels, predicate keys and section names, and the best match wins. That mapping is deliberately generous: a citation that merely mentions the right subject scores. A strawman baseline would make this page worthless.

What decision accuracy here does and does not measure. Both systems are scored on the same gold facts, so this figure isolates whether the coverage logic is applied correctly. It does not measure extraction. An end-to-end number would require scoring LLM-extracted facts, which is a separate run and a separate claim.

Prompts and code. The extraction prompt is in pipeline/extract_facts.py, the policy-parsing prompt in pipeline/parse_policy.py, and the baseline prompt in pipeline/baseline_rag.py.