Why "no" is the hardest answer to get right

This post is part of Pattern Labs, our ongoing research series focused on translating real-world experimentation into defensible, production-ready workflows for mass tort litigation. This installment examines a specific and underappreciated failure mode in AI-assisted case evaluation: the structural tendency of large language models to return a finding even when the evidentiary record doesn't support one.

Most litigators who have experimented with general-purpose AI for record review come away with the same impression: the model found something in almost every file. That volume of findings is only useful if the findings are right.

Medical records contain a lot of material that doesn't determine case outcome. Incidental findings. Mentioned-but-unconfirmed conditions. Diagnoses that appeared in a specialist note and were never followed up. A general-purpose LLM processing that kind of record is solving a different problem than the one a reviewer is actually trying to answer.

These models are trained to return information; declining to answer is materially more difficult for them than returning an incorrect response. At mass tort scale, that produces a reliable pattern: findings that are technically present in the record but don't constitute confirmed, qualifying evidence. The model found something. Whether that something holds up under scrutiny is a question the model has no mechanism to ask.

The distinction that gets compressed

Any litigator who has spent time in medical records knows the difference between a confirmed diagnosis and a provider noting something in passing. Settlement criteria are built around exactly that distinction. A general-purpose LLM given no guidance on how to weight those differences tends to collapse them into a single category. Confirmed diagnoses, conditions mentioned incidentally by a prior provider, and findings explicitly ruled out after workup all register as positive signal, varying in confidence perhaps, but not in kind.

This is where the cost accumulates. A false positive moving through review without detection doesn't represent a single error in isolation. Across thousands of claimants, that pattern adds up fast. Review time spent on cases built on a mentioned-but-unconfirmed finding is capacity that doesn't come back.

Teaching a Model to Mean No

The standard prompting approach frames the task as binary: does this record support the qualifying criterion? The model has learned that returning a finding is more useful than returning nothing, so when evidence is thin, it leans toward yes. A definitive positive is informative. A definitive negative is an absence of information.

Restructuring the task so that a negative finding carries equal structural weight changes that behavior. Rather than allowing the model to default to withholding a positive, we build explicit negative categories directly into the prompt. Genuine ambiguity gets its own labeled path, a structured acknowledgment that the record doesn't provide sufficient basis to conclude either way. When the model has a clearly defined way to say nothing, it stops borrowing from yes.

When certain and correct are not the same thing

A reviewer triaging hundreds of cases in a day is making decisions based partly on how certain a finding appears. That's reasonable. Surface confidence in an LLM output and the quality of the underlying evidence move independently. A model can return a well-supported finding at low apparent certainty and a weakly-evidenced one with high apparent certainty, and nothing in the output signals which is which.

What the second model in our current evaluation system really does is check the first model's work. Does the answer the first model gave make sense given the citation it generated? Does that citation text provide enough context to justify the answer? Those are the questions the second model is asking. When the answer is no, it can flag, adjust, or overturn the output before it reaches reviewers.

Why getting it right takes more than better prompting

Getting the architecture right on paper is the simpler part. The edge cases that require calibration are litigation-specific, record-type-specific, and in some cases provider-specific.

What a confirmed diagnosis looks like in a Roundup record differs from what it looks like in a Paraquat or CPAP record. Documentation norms vary by specialty, institution, and era. Prompts that handle those variations correctly were built by working through them with litigation experts and medical reviewers who could identify when the model was getting it wrong before those errors reached reviewers. That process doesn't transfer automatically from one litigation to the next, and it doesn't compress into a prompting session.

Ryan Schmitt is a Product Manager on the AI team at Pattern Data.

FAQs

Why do AI tools return false positives when reviewing medical records?
Large language models are trained to return information, so declining to answer is harder for them than producing one. Given a medical record, a general-purpose model tends to treat a confirmed diagnosis, a condition mentioned in passing, and a finding later ruled out as the same positive signal. At mass tort scale, that produces findings that are present in the record but don't qualify under the litigation's criteria.

How does Pattern reduce false positives in case evaluation?
Pattern builds explicit negative categories into how each case is evaluated, giving the model a defined path to conclude that a record does not support a finding instead of defaulting to a positive when evidence is thin. A second model then checks the first model's answer against the citation it produced, flagging or overturning conclusions the evidence doesn't justify before they reach a reviewer.

Why does AI case evaluation have to be built for each litigation?
What counts as a confirmed diagnosis varies by litigation, record type, and even provider. Documentation norms differ across specialties, institutions, and eras, so the calibration that works in a Roundup record does not automatically transfer to Paraquat or CPAP. Pattern's evaluation logic is built by working through those edge cases with litigation experts and medical reviewers against each litigation's criteria.