Why one prompt can't evaluate a case

This post is part of Pattern Labs, our ongoing research series focused on translating real-world experimentation into defensible, production-ready workflows for mass tort litigation. In the last installment, we examined why large language models have a structural tendency to return positive findings even when the record doesn't support one. This installment looks at a related problem: why asking a single question of a model, however well-worded, produces unreliable results for something as complex as a case evaluation.

When a firm experiments with general-purpose AI for record review, the typical starting point looks something like this: take a claimant's records, paste them into a chat interface, and ask whether the file supports a qualifying diagnosis. The model returns an answer. It often sounds confident and specific. The problem is that a case evaluation isn't a single question. It's twenty questions asked in a particular order, where the answer to each one shapes how the next should be interpreted.

Asking one prompt to do that work is like handing a paralegal a file and asking them to evaluate it in a single sentence. The answer they give you will technically respond to what you asked. It won't reflect what the file actually contains.

Twenty questions in the right order

A qualifying determination in mass tort depends on a chain of clinical and factual conclusions. Was the claimant exposed during the relevant window? Does the record contain a diagnosis from a qualifying provider? Is that diagnosis confirmed or noted incidentally? Does the claimant have a condition that affects eligibility? Each of those questions has its own evidentiary requirements, and the answer to one frequently changes how the next should be weighted.

A general-purpose LLM asked to evaluate a case in a single prompt has to compress all of that into one response. It does so by averaging across the questions implicitly, which means it gets some right, misses others, and provides no visibility into which is which.

One job at a time

The more reliable approach treats each component of the evaluation as a discrete task. Rather than asking the model to assess a case, you ask it to assess exposure. Then diagnosis. Then provider qualification. Then secondary conditions. Each question is structured for that specific determination, with the evidentiary context it needs and explicit guidance on what a negative or ambiguous finding looks like.

This is prompt engineering, but not in the way most people encounter the term. Asking ChatGPT the right way to get a better answer is one version of prompt engineering. Building a system where complex evaluations are automatically decomposed into the right sequence of discrete, calibrated questions requires knowing the litigation well enough to define what those questions should be.

The harder problem: knowing what to ask

Task decomposition is straightforward to implement. Knowing what to decompose, and how, for a specific litigation requires a different kind of work entirely.

The questions that matter in a Roundup evaluation are not the same as the questions that matter in a Talc or CPAP evaluation. The evidentiary standards differ. The qualifying diagnoses differ. The documentation patterns in the relevant medical records differ by specialty, by era, and by the type of provider most claimants are likely to have seen. Building a decomposed evaluation framework for a new litigation requires working through those specifics with people who understand both the settlement criteria and the clinical record types involved.

That work is what separates a prompt from a system. A firm that pastes records into a general-purpose AI and asks for a case evaluation is getting one model's best attempt at compressing a complex determination into a single output. What gets built before the model sees a real record, the question structure, the sequencing, the criteria mapping, is where the accuracy actually comes from.

Where the errors go

At the case level, the difference between a decomposed evaluation and a single-prompt approach can be difficult to detect. A well-worded prompt will get a lot of cases approximately right, and the errors may not surface until a file reaches a stage where the stakes are higher.

Errors in early evaluation logic don't stay contained to individual cases. They propagate through triage decisions, workup prioritization, and eventually settlement positioning.

A system built on decomposed, litigation-specific evaluation logic produces outputs that carry forward reliably. Single-prompt approaches require re-verification every time the question changes.

Ryan Schmitt is a Product Manager on the AI team at Pattern Data.

FAQs

Why does a single AI prompt fail at mass tort case evaluation?
A case evaluation is a chain of clinical and factual determinations asked in a specific order, where the answer to each question changes how the next should be weighted. A single prompt has to compress all of that into one response, averaging across the questions implicitly. It gets some right, misses others, and gives no visibility into which is which.

What does it mean to decompose a case evaluation?
Decomposition treats each part of the evaluation as its own task. Instead of asking a model to assess a whole case, you ask it to assess exposure, then diagnosis, then provider qualification, then secondary conditions. Each question is structured for that determination, with the evidentiary context it needs and explicit guidance on what a negative or ambiguous finding looks like.

How is this different from prompt engineering in ChatGPT?
Prompting a chat tool for a better answer is one form of prompt engineering. Building a case evaluation system is a different kind of work: it automatically breaks a complex determination into the right sequence of discrete, calibrated questions. Defining what those questions should be for a given litigation requires knowing its settlement criteria and clinical record types well enough to map them in advance.