How EvenUp Uses AI Agents to Build Ground Truth Data With More Productive and Accurate Labeling

This work was completed under the supervision of Haixun Wang, VP for AI at EvenUp, with contributions from Min Li, Ethan Bailie, Yuang Gao, Taesik Na, and Vishwakarma Singh.

AI accuracy is paramount for our personal injury firm customers. We use ground truth data as a benchmark to measure whether our AI systems are accurate. But ground truth generation for personal injury cases is slow because a single label can depend on evidence scattered across thousands of pages. 

That’s why EvenUp built a smart ground truth data framework. 

Agentic AI Data Labeling for Smart Ground Truth Generation

We needed an approach to AI data labeling that puts an AI audit agent between preliminary labelers and human expert reviewers. The agent checks every preliminary label against the source record and hands the reviewer a source-cited list of discrepancies instead of a full case file, ensuring data labeling accuracy. 

In measured review workflows, this increased labeling productivity roughly 10x and improved labeling precision by 15%.

Why Does Ground Truth Data Matter For Legal AI?

Ground truth is the answer key. Without it, you cannot measure whether a model is right, and you cannot tell whether a change made things better or worse.

AI tools are now embedded in personal injury workflows, including Medical Bill Summaries, Medical Chronologies, and AI Drafting. These systems read long medical records, summarize bills, and organize case information. Because they support high-stakes legal work, accuracy is a must-have. Every accuracy claim a legal AI vendor makes is only as good as the dataset it was measured against.

That makes the quality of the ground truth dataset the upstream constraint on everything else.

Why Is Data Labeling for Personal Injury Cases So Hard?

Because the answer is rarely in one clean place.

Case files often span thousands of pages. A reviewer may need to connect information across medical records, bills, provider letters, and payment records before deciding what the correct label is. Using medical billing as the example, a single billing label can depend on provider identity, billed amounts, payments, adjustments, duplicate bills, and whether a charge is even related to the injury.

Two common failure patterns make this harder. 

• A charge may appear in one part of the file but need context from another part to interpret correctly. 
• A total can be exactly right while the provider label attached to it is wrong.

The work is slow, judgment-heavy, and dependent on reviewer expertise. Reviewers spend significant time searching for the right evidence before they can make a call.

What Does AI Data Labeling Involve for Medical Bill Summaries?

It is a four-step process, and each step can introduce a different class of error:

1. Locate the pages in the case file that contain healthcare costs
2. Annotate provider-level information, including adjustments, payments, and individual line items
3. Group charges by provider
4. Finalize the total charge per provider by removing irrelevant or duplicate charges and reconciling all financials

Here is what that looks like on a synthetic billing review. Every example in this post is synthetic.

Current Label	Source Evidence To Check	Verified Output
Provider A, $6,400 billed	Same bill appears twice under different file names	Duplicate removed
	One charge is unrelated to the injury	Unrelated charge excluded
	Payment record charges remaining balance	Payment reconciled
	Provider name appears in two forms	Provider identity reconciled

The reviewer is not labeling based on a single number from a single page. They are checking multiple pieces of source evidence before the label can be trusted, since the quality of that ground truth data is what every downstream accuracy claim rests on. That is the cost we set out to reduce.

How Does an Audit Agent Improve EvenUp’s AI Data Labeling?

By turning ground truth generation from a static manual task into a closed feedback loop. The framework has three components.

1. Preliminary Data Labelers

Preliminary labelers generate the first set of labels and supplemental notes from case files or OCR-extracted text. In a billing context, they produce provider-specific line items and total charges. These labels come from a mix of machine and human labeling. Critically, we do not treat them as answers. We treat them as claims to be verified.

2. AI Audit Agent

The audit agent ingests case files, OCR text, and the preliminary labels, then checks each claim against the source record. This is an application of the LLM-as-a-judge pattern, with one difference that matters: the agent does not just score correctness; it investigates root cause. For billing, that means checking provider identity, relevance, duplicate handling, and source-date interpretation.

The output is a source-cited review packet, not a transcript and not a black-box verdict. Each entry shows the claim being checked, the audit finding, the reasoning, and the source evidence page.

3. Expert Reviewer

A human expert evaluates the agent’s findings. Validated findings produce a source-backed artifact. Disputed findings help inform refinements to audit behavior. The human is still the decision-maker. What changed is what they are handed at the start.

What Are the Framework’s Two Operating Modes?

The ground truth generation framework supports two operating modes:

	Base Mode	Auto Research
What it does	Verifies preliminary labeler claims against the original documents	Investigates the underlying system mechanics, not just the corrected values
What it needs	Initial labels plus case documentation	Same inputs, deeper reasoning pass
What it outputs	High-accuracy audits of each claim	Failure modes, improvement backlogs, and diagnostic insight into why errors occurred
Billing example	Treats each provider, total, and relevance call as a claim to verify	Identifies why a provider was split or renamed, why a relevance classifier dropped a charge, or what regression case should exist next

What Results Did We Measure for Data Labeling Accuracy?

Two metrics improved with our smart labeling, both measured against our own expert annotation workflow as the baseline.

1. Labeling productivity increased about 10x. We measure this as the time for a reviewer to produce a verified label the old way. This includes reading the case file and searching for the relevant evidence from scratch. We compared this against the time required to verify an AI-generated, source-cited list of discrepancies. The improvement comes from eliminating search time and reducing manual review effort. The reviewer still makes every call; they just start from AI-cited findings instead of a case file.
2. Labeling precision improved by 15%. Adding the AI audit layer between preliminary labeling and expert review raised the precision of the final ground truth dataset by 15%, compared to the workflow without it. The lift comes from catching a specific class of errors that manual review often misses: labels where the number is right but the underlying evidence is wrong. The audit agent checks every claim against the source record, so those errors surface before they reach the dataset instead of after.

The second-order effect matters more than either number. When reviewers confirm or reject audit findings, those decisions help us identify recurring error patterns and  improve future audit behavior, model evaluation, and the labeling process itself. The system compounds.

The framework also surfaces quiet failures that aggregate metric misses. For example:

• The agent found a case where the total dollar amount was correct, but the provider name was wrong. An aggregate accuracy metric would have scored this as a pass.
• The agent cross-referenced pipeline data against the original bills and caught chiropractic charges, dated after the injury, that a labeling model had wrongly marked as unrelated.

Both are cases where the number looked right and the label was wrong. Those are the errors that survive review and quietly degrade the quality of the ground-truth dataset.

Where Does This Approach Not Work?

Three limits are worth stating plainly.

1. The framework is only as good as the source documents. If a charge is genuinely ambiguous in the underlying record, the agent surfaces the ambiguity but cannot resolve it. The reviewer still makes that call. 
2. It requires preliminary labels to audit. This is a verification architecture, not a cold-start labeling system. On a workflow with no existing labeling process, you have to build the first stage before this adds value.
3. It works best where correctness is checkable against a source. Billing has that property, because a charge either appears in the record or does not. Labels that rest primarily on legal judgment rather than document evidence are a weaker fit, and we would not claim the same productivity gains there.

What We Are Building Next

We are extending the framework beyond medical bill summaries to other personal injury workflows. We are also adding audit memory, so the system can recognize recurring evidence patterns and apply them consistently across repeated tasks.

The broader point generalizes past personal injury. Any domain where ground truth requires expert judgment across long documents faces the same bottleneck. Putting a reasoning agent between the labeler and the reviewer, and requiring it to cite its evidence, is a reusable answer to that bottleneck.