Billing Extraction for PI Cases: Can Open Models Outperform Frontier Quality at Lower Cost?

Billing summaries turn thousands of pages of medical and legal records into structured evidence: provider identities, treatment dates, aggregate billing amounts, and procedure codes. Attorneys use them to evaluate damages and negotiate settlements, which means an error in extraction becomes an error in the case.

Most legal AI does this work with a general-purpose frontier model, and most of the time that’s a reasonable default. Frontier models are capable, they’re available, and nobody gets criticized for choosing one.

We wanted to know whether fine-tuning an open-weight model on personal injury billing could do better. Not cheaper at the expense of quality, which is the usual tradeoff, but better on quality, cost, and speed at once.

It can. A fine-tuned Qwen model improved total match rate by 6% over our frontier baseline, while cutting cost per case by 50% and reducing p50 latency by roughly 68%. Here’s how we got there, and where the approach stops working.

Why Frontier Models Fall Short Here

Frontier models produce decent billing summaries. The failures are specific and repeatable, which is what makes them a problem.

Frontier models miss quantity multipliers when converting unit prices into total charges; treat cost estimates as if they were medical bills; and group charges under the wrong provider. Each of those produces a number that looks plausible and is wrong, which is the kind of error that survives review.

Personal injury cases carry enormous page volume and produce long structured outputs. Per-token pricing scales with both.

Why Fine-Tuning Open-Weight Models?

Open-weight models like Qwen, Kimi, MiniMax, GLM, and Inkling perform competitively with frontier models across public benchmarks. More importantly, domain-specific fine-tuning of an open model can beat a general-purpose frontier model on a targeted task.

That combination is what makes them worth investigating. A smaller architecture, tuned for one job, can deliver better quality at lower cost and latency than a larger general one.

How well that holds inside specialized legal workflows hasn’t been examined much. We started by testing out-of-the-box open-weight models against frontier models on billing extraction. Open-weight performance was competitive but didn’t beat the frontier baseline. That told us the architecture was viable and gave us a starting point to fine-tune from.

How We Chose Which Model to Fine-Tune

Choosing a model to fine-tune is expensive in time, and there are a lot of promising candidates.

We narrowed on two criteria:

1. Maximum context length, since the model has to process large cases
2. Published benchmark performance

After testing the baseline extraction quality of several long-context models, we chose the Qwen family. These models offers a range of sizes with context windows that rival frontier models.

Weighing model size, baseline quality, and inference cost together, the 397B variant gave us the best starting point.

How the Fine-Tuning Worked

For fine-tuning our open-weight model, we built a training set of a few hundred labeled examples, chosen to reflect the complexities of real-world cases. The model’s output is a JSON schema with billing details. 

We ran supervised fine-tuning (SFT) on the base Qwen3.5-397B-A17B model applying Low-Rank Adaptation (LoRA) and standard next-token prediction objectives on Nvidia B300 GPUs. We preserved checkpoints after each epoch to facilitate performance analysis.

With a dataset this small, overfitting is a real risk. We picked the checkpoint with the lowest validation loss rather than the final training iteration, which gave us the point before the model started memorizing.

How Much Extraction Quality Improved

On total match rate, where the predicted total billing amount falls within 1% of ground truth, the fine-tuned model improved 6% over the frontier baseline.

We measured against frontier models on a holdout set excluded from training. Combining task-specific fine-tuning with precise prompt instructions, the model became more reliable at exactly what frontier models miss: spotting cost estimates, factoring quantity multipliers into unit-price calculations, and grouping multi-provider bills correctly.

Quality improved with each training epoch up to the selected checkpoint, then plateaued as overfitting set in.

How Much Cost and Latency Improved

Against the frontier baseline, the fine-tuned model delivered a 50% reduction in cost per case and cut p50 latency by nearly 68%.

Getting there meant moving off per-token pricing and onto usage-based GPU billing with third-party hosting, then optimizing the serving path:

• Vision tower removal. Our input is text, so we removed the vision component, which reduced model size and compute.
• Quantization. We quantized the merged model to FP8 using block-wise quantization.
• Efficient serving. Served on Nvidia B200 with SGLang, using DFlash to accelerate decoding.

We also added auto-scaling, so GPUs aren’t sitting idle during low traffic.

Where This Approach Doesn’t Work

Three limits are worth stating plainly.

1. It needs labeled training data. A few hundred examples sounds small until you have to produce them for a domain where a single label depends on evidence scattered across thousands of pages. Building that data is its own engineering problem, and we wrote about how we solved it separately. Without that pipeline, this approach doesn’t start.
2. It’s task-specific by design. This model does billing extraction. That focus is where the quality gain comes from, and it means the model won’t generalize to adjacent tasks without its own fine-tuning run. A frontier model is worse at this job and better at everything else.
3. Self-hosting means owning the infrastructure. Per-token pricing buys you someone else’s operational burden. Moving to GPU billing means owning quantization, serving, auto-scaling, and utilization. The cost reduction is real, and it isn’t free.

Where the Next Gains Are

We’re exploring open-weight models for other use cases, most of which involve large inputs and large outputs. That points the work toward context compression and efficient serving, specifically scaling throughput, improving GPU utilization, and driving cost down further.

We’re also investigating a Multi-LoRA approach, where a single foundational open-weight model supports several use cases at once.

Contributors

This project, led by Vishwakarma Singh and supervised by Haixun Wang, was accomplished with key contributions from Jay Dong, Saketh Bachu, and Sarina Xi. We also acknowledge the contributions of Prathamesh Korane, Ethan Bailie, Yuang Gao, Ronglian Yang, and Taesik Na.