Blog

How Much Evidence Is Enough? Rethinking Retrieval in RAG

Emre Yamangil

Principal Machine Learning Engineer, EvenUp

November 16, 2026

How Much Evidence Is Enough? Rethinking Retrieval in RAG

Some questions need one fact. Others need hundreds of records. Here’s how a RAG system can tell from the query alone how much to retrieve before it starts.

The Problem

Every RAG system has to answer one question for itself before answering the user’s question: how many documents should it retrieve to conduct the work?

That number is called top-k, or just k. In many RAG systems, it’s set once at configuration time and used for every query after that. This fixed approach is simple and predictable, but it doesn’t match how people actually ask questions.

In legal tech, a single assistant receives questions whose right k can differ by an order of magnitude:

  • “What is the statute of limitations for personal injury in California?”
    • One authoritative source may be enough.
  • “Explain how comparative negligence affects damages when the plaintiff is partially at fault.”
    • It needs doctrine plus jurisdiction-specific nuance.
  • “Given these facts, draft a demand summary and explain what evidence supports each damages category.”
    • It needs medical records, billing, prior demands, policy language, and deposition excerpts.

One fixed number can’t serve all three. Set it too low, and the system misses critical documents and answers incompletely. Set it too high, and the context fills with redundant material, which raises cost and latency and can degrade the answer.

This challenge is at the heart of our recently published paper, Pre-retrieval Query Clustering for Adaptive Top-k Document Retrieval in RAG Systems, co-authored by Ye Xia, Emre Yamangil, and Haixun Wang, and published in the proceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM 2026) in Rome, Italy.

Our research explores how retrieval systems can adapt the number of documents they retrieve based on the characteristics of a query, rather than relying on a fixed retrieval budget for every question.

So can we tell how much a query needs before retrieving anything? Two common approaches get part of the way.

Keyword Rules Cover About a Fifth of Queries

Some queries state their own scope in their words: “list all,” “for each provider,” “top 5.” A pattern match routes those correctly. Most queries don’t. 

Take these two:

  • “Who was the driver at fault?” A single police report or witness statement usually answers it.
  • “Explain how comparative negligence affects damages in this case.” It may need doctrine, jurisdiction-specific rules, case facts and prior rulings.

Neither contains a scope marker, so a lexical rule treats them the same. Yet one is answered by a single document, and the other needs a broad sweep of the file. That difference comes from what the question means, and it has to be learned from domain data.

Trimming After the Fact Means Paying First

The other established approach is to retrieve broadly and cut the results down afterward. It works, but by the time you decide what to keep, you’ve already paid for the large retrieval. That’s a real cost when retrieval is expensive, when the corpus is full of near-duplicates, or when you need a predictable budget before calling downstream components.

Retrieval depth should vary by query, and the system should decide it before retrieval runs. This post describes QCAR (Query-Clustering Adaptive Retrieval) and how we estimate that depth from the query alone.

What the Question Tells Us

We estimate retrieval depth from two signals: how the question is phrased, and what answering it requires.

QueryTypeWhat the answer requiresDepth
“What is the plaintiff’s date of birth?”EasyOne authoritative source; highly localizedLow
“List all treatments with dates and providers.”Hard, lexicalExhaustive coverage of the medical recordk_max
“Who was the driver at fault?”Easy, semanticA single police report or witness statementLow
“How does comparative negligence affect damages here?”Hard, semanticDoctrine, jurisdiction rules, case facts, prior rulingsHigh

Takeaway. Phrasing settles depth for about a fifth of queries. For the rest, depth depends on how much evidence the answer requires, and in a bounded domain, that’s predictable from the query itself.

How QCAR Works

The expensive work happens offline and produces a Query-Cluster Database. At query time, QCAR is a lookup.

1. Cluster the Queries

We embed historical queries with text-embedding-ada-002 (1,536 dimensions) and reduce the dimensions before clustering. PCA needed 186 components to explain 80% of the variance, and clusters built on it scored silhouette values near zero. UMAP preserved local structure far better, so we used it.

For clustering, we compared K-means, Ward-linkage agglomerative clustering and HDBSCAN. HDBSCAN left about a third of queries unclassified as noise, so we used it to find the natural number of clusters and stable centroids, then ran K-means to assign every query.

The Configuration We Rejected

Our best numerical configuration, UMAP at two components with 60 clusters, scored a silhouette of 0.6 and a Davies-Bouldin index of 0.46. Excellent numbers.

We threw it away. A qualitative review, with Gemini 3 Pro reading the clusters, showed that two dimensions collapsed distinct legal topics into single groups. The clusters were compact, and compactness was all the metrics measured.

We shipped 190 components and 80 clusters instead: silhouette 0.369, Davies-Bouldin 1.019. Worse metrics, better semantics.

2. Measure How Deep Each Cluster Needs to Go

Depth needs ground truth. We sample 21 queries per cluster and use Claude Sonnet 4.5 as a judge to label every retrieved page 0 (not relevant), 1 (partially relevant) or 2 (fully relevant), about 190 pages per case. It’s expensive, but it runs offline, once.

Then we retrieve at the maximum k of 100 and compute NDCG at every depth from 1 to 100. The elbow is the deepest k at which NDCG is still meaningfully gaining:

k*(q) = max { k | NDCG_k+1(q) − NDCG_k(q) ≥ ε }, ε = 0.001

A simple query such as “Who was the driver at fault?” converges at k = 25 and gains only 0.1 NDCG across the remaining 75 documents. Everything past the elbow is cost without benefit. A hard query such as “List all treatments with dates and providers” climbs all the way to 100 with no plateau, because its evidence is spread across the record.

For each cluster, we take the mean and standard deviation of those elbows:

k*_cluster = μ + 2σ

That targets about 95% coverage of the relevant documents in the cluster.

3. Decide at Query Time

At inference, QCAR is a thin layer in front of standard RAG:

  1. Lexical check. If the query carries a scope marker, set k* = k_max and retrieve.
  2. Cluster assignment. Otherwise, embed the query, project it, find the nearest centroid and read off that cluster’s k*.
  3. Retrieve and generate. Everything downstream stays the same.

The added overhead is small: under 1 ms for the lexical check, about 1 ms for the projection and under 1 ms for the nearest-centroid search. The one embedding call (30 to 50 ms) is already required for retrieval. Cluster assignment depends on the dimension of the clustering space and the number of clusters, and the number of documents in a case doesn’t affect it, so it runs in constant time per query.

Results

We started with 20,000 raw queries, then cleaned and PII-filtered them down to 6,820. The baseline is a static k = 50, our optimized production setting at the time. Adaptive k* is capped at 100.

In offline evaluation against k = 50, simple fact-retrieval queries reached stable NDCG at lower depth, using 35.95% fewer tokens while F1 improved. Complex queries pushed k* to the 100-document cap and improved F1 at the cost of 85.95% more tokens. On those queries, accuracy rose with depth, and that’s the core argument against a static setting.

Adaptivity drives the gains. Against a randomly chosen k, adaptive retrieval holds a plateau, while random selection degrades immediately, because random cuts drop essential documents.

One caveat: QCAR doesn’t save money overall. In the A/B test, total token use rose 51.19%, because the expansion on complex queries outweighs the savings on simple ones. QCAR spends retrieval budget where it improves answers.

Where This Works

QCAR suits bounded-domain RAG: stable, recurring query distributions, a query history to learn from, and documents drawn from a coherent corpus such as case files or medical records. That covers most production deployments, including enterprise search, customer support, legal research, clinical decision support and financial analysis.

Porting it to a new vertical takes three things: a historical query log, a way to label relevance (LLM-as-a-judge or human annotation) and, optionally, some domain lexical patterns. The clustering and depth estimation transfer unchanged.

Open-domain search is the weaker case. Query diversity is unbounded, and corpus characteristics vary unpredictably. QCAR can still handle recognized query types and send the rest to a default k or a post-retrieval method, but its value is in bounded domains.

Complementary to Post-Retrieval Methods

QCAR is a prior: it sets the budget before retrieval. Post-retrieval methods are a posterior: they trim what came back. Their failure modes complement each other. One prevents unnecessary retrieval, and the other refines an existing pool. The natural extension uses QCAR for the initial budget and post-retrieval cutoff detection on the result, and that’s the direction we find most promising.

Query distributions also drift. QCAR handles that by rebuilding the Query-Cluster Database on a schedule, or when monitoring detects a shift. Between rebuilds, the mapping stays fixed, so behavior stays predictable.

Scale Your Firm, Not Your Payroll

Schedule a call today to see how EvenUp's AI tools automate repetitive tasks, streamline custom drafting, and empower staff to focus on case strategy and client engagement.

Schedule a call

Explore More


All-In-One, Case-Based Pricing

Schedule a Call
Win bigger and settle faster. Reduce time on desk. Clear your demand backlog. Automate your intake process.