Emre Yamangil
Principal Machine Learning Engineer, EvenUp
November 16, 2026
Some questions need one fact. Others need hundreds of records. Here’s how a RAG system can tell from the query alone how much to retrieve before it starts.
Every RAG system has to answer one question for itself before answering the user’s question: how many documents should it retrieve to conduct the work?
That number is called top-k, or just k. In many RAG systems, it’s set once at configuration time and used for every query after that. This fixed approach is simple and predictable, but it doesn’t match how people actually ask questions.
In legal tech, a single assistant receives questions whose right k can differ by an order of magnitude:
One fixed number can’t serve all three. Set it too low, and the system misses critical documents and answers incompletely. Set it too high, and the context fills with redundant material, which raises cost and latency and can degrade the answer.
This challenge is at the heart of our recently published paper, Pre-retrieval Query Clustering for Adaptive Top-k Document Retrieval in RAG Systems, co-authored by Ye Xia, Emre Yamangil, and Haixun Wang, and published in the proceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM 2026) in Rome, Italy.
Our research explores how retrieval systems can adapt the number of documents they retrieve based on the characteristics of a query, rather than relying on a fixed retrieval budget for every question.
So can we tell how much a query needs before retrieving anything? Two common approaches get part of the way.
Some queries state their own scope in their words: “list all,” “for each provider,” “top 5.” A pattern match routes those correctly. Most queries don’t.
Take these two:
Neither contains a scope marker, so a lexical rule treats them the same. Yet one is answered by a single document, and the other needs a broad sweep of the file. That difference comes from what the question means, and it has to be learned from domain data.
The other established approach is to retrieve broadly and cut the results down afterward. It works, but by the time you decide what to keep, you’ve already paid for the large retrieval. That’s a real cost when retrieval is expensive, when the corpus is full of near-duplicates, or when you need a predictable budget before calling downstream components.
Retrieval depth should vary by query, and the system should decide it before retrieval runs. This post describes QCAR (Query-Clustering Adaptive Retrieval) and how we estimate that depth from the query alone.
We estimate retrieval depth from two signals: how the question is phrased, and what answering it requires.
| Query | Type | What the answer requires | Depth |
| “What is the plaintiff’s date of birth?” | Easy | One authoritative source; highly localized | Low |
| “List all treatments with dates and providers.” | Hard, lexical | Exhaustive coverage of the medical record | k_max |
| “Who was the driver at fault?” | Easy, semantic | A single police report or witness statement | Low |
| “How does comparative negligence affect damages here?” | Hard, semantic | Doctrine, jurisdiction rules, case facts, prior rulings | High |
Takeaway. Phrasing settles depth for about a fifth of queries. For the rest, depth depends on how much evidence the answer requires, and in a bounded domain, that’s predictable from the query itself.
The expensive work happens offline and produces a Query-Cluster Database. At query time, QCAR is a lookup.
We embed historical queries with text-embedding-ada-002 (1,536 dimensions) and reduce the dimensions before clustering. PCA needed 186 components to explain 80% of the variance, and clusters built on it scored silhouette values near zero. UMAP preserved local structure far better, so we used it.
For clustering, we compared K-means, Ward-linkage agglomerative clustering and HDBSCAN. HDBSCAN left about a third of queries unclassified as noise, so we used it to find the natural number of clusters and stable centroids, then ran K-means to assign every query.
Our best numerical configuration, UMAP at two components with 60 clusters, scored a silhouette of 0.6 and a Davies-Bouldin index of 0.46. Excellent numbers.
We threw it away. A qualitative review, with Gemini 3 Pro reading the clusters, showed that two dimensions collapsed distinct legal topics into single groups. The clusters were compact, and compactness was all the metrics measured.
We shipped 190 components and 80 clusters instead: silhouette 0.369, Davies-Bouldin 1.019. Worse metrics, better semantics.
Depth needs ground truth. We sample 21 queries per cluster and use Claude Sonnet 4.5 as a judge to label every retrieved page 0 (not relevant), 1 (partially relevant) or 2 (fully relevant), about 190 pages per case. It’s expensive, but it runs offline, once.
Then we retrieve at the maximum k of 100 and compute NDCG at every depth from 1 to 100. The elbow is the deepest k at which NDCG is still meaningfully gaining:
k*(q) = max { k | NDCG_k+1(q) − NDCG_k(q) ≥ ε }, ε = 0.001
A simple query such as “Who was the driver at fault?” converges at k = 25 and gains only 0.1 NDCG across the remaining 75 documents. Everything past the elbow is cost without benefit. A hard query such as “List all treatments with dates and providers” climbs all the way to 100 with no plateau, because its evidence is spread across the record.
For each cluster, we take the mean and standard deviation of those elbows:
k*_cluster = μ + 2σ
That targets about 95% coverage of the relevant documents in the cluster.
At inference, QCAR is a thin layer in front of standard RAG:
The added overhead is small: under 1 ms for the lexical check, about 1 ms for the projection and under 1 ms for the nearest-centroid search. The one embedding call (30 to 50 ms) is already required for retrieval. Cluster assignment depends on the dimension of the clustering space and the number of clusters, and the number of documents in a case doesn’t affect it, so it runs in constant time per query.
We started with 20,000 raw queries, then cleaned and PII-filtered them down to 6,820. The baseline is a static k = 50, our optimized production setting at the time. Adaptive k* is capped at 100.
In offline evaluation against k = 50, simple fact-retrieval queries reached stable NDCG at lower depth, using 35.95% fewer tokens while F1 improved. Complex queries pushed k* to the 100-document cap and improved F1 at the cost of 85.95% more tokens. On those queries, accuracy rose with depth, and that’s the core argument against a static setting.
Adaptivity drives the gains. Against a randomly chosen k, adaptive retrieval holds a plateau, while random selection degrades immediately, because random cuts drop essential documents.
One caveat: QCAR doesn’t save money overall. In the A/B test, total token use rose 51.19%, because the expansion on complex queries outweighs the savings on simple ones. QCAR spends retrieval budget where it improves answers.
QCAR suits bounded-domain RAG: stable, recurring query distributions, a query history to learn from, and documents drawn from a coherent corpus such as case files or medical records. That covers most production deployments, including enterprise search, customer support, legal research, clinical decision support and financial analysis.
Porting it to a new vertical takes three things: a historical query log, a way to label relevance (LLM-as-a-judge or human annotation) and, optionally, some domain lexical patterns. The clustering and depth estimation transfer unchanged.
Open-domain search is the weaker case. Query diversity is unbounded, and corpus characteristics vary unpredictably. QCAR can still handle recognized query types and send the rest to a default k or a post-retrieval method, but its value is in bounded domains.
QCAR is a prior: it sets the budget before retrieval. Post-retrieval methods are a posterior: they trim what came back. Their failure modes complement each other. One prevents unnecessary retrieval, and the other refines an existing pool. The natural extension uses QCAR for the initial budget and post-retrieval cutoff detection on the result, and that’s the direction we find most promising.
Query distributions also drift. QCAR handles that by rebuilding the Query-Cluster Database on a schedule, or when monitoring detects a shift. Between rebuilds, the mapping stays fixed, so behavior stays predictable.
Schedule a call today to see how EvenUp's AI tools automate repetitive tasks, streamline custom drafting, and empower staff to focus on case strategy and client engagement.
Schedule a callRead Now
Read Now
Read Now
Read Now