Grounding, long context, and retrieval evaluation
Compare broad context and retrieval on measured tasks, separate ranking from claim support, and explain evaluator limitations.
- Compare context and retrieval designs without fixed latency promises.
- Interpret ranking metrics with explicit conventions.
- Distinguish evidence support from truth and task usefulness.
Why this matters
Section titled “Why this matters”Lee asks the shop assistant whether a lamp can be returned after 30 days. The assistant retrieves a policy and gives a confident answer. Three things could still go wrong: it found the wrong policy, misunderstood the right policy, or answered a different question.
Grounding means connecting an answer to supplied evidence. Retrieval-Augmented Generation (RAG) retrieves evidence before generating an answer. Neither process proves the answer correct. This chapter compares retrieval with long context and shows how to measure the separate steps.
Simple mental model
Section titled “Simple mental model”The assistant can read a complete policy handbook or search for selected passages. Reading more may preserve useful connections. Searching may remove distractions, but can omit an important exception. A third option searches first and then supplies complete matching documents.
The choice depends on the questions, document set, model and execution budget. Keep the same test questions when comparing designs.
How it works
Section titled “How it works”Compare designs on the actual task
Section titled “Compare designs on the actual task”A context window limits how much input and output a model request can hold. Large windows make broad document input possible, but advertised capacity does not measure how well a model uses every passage. The Gemini 1.5 paper demonstrates long-context capabilities for specific models and tasks; it is historical evidence, not a current model catalog. Gemini 1.5.
| Question to measure | Broad document context | Retrieval before generation |
|---|---|---|
| What preparation is needed? | Parse and select documents that fit | Also build and maintain a searchable index |
| Where does time go? | Input processing and generation | Search, possible reranking, input processing and generation |
| What costs recur? | Document input, with provider-dependent caching | Index maintenance, search and selected document input |
| What can be missed? | Relevant details among other content | Evidence excluded by search or selection |
| How are changes included? | Supply updated documents or references | Update the source and its indexed representation |
| How is broad synthesis handled? | Supply related documents together | Retrieve larger units or assemble evidence through several searches |
Measure time to first token and time to the complete answer separately. Smaller inputs can reduce model work while search adds another step. Neither design promises sub-second responses.
Figure 1. Both robots need the same task-level checks. Compare latency, cost and missing evidence rather than choosing from context size alone.
The 2023 Lost in the Middle study found position-sensitive performance in the models and tasks it tested. Use that finding to design position and distractor tests for your chosen model, rather than assuming every current model has the same degradation curve. Study.
Assemble evidence and preserve its meaning
Section titled “Assemble evidence and preserve its meaning”Each selected passage needs a document identifier, source version and text. For Lee’s question, record the return-policy version and the exception paragraph alongside the main return rule. Instructions can ask the model to cite these references and acknowledge missing evidence.
A citation marker shows which source the model claims to use. Check separately that the referenced passage was supplied and that it supports the statement. Labels and document delimiters help organize input; they do not enforce model behavior or make the source trustworthy.
Measure search independently
Section titled “Measure search independently”A relevance judgment labels whether a document helps answer a test question. A ranking orders search results. With a fixed set of judgments, these metrics answer different questions:
| Metric | Question |
|---|---|
| Precision@K | How many of the first K positions contain relevant results? |
| Recall@K | How much of the known relevant set appears in those positions? |
| Reciprocal rank | How early is the first relevant result? |
| NDCG@K | Does the order put the most useful graded results first? |
Here K is the evaluation cutoff. Mean reciprocal rank (MRR) averages reciprocal rank across questions. Normalized discounted cumulative gain (NDCG) compares an observed ranking with an ideal ranking using the same relevance grades. Record judgments, cutoff and aggregation alongside scores. BEIR illustrates evaluation across varied retrieval tasks. BEIR.
Measure the generated answer separately
Section titled “Measure the generated answer separately”Ask whether the retrieved context is relevant, the claims are supported and the answer addresses the request. Ragas offers automated evaluation for such dimensions, including approaches that do not require a reference answer. Ragas paper.
Faithfulness estimates the proportion of answer claims supported by retrieved context. Its value depends on claim extraction and the evaluator. A score below one suggests unsupported claims under that evaluation; it does not establish that they came from model training data. A score of one can still support a wrong answer if the source itself is wrong. Ragas faithfulness.
Figure 2. Inspect the question, evidence and answer separately. Automated scores need calibration against human judgments and do not prove truth.
Worked example
Section titled “Worked example”Suppose the correct policy allows ordinary returns for 30 days but requires a separate check for damaged goods. Retrieval finds the main rule and misses the exception. An answer about damaged goods could be faithful to the selected text yet fail Lee’s task. Search coverage and answer usefulness therefore need separate checks.
Minimal implementation
Section titled “Minimal implementation”Run the offline evaluation example. It calculates ranking metrics and demonstrates why word overlap cannot establish support: a negated statement can share nearly all the source’s words.
Expand minimal Python implementation
hits = len(set(retrieved[:k]) & relevant)precision = hits / krecall = hits / len(relevant) if relevant else 0.0The full script requires unique result identifiers and a positive cutoff. Missing result positions count as non-relevant for Precision@K. Its lexical comparison is a deliberately limited heuristic, not an implementation of Ragas. No model or semantic evaluator is called.
Limitations and trade-offs
Section titled “Limitations and trade-offs”Ground-truth relevance sets can be incomplete. Automated judges can disagree with people. Evaluate a representative human-reviewed sample and report the evaluator configuration. Online evaluation adds work; sampled offline evaluation delays feedback. Choose the cadence according to the debugging question.
Synthetic tests of a single fact do not establish performance on questions that need conflicting sources or several reasoning steps. Compare memory, retrieval and tool outcomes in the complete workflow.
Security preview
Section titled “Security preview”Source content, citations and evaluator inputs introduce boundaries examined in retrieval, memory and data security. This chapter establishes what is measured before that security analysis.
Key takeaways
Section titled “Key takeaways”- Compare broad context, retrieval and hybrid designs on the same tasks.
- Separate search ranking, claim support and task usefulness.
- A citation or evaluation score does not prove truth.
- State metric conventions and the limits of mocked examples.
Check your understanding
Can an answer be faithful to retrieved text and still be wrong? Yes: the text may be outdated, incomplete or incorrect, or the answer may miss the user’s request.
References
Section titled “References”- Liu et al. Lost in the Middle. Research on the tested 2023 models and tasks.
- Es et al. Ragas; Ragas faithfulness documentation, checked 2026-10-10.
- Gemini Team. Gemini 1.5. Historical long-context study.
- Thakur et al. BEIR.
