Evaluation levels, test design, and metrics
Measure components, observed steps, actual outcomes and repeated-run reliability using explicit acceptance rules.
- Define cases, environments, oracles, evaluators and metrics.
- Separate trajectory quality from task outcomes.
- Interpret repeated-trial metrics and full-configuration comparisons.
Why this matters
Section titled “Why this matters”Lee asks the shop assistant to reserve two blue mugs. It calls the stock tool correctly, writes a convincing reply and ends without an error. But the reservation database contains no reservation. A healthy trace tells us that software ran; it does not establish that Lee’s goal was met.
Evaluation means checking behavior against stated expectations. Begin with the task: which mugs, how many, for whom, and what should happen if stock is unavailable? The observability records supply evidence; acceptance rules decide what that evidence means.
How it works
Section titled “How it works”Turn a request into a test case
Section titled “Turn a request into a test case”A test case specifies an input, starting conditions and acceptance rules. A test set collects cases. An environment contains the tools and state in which the agent acts. A simulator imitates part of that environment, such as a customer or stock service. Each fresh attempt is a trial.
An oracle is the reference used to decide correctness: for example, an expected reservation record. An evaluator, also called a grader, applies checks. A metric summarizes a measurement, such as the proportion of trials that pass. A benchmark packages tasks and an evaluation procedure so results can be compared under stated conditions. These are different jobs, even when one script performs several of them.
For our shop, restore stock to three mugs before each trial. Ask for two. Require one reservation for exactly two mugs and remaining stock of one. Also include insufficient stock, ambiguous color, a failed lookup and interruption before the write. A correct clarification or failure report can be the expected outcome of a case.
Keep the task understandable to both the agent and reviewer. Do not grade an unstated requirement. Check that a known valid solution passes and a known wrong solution fails. Isolate trial state, including files and durable memories, unless persistence is deliberately part of the test. Agent evaluation guidance.
Choose the level that answers your question
Section titled “Choose the level that answers your question”These four levels are a teaching map, not a prescribed architecture:
| Level | Shop question | Useful measurement | What it cannot establish alone |
|---|---|---|---|
| Component | Did the tool receive a valid quantity? | Argument validity, lookup accuracy | Whether a reservation exists |
| Trajectory | Did the assistant read evidence before acting? | Required observations, unnecessary calls | Whether the final write worked |
| Task | Is the requested reservation present? | Verified task pass or failure | Reliability across other requests |
| System | Does the service keep working across tasks and runs? | Failure rate, time, cost, recovery | Behavior outside the sampled conditions |
A trajectory is the observed sequence of messages, calls and results. It is not access to hidden model reasoning. ADK exposes trajectory and response evaluation separately. An exact tool sequence is useful when order is required, but a broader condition can allow equivalent valid routes. ADK evaluation.
Figure 1. Read from a local check toward the whole service. A success at one shelf does not automatically pass the next shelf. The task shelf checks the shop’s records, rather than the robot’s claim. Numbers and item IDs in the illustration are invented examples, separate from the blue-mug trial below; they are not measured performance.
Use the right kind of grader
Section titled “Use the right kind of grader”A deterministic check applies a fixed rule, such as comparing database fields. A judge model scores an answer using a rubric, an explicit set of criteria. Human review can resolve ambiguous cases and check whether the rubric represents the intended task.
For the shop, code verifies stock and reservations; a reviewer assesses whether the explanation is clear. A judge’s favorable wording score cannot replace a missing reservation. Calibrate model judgments against human-reviewed examples, including wrong answers that sound persuasive, and permit an unknown result when evidence is missing. Grader guidance.
Repeat trials and keep the denominators visible
Section titled “Repeat trials and keep the denominators visible”Pass rate is passed trials divided by attempted trials under a defined rule. Variance describes how measurements spread across runs or cases. Report case counts, attempts per case and failed or ungraded attempts, alongside the score. Do not quietly remove timeouts from the denominator.
pass@k asks whether at least one of k attempts succeeds. pass^k asks whether all k attempts succeed. If independent attempts on one task each succeed with probability p, these probabilities are 1 - (1 - p)^k and p^k respectively. These are explanatory formulas under an independence assumption, not automatic estimators for every dataset. For heterogeneous tasks, compute the per-task quantity before averaging; a shared outage or retained state can break independence. τ-bench introduced pass^k to expose inconsistency across repeated tool-user interactions. τ-bench.
For an illustrative p = 0.8 and k = 3, at least one success is 99.2%, while all three successes is 51.2%. Neither number means users can obtain a verified reservation 99.2% of the time in one interaction. Retrying also requires a way to detect failure and avoid duplicate effects.
Compare complete configurations
Section titled “Compare complete configurations”Record the model version, agent harness, tool schemas, instructions or skill versions, environment snapshot, memory policy, budgets and grader version. The agent harness is the code that runs the model and tools. The evaluation harness runs the tests and grading. Changes to either can change a result.
Measure end-to-end elapsed time, including waits, and state whether human waiting is included. Report the median, the middle value of ordered measurements, and a tail percentile, such as the 95th percentile, a value at or below which roughly 95% of observations fall. Include enough observations to interpret these measurements. Count model and tool costs for failed trials too. Cost per verified success is total trial cost divided by verified successes; with none, report that it is undefined. Compare direct tool loops and code orchestration on the same cases and budgets. These are our reporting conventions, not provider guarantees.
Worked example
Section titled “Worked example”The offline grader simulates four scripts against fresh shop state. It checks final records separately from required observations. One script says “Reserved” without writing anything. Another writes correctly but skips the required lookup. Both distinctions appear in the output.
Expand minimal Python implementation
outcome_ok = reservations == [{"sku": "blue-mug", "quantity": 2}]trajectory_ok = "read_stock" in observed_stepsaccepted = outcome_ok and trajectory_okThe runnable example also checks remaining stock. Its required lookup is a stated condition of this teaching task, not a rule that every reservation workflow must use that exact call. No model is invoked; scripted cases demonstrate the grader, not model capability.
Limitations and trade-offs
Section titled “Limitations and trade-offs”Mocks make failures repeatable but may omit real tool behavior, user ambiguity or changing data. A large score on a narrow test set is evidence about that set. Review failures by category, compare representative tasks and include interrupted work with recovery. Detailed benchmark and release practice follows in the optional deep dive. To stay on the main path, continue to multi-agent systems.
Check your understanding
A judge rates the reply highly and all tool calls returned successfully. The reservation table is empty. Should the reservation task pass?
No. Those checks measure communication and execution. The stated task requires an actual reservation. Investigate whether the tool result was misunderstood, the write was omitted or the outcome check is looking at the wrong state.
Security preview
Section titled “Security preview”Functional evaluations show whether intended tasks work. They do not establish resistance to hostile inputs. Pass 2 adds security testing and assurance with explicit threat assumptions and separate checks.
Key takeaways
Section titled “Key takeaways”- State success before choosing metrics, and check actual artifacts or external state.
- Measure components, observed trajectories, tasks and repeated service behavior separately.
- Report complete configurations, denominators, uncertainty, latency and cost.
- A repeatable simulator and a calibrated grader are useful evidence with visible limits.
