Key points
- The eval optimize command runs automated hill-climbing passes on agent prompts to systematically improve quality scores.
- Developers can use eval compare to generate structured JSON diffs that highlight per-case changes between baseline and candidate runs.
- The tooling supports integrating nightly scheduled runs with threshold-based alerting to track quality over time.
Google published a walkthrough on its Developers blog detailing how Agents CLI v1.8.0 facilitates systematic agent improvement through continuous testing.confirmed The post describes an evaluation loop that allows developers to catch prompt regressions and evaluate backend model swaps using side-by-side trace comparisons. This workflow supports automated evaluation passes and integrates with scheduled runs and alerting mechanisms to monitor quality trends.
What it does
The Agents CLI eval namespace provides framework-agnostic commands for managing this quality engineering loop. The eval generate command runs the agent against a dataset of test cases, while eval grade scores the resulting traces. The eval compare command produces a structured JSON diff showing which keys changed between two runs. The eval analyze command clusters failures into themes, and eval optimize executes an automated hill-climbing pass on the agent's prompt.
How to get it
The documented workflow uses a LangChain template that ships with Agents CLI v1.8.0.reported Setup requires a single command to initialize the scaffold. This scaffold includes an evaluation dataset at tests/eval/datasets/basic-dataset.json, a configuration file at tests/eval/eval_config.yaml, and a custom LLM-as-judge metric located at tests/eval/response_quality.py. The configuration declares the specific metric to run during the evaluation process.
Limits
The default dataset provided in the scaffold contains only a single trivial case, which the source material notes is insufficient for effective hill-climbing. The example metric grades responses on a 1-to-5 scale for accuracy, relevance, and clarity using gemini-3.8-flash as the judge.reported The walkthrough demonstrates that a small dataset is preferable for the hill-climbing loop, as it ensures the evaluation runs quickly on every change made to the agent.
The takeaway
The blog post illustrates how this loop surfaces regressions that traditional pre-eval pipelines might miss. In the example, a developer accidentally pasted a chef persona system prompt into a weather agent. Standard lint, unit tests, and type checks did not catch this error because the code compiled and the agent still ran. However, re-running the evaluation sequence dropped the mean score from 4.0 to 2.75, quantifying the regression before the agent reached production.reported
Building on this
Developers can build automated quality gates that run eval generate and eval grade on every commit. The eval compare output allows developers to inspect the differences object for full per-case breakdowns with baseline and candidate values side by side. Teams can configure the system to trigger alerts when mean scores drop below a threshold, such as the 1.25-point drop shown in the example. The eval optimize command runs an automated hill-climbing pass on the agent's prompt.
Why it matters
- Automated hill-climbing via eval optimize reduces manual effort in prompt engineering by systematically testing variations until quality scores improve.
- The eval compare JSON diff provides granular visibility into per-case performance changes, allowing teams to pinpoint exactly which behaviors regressed after a code change.
- Integrating evaluation into pre-production checks catches logical errors like persona mismatches that static analysis tools and unit tests cannot detect.
- Scheduled nightly runs with threshold alerting allow developers to track quality over time, as described in the Google Developers blog post.
Topics: Google, Agents CLI, evaluation, prompt engineering, MLOps
Updated
Ask the article
Local keyword search of this article; no generated answers.Which of these is confirmed?
Check the answer
Google published a walkthrough on its Developers blog detailing how Agents CLI v1.8.0 facilitates systematic agent improvement through continuous testing.
The matched primary source and announcement wording support the confirmed label.
Did this article give you the evidence you needed?
Your choice stays on this device. No public tally.Questions readers ask
How does the eval optimize command improve agent prompts?
It runs an automated hill-climbing pass on the agent's prompt, systematically testing changes and keeping those that improve evaluation scores.
What metric is used in the example evaluation workflow?
The example uses a custom LLM-as-judge metric that grades responses on a 1-to-5 scale for accuracy, relevance, and clarity using gemini-3.8-flash.
Can the evaluation loop detect errors that unit tests miss?
Yes, the example shows it caught a wrong system prompt that compiled successfully and passed standard lint and type checks but degraded agent performance.
Connected stories
Reading receipt
- The eval optimize command runs automated hill-climbing passes on agent prompts to systematically improve quality scores.
- Developers can use eval compare to generate structured JSON diffs that highlight per-case changes between baseline and candidate runs.
- The tooling supports integrating nightly scheduled runs with threshold-based alerting to track quality over time.