Google adds silent AI failure detection to Gemini platform

Google has announced the general availability of Agent and Model Evaluations within its Gemini Enterprise Agent Platform. This release aims to bridge the gap between development-stage testing and live production monitoring by providing a unified evaluation engine. By applying identical metric definitions across both environments, developers can reliably identify performance drift without worrying if discrepancies stem from inconsistent measurement techniques.

The platform includes more than 20 pre-built metrics categorized into two distinct families. Deterministic, computation-based metrics are free to use and include ROUGE for summarization, exact match for extractive QA, and BLEU, MetricX, and COMET for translation. Conversely, LLM-as-a-judge metrics, which are billed based on token usage, handle subjective assessments such as task success, safety, hallucination, grounding, tool use quality, and trajectory quality. To assist with testing, Google provides simulation tools including a case generator, a multi-turn user simulator, and an environment simulator for mock backends.

A key feature of the platform is its adaptive rubrics, co-developed with Google DeepMind, which generate customized pass-and-fail criteria for each evaluation input rather than relying on static judge prompts. For live systems, the platform supports continuous online monitoring of production traffic, complete with score-over-time dashboards and automated drift alerts delivered through email or Slack. New accounts can get started with $300 in free credits to test these capabilities.

For AI practitioners, this release addresses the notorious difficulty of detecting silent failures once an agent goes live. Instead of manually writing extensive test suites or building custom monitoring pipelines, developers can leverage automated simulations and consistent scoring. This ensures that any drop in an agent's performance score indicates an actual regression in user experience rather than a mere benchmarking artifact, making production deployments significantly safer and more predictable.