Study guide

6.5 Monitoring, Testing and Evaluating Generative AI in Production

Google Cloud Professional Machine Learning Engineer Exam

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

0.1 Exam Overview, Blueprint & Certification Logistics 0.2 Study Strategy, Scenario Mechanics & 8-Week Roadmap

2 Section 1: Architecting Low-Code AI Solutions (~13%)

1.1 BigQuery ML: Model Types, SQL Syntax and In-Database Evaluation 1.2 Feature Engineering and Feature Selection in BigQuery ML 1.3 Generating Predictions with BigQuery ML 1.4 AutoML on Agent Platform: Tabular, Image, Text and Video Training 1.5 Fine-Tuning Gemini Models from BigQuery 1.6 Model Garden: Evaluating and Selecting the Right Model 1.7 Industry-Specific AI APIs: Document AI, Vision, Translate and Speech 1.8 Generative Media and Models as a Service: Gemini, Imagen and Veo 1.9 Optimizing Gemini Applications for Cost, Latency and Availability

3 Section 2: Collaborating Within and Across Teams to Manage Data and Models (~16%)

2.1 Organizing and Exploring Tabular, Text, Image and Video Data 2.2 Choosing the Preprocessing Tool: BigQuery, Dataflow, Spark and In-Memory Python 2.3 Creating and Consolidating Features in Agent Platform Feature Store 2.4 Data Privacy, PII Handling and Sensitive Information 2.5 Notebook Environments: Collaboration and Security Best Practices 2.6 Prototyping in Workbench and Colab Enterprise with PyTorch, scikit-learn and JAX 2.7 Prototyping with Model Garden Foundational and Open Models in Notebooks 2.8 Choosing the Development and Experimentation Environment 2.9 Evaluating Generative Solutions: LLM-as-a-Judge and RAG Evaluation 2.10 Experiment Tracking, Artifact Lineage and Model Registry Governance

4 Section 3: Scaling Prototypes into ML Models (~21%)

3.1 Choosing the Model Type: ARIMA, DNN, LLM and Classical Baselines 3.2 Choosing the Product: AutoML, BigQuery ML, Custom Training and Pipelines 3.3 Choosing the Deployment Strategy at Design Time 3.4 Modeling Techniques for Interpretability Requirements 3.5 Organizing Training Data on Cloud Storage and BigQuery 3.6 Ingesting Structured and Unstructured Data into Training Pipelines 3.7 Training SDKs and Services: Custom Training, Kubeflow on GKE, AutoML and Tabular Workflows 3.8 Troubleshooting ML Model Training Failures 3.9 Hyperparameter Tuning, Bayesian Optimization and Vizier 3.10 Model Evaluation Metrics, Transfer Learning and Fine-Tuning 3.11 Evaluating Compute and Accelerator Options: CPU, GPU and TPU 3.12 Distributed Training Strategies, Hardware Acceleration and Performance Optimization

5 Section 4: Serving and Scaling Models (~20%)

4.1 Batch and Online Inference: Agent Platform, Model Garden, Cloud Run and GKE 4.2 Rollout Strategies: Canary, Traffic Splitting, Shadow and Blue-Green 4.3 Tuning Models for Production: Vector Search and Model Compression 4.4 Deploying Models to Public and Private Endpoints 4.5 Choosing Serving Hardware: CPU, GPU, TPU and Edge 4.6 Scaling the Serving Backend for Throughput

6 Section 5: Automating and Orchestrating ML Pipelines (~18%)

5.1 Validating Data and Models Inside Pipelines 5.2 Agent Platform Pipelines: Kubeflow Pipelines (KFP) and TFX Architecture 5.3 Reusable Pipeline Components, Artifacts and ML Metadata Lineage 5.4 Ray on Agent Platform for Distributed Python Workloads 5.5 MLOps CI/CD Automation, Retraining Triggers and Skew Prevention

7 Section 6: Monitoring AI Solutions (~13%)

6.1 Securing AI Systems: Exfiltration, Malicious Prompting and Model Armor 6.2 Responsible AI, Explainable AI, Fairness Metrics and Model Governance 6.3 Model Monitoring: Feature Skew and Drift Detection 6.4 Operational Troubleshooting, Performance Profiling and Cost Optimization 6.5 Monitoring, Testing and Evaluating Generative AI in Production

6.5 Monitoring, Testing and Evaluating Generative AI in Production

Key Takeaways Explain This Topic AI

Generative systems have no ground-truth label stream, so production quality is tracked through continuous automated evaluation on sampled traffic rather than through accuracy against outcomes.

The production signal set is groundedness, safety-filter and Model Armor trigger rates, refusal rate, latency, token cost per request, and user feedback.

Managed base models are updated by the provider, so a versioned regression suite must be re-run against any new version before it is adopted.

Retrieval health is monitored separately from generation quality, because a fall in answer quality is usually a retrieval regression.

Sampled human review remains necessary to keep an automated judge calibrated and to catch failure modes no rubric anticipated.

Last updated: September 2026

  • 6.5 Monitoring, Testing and Evaluating Generative AI in Production
  • Blueprint reference: Section 6.2, "Monitoring, testing, and evaluating gen AI solutions."

Section 6.3 covered monitoring for a predictive model: a baseline distribution, statistical distance metrics, skew and drift alerts. Generative systems break most of those assumptions, and the exam expects you to know why and what replaces them.

Why Classical Monitoring Does Not Transfer

  • Predictive model Generative system
  • Fixed input schema Free-form text, images, documents
  • Bounded output space Unbounded output space
  • Ground-truth labels arrive eventually Usually no label ever arrives
  • Feature distribution comparison is meaningful "Distribution of the output text" is not a usable statistic
  • Accuracy computable from outcomes Quality must be judged , not computed

The consequence: production quality is established by continuously evaluating a sample of live traffic with the same rubric-based methods used offline, rather than by waiting for labels that never come. Offline evaluation methodology — judge design, rubric anchoring, calibration, RAG's two halves — is covered in Section 2.9 ; this section is about running it against production.

What to Monitor

Signal Why it matters Typical alert

Groundedness / faithfulness on sampled traffic The primary quality signal for RAG and summarization Rolling score falls below threshold

Safety filter and Model Armor trigger rates A spike means an attack campaign or a policy regression Rate exceeds baseline band

Refusal rate Rising refusals mean the model has become over-cautious or prompts have changed Sustained increase

Latency p50/p95 User experience and capacity Breach of budget

Token cost per request Prompt growth and verbosity creep are silent budget killers Rolling cost per request rises

Retrieval hit rate / recall proxy The most common root cause of quality loss Fewer relevant documents retrieved

  • User feedback (thumbs, escalations, edits) The only direct signal of perceived quality Ratio deteriorates
  • Empty or malformed structured output Schema violations break downstream systems Any sustained rate

Cost per request deserves specific attention. Prompts accumulate: an extra few-shot example here, a longer system instruction there, a larger retrieved context. None of it fails a test, and the bill grows steadily. Tracking tokens per request as a monitored metric catches it.

Continuous Evaluation on Sampled Traffic

The mechanics:

Sample production requests and responses at a rate that is statistically useful and affordable. Sample deliberately across segments rather than taking the first N.

Score the sample with the automated judge and rubric, and with any deterministic checks available — did the structured output parse, did every citation resolve to a retrieved document, did the response respect the required format.

Log the scores as a time series so trends are visible and alertable.

Route low scores to human review , both to confirm the finding and to build the next round of regression cases.

Deterministic checks are underrated because they are cheap and unambiguous. Citation resolution in particular — does every cited source actually appear in the retrieved set — catches a large fraction of grounding failures without any judge at all.

Regression Testing Against Model Updates

This is the generative-specific concern that has no analogue in classical monitoring. Managed base models are updated by the provider. A version change can alter tone, formatting, refusal behaviour, or reasoning in ways that break a downstream application built around the previous behaviour.

The discipline:

Keep a versioned regression suite — the eval set from model selection, plus every incident that has been fixed, retained as a test case.

Pin the model version in production where the platform allows it, so changes are adopted deliberately.

Re-run the suite against any new version before adopting it, comparing per-dimension scores rather than a single aggregate.

Re-run it after any prompt change too. A prompt edit is a code change to the system's behaviour and deserves the same gate.

Monitor Retrieval Separately

For a retrieval-augmented system, answer quality falling is usually a retrieval regression: the index grew stale, an embedding model changed, a document set was re-chunked, or a filter is now excluding relevant material. Monitoring only end-to-end quality tells you something broke without telling you which half.

Track on the retrieval side: index freshness and document count, retrieved-document relevance on a labelled probe set run periodically, the rate of queries returning nothing, and latency of the retrieval step.

Human Review and Feedback Loops

Automated judges drift out of calibration as traffic changes and cannot recognize failure modes their rubric never anticipated.

Sample for human review continuously , not only during incidents.

Re-measure judge-human agreement periodically ; falling agreement means the judge needs recalibration or the rubric needs extending.

Feed confirmed failures back into the regression suite so the same defect cannot silently return.

Treat explicit user feedback as a monitored signal , remembering it is biased toward extremes — most satisfied users say nothing.

Exam Traps

Applying feature-drift monitoring to free-form text. The statistic is not meaningful.

Waiting for ground-truth labels. They usually never arrive.

Adopting a new base model version without re-running the regression suite.

Monitoring only end-to-end quality in a RAG system; measure retrieval separately.

Trusting an automated judge indefinitely without re-checking human agreement.

Ignoring token cost per request as prompts quietly grow.