Study guide
19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production
Google Cloud Professional Machine Learning Engineer PMLE
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Overview & the Agent Platform Map
1.1 PMLE Exam Facts, Policies & the June 2026 Blueprint 1.2 From Vertex AI to Gemini Enterprise Agent Platform: The Product Name Map
2 Chapter 2: Low-Code ML with BigQuery ML
2.1 Choosing, Training & Predicting with BigQuery ML Models 2.2 Feature Engineering & Feature Selection in BigQuery ML 2.3 Fine-Tuning & Evaluating Gemini Models from BigQuery
3 Chapter 3: AutoML & Google Cloud AI APIs
3.1 Training Models with Agent Platform AutoML 3.2 Building with Document AI, Vision, Translation & Other AI APIs
4 Chapter 4: Foundation Models in Model Garden
4.1 Evaluating & Selecting Models in Model Garden 4.2 Building & Tuning Solutions with Gemini, Imagen, Veo & MaaS 4.3 Optimizing Gemini Applications for Cost, Latency & Availability
5 Chapter 5: Exploring, Preprocessing & Protecting Data
5.1 Organizing & Exploring Tabular, Text & Image Data 5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python 5.3 Creating & Consolidating Features in Agent Platform Feature Store 5.4 Protecting PII & Sensitive Data in ML Workflows
6 Chapter 6: Notebooks, Prototyping & Experiment Tracking
6.1 Workbench vs. Colab Enterprise: Secure, Collaborative Notebooks 6.2 Prototyping with PyTorch, scikit-learn, JAX & Model Garden in Notebooks 6.3 Experiment Environments, Tracking & Lineage with Experiments and ML Metadata
7 Chapter 7: Evaluating Predictive & Generative AI Solutions
7.1 Evaluation Metrics for Predictive Models 7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge
8 Chapter 8: Choosing the Modeling Approach
8.1 Choosing the Model Type: ARIMA, Trees, DNNs & LLMs 8.2 Choosing the Product & Deployment Strategy 8.3 Modeling Techniques for Interpretability Requirements
9 Chapter 9: Training Data & Training SDKs
9.1 Organizing & Ingesting Training Data from Cloud Storage & BigQuery 9.2 Custom Training on Agent Platform: Containers, Jobs & Worker Pools 9.3 Kubeflow on GKE, AutoML & Tabular Workflows as Training Choices
10 Chapter 10: Troubleshooting, Hyperparameter Tuning & Fine-Tuning
10.1 Troubleshooting ML Training Failures 10.2 Hyperparameter Tuning on Agent Platform 10.3 Fine-Tuning Foundation Models & When to Tune
11 Chapter 11: Training Hardware & Distributed Training
11.1 Choosing CPUs, GPUs & TPUs for Training 11.2 Distributed Training: Data & Model Parallelism
12 Chapter 12: Deploying Models for Inference
12.1 Batch vs. Online Inference on Agent Platform 12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE 12.3 Prebuilt & Custom Serving Containers
13 Chapter 13: Model Versions, Rollouts & Inference Logic
13.1 Organizing & Versioning Models in Model Registry 13.2 Rollout Strategies: A/B Tests, Canaries & Traffic Splits 13.3 Inference Preprocessing & Postprocessing
14 Chapter 14: Scaling Online Serving
14.1 Serving Features Online with Feature Store 14.2 Public, Dedicated & Private Endpoints 14.3 Serving Hardware & Autoscaling for Throughput 14.4 Tuning Models for Production Training & Serving
15 Chapter 15: End-to-End ML Pipelines
15.1 Validating Data & Models in Pipelines 15.2 Agent Platform Pipelines with KFP: Components, Templates & Caching 15.3 Ensuring Consistent Preprocessing Between Training & Serving
16 Chapter 16: Orchestration with Managed Airflow & Ray
16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow 16.2 Ray on Agent Platform for Distributed ML Workloads
17 Chapter 17: Automating Retraining & CI/CD/CT
17.1 Choosing a Retraining Policy 17.2 CI/CD/CT for ML with Cloud Build
18 Chapter 18: Securing AI Systems, Responsible AI & Explainability
18.1 Protecting AI Systems from Data & Model Exfiltration 18.2 Defending Gen AI Apps: Safety Filters, Model Armor & Regex Guardrails 18.3 Responsible AI: Detecting & Monitoring Bias 18.4 Model Explainability on Agent Platform
19 Chapter 19: Monitoring AI Solutions
19.1 Configuring Model Monitoring on Agent Platform 19.2 Training-Serving Skew, Data Drift, Concept Drift & Attribution Drift 19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production
19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production
Key Takeaways Explain This Topic AI
The Agent Platform model observability dashboard reports metrics for fully managed models, such as requests per second, token throughput, first token latency, and API error rates.
Request-response logging (preview) saves samples of Gemini and supported partner model requests and responses to BigQuery for analysis and evaluation.
Online monitors for agents continuously evaluate production traces and export evaluation scores to Cloud Monitoring, where quality alerts can notify teams through Slack, email, or Pub/Sub.
Model Armor logs and insights show prompt injection attempts, sensitive data detections, and harmful content blocks for monitored AI applications.
Gen AI regression testing reruns a fixed evaluation set with Gen AI evals whenever prompts, models, tools, or retrieval settings change.
Last updated: September 2026
The last consideration in the exam guide is monitoring, testing, and evaluating gen AI solutions . A gen AI app needs the same operational monitoring as any service, plus quality , safety , and cost monitoring specific to LLMs.
What to Monitor
Dimension Signals Google Cloud tools
Operational health Requests per second, error rates (429, 5xx), time to first token, end-to-end latency, token throughput Model observability dashboard (fully managed Gemini and partner models), Cloud Monitoring
Cost Input, output, and cached tokens per request, requests per user or feature Token usage metrics, billing exports, usageMetadata in responses
Quality Rubric pass rates, groundedness, instruction following, task success, hallucination rate Gen AI evals on sampled logs, online monitors for agents, human review
Safety and security Safety filter blocks, prompt injection detections, sensitive data detections, malicious URLs Model Armor logs and insights, Gemini safety ratings
User outcomes Thumbs up or down, escalations to humans, task completion Application analytics, Feedback service (preview) for agents
Behavior drift Changes in topics, languages, prompt lengths, tool call patterns Logged request analysis in BigQuery, agent traces
Operational Monitoring
The model observability dashboard covers fully managed models (Gemini and partner MaaS). It shows model requests per second, token throughput , first token latencies , and API error rates . Self-hosted model metrics aren't included, so use endpoint metrics for those (Chapter 14). Console usage such as Agent Studio isn't counted.
Alert on 429 rates (capacity), latency percentiles , and token spikes , such as a prompt change that doubles context size.
Logging Requests and Responses
Request-response logging (preview) saves samples of requests and responses for Gemini models ( generateContent , streamGenerateContent ) and supported partner models such as Anthropic Claude to a BigQuery table, for base and tuned models. Uses:
Build evaluation datasets from real traffic.
Investigate complaints and incidents.
Analyze topic and usage drift.
Treat these logs as sensitive data : restrict access, set retention, and de-identify where possible (Chapter 5).
Quality Monitoring and Evaluation in Production
Sample production interactions from logs or traces.
Evaluate them with Gen AI evals : adaptive rubrics such as GENERAL_QUALITY with your guidelines, GROUNDING for RAG answers, SAFETY, and custom functions such as JSON validity or citation presence (Chapter 7).
Track scores over time in dashboards, sliced by feature, language, and user segment.
Alert when scores fall below thresholds.
Online monitors for agents
For agents deployed on Agent Platform, online monitors continuously evaluate production traces, which requires Cloud Trace and OpenTelemetry gen AI signals such as agent name, conversation ID, input and output messages, system instructions, and tool definitions. They detect quality drift , meaning performance that declines over time even when the model is unchanged, because user behavior or data shifts. Evaluation scores are exported to Cloud Monitoring , where quality alerts notify teams through Slack, email, or Pub/Sub and link back to the traces that caused them.
Safety and Security Monitoring
Model Armor logs every detection to Cloud Logging. In Inspect only mode this measures violation rates without blocking, and Model Armor insights summarize prompt injection attempts, sensitive data leakage, and harmful content across AI agents.
Track Gemini safety ratings and finishReason values ( SAFETY , RECITATION , SPII , PROHIBITED_CONTENT ). A spike can signal an attack or a prompt regression.
Watch for false positives too: legitimate requests being blocked hurt users and suggest thresholds need tuning.
Testing Gen AI Before Release
Test type Purpose
Regression evaluation Rerun a fixed evaluation set on every change to prompts, model versions, tools, retrieval settings, or safety thresholds
- Pairwise comparison Candidate vs. current version with an LLM judge and response flipping
- Adversarial and red-team tests Prompt injection, jailbreaks, data extraction attempts, off-topic abuse
- Grounding tests Questions whose answers must come only from provided documents
- Load and cost tests Latency and token usage at expected traffic
- Canary release A small share of users, with quality and safety monitoring before full rollout (Chapter 13)
- Common Gen AI Production Issues
Symptom Likely cause Response
Groundedness scores fall after a document update Retrieval index stale or chunking changed Re-index, check retrieval quality, add retrieval evaluation
Token cost per request doubles Prompt template change added context, or caching stopped hitting Review prompt diffs and cachedContentTokenCount
Quality drops after a model version upgrade Behavior change in the new version Roll back the version, re-tune prompts, rerun evaluations
Spike in prompt injection detections Active attack Switch Model Armor to Inspect and block, add rate limits, investigate sources
Rising 429 errors at peak Capacity limits Provisioned Throughput, global endpoint (if residency allows), backoff (Chapter 4)
User thumbs-down on one topic Missing knowledge or prompt gap Add grounding content, update instructions, expand the evaluation set
Worked Scenario
A health insurer's member-support assistant (Gemini with RAG) launches with:
Dashboards: first token latency p95, 429 rate, tokens per conversation, and cost per day.
Daily evaluation: 500 sampled conversations from request-response logs scored with GENERAL_QUALITY (guidelines: no medical advice, cite plan documents), GROUNDING, and a custom citation-presence check. Alerts fire if the grounding pass rate drops below 90%.
Security: Model Armor input and output templates with Cloud Logging, with a weekly review of injection and sensitive-data detections.
Change control: any prompt, model, or retrieval change must pass the regression evaluation set and a 5% canary before full rollout.