Study guide

7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge

Google Cloud Professional Machine Learning Engineer PMLE

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview & the Agent Platform Map

1.1 PMLE Exam Facts, Policies & the June 2026 Blueprint 1.2 From Vertex AI to Gemini Enterprise Agent Platform: The Product Name Map

2 Chapter 2: Low-Code ML with BigQuery ML

2.1 Choosing, Training & Predicting with BigQuery ML Models 2.2 Feature Engineering & Feature Selection in BigQuery ML 2.3 Fine-Tuning & Evaluating Gemini Models from BigQuery

3 Chapter 3: AutoML & Google Cloud AI APIs

3.1 Training Models with Agent Platform AutoML 3.2 Building with Document AI, Vision, Translation & Other AI APIs

4 Chapter 4: Foundation Models in Model Garden

4.1 Evaluating & Selecting Models in Model Garden 4.2 Building & Tuning Solutions with Gemini, Imagen, Veo & MaaS 4.3 Optimizing Gemini Applications for Cost, Latency & Availability

5 Chapter 5: Exploring, Preprocessing & Protecting Data

5.1 Organizing & Exploring Tabular, Text & Image Data 5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python 5.3 Creating & Consolidating Features in Agent Platform Feature Store 5.4 Protecting PII & Sensitive Data in ML Workflows

6 Chapter 6: Notebooks, Prototyping & Experiment Tracking

6.1 Workbench vs. Colab Enterprise: Secure, Collaborative Notebooks 6.2 Prototyping with PyTorch, scikit-learn, JAX & Model Garden in Notebooks 6.3 Experiment Environments, Tracking & Lineage with Experiments and ML Metadata

7 Chapter 7: Evaluating Predictive & Generative AI Solutions

7.1 Evaluation Metrics for Predictive Models 7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge

8 Chapter 8: Choosing the Modeling Approach

8.1 Choosing the Model Type: ARIMA, Trees, DNNs & LLMs 8.2 Choosing the Product & Deployment Strategy 8.3 Modeling Techniques for Interpretability Requirements

9 Chapter 9: Training Data & Training SDKs

9.1 Organizing & Ingesting Training Data from Cloud Storage & BigQuery 9.2 Custom Training on Agent Platform: Containers, Jobs & Worker Pools 9.3 Kubeflow on GKE, AutoML & Tabular Workflows as Training Choices

10 Chapter 10: Troubleshooting, Hyperparameter Tuning & Fine-Tuning

10.1 Troubleshooting ML Training Failures 10.2 Hyperparameter Tuning on Agent Platform 10.3 Fine-Tuning Foundation Models & When to Tune

11 Chapter 11: Training Hardware & Distributed Training

11.1 Choosing CPUs, GPUs & TPUs for Training 11.2 Distributed Training: Data & Model Parallelism

12 Chapter 12: Deploying Models for Inference

12.1 Batch vs. Online Inference on Agent Platform 12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE 12.3 Prebuilt & Custom Serving Containers

13 Chapter 13: Model Versions, Rollouts & Inference Logic

13.1 Organizing & Versioning Models in Model Registry 13.2 Rollout Strategies: A/B Tests, Canaries & Traffic Splits 13.3 Inference Preprocessing & Postprocessing

14 Chapter 14: Scaling Online Serving

14.1 Serving Features Online with Feature Store 14.2 Public, Dedicated & Private Endpoints 14.3 Serving Hardware & Autoscaling for Throughput 14.4 Tuning Models for Production Training & Serving

15 Chapter 15: End-to-End ML Pipelines

15.1 Validating Data & Models in Pipelines 15.2 Agent Platform Pipelines with KFP: Components, Templates & Caching 15.3 Ensuring Consistent Preprocessing Between Training & Serving

16 Chapter 16: Orchestration with Managed Airflow & Ray

16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow 16.2 Ray on Agent Platform for Distributed ML Workloads

17 Chapter 17: Automating Retraining & CI/CD/CT

17.1 Choosing a Retraining Policy 17.2 CI/CD/CT for ML with Cloud Build

18 Chapter 18: Securing AI Systems, Responsible AI & Explainability

18.1 Protecting AI Systems from Data & Model Exfiltration 18.2 Defending Gen AI Apps: Safety Filters, Model Armor & Regex Guardrails 18.3 Responsible AI: Detecting & Monitoring Bias 18.4 Model Explainability on Agent Platform

19 Chapter 19: Monitoring AI Solutions

19.1 Configuring Model Monitoring on Agent Platform 19.2 Training-Serving Skew, Data Drift, Concept Drift & Attribution Drift 19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production

7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge

Key Takeaways Explain This Topic AI

Gen AI evals on Agent Platform support adaptive rubrics, static rubrics, computation-based metrics, and custom function metrics.

Adaptive rubrics generate a unique set of pass/fail tests for each prompt, and Google recommends them as the default starting point.

Computation-based metrics such as exact_match, BLEU, and ROUGE need a ground-truth reference and use deterministic algorithms.

Response flipping reduces position bias in pairwise judging by swapping baseline and candidate responses in half of the judge calls.

A judge model should be validated against human ratings using metrics such as balanced accuracy and balanced F1 before its scores are trusted.

Last updated: September 2026

The exam guide explicitly names LLM-as-a-judge . Gen AI evaluation matters for model selection (Chapter 4), prompt changes, tuning, migration to new model versions, and production monitoring (Chapter 19). Agent Platform's Gen AI evals (formerly the Gen AI evaluation service) provides the tools.

Step 1: Build the Evaluation Dataset

Evaluation is only as good as its dataset. Gen AI evals supports three ways to build one:

Upload complete prompts, or a prompt template plus a file of variable values.

Sample production logs to reflect real usage.

Generate synthetic data from a prompt template to get many consistent examples.

Include reference answers (ground truth) if you plan to use computation-based metrics. Cover edge cases: long inputs, ambiguous requests, adversarial prompts, multiple languages, and anything that previously failed.

Step 2: Choose Metric Types

Metric type How it works Output Best for

Adaptive rubrics (recommended) A judge generates pass/fail tests unique to each prompt , then checks the response against each Pass rate per response, with a rationale per rubric Fast, task-relevant evaluation of open-ended outputs

Static rubrics One fixed rubric applied to every prompt A numeric score (such as 1-5) Consistent benchmarks on a specific dimension

Computation-based Deterministic comparison with a reference Score such as 0.0-1.0 Tasks with a well-defined correct answer

Custom function (SDK) Your Python logic, run locally or in a remote sandbox Your score Format checks, business rules, domain validators

Adaptive rubric example

Prompt: "Write a four-sentence summary of the article, keeping an optimistic tone." Generated rubrics: (1) it summarizes the article, (2) it has exactly four sentences, (3) it keeps an optimistic tone. A response that passes the first two but ends on a negative note fails the third, for a 66.7% pass rate . Comparing two models on the same rubrics shows exactly which requirement each one misses.

Predefined adaptive metrics include GENERAL_QUALITY (the recommended default, which can be guided with natural-language guidelines such as "must not give financial advice"), INSTRUCTION_FOLLOWING , and TEXT_QUALITY , plus multi-turn and agent metrics (for example, hallucination and tool_use_quality ).

Static rubric and template metrics

GROUNDING checks factual consistency against provided source text, which is essential for RAG. SAFETY checks policy violations such as hate speech or dangerous content. Template metrics such as FLUENCY are also available. You can define custom static rubrics with your own criteria and rating scale, but Google recommends trying GENERAL_QUALITY with guidelines first.

Computation-based metrics

  • Metric Measures Typical task
  • exact_match Share of responses identical to the reference Short factual answers, labels, IDs
  • BLEU Matching n-grams against a reference Translation
  • ROUGE ( rouge_1 , rouge_l ) N-gram or longest-common-subsequence overlap Summarization

Overlap metrics reward wording similarity, not correctness. A correct paraphrase can score low, and a fluent wrong answer that reuses reference words can score high. Pair them with rubric-based or human checks.

Custom function metrics

A custom function receives each instance (prompt, response, reference, and other data) and returns a score, for example "is the output valid JSON with all required fields?" Remote custom functions run in a sandbox with no network access , a 1-minute execution limit, and a 1.5 GB limit on code plus loaded data.

Step 3: Pointwise vs. Pairwise Judging

Pointwise: score each response on its own against criteria. Good for tracking absolute quality over time.

Pairwise (side-by-side): a judge compares a baseline and a candidate response and picks the better one or a tie. Good for "is the new prompt or model better than the current one?"

LLM-as-a-Judge: Making Scores Trustworthy

Using a model as the judge scales far beyond human review, but judges have biases. Agent Platform provides configuration options:

  • Option Purpose
  • System instructions for the judge Set the evaluator role and rules, such as "You are an expert claims auditor"

Response flipping In pairwise judging, swap baseline and candidate positions in half of the calls to reduce position bias

  • Multi-sampling Call the judge several times per item and aggregate, which improves consistency
  • Tuned judge model Use a judge tuned on your domain's human ratings
  • Validate the judge against humans

Before trusting a judge, collect human ratings for a sample and compare. Add columns such as {metric}/human_rating (pointwise) or {metric}/human_pairwise_choice (pairwise) and compute agreement:

Balanced accuracy and balanced F1 for two-class results (pass/fail, A/B).

Multi-class balanced accuracy and F1 for 1-5 scales or A/B/tie.

Confusion matrices to see how the judge disagrees. For example, it may call too many ties.

If agreement is weak, refine the rubric or guidelines, add judge system instructions, enable multi-sampling, or tune the judge.

Matching Metrics to Common Gen AI Tasks

  • Task Primary metrics
  • Classification or extraction with fixed labels exact_match or custom field-level accuracy functions
  • Translation BLEU plus a fluency or rubric check
  • Summarization ROUGE for overlap, GROUNDING for faithfulness, rubrics for length and tone
  • RAG question answering GROUNDING, adaptive rubrics, and a custom check for citations
  • Chat assistant GENERAL_QUALITY and INSTRUCTION_FOLLOWING, multi-turn quality, SAFETY
  • Agent with tools tool_use_quality, hallucination, final response quality
  • Putting It Together: Evaluation-Driven Development
  • Define success criteria with stakeholders, such as "cites policy, under 120 words, no financial advice."

Build a dataset from production samples plus curated edge cases.

Use GENERAL_QUALITY with guidelines, plus GROUNDING for RAG and a custom JSON validity function.

Validate the judge on 200 human-rated items.

Run baseline vs. candidate pairwise comparisons with response flipping for every prompt, model, or tuning change.

Gate releases on pass rates in CI/CD (Chapter 17), and sample production traffic for ongoing evaluation (Chapter 19).

BigQuery ML also offers SQL-native evaluation: ML.EVALUATE on remote LLM models returns BLEU and ROUGE-L for text generation (Chapter 2), and AI.EVALUATE scores TimesFM forecasts.

Loading diagram... Choosing a Gen AI Evaluation Method