Study guide

6.4 AI Data Enrichment inside Data Pipelines

Google Cloud Professional Data Engineer Exam

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

0.1 About the Google Cloud Professional Data Engineer Exam 0.2 Study Strategy & Cloud Data Engineering Methodology

2 Chapter 1: Designing for Security and Compliance

1.1 IAM Roles, Service Accounts, and Least Privilege for Data 1.2 Data Encryption, Key Management, and Privacy with Cloud DLP 1.3 Compliance, Data Sovereignty, and Geographic Placement 1.4 Multi-Environment Architecture: Development, Staging, and Production

3 Chapter 2: Designing for Reliability and Resilience

2.1 High Availability, Disaster Recovery, and Failover Patterns 2.2 Fault Tolerance and ACID Guarantees in Distributed Systems 2.3 Data Validation, Cleansing, and Quality Assurance with Dataform 2.4 AI-Assisted Data Preparation and LLM Query Generation

4 Chapter 3: Architecture Patterns and Flexibility

3.1 Modern Data Architectures: Data Lake, Warehouse, Lakehouse, and Mesh 3.2 Data Migration Strategies and Hybrid/Multi-Cloud Integration 3.3 Database Migration Service and Heterogeneous Database Migrations

5 Chapter 4: Ingestion Architectures: Pub/Sub and Streaming Sources

4.1 Cloud Pub/Sub Core Architecture and Ingestion 4.2 Pub/Sub Advanced Patterns: Dead-Lettering, Ordering, and Schemas 4.3 Change Data Capture (CDC) with Datastream and Batch Ingestion

6 Chapter 5: Stream and Batch Processing with Apache Beam and Cloud Dataflow

5.1 Apache Beam Core Concepts and Pipeline Design 5.2 Advanced Windowing, Watermarks, and Triggers 5.3 Dataflow Pipeline Optimization, Scaling, and Templates

7 Chapter 6: Data Transformation with Dataproc, Data Fusion, and Workflow Orchestration

6.1 Apache Spark and Hadoop with Cloud Dataproc 6.2 Visual Data Pipelines with Cloud Data Fusion 6.3 Workflow Orchestration with Cloud Composer 6.4 AI Data Enrichment inside Data Pipelines

8 Chapter 7: Storage Service Selection and Cloud Storage Architecture

7.1 Google Cloud Storage Service Selection Decision Matrix 7.2 Cloud Storage Architecture, Classes, and Lifecycle Management

9 Chapter 8: Deep Dive into Google BigQuery Storage

8.1 BigQuery Storage Architecture and Data Organization 8.2 Partitioning and Clustering for Performance and Cost 8.3 BigLake and Hybrid Querying across Clouds 8.4 Data Warehouse Modeling: Star Schemas, Denormalization, and Slowly Changing Dimensions

10 Chapter 9: Operational and NoSQL Databases: Bigtable and Cloud Spanner

9.1 Cloud Bigtable Architecture and Row Key Design 9.2 Cloud Bigtable Schema Management and Performance Tuning 9.3 Cloud Spanner: Globally Scalable Relational Database

11 Chapter 10: BigQuery Query Performance and Optimization

10.1 BigQuery SQL Tuning and Performance Best Practices 10.2 BigQuery BI Engine and Materialized Views 10.3 Data Sharing and Analytics Hub 10.4 Preparing Data for Visualization and BI Tool Connectivity

12 Chapter 11: Machine Learning Readiness and Data Governance

11.1 Machine Learning with BigQuery ML 11.2 Feature Engineering and Data Preparation for AI/ML 11.3 Data Governance, Cataloging, and Discovery with Dataplex 11.4 Unstructured Data, Embeddings, and Retrieval-Augmented Generation

13 Chapter 12: Site Reliability Engineering and Monitoring for Data Systems

12.1 SRE Principles for Data: SLIs, SLOs, and SLAs 12.2 Observability: Cloud Monitoring, Cloud Logging, and Alerting 12.3 Troubleshooting Data Pipelines and Workload Failures

14 Chapter 13: Workload Automation, CI/CD, and Cost Optimization

13.1 CI/CD and Version Control for Data Pipelines 13.2 BigQuery Cost Management and Capacity Planning 13.3 Holistic Cost Optimization across GCP Data Services 13.4 Query Workload Management: Interactive vs. Batch Jobs, Queues, and Quotas

6.4 AI Data Enrichment inside Data Pipelines

Key Takeaways Explain This Topic AI

AI data enrichment is a named transformation type in the v4.2 blueprint, and the core decision is whether to enrich at rest in BigQuery or in flight in Dataflow.

BigQuery enrichment runs through remote models created with CREATE MODEL REMOTE WITH CONNECTION over a Cloud resource connection, exposing ML.GENERATE_TEXT, AI.GENERATE, ML.UNDERSTAND_TEXT, ML.TRANSLATE, and ML.GENERATE_EMBEDDING.

Unstructured files in Cloud Storage must be exposed through an object table before ML.PROCESS_DOCUMENT, ML.ANNOTATE_IMAGE, or ML.TRANSCRIBE can read them.

BigQuery remote functions call Cloud Run or Cloud Run functions endpoints for custom enrichment; they do not support ARRAY, STRUCT, INTERVAL, or GEOGRAPHY types and are tuned with max_batching_rows.

The Apache Beam Enrichment transform, available from version 2.54.0, provides batching, client-side throttling, exponential backoff, and optional Redis caching with handlers for Bigtable, BigQuery, Cloud SQL, and Vertex AI Feature Store.

Last updated: September 2026

6.4 AI Data Enrichment inside Data Pipelines

The v4.2 exam guide lists four kinds of transformation under Building the pipelines : Batch , Streaming , Processing logic , and AI data enrichment . The last one is the newest addition and the one most candidates arrive without. It asks a specific architectural question: when a pipeline needs to attach information that the source data does not contain — a sentiment score, a translated string, an entity extracted from a scanned invoice, a semantic embedding, a feature value from a store — where in the architecture does that call happen, and what does it cost you in latency, quota, and failure modes?

There are two legitimate topologies, and picking between them is the whole skill.

Topology 1: Enrich at Rest, in BigQuery

BigQuery can call models and services directly from SQL. This is the low-operations path, and it is the right answer whenever the enrichment can wait for the next batch and the data already lives in BigQuery.

The mechanism is a remote model : CREATE MODEL ... REMOTE WITH CONNECTION , backed by a Cloud resource connection whose auto-created service account is granted access to the target Vertex AI or Cloud AI service. Once the model exists, enrichment is a SELECT .

  • Function What It Produces Requires an Object Table
  • ML.GENERATE_TEXT Model-generated text from Gemini or Gemma models No
  • AI.GENERATE Scalar generated text inside a query No
  • AI.GENERATE_TABLE Structured, tabular generated output No
  • AI.GENERATE_BOOL A boolean classification result No
  • ML.GENERATE_EMBEDDING Vector embeddings of text, image, or video Only for unstructured inputs
  • ML.UNDERSTAND_TEXT Natural-language tasks such as classification and sentiment analysis No
  • ML.TRANSLATE Translated text No
  • ML.ANNOTATE_IMAGE Cloud Vision labels, face detection, object detection Yes
  • ML.PROCESS_DOCUMENT Structured fields extracted from documents via Document AI Yes
  • ML.TRANSCRIBE Text transcribed from audio via Speech-to-Text Yes

The pattern to internalize: structured text in a column goes straight into the generative and NLP functions, while unstructured files in Cloud Storage must first be exposed through an object table — a BigQuery table of references to the underlying objects — before ML.ANNOTATE_IMAGE , ML.PROCESS_DOCUMENT , or ML.TRANSCRIBE can read them.

Custom Logic: Remote Functions

When the enrichment is not a model call at all — a proprietary scoring service, an internal address-normalization API, a licensed data vendor — the tool is a BigQuery remote function . A remote function is a GoogleSQL function backed by an HTTP endpoint running on Cloud Run functions or Cloud Run , wired up through the same CLOUD_RESOURCE connection type.

Two limits matter operationally:

Data types are restricted. Remote functions accept and return Boolean, Bytes, Numeric, String, Date, Datetime, Time, Timestamp, and JSON. ARRAY , STRUCT , INTERVAL , and GEOGRAPHY are not supported directly — complex payloads must be serialized into JSON or STRING .

Batching is a tuning knob. max_batching_rows in the function's OPTIONS controls how many rows BigQuery packs into each HTTP request. Set it too low and you pay per-request overhead on every row; set it too high and you risk the backing service timing out.

Topology 2: Enrich in Flight, in Dataflow

When enrichment must happen before the data lands — a fraud score that has to accompany the event, a lookup that must reflect state at event time, a streaming pipeline with a seconds-level freshness SLO — the call belongs inside the Beam pipeline.

Beam gives you three escalating options:

Side inputs for small, slowly changing reference data. The lookup table is broadcast to every worker, so there is no per-element network call at all. This is the cheapest enrichment there is, and it is correct whenever the reference data fits comfortably in worker memory.

The Enrichment transform for key-value lookups against a remote service. Available from Apache Beam 2.54.0 in the Python SDK, it ships handlers for Cloud Bigtable , BigQuery , Cloud SQL (PostgreSQL, MySQL, SQL Server), and Vertex AI Feature Store . Critically, it provides the production behavior you would otherwise have to hand-roll: client-side throttling so the remote service is not overwhelmed, exponential backoff on service-side errors, batching that groups elements into a single lookup per batch, and optional Redis caching for frequently requested keys.

A custom DoFn calling an external model endpoint when no handler fits. This is where pipelines go wrong, because every production concern becomes your responsibility: batching inside @ProcessElement , connection reuse via @Setup , retry with backoff, a dead-letter side output for elements the service rejects, and idempotency so that a retried bundle does not double-charge a metered API.

Why Batching Dominates the Cost Model

A per-element call to a model endpoint on a 50,000 events-per-second stream is 50,000 API calls per second, which will hit a quota long before it hits a budget. Both the Enrichment transform's built-in batching and max_batching_rows on a remote function exist for the same reason: the unit of cost and quota is the request , not the row. Any exam scenario reporting rate-limit errors, RESOURCE_EXHAUSTED , or stuck watermarks on an enrichment step is asking you to batch, throttle, or cache — not to add workers.

Choosing Between the Two Topologies

  • Requirement in the Scenario Enrich at Rest (BigQuery) Enrich in Flight (Dataflow)
  • Data already lands in BigQuery and batch latency is acceptable Yes Overkill
  • Enriched value must accompany the event downstream in real time No Yes
  • Lowest operational overhead, SQL-only team Yes No
  • Lookup must reflect feature values at event time No Yes (Vertex AI Feature Store handler)

Unstructured files in Cloud Storage need document, image, or audio extraction Yes (object table plus ML.PROCESS_DOCUMENT , ML.ANNOTATE_IMAGE , ML.TRANSCRIBE ) Possible but heavier

  • Proprietary internal scoring API Remote function on Cloud Run Custom DoFn
  • Reference data is small and rarely changes Join to a table Side input
  • Governance and Regional Constraints

Three rules that show up as one-line disqualifiers in exam answers:

De-identify before you enrich. If the column contains regulated identifiers, run Cloud DLP de-identification first. Sending raw PII to a model endpoint moves that data across a boundary your data-sovereignty controls may not cover.

Keep the connection, the model, and the dataset regionally consistent. A Cloud resource connection and the dataset it serves must be regionally compatible; a scenario that mixes an EU dataset with a US endpoint is describing a residency violation, not just a latency problem.

Enrichment failures need a destination. Whether it is a BigQuery error column, a Beam tagged side output, or a Pub/Sub dead-letter topic, a pipeline that silently drops elements the model could not process has no completeness SLI worth reporting.

Exam Traps and Antipatterns Summary

Scenario Cue Wrong Answer Correct Architecture

"Extract fields from 2 million scanned invoices in Cloud Storage" Read the files with a Dataflow TextIO source Object table over the bucket plus ML.PROCESS_DOCUMENT with a Document AI remote model

"Sentiment score every support ticket nightly" Build a streaming Dataflow pipeline ML.UNDERSTAND_TEXT in a scheduled BigQuery query

"Fraud score must travel with the event in real time" Enrich in BigQuery after loading Enrich in flight in the Beam pipeline before the sink

"Enrichment step is throwing rate-limit errors" Increase maxNumWorkers Batch the calls, enable client-side throttling, and cache hot keys

"Attach a 5,000-row currency table to every event" Call an external API per element Beam side input broadcast to workers

"Feature values must match what the model saw at event time" Query the warehouse table Vertex AI Feature Store enrichment handler

"Vendor scoring service returns a nested payload" Declare a STRUCT return type on the remote function Return JSON or STRING and parse it in SQL; remote functions do not support STRUCT

Loading diagram... Placing AI Enrichment: At Rest in BigQuery vs. In Flight in Dataflow