7.1 Watermarks, Triggers, Accumulating vs Discarding Panes, and Late Data Handling

Google Cloud Certified Professional Data Engineer Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Security, IAM, and Data Protection

1.1 Cloud IAM, Service Accounts, and Resource Hierarchy in GCP Data Systems 1.2 Data Encryption, Key Management (CMEK), and Cloud KMS 1.3 Sensitive Data Protection: Cloud DLP, De-identification, and Masking

2 Chapter 2: Reliability and Compliance Architecture

2.1 High Availability, Regional vs Multi-Regional Deployments, and Disaster Recovery 2.2 Data Residency, Sovereignty, and Regulatory Compliance (HIPAA, GDPR) 2.3 Modern Data Architecture Patterns: Data Mesh, Lakehouse, and Modern Data Stack

3 Chapter 3: Cross-Cloud and Federated Governance

3.1 BigLake Architecture: Unified Multicloud Storage and Fine-Grained Access Control 3.2 BigQuery Omni: Multicloud Analytics Across AWS S3, Azure ADLS, and Google Cloud 3.3 Analytics Hub and Federated Computational Governance

4 Chapter 4: Data Ingestion and Pipeline Planning

4.1 Data Pipeline Planning: Sources, Sinks, and Network Architecture (VPC, PSC, Interconnect) 4.2 Batch vs Streaming Processing Trade-Offs and Architecture Selection 4.3 Change Data Capture (CDC) with Datastream for Continuous Database Ingestion

5 Chapter 5: Streaming Ingestion with Cloud Pub/Sub

5.1 Cloud Pub/Sub Core Architecture: Topics, Subscriptions, and Message Lifecycle 5.2 Pub/Sub Message Ordering, Exactly-Once Delivery, and Deduplication 5.3 Pub/Sub BigQuery/Cloud Storage Direct Subscriptions and Dead-Letter Topics

6 Chapter 6: Pipeline Processing with Apache Beam and Cloud Dataflow

6.1 Apache Beam Programming Model: Pipelines, PCollections, and Core PTransforms 6.2 Cloud Dataflow Architecture: Autoscaling, Worker Allocation, and Dataflow Prime 6.3 Windowing Strategies in Streaming Dataflow: Fixed, Sliding, and Session Windows

7 Chapter 7: Advanced Dataflow: Robustness and Exactly-Once Semantics

7.1 Watermarks, Triggers, Accumulating vs Discarding Panes, and Late Data Handling 7.2 Side Inputs, Broadcast Joins, and Stateful Stream Processing 7.3 Pipeline Robustness: Dead-Letter Queues, Error Handling, and Pipeline Lifecycle Management

8 Chapter 8: Managed Data Processing: Cloud Dataproc and Cloud Data Fusion

8.1 Cloud Dataproc: Architecture, Ephemeral vs Persistent Clusters, and Spark Tuning 8.2 Dataproc Serverless and Autoscaling Policies for Batch Processing 8.3 Cloud Data Fusion: Visual ETL/ELT Pipelines, CDAP Plugins, and Data Lineage

9 Chapter 9: Storage Selection and Cloud Storage Architecture

9.1 GCP Storage Selection Decision Matrix: Relational, NoSQL, Object, and Analytical Stores 9.2 Cloud Storage Architecture: Storage Classes, Lifecycle Rules, and Object Versioning 9.3 Cloud Storage Security: Bucket Lock (WORM), Access Control, and Data Migration Tools

10 Chapter 10: Relational Cloud Databases: Cloud Spanner and Cloud SQL

10.1 Cloud Spanner Architecture: TrueTime, External Consistency, and Global Scalability 10.2 Cloud Spanner Schema Design: Primary Keys, Interleaved Tables, and Query Tuning 10.3 Cloud SQL: High Availability, Read Replicas, Maintenance, and Point-in-Time Recovery

11 Chapter 11: Scalable NoSQL Databases: Cloud Bigtable and Firestore

11.1 Cloud Bigtable Architecture: Storage Model, Tablet Splitting, and Cluster Management 11.2 Cloud Bigtable Row Key Design: Lexicographical Ordering, Salt Keys, and Anti-Patterns 11.3 Cloud Firestore: Document Model, Real-Time Sync, and ACID Collections

12 Chapter 12: Enterprise Data Warehousing with BigQuery

12.1 BigQuery Architecture: Capacitor Columnar Format, Jupiter Network, and Borg Slots 12.2 BigQuery Partitioning: Ingestion-Time, Date/Timestamp, and Integer Range Strategies 12.3 BigQuery Clustering: Cardinality, Multi-Column Ordering, and Performance Optimization 12.4 Data Warehouse Modeling: Normalization, Star Schemas, and Slowly Changing Dimensions

13 Chapter 13: Advanced Analytical Queries and Transformations in BigQuery

13.1 Advanced SQL in BigQuery: Window Analytic Functions, ARRAY/STRUCT, and PIVOT/UNPIVOT 13.2 Modern SQL Orchestration with Dataform: DAG Modeling, Testing, and Documentation 13.3 BI Engine In-Memory Acceleration and Enterprise Looker/LookML Semantic Modeling

14 Chapter 14: Machine Learning and Feature Engineering with BigQuery ML

14.1 BigQuery ML Architecture: In-Database Training, Model Topologies, and SQL Analytics 14.2 Feature Engineering: ML.TRANSFORM, Preprocessing Pipelines, and Vertex AI Feature Store 14.3 MLOps, Model Governance, Continuous Monitoring, and Deployment Lifecycles

15 Chapter 15: Operations, Orchestration, and Data Governance

15.1 Workflow Orchestration with Cloud Composer: Managed Airflow, DAG Design, and Dependencies 15.2 Unified Data Governance with Dataplex: Lakes, Zones, Data Quality, and Lineage 15.3 Pipeline Observability, FinOps, and Cost Optimization Across GCP Data Services

7.1 Watermarks, Triggers, Accumulating vs Discarding Panes, and Late Data Handling

Key Takeaways Explain This Topic AI

Watermarks are monotonic event-time clocks that track processing completeness; heuristic watermarks used in unbounded streams (such as Cloud Pub/Sub) estimate completeness and can produce false assertions, requiring explicit late data handling.

Dataflow's default allowed lateness is zero; any record arriving after the watermark passes the window end is silently and permanently dropped unless configured with withAllowedLateness().

Allowed lateness instructs Streaming Engine to preserve window state in memory and storage, enabling late firings until the allowed lateness horizon (Window End + Allowed Lateness) expires.

Composite triggers combining AfterWatermark.pastEndOfWindow() with early speculative firings (processing-time or element-count) and late firings balance real-time dashboard responsiveness with final data completeness.

Accumulating mode (accumulatingFiredPanes) retains all window elements across firings for idempotent upserts into sinks like BigQuery or Bigtable, whereas discarding mode (discardingFiredPanes) emits only deltas for additive counter sinks.

Last updated: September 2026

7.1 Watermarks, Triggers, Accumulating vs Discarding Panes, and Late Data Handling

Exam Focus: The Google Cloud Professional Data Engineer exam rigorously tests streaming window lifecycles: how Dataflow tracks time via watermarks , how allowed lateness retains state in Streaming Engine, when to emit intermediate vs. final results using triggers , and how to choose between accumulating and discarding pane accumulation modes to match downstream sink semantics.

In batch processing, datasets are bounded and fully known prior to execution. In contrast, real-time stream processing operates over infinite, unbounded datasets where records frequently arrive out of order, delayed by intermittent network connectivity, client buffering, or cross-regional transit delays. To compute deterministic aggregations over streaming data, Apache Beam and Cloud Dataflow decouple four fundamental questions: What is being computed (transformations), Where in event time (windowing), When in processing time results are materialized (triggers and watermarks), and How subsequent results relate to previous outputs (accumulation modes).

1. The Dual Time Domains: Event Time vs. Processing Time

Every distributed streaming system manages two fundamentally independent temporal domains: +-----------------------------------------------------------------------------+

• | EVENT TIME DOMAIN (When the real-world action occurred on the device) |
• | Example: Sensor records temperature at 12:01:15 UTC |
• +-----------------------------------------------------------------------------+
• │
• │ Variable Network Latency / Offline Queue
• ▼
• +-----------------------------------------------------------------------------+
• | PROCESSING TIME DOMAIN (When the Dataflow worker executes the computation) |
• | Example: Dataflow worker processes the record at 12:05:42 UTC |
• +-----------------------------------------------------------------------------+

Event Time ( T e T_e T e ​ ): The timestamp embedded within the record payload when the physical event occurred at the source (e.g., mobile click, financial trade, IoT reading). Event time never changes regardless of pipeline lag.

Processing Time ( T p T_p T p ​ ): The wall-clock time of the Dataflow worker VM currently processing the record. Processing time advances continuously and is influenced by machine load, network latency, and worker scaling.

Time Skew: The difference between processing time and event time ( T p − T e T_p - T_e T p ​ − T e ​ ). In real-world environments, time skew fluctuates wildly due to mobile devices transitioning through airplane mode, network partitions, or producer backlogs.

2. Event-Time Watermarks: Mechanics and Mathematics

A watermark is a monotonically advancing threshold that models the system's perception of completeness in the event-time domain. When a watermark reaches timestamp W = t W = t W = t , the pipeline asserts: "We believe no further records with event timestamp T e ≤ t T_e \le t T e ​ ≤ t will arrive." Event Time Timeline (Hours:Minutes)

• 12:00 12:05 12:10 12:15 12:20 12:25
• ────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────>
• ▲
• │ Watermark = 12:07:00
• [ Historical Data (Complete) ] │ [ Potential Incoming Data ]

Events with Te <= 12:07 arrived. │ Events with Te > 12:07 expected.

Perfect vs. Heuristic Watermarks

Beam categorizes watermarks into two distinct implementations based on the underlying ingestion source:

Watermark Type Source Characteristics Predictability & Behavior Late Data Possible?

Perfect Watermark Bounded sources (Cloud Storage files, BigQuery exports) or strict sequentially ordered single-channel logs with monotonically increasing timestamps. The runner knows the exact maximum event time of all remaining unprocessed items. The watermark is mathematically guaranteed never to lag behind incoming data. No. Late-arriving data is mathematically impossible because all inputs are known upfront.

Heuristic Watermark Unbounded distributed sources (Cloud Pub/Sub, Apache Kafka across multiple partitions, mobile client SDKs). The runner cannot inspect unread messages distributed across millions of mobile clients. It computes a statistical estimate of event-time completeness based on partition offsets, producer message rates, network propagation history, and subscription backlog age. Yes. Because the watermark is a statistical estimate, events occasionally arrive with T e ≤ W T_e \le W T e ​ ≤ W . Such records are designated as late data .

Watermark Lag and Data Freshness

In Cloud Dataflow, Watermark Lag (also exposed in Cloud Monitoring as Data Freshness ) measures the duration between real-world wall-clock time and the current stage watermark: Watermark Lag = Current Wall-Clock Time − Current Stage Watermark \text{Watermark Lag} = \text{Current Wall-Clock Time} - \text{Current Stage Watermark} Watermark Lag = Current Wall-Clock Time − Current Stage Watermark

If a streaming pipeline has a Data Freshness metric of 15 minutes, the downstream consumers are seeing aggregations that are guaranteed complete only up to 15 minutes ago. High watermark lag is caused by:

Upstream Source Inactivity: If a Pub/Sub topic or Kafka partition stops receiving data, the watermark cannot advance past the last received record without idle-source timeouts.

Stuck Worker / Unprocessed Backlog: If a worker crashes or encounters CPU bottlenecks, unconsumed bundles hold back the stage watermark.

Backfill Producers: If an external system re-ingests 3-day-old logs into a live Pub/Sub topic, the heuristic watermark can stall or advance slowly to accommodate the historical data.

3. Late Data Handling and Allowed Lateness ( withAllowedLateness )

When using event-time fixed, sliding, or session windows, the end of a window ( W end W_{\text{end}} W end ​ ) defines its boundary. For example, a 10-minute fixed window spanning [12:00, 12:10) expects all records with event timestamps between 12:00:00 and 12:09:59.999.

The Default Behavior (Allowed Lateness = 0)

By default in Apache Beam, allowed lateness is zero . When the stage watermark advances past the window end ( W ≥ W end W \ge W_{\text{end}} W ≥ W end ​ ), the window fires its ON_TIME pane. The window's metadata and accumulator state are immediately purged from the worker memory and Streaming Engine. Any record arriving subsequently with T e < 12 : 10 T_e < 12:10 T e ​ < 12 : 10 is silently dropped .

Exam Trap: In streaming pipelines, developers often assume late-arriving events are automatically merged into existing windows. Without explicitly setting withAllowedLateness() , Dataflow drops all records that arrive after the watermark passes the window end. No exception is thrown, but records are lost from final business aggregates.

Extending Window Lifespan with withAllowedLateness()

To capture delayed data, pipelines configure .withAllowedLateness(Duration) on the windowing transform: PCollection<Transaction> windowed = input.apply(

• Window.<Transaction>into(FixedWindows.of(Duration.standardMinutes(10)))
• .triggering(
• AfterWatermark.pastEndOfWindow()
• .withLateFirings(AfterPane.elementCountAtLeast(1))
• )
• .withAllowedLateness(Duration.standardHours(2))
• .accumulatingFiredPanes()
• ); Window: [12:00 - 12:10)
• Watermark advances past 12:10 ──> ON_TIME Pane Fires
• │
• ├── Record arrives at 12:25 with Te = 12:08 (LATE, but within 2h allowed lateness)
• │ └──> LATE Pane Fires (Window state was preserved in Streaming Engine)
• │
• Watermark reaches 14:10 (Window End 12:10 + Allowed Lateness 2h)
• │

└── Window state is PERMANENTLY PURGED.

Any record arriving with Te < 12:10 after this point is DROPPED.

State Retention Overhead in Streaming Engine

Setting an excessively large allowed lateness (e.g., 30 days) forces Cloud Dataflow's Streaming Engine to retain the state, accumulators, and timer structures for millions of historical windows. This increases persistent storage footprint and state lookup overhead. Best practice is to set allowed lateness to match your realistic network delay distribution (e.g., 1 to 6 hours for mobile telemetry), routing any data that arrives after the lateness cutoff to an audit log or dead-letter storage.

4. Triggers: Governing Window Emission Lifecycles

A trigger determines precisely when a window emits its accumulated contents as an output pane. While windowing determines which events belong together in event time, triggers control when those aggregations materialize in processing time.

Core Trigger Categories

Event-Time Triggers ( AfterWatermark ):

Driven by the watermark passing a specific temporal threshold (typically the end of the window: AfterWatermark.pastEndOfWindow() ).

Emits the definitive, official pane when the system believes all data for the window has arrived.

Processing-Time Triggers ( AfterProcessingTime ):

Driven by the worker's wall-clock time ( AfterProcessingTime.pastFirstElementInPane().plusDelayOf(Duration.standardMinutes(1)) ).

Independent of event timestamps. Useful for producing speculative early results for real-time dashboards before the watermark advances.

Data-Driven Triggers ( AfterPane ):

Driven by element volume ( AfterPane.elementCountAtLeast(100) ).

Fires as soon as N N N elements accumulate in the pane buffer, ensuring high-throughput streams emit intermediate batches without waiting for time delays.

Composite Triggers:

Combine multiple triggers using logical operators: Repeatedly.forever(...) , AfterEach.inOrder(...) , AfterFirst(...) , AfterAll(...) .

• The Production Standard: Early, On-Time, and Late Trigger Pipeline
• The canonical enterprise pattern combines all three phases into a single composite trigger: .triggering(
• AfterWatermark.pastEndOfWindow()
• // 1. EARLY FIRINGS: Speculative updates for real-time dashboards
• .withEarlyFirings(
• AfterProcessingTime.pastFirstElementInPane()
• .plusDelayOf(Duration.standardSeconds(30))
• )
• // 2. LATE FIRINGS: Corrective updates for out-of-order data
• .withLateFirings(
• AfterPane.elementCountAtLeast(1)
• )
• )

In this configuration:

Early Panes: As soon as data enters the window, Dataflow emits speculative aggregates every 30 seconds of processing time.

On-Time Pane: Exactly when the watermark passes the window end, Dataflow emits the official aggregate representing complete data.

Late Panes: If any late records arrive before the allowed lateness expires, Dataflow emits an immediate corrective pane for every newly received element.

5. Accumulation Modes: Accumulating vs. Discarding Panes

When a window triggers multiple times (due to early speculative firings or late-arriving records), the developer must specify how each new firing relates to previous firings for that same window. This is configured via the accumulation mode . Window [10:00 - 10:05) receives: Event A (value=10), Event B (value=20)

Early Firing 1 occurs:

- Both Modes emit: 30

Next, Event C (value=15) arrives for the same window.

Early Firing 2 occurs:

• ACCUMULATING MODE emits: 45 (10 + 20 + 15 -> Running Total)
• DISCARDING MODE emits: 15 (Delta since last firing)
• Architectural Trade-Off Matrix

Feature accumulatingFiredPanes() discardingFiredPanes()

Emitted Content The complete cumulative result of all elements received in the window from the beginning of time up to the current pane. Only the incremental delta representing elements that arrived since the immediately preceding pane.

State Retention Dataflow must preserve all accumulated values or buffer intermediate combiner state across firings. Dataflow flushes the pane buffer upon emission; memory footprint between firings is minimized.

Downstream Sink Requirement Idempotent Upsert / Overwrite: Sinks that overwrite previous state by primary key (e.g., BigQuery tables with MERGE , Cloud Bigtable row mutations, Cloud Spanner INSERT OR UPDATE ). Additive / Delta Accumulator: Sinks that sum deltas (e.g., Cloud Bigtable atomic increment counters via ReadModifyWriteRow , Pub/Sub delta streams).

Catastrophic Failure Scenario Connecting an accumulating stream to an additive sink (e.g., executing an atomic database increment of +45 after already executing +30 , causing double-counting of values 10 and 20). Connecting a discarding stream to an overwriting sink without a downstream aggregator (the sink ends up storing only the final delta 15 instead of the true total 45 ).

6. End-to-End Code Implementation

Here is a complete, production-grade Apache Beam streaming pipeline configuration combining event-time windowing, composite triggers, allowed lateness, and accumulation modes: PCollection<KV<String, Long>> aggregatedMetrics = rawStream

• .apply("ExtractKeyAndTimestamp", ParDo.of(new ExtractMetricDoFn()))
• // Apply a 5-minute fixed window with advanced streaming semantics
• .apply("WindowWithLateDataHandling", Window.<KV<String, Long>>into(
• FixedWindows.of(Duration.standardMinutes(5)))
• .triggering(
• AfterWatermark.pastEndOfWindow()
• // Speculative early pane every 1 minute of wall-clock time
• .withEarlyFirings(
• AfterProcessingTime.pastFirstElementInPane()
• .plusDelayOf(Duration.standardMinutes(1))
• )
• // Corrective late pane for every late arrival
• .withLateFirings(AfterPane.elementCountAtLeast(1))
• )
• // Keep window state alive in Streaming Engine for 1 hour after window closes
• .withAllowedLateness(Duration.standardHours(1))
• // Emit cumulative aggregates so downstream BigQuery MERGE can overwrite state
• .accumulatingFiredPanes()
• )
• .apply("SumPerKey", Combine.perKey(Sum.ofLongs()));
• 7. Comparative Production Scenarios & Exam Traps

Production Scenario Common Anti-Pattern Correct Google Cloud Architecture

Mobile Ad-Click Attribution ; Mobile apps buffer user ad-clicks when offline (subway, airplane) and flush them up to 3 hours later. The billing pipeline counts clicks in 10-minute windows. Leaving default windowing ( allowedLateness = 0 ). When mobile devices sync, their clicks are silently dropped because the watermark has already advanced past the window end. Configure .withAllowedLateness(Duration.standardHours(4)) and .triggering(AfterWatermark.pastEndOfWindow().withLateFirings(...)) . Downstream sinks update advertiser attribution tables using accumulating panes.

Real-Time Fraud Dashboard ; A financial institution requires sub-second fraud alerts, but event-time watermarks lag by 2 minutes due to cross-region batch synchronization. Relying exclusively on default AfterWatermark.pastEndOfWindow() triggers, delaying fraud detection alerts by 2+ minutes until the watermark catches up. Implement composite triggers with early firings: .withEarlyFirings(AfterProcessingTime.pastFirstElementInPane().plusDelayOf(Duration.standardSeconds(5))) . Speculative alerts fire immediately via processing time without waiting for watermark confirmation.

Bigtable Counter Aggregation ; A streaming pipeline increments real-time counter columns in Cloud Bigtable using ReadModifyWriteRow atomic increment mutations. Using .accumulatingFiredPanes() . Each early and late firing re-emits the cumulative total, causing Bigtable to repeatedly add historical counts, resulting in massive overcounting. Configure .discardingFiredPanes() . Each firing emits only the newly arrived delta, which the Bigtable atomic increment safely adds to the running total.

Loading diagram... Watermark Timeline, Trigger Firings, and Allowed Lateness Horizon