Study guide

8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs

AWS Certified Data Engineer - Associate DEA-C01

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Streaming Data Ingestion

1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State

2 Chapter 2: Batch Data Ingestion & Migration

2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion

3 Chapter 3: ETL Transformation with AWS Glue

3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage

4 Chapter 4: Big Data Processing & Workflow Orchestration

4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD

5 Chapter 5: Object Storage & Data Lake Architecture

5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations

6 Chapter 6: Data Warehousing with Amazon Redshift

6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum

7 Chapter 7: NoSQL, Relational & Specialty Data Stores

7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing

8 Chapter 8: Pipeline Operations, Monitoring & Observability

8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail

9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning

9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew

10 Chapter 10: High Availability, Backup & Cost Optimization

10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads

11 Chapter 11: Data Security, Encryption & Identity Management

11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints

12 Chapter 12: Data Governance, Privacy & Compliance

12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance

8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs

Key Takeaways Explain This Topic AI

Amazon CloudWatch serves as the centralized observability platform for AWS data pipelines, providing real-time metric collection, alarm thresholds, and structured log analytics.

Key pipeline performance indicators include IteratorAgeMilliseconds for Kinesis streaming latency, bytesRead and jvm.heap.used for AWS Glue memory profiling, and YARNMemoryAvailablePercentage for EMR cluster capacity.

CloudWatch Embedded Metric Format (EMF) extracts custom metrics from structured JSON logs without synchronous PutMetricData calls in application code; Logs and metric quotas and high-cardinality custom-metric costs still apply.

CloudWatch Logs Insights provides a serverless SQL-like query interface to instantly aggregate, filter, and isolate exception tracebacks across distributed Glue, Lambda, and EMR execution logs.

Last updated: August 2026

8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs

In modern enterprise data architectures, operational health is measured not only by pipeline completion but also by data freshness, system throughput, resource utilization, and error frequency. Amazon CloudWatch acts as the central nerve system for monitoring AWS data operations. It collects raw operational data from AWS data processing services—such as AWS Glue , Amazon EMR , Amazon Kinesis , AWS Lambda , and Amazon Redshift —converting raw logs and system signals into actionable metrics, visual dashboards, and queryable analytical streams.

Building an effective monitoring strategy requires understanding CloudWatch's foundational components: metrics, dimensions, namespaces, resolution tiers, metric filters, Embedded Metric Format (EMF), and CloudWatch Logs Insights.

  • Core CloudWatch Concepts for Data Pipelines
  • Metrics, Namespaces, and Dimensions

Metrics : Time-series data points representing a single monitored variable (e.g., CPU utilization, record count, memory usage). Metrics are fundamentally defined by a name, a timestamp, a unit of measure, and one or more key-value attributes called dimensions.

Namespaces : Isolated containers for CloudWatch metrics. AWS services use standardized default namespaces (e.g., AWS/Glue , AWS/Kinesis , AWS/Lambda , AWS/EMR , AWS/Redshift ). Custom pipeline metrics must be published under custom namespaces (e.g., CustomDataPipeline/Production ).

Dimensions : Key-value pairs that uniquely identify a metric instance. For example, the AWS/Lambda namespace includes dimensions such as FunctionName and Resource . Aggregating metrics across specific dimension combinations allows engineers to analyze overall system performance vs. individual component health.

Metric Resolution Tiers

Retention tiers : Data points with periods under 60 seconds are retained for 3 hours. One-minute points are retained for 15 days, then rolled up to 5-minute points through 63 days and 1-hour points through 455 days.

High-Resolution Metrics : Custom metrics can be published with sub-minute granularity, including 1-second points. High-resolution alarms can evaluate at 10- or 30-second periods, while the sub-minute data itself follows the 3-hour retention tier.

Key Pipeline Metrics by Service

Data engineers must know which specific CloudWatch metrics signal pipeline health or impending failure across AWS analytics services:

AWS Service Metric Name Namespace Operational Significance & Alert Threshold

Kinesis Data Streams GetRecords.IteratorAgeMilliseconds AWS/Kinesis Measures the age (in ms) of the last record read by a consumer. A rising iterator age indicates consumer lag or processing bottlenecks.

Kinesis Data Streams ReadProvisionedThroughputExceeded AWS/Kinesis Triggers when consumer requests exceed shard limits (2 MB/s or 5 transactions/s). Signals a need to reshard or add Enhanced Fan-Out.

Kinesis Data Streams WriteProvisionedThroughputExceeded AWS/Kinesis Triggers when ingestion rate exceeds shard limits (1,000 records/s or 1 MB/s). Indicates producer throttling.

Kinesis Data Firehose DeliveryToS3.Success AWS/Firehose Percentage of successful data delivery attempts to S3. Drops below 100% indicate IAM permission issues, S3 bucket limits, or transformation Lambda timeouts.

AWS Lambda IteratorAge AWS/Lambda Equivalent to Kinesis iterator age for Lambda stream event source mappings. High values indicate Lambda invocation backlog or function timeouts.

AWS Lambda Errors / Throttles AWS/Lambda Unhandled code exceptions ( Errors ) or concurrency execution limit exhaustion ( Throttles ).

AWS Glue (ETL) glue.driver.aggregate.bytesRead Glue (Custom Metrics) Total bytes read across S3/JDBC sources by the Spark driver. Useful for profiling input volume.

AWS Glue (ETL) glue.driver.jvm.heap.used Glue (Custom Metrics) JVM heap memory consumption in Spark driver/executors. Spikes near 100% precede OutOfMemory (OOM) errors.

AWS Glue (ETL) glue.ALL.s3.filesystem.read_bytes Glue (Custom Metrics) Measures S3 data read volume. Discrepancies against target datasets highlight partition pruning efficiency.

Amazon EMR YARNMemoryAvailablePercentage AWS/ElasticMapReduce Percentage of YARN memory available to applications. Values under 15% indicate cluster under-provisioning or container leakage.

Amazon EMR ContainerPendingRatio AWS/ElasticMapReduce Ratio of pending YARN containers to allocated containers. Values > 0 signal resource contention and queuing delay.

Amazon Redshift PercentageDiskSpaceUsed AWS/Redshift Disk usage percentage across nodes. Sustained values > 80% degrade query sorting performance and risk table lockouts.

CloudWatch Logs & Logs Insights

While CloudWatch Metrics track quantitative trends, CloudWatch Logs captures execution details, error tracebacks, and operational event records. Log destinations depend on the service and configuration:

  • AWS Lambda: /aws/lambda/<function-name> when the execution role can write logs
  • AWS Glue Jobs: /aws-glue/jobs/output and /aws-glue/jobs/error for continuous logging

Amazon EMR: for EMR 7.11 or later, configure native CloudWatch Logs integration; its default group is /aws/emr/<cluster-id> . S3 log archiving is a separate configured destination, not a CloudWatch log-group name.

CloudWatch Logs Insights Querying

CloudWatch Logs Insights is an interactive, serverless log analytics tool that uses a specialized SQL-like query language. Data engineers can write queries to analyze gigabytes of raw log data in seconds.

Key Query Commands:

fields : Specifies which log attributes to display.

filter : Filters records based on logical expressions or regular expressions.

stats ... by ... : Performs aggregations (e.g., count() , avg() , sum() , percentile() ) over group-by fields.

sort : Orders results in ascending ( asc ) or descending ( desc ) order.

parse : Extracts ephemeral fields from unstructured log strings using regex or format matchers.

  • Example 1: Isolating AWS Glue Spark Exception Tracebacks fields @timestamp, @message
  • | filter @message like /Exception|Error|OOM/
  • | parse @message '*: *' as ErrorType, ErrorDetails
  • | stats count(*) as FailureCount by ErrorType
  • | sort FailureCount desc
  • | limit 20

Explanation : This query scans AWS Glue job error logs, filters for fatal runtime exceptions, extracts the specific Java/Python exception type, and aggregates failure frequencies to identify root causes.

Example 2: Analyzing Lambda Stream Processing Performance fields @timestamp, @duration, @billedDuration, @maxMemoryUsed

| filter @type = 'REPORT'

| stats avg(@duration) as AvgDuration, max(@duration) as MaxDuration, pct(@duration, 99) as p99Duration by bin(5m)

| sort @timestamp desc

Explanation : This query analyzes Lambda execution report logs, calculating average, maximum, and 99th percentile execution durations in 5-minute time bins to detect micro-batch degradation.

Metric Filters vs. Embedded Metric Format (EMF)

Publishing custom operational metrics from application code (such as records processed per second, bad row counts, or data drift scores) can be implemented through two distinct paradigms: [ Data Pipeline Execution (Glue / Lambda / EMR) ]

  • │
  • ├──> Standard Logs ──> CloudWatch Logs Group ──> CloudWatch Metric Filter ──> Metric
  • │
  • └──> EMF Structured JSON ──> CloudWatch Logs Group ──> Automatic Extraction ──> Metric
  • 1. CloudWatch Metric Filters

Metric filters monitor log events in real time as they arrive in CloudWatch Logs. They search for defined term patterns or JSON fields and increment a CloudWatch metric value whenever a match occurs.

Pros : Requires zero code modifications to existing applications that write structured console output.

Cons : Metrics are extracted synchronously upon log ingestion. Complex pattern parsing can incur latency, and retroactive log historical parsing does not backfill past metrics.

2. Embedded Metric Format (EMF)

CloudWatch Embedded Metric Format (EMF) is a JSON specification that allows data applications to emit custom metrics asynchronously within standard log streams. When CloudWatch Logs ingests a log event matching the EMF JSON schema, it automatically extracts and publishes the metrics to CloudWatch Metrics in real time.

  • Sample EMF Log Payload: {
  • "_aws": {
  • "Timestamp": 1786603100000,
  • "CloudWatchMetrics": [
  • {
  • "Namespace": "DataPipeline/Ingestion",
  • "Dimensions": [["PipelineName", "TargetTable"]],
  • "Metrics": [
  • { "Name": "RecordsIngested", "Unit": "Count" },
  • { "Name": "ValidationErrors", "Unit": "Count" },
  • { "Name": "ProcessingLatencyMs", "Unit": "Milliseconds" }
  • ]
  • }
  • ]
  • },
  • "PipelineName": "ClickstreamETL",
  • "TargetTable": "fact_user_events",
  • "RecordsIngested": 45000,
  • "ValidationErrors": 12,
  • "ProcessingLatencyMs": 1420
  • }

Architectural Advantages of EMF for Data Pipelines:

Decoupled emission : Writing EMF through the application's existing log path avoids a synchronous PutMetricData call in the processing loop. Log ingestion still consumes CloudWatch Logs throughput and storage.

Different quota path : EMF avoids direct PutMetricData request pressure (whose default quota is adjustable), but CloudWatch Logs and metric quotas still apply. High-cardinality dimension combinations create distinct custom metrics and can materially increase cost.

Contextual Correlation : Metrics are permanently bound to the surrounding log event context. When investigating a metric spike, engineers can jump directly from the metric point to the exact log record containing full payload context.

Loading diagram... CloudWatch Pipeline Observability Flow