Study guide
9.3 Implementing Data Quality Frameworks & Automated Validation Checks
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
9.3 Implementing Data Quality Frameworks & Automated Validation Checks
Key Takeaways Explain This Topic AI
AWS Glue Data Quality evaluates datasets using Data Quality Definition Language (DQDL) rulesets (Completeness, Uniqueness, ColumnValues) embedded directly into Glue ETL jobs and Glue Data Catalog metrics.
Open-source data quality libraries like Apache Deequ (Spark-native) and Great Expectations enable statistical profiling, automated unit testing, and historical data metric repositories (in DynamoDB or S3).
The Data Quarantine Pattern segregates corrupted or non-compliant records into designated S3 locations (DLQ/quarantine path) with error metadata while allowing clean records to flow downstream into consumption layers.
Automated data quality circuit breakers built into AWS Step Functions halt downstream ETL tasks, trigger EventBridge events, and dispatch Amazon SNS notifications upon DQDL rule threshold violations.
CloudWatch metrics generated by AWS Glue Data Quality provide real-time dashboard visibility into dataset completeness, column uniqueness drift, and SLA adherence across enterprise data lakes.
Last updated: August 2026
Implementing Data Quality Frameworks & Automated Validation Checks
Maintaining high data fidelity across lakehouse environments requires embedding continuous data quality checks directly into automated data pipelines. Silent data corruption, schema mutations, and null-value spikes degrade downstream BI dashboards and machine learning models. This section details native and open-source data quality frameworks, quarantine architecture patterns, and automated pipeline alerting mechanisms on AWS.
AWS Glue Data Quality & Data Quality Definition Language (DQDL)
AWS Glue Data Quality provides managed, serverless automated data validation embedded into AWS Glue ETL jobs and the AWS Glue Data Catalog. It evaluates datasets against rules defined in Data Quality Definition Language (DQDL) .
1. DQDL Syntax & Core Rule Categories
DQDL rulesets are declarative strings defining validation assertions over columns, row counts, and statistical metrics: Rules = [
- Completeness "customer_id" = 1.0,
- Uniqueness "customer_id" = 1.0,
- Completeness "email" >= 0.98,
- ColumnValues "account_status" in ["ACTIVE", "SUSPENDED", "CLOSED"],
- ColumnLength "zip_code" = 5,
- RowCount > 10000,
- Mean "transaction_amount" between 10.0 and 500.0
- ]
Completeness: Measures the proportion of non-null values in a column.
Uniqueness: Evaluates duplicate ratios across one or more columns.
ColumnValues: Validates column values against permitted domain enumerations or regex expressions ( ColumnValues "phone" matches "^\\d{3}-\\d{3}-\\d{4}$" ).
Mean / StandardDeviation / Minimum / Maximum: Assert aggregate properties of a numeric column to detect drift or implausible ranges.
2. AWS Glue Job Integration ( EvaluateDataQuality )
In PySpark Glue jobs, the EvaluateDataQuality transform executes rulesets directly against DynamicFrames: from awsgluedq.transforms import EvaluateDataQuality
- # Define DQDL Ruleset
- dq_ruleset = """
- Rules = [
- Completeness "customer_id" = 1.0,
- Uniqueness "customer_id" = 1.0
- ]
- """
- # Execute Data Quality Evaluation
- dq_results = EvaluateDataQuality().process_rows(
- frame=input_dynamic_frame,
- ruleset=dq_ruleset,
- publishing_options={
- "dataQualityEvaluationContext": "CustomerIngestionJob",
- "enableDataQualityCloudWatchMetrics": True,
- "enableDataQualityResultsPublishing": True,
- "resultsS3Prefix": "s3://my-data-lake-dq-results/"
- },
- additional_options={"performanceTuning.caching": "CACHE_NOTHING"}
- )
Outcome Actions: Publishing options can send metrics to CloudWatch and detailed results to S3. To enforce a gate, configure the Glue Studio Data Quality outcome action or inspect ruleOutcomes in the script and explicitly stop or branch the workflow; fail_job_if_rules_fail is not an EvaluateDataQuality parameter.
Open-Source Frameworks: Apache Deequ & Great Expectations
For complex data quality requirements on Amazon EMR or AWS MWAA (Airflow), open-source quality frameworks offer advanced statistical checks and custom metric repositories.
1. Apache Deequ (Spark-Native Validation)
Apache Deequ is an open-source library built on Apache Spark by AWS to measure data quality at scale.
Key Features:
Unit Tests for Data: Define checks programmatically in Scala or Python (PyDeequ): from pydeequ.checks import Check, CheckLevel
from pydeequ.verification import VerificationSuite
- check = Check(spark, CheckLevel.Error, "Integrity Check")
- checkResult = VerificationSuite(spark) \
- .onData(df) \
- .addCheck(
- check.hasCompleteness("customer_id", lambda c: c == 1.0) \
- .isUnique("customer_id") \
- .hasPattern("email", r"^[\w-\.]+@([\w-]+\.)+[\w-]{2,4}$")
- ) \
- .run()
Metrics Repository: Deequ persists historical metrics (e.g., daily row counts, mean transaction values) to Amazon DynamoDB or Amazon S3 , enabling automated anomaly detection based on statistical data drift.
2. Great Expectations
Great Expectations is a Python-based data quality library widely deployed on EMR or managed by AWS MWAA.
Key Concepts:
Expectation Suites: JSON declarative rules defining expected data states ( expect_column_values_to_be_unique ).
Data Docs: Automatically renders human-readable HTML data quality documentation and validation reports, hosted directly on static Amazon S3 website buckets.
Data Quarantine & Conditional Routing Pattern
Failing an entire batch ETL job due to a small percentage of malformed records can violate business SLAs. The Data Quarantine Design Pattern routes valid records downstream while segregating bad records for auditing. # Split DataFrame into Valid and Quarantined datasets
from pyspark.sql.functions import coalesce, col, lit
- valid_condition = (
- col("customer_id").isNotNull()
- & coalesce(col("account_status").isin(["ACTIVE", "SUSPENDED", "CLOSED"]), lit(False))
- )
- valid_df = input_df.filter(valid_condition)
- quarantine_df = input_df.filter(~valid_condition)
- # Write Valid Records to Lakehouse Gold Layer
- valid_df.write.format("parquet").mode("append").save("s3://lakehouse-gold/customers/")
- # Enriched Quarantined Records with Failure Metadata
- from pyspark.sql.functions import current_timestamp
- quarantine_enriched = quarantine_df \
- .withColumn("quarantine_timestamp", current_timestamp()) \
- .withColumn("failure_reason", lit("NULL_CUSTOMER_ID_OR_INVALID_STATUS"))
- # Write Quarantined Records to S3 Quarantine Storage
- quarantine_enriched.write.format("parquet").mode("append").save("s3://lakehouse-quarantine/customers/")
Dead Letter Queues (DLQ): For streaming workloads (Kinesis / MSK), bad records are routed to an Amazon SQS DLQ or an S3 quarantine prefix, triggering an Amazon SNS alert for data stewardship review.
Pipeline Governance, Orchestration & Alerting
To enforce enterprise data governance, data quality checks must be integrated into orchestration workflows using AWS Step Functions and Amazon EventBridge .
1. Step Functions Circuit Breaker Pattern
Execute Validation Task: Step Functions invokes a Glue Data Quality task.
Evaluate Outcome (Choice State):
If QualityScore >= 95% : Transition to the Load to Redshift state.
If QualityScore < 95% : Transition to the Circuit Breaker state.
Circuit Breaker State: Halts pipeline execution, executes a rollback transaction, and dispatches a critical SNS alert to Slack or PagerDuty.
2. CloudWatch & EventBridge Governance
When CloudWatch publishing is enabled, AWS Glue Data Quality publishes evaluation metrics under the configured data-quality evaluation context. Build alarms from the actual emitted dimensions and metrics for that run, and route alarm state changes through EventBridge or SNS for incident response.
Data Quality Framework Comparison Matrix
- Attribute AWS Glue Data Quality Apache Deequ Great Expectations
- Engine Dependency AWS Glue (PySpark / Spark) Apache Spark (EMR, Glue) Python (Pandas, Spark, SQL)
- Rule Language DQDL (Declarative text) Scala / Python API JSON Expectation Suites
Historical Metrics CloudWatch & Glue Data Catalog DynamoDB / S3 Metrics Repository Data Docs & Validation Store
Setup Overhead Serverless (Zero setup) Requires library JAR inclusion Requires expectation suite setup
Best Use Case Native AWS Glue ETL pipelines Large-scale Spark anomaly detection Cross-platform Python/Airflow data validation
Reconciliation and data contracts
Quality is not only a row-level rule. Reconcile source and target counts, control totals, and business aggregates for every load. Persist the ruleset version, run identifier, source watermark, result, and quarantine location so an operator can reproduce a failure. A producer data contract should define field meaning, type, nullability, allowed evolution, and ownership; a schema-valid record can still violate a business invariant. For streaming pipelines, emit quality metrics by event time and isolate malformed records without silently advancing a checkpoint that makes recovery impossible.