Study guide
3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality
Key Takeaways Explain This Topic AI
AWS Glue DataBrew is a visual, no-code data preparation tool offering over 250 pre-built transformations for data cleaning, normalization, and profiling.
DataBrew Profile Jobs compute over 40 distinct data quality statistics, including missing values, entropy, column correlations, data distributions, and duplicate counts.
AWS Glue Data Quality evaluates rules defined in Data Quality Definition Language (DQDL), powered by the open-source Amazon Deequ framework on Apache Spark.
DQDL rulesets enforce data integrity checks like Completeness, IsUnique, ColumnValues, RowCount, and CustomSql within Glue ETL pipelines or Catalog tables.
Conditional routing in Glue Data Quality allows splitting datasets into valid records (passed to data lakes) and quarantined records (routed to S3 audit buckets for remediation).
Last updated: August 2026
3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality
Ensuring data cleanliness, schema compliance, and statistical validity is essential for enterprise data lakes and downstream machine learning models. AWS Glue provides two complementary tools for data quality management:
AWS Glue DataBrew : A visual, interactive data preparation tool designed for analysts and engineers to clean, transform, and profile data without writing code.
AWS Glue Data Quality : A programmatic framework powered by open-source Amazon Deequ that enables data engineers to define, monitor, and enforce data quality rulesets using Data Quality Definition Language (DQDL) directly inside Glue ETL pipelines.
AWS Glue DataBrew: Visual Data Cleaning & Profiling
AWS Glue DataBrew accelerates data preparation by allowing users to visually inspect datasets, build reusable transformation recipes, and run automated data profiling jobs.
DataBrew Key Concepts & Components
Datasets : Connections to data stored in Amazon S3, AWS Glue Data Catalog, Amazon Redshift, or Amazon RDS.
Projects & Interactive Sessions : An interactive workspace where users view sample data records, evaluate column statistics, and visually apply transformations.
Recipes : An ordered sequence of data transformation instructions (e.g., replace missing values, unpivot columns, apply one-hot encoding, extract regex patterns). Recipes can be published and versioned.
Recipe Jobs : Automated background batch jobs that apply a published Recipe to entire target datasets in S3 or Redshift.
Profile Jobs : Batch jobs that evaluate an entire dataset to generate comprehensive statistical quality metrics.
Statistical Metrics Generated by DataBrew Profile Jobs
DataBrew Profile Jobs calculate over 40 detailed statistical indicators across datasets:
Metric Category Calculated Statistics Practical Application
Completeness & Missingness Null counts, missing percentage, empty string frequency Identify incomplete fields requiring imputation or filter rules.
Uniqueness & Cardinality Distinct values count, duplicate rows count, uniqueness ratio Detect duplicate primary keys or high-cardinality categorical fields.
Statistical Distributions Min, max, mean, median, standard deviation, skewness, entropy Detect numerical outliers, skewed data distributions, or abnormal values.
Data Types & Validation Data type distribution, top string lengths, schema match % Uncover mixed data types or invalid string formats (e.g., phone numbers).
Correlations Pearson correlation matrix across numerical columns Identify collinear features for feature engineering in ML pipelines.
AWS Glue Data Quality & Data Quality Definition Language (DQDL)
AWS Glue Data Quality allows data engineers to measure and monitor the quality of data stored in S3, Glue Data Catalog tables, or flowing through Glue ETL pipelines. Under the hood, Glue Data Quality leverages Amazon Deequ , an open-source library built on top of Apache Spark that computes data quality metrics at scale.
Authoring DQDL Rulesets
Data quality expectations are declared using Data Quality Definition Language (DQDL) . A DQDL document consists of a list of rules evaluating specific attributes of a dataset: Rules = [
- Completeness "customer_id" >= 0.99,
- IsUnique "transaction_id",
- ColumnLength "zip_code" = 5,
- ColumnValues "account_status" in ["ACTIVE", "SUSPENDED", "CLOSED"],
- RowCount > 10000,
- DistinctValuesCount "country" <= 250,
- CustomSql "SELECT COUNT(*) FROM primary_table WHERE transaction_amount < 0" = 0
- ]
- Core DQDL Rule Types & Syntax Matrix
DQDL Rule Type Syntax Example Rule Purpose
Completeness Completeness "email" > 0.95 Ensures at least 95% of rows contain non-null values in the target column.
IsUnique IsUnique "ssn" Enforces that all non-null values in the specified column are 100% unique.
ColumnLength ColumnLength "state_code" = 2 Validates exact string length for fixed-length attributes.
ColumnValues ColumnValues "age" between 18 and 120 Restricts numerical or categorical values to an explicit allowed range or set.
RowCount RowCount between 1000 and 500000 Verifies that total dataset volume falls within expected batch thresholds.
CustomSql CustomSql "SELECT COUNT(*) FROM primary_table WHERE tax < 0" = 0 Executes arbitrary Spark SQL queries to validate complex business logic rules.
Integrating Data Quality into AWS Glue ETL Pipelines
Glue Data Quality can be executed directly inside PySpark Glue ETL scripts using the EvaluateDataQuality transform. This enables automated decision-making based on rule evaluations.
Actions on Data Quality Failure & Quarantine Pattern
A rule result does not silently enforce a pipeline decision. In Glue Studio, configure the Data Quality transform's outcome action to continue or stop the job at the intended point. In a script, inspect the returned rule or row outcomes and deliberately raise an error, branch, or quarantine rows according to the contract. Publishing options can send evaluation metrics to CloudWatch and detailed results to S3.
Use EvaluateDataQuality().process_rows when row-level routing is required. Its DynamicFrameCollection contains ruleOutcomes and rowLevelOutcomes ; the latter includes DataQualityEvaluationResult plus pass, fail, and skip rule arrays. Filter those explicit fields rather than relying on a nonexistent failJob parameter or ConditionalDataQualityResult transform.
- Glue PySpark DQDL Integration Example import sys
- from awsglue.context import GlueContext
- from awsglue.dynamicframe import DynamicFrame
- from awsglue.job import Job
- from awsglue.transforms import SelectFromCollection
- from awsglue.utils import getResolvedOptions
- from awsgluedq.transforms import EvaluateDataQuality
- from pyspark.context import SparkContext
- from pyspark.sql.functions import col
- args = getResolvedOptions(sys.argv, ["JOB_NAME"])
- glueContext = GlueContext(SparkContext())
- job = Job(glueContext)
- job.init(args["JOB_NAME"], args)
- input_dyf = glueContext.create_dynamic_frame.from_catalog(
- database="raw_db", table_name="incoming_orders"
- )
- rules = """
- Rules = [
- Completeness "order_id" = 1.0,
- IsUnique "order_id",
- ColumnValues "total_amount" > 0
- ]
- """
- outcomes = EvaluateDataQuality().process_rows(
- frame=input_dyf,
- ruleset=rules,
- publishing_options={
- "dataQualityEvaluationContext": "orders_dq_context",
- "enableDataQualityCloudWatchMetrics": True,
- "enableDataQualityResultsPublishing": True,
- "resultsS3Prefix": "s3://company-dq-reports/orders/",
- },
- additional_options={"performanceTuning.caching": "CACHE_NOTHING"},
- )
- row_outcomes = SelectFromCollection.apply(
- dfc=outcomes, key="rowLevelOutcomes"
- ).toDF()
- metadata = [
- "DataQualityRulesPass", "DataQualityRulesFail",
- "DataQualityRulesSkip", "DataQualityEvaluationResult",
- ]
- valid_df = row_outcomes.filter(
- col("DataQualityEvaluationResult") == "Passed"
- ).drop(*metadata)
- quarantine_df = row_outcomes.filter(
- col("DataQualityEvaluationResult") == "Failed"
- )
- glueContext.write_dynamic_frame.from_options(
- frame=DynamicFrame.fromDF(valid_df, glueContext, "valid"),
- connection_type="s3",
- connection_options={"path": "s3://company-analytics-lake/curated/orders/"},
- format="parquet",
- )
- glueContext.write_dynamic_frame.from_options(
- frame=DynamicFrame.fromDF(quarantine_df, glueContext, "quarantine"),
- connection_type="s3",
- connection_options={"path": "s3://company-quarantine/orders/"},
- format="parquet",
- )