Study guide
4.6 Data Engineering Code, IaC & CI/CD
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
4.6 Data Engineering Code, IaC & CI/CD
Key Takeaways Explain This Topic AI
Use Python, SQL, Scala, Java, R, Bash, or PowerShell according to runtime and team needs, but keep data contracts, tests, logging, configuration, and deployment practices language-independent.
CloudFormation and AWS CDK define repeatable infrastructure, while AWS SAM specializes the CloudFormation workflow for serverless resources and local packaging or testing.
A pipeline delivery workflow should lint, unit-test, contract-test, synthesize templates, scan changes, deploy to an isolated environment, run integration tests, and promote an immutable artifact.
Safe data-pipeline releases require idempotent jobs, versioned artifacts, schema compatibility, rollback or roll-forward plans, and separation of code from secrets and environment configuration.
Last updated: August 2026
4.6 Data Engineering Code, IaC & CI/CD
Data engineering code includes SQL transformations, Python orchestration, Scala or Java Spark jobs, shell automation, and infrastructure definitions. DEA-C01 does not reward memorizing syntax from every language; it tests whether the solution is repeatable, observable, testable, and safe to change.
Language and framework choice
Use SQL for set-based filtering, aggregation, windows, and joins that the engine can optimize. Use Python for orchestration, Glue scripts, Lambda handlers, and broad library support. Scala or Java can fit Spark or streaming teams that need JVM integration. R serves statistical workflows. Bash and PowerShell are useful for small deployment or operational steps but become fragile when they accumulate complex state.
Move large computation to the distributed engine. A Python loop that retrieves every row and transforms it locally discards Redshift, Athena, or Spark parallelism. Parameterize queries, project only needed columns, and avoid driver-side collect operations.
Software engineering baseline
Store code in version control with small reviewed changes. A useful repository separates source, tests, infrastructure, schemas, and environment configuration. Never commit passwords or access keys. Review data migrations independently from application code: a reversible code deployment can still launch an irreversible backfill, so require an explicit migration plan, bounded blast radius, reconciliation checks, and a recovery decision before promotion.
Test at several levels:
Unit tests exercise pure transforms with small deterministic inputs, including nulls and boundary cases.
Contract tests check producer and consumer schema compatibility.
Data-quality tests validate uniqueness, completeness, referential integrity, and business rules.
Integration tests use an isolated AWS environment to verify IAM, networking, service APIs, and actual formats.
Reconciliation tests compare source and target counts and control totals.
Structured logs should include run ID, dataset, partition or watermark, source version, record counts, and error category. Do not log secrets or raw PII. Emit metrics for freshness, backlog age, rejected rows, and duration so an operator can detect a logically successful but incomplete run.
Infrastructure as Code
AWS CloudFormation declares resources and dependencies in JSON or YAML and manages them as stacks. It detects stack-level changes and can roll back a failed update, but data migrations and destructive replacements still need explicit review.
AWS Cloud Development Kit (AWS CDK) defines constructs in supported programming languages and synthesizes CloudFormation templates. CDK is not a separate deployment control plane; inspect the synthesized template and asset changes.
AWS Serverless Application Model (AWS SAM) extends CloudFormation with concise serverless resource types and tooling for packaging, local testing, and deployment. It is well suited to a Lambda, API Gateway, Step Functions, and DynamoDB pipeline. Use parameters, mappings, and separate stacks or stages rather than copying templates and hand-editing each environment.
Infrastructure as Code should define IAM roles, KMS keys, buckets, event mappings, log retention, alarms, and failure destinations together. A pipeline is incomplete if only its happy-path compute is declared.
CI/CD stages
A robust delivery path looks like this:
A pull request runs formatting, linting, unit tests, and secret scanning.
Schema compatibility and sample-data tests run.
CDK synth or SAM/CloudFormation validation produces a reviewable template.
Security checks identify wildcard IAM, public networking, and unencrypted resources.
The pipeline builds an immutable artifact identified by commit and digest.
A deployment role updates a development stack; integration and quality tests run.
The same artifact is promoted to later environments with approval gates.
Post-deployment canaries and freshness alarms verify the data path.
AWS CodeBuild can execute builds, CodePipeline can orchestrate stages, and CodeDeploy can manage supported deployment strategies. Equivalent third-party tools are valid when they provide the same controls.
Data-aware release safety
Rolling back code does not automatically roll back data. A bad transform may already have overwritten a partition. Prefer write-audit-publish: write to a run-specific staging location, validate it, then atomically update a catalog pointer, view, alias, or manifest. Preserve the prior version long enough to recover.
For schema changes, use expand-and-contract. Add a compatible field, deploy readers that tolerate both forms, migrate producers, backfill, then remove the old field only after dependency evidence permits it.
Jobs should be idempotent by partition and run ID. Record the code version, input watermark, schema version, and output location. When a deployment fails halfway, rerunning the same artifact against the same inputs should converge on one correct result.
Exam traps
Manual console creation is not repeatable deployment. A CDK construct without tests is not automatically safe. A green application deployment does not prove data completeness. And CI/CD credentials should come from short-lived assumed roles, not static keys stored in build variables.