2.4 Application, API & Managed File Ingestion

AWS Certified Data Engineer - Associate DEA-C01

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Streaming Data Ingestion

1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State

2 Chapter 2: Batch Data Ingestion & Migration

2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion

3 Chapter 3: ETL Transformation with AWS Glue

3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage

4 Chapter 4: Big Data Processing & Workflow Orchestration

4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD

5 Chapter 5: Object Storage & Data Lake Architecture

5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations

6 Chapter 6: Data Warehousing with Amazon Redshift

6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum

7 Chapter 7: NoSQL, Relational & Specialty Data Stores

7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing

8 Chapter 8: Pipeline Operations, Monitoring & Observability

8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail

9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning

9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew

10 Chapter 10: High Availability, Backup & Cost Optimization

10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads

11 Chapter 11: Data Security, Encryption & Identity Management

11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints

12 Chapter 12: Data Governance, Privacy & Compliance

12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance

2.4 Application, API & Managed File Ingestion

Key Takeaways Explain This Topic AI

Amazon AppFlow provides managed scheduled or event-triggered transfers between supported SaaS applications and AWS destinations, with mapping and filtering but without arbitrary connector logic.

API ingestion must handle pagination, authentication, quotas, retries, idempotency, incremental watermarks, and a durable raw landing zone before transformation.

AWS Transfer Family exposes managed SFTP, FTPS, FTP, or AS2 endpoints backed by S3 or EFS, separating partner protocol access from storage processing.

JDBC and ODBC connections need network reachability, allowlists or security groups, credential rotation, and bounded parallel reads that do not overload the source.

Last updated: August 2026

2.4 Application, API & Managed File Ingestion

Not every source is a stream or an S3 bucket. DEA-C01 scenarios include SaaS applications, partner file protocols, database drivers, and HTTP APIs. The design goal is to land a complete, auditable raw copy without overwhelming the source or hiding partial failures.

Amazon AppFlow

Amazon AppFlow moves data between supported SaaS applications and AWS services through managed connectors. A flow identifies a source, destination, trigger, field mappings, filters, and optional validations or transformations. Depending on the connector, a flow can run on demand, on a schedule, or in response to an application event.

AppFlow is a strong choice when the exact source and destination are supported and the requirement is mostly mapping and transfer. It is not a universal replacement for Glue or Lambda: custom protocols, complex joins, and arbitrary libraries still require code. Protect connector credentials, choose an incremental field when supported, and send rejected records or execution details to an observable location.

Consuming an external API

An API extractor is a state machine even if each request is stateless. It must remember a cursor, page token, or high-water mark and advance that state only after durable output is committed.

Retrieve the credential from Secrets Manager or use a temporary role-based mechanism.

Request one page with a bounded timeout.

Validate the HTTP status and response schema.

Write the unmodified response plus request time, source, and cursor metadata to a raw S3 prefix.

Persist the next cursor with a conditional update.

Continue until the page is empty or the provider signals completion.

Use exponential backoff with jitter for transient 429 and 5xx responses, and honor Retry-After when the provider supplies it. Do not retry authentication or schema errors indefinitely. Reserve concurrency or place requests behind SQS so a Lambda scale-out does not create a retry storm against the provider.

For an API you expose, Amazon API Gateway can authenticate, validate, throttle, and route calls to Lambda or another backend. A synchronous API should quickly accept and durably enqueue long work rather than hold a client connection while a multi-minute ETL job runs. Return a request ID and expose status separately.

SDK calls and data APIs

AWS SDKs implement signed requests, pagination helpers, retries, and service-specific APIs. Use paginators rather than assuming one response is complete. Services such as the Redshift Data API allow SQL submission without a persistent client connection; execution is asynchronous, so store the statement ID, poll or react to completion, and retrieve results in pages.

An IAM permission to call the API does not grant every downstream data permission. For example, a Redshift Data API caller also needs an appropriate database identity and database privileges.

JDBC and ODBC sources

Java Database Connectivity (JDBC) and Open Database Connectivity (ODBC) provide driver-based access to relational and other tabular systems. Connectivity requires more than a connection string:

Route tables, security groups, network ACLs, DNS, and possibly Direct Connect or VPN must allow the path.

A source firewall or database may require an IP allowlist. Prefer stable private network ranges or controlled egress rather than brittle worker IP lists.

Store credentials outside job arguments and rotate them safely.

Partition parallel reads on a selective numeric or date column, but cap concurrency so extraction does not exhaust source connections or I/O.

Use predicate pushdown and incremental watermarks to reduce transfer volume.

Glue connections can centralize JDBC properties and network settings. A job that accesses a VPC source needs available subnet IP addresses and suitable security-group rules for the Elastic Network Interfaces it creates.

AWS Transfer Family

Transfer Family provides managed endpoints for partner protocols such as SFTP, FTPS, FTP, and Applicability Statement 2, backed by Amazon S3 or Amazon EFS. It is useful when a partner cannot change its file-transfer client. Identity can come from a supported managed or custom provider, and each user can be scoped to a logical directory.

Landing a file is only the ingestion boundary. Use S3 events or EventBridge to start validation after upload completion, quarantine files with bad checksums or schemas, and record the partner, logical file date, and immutable object version. For protocols without an atomic rename convention, agree on a completion marker so processing does not read a partial upload.

Source protection and auditability

Throttle at both worker and account levels. CloudWatch should show request rate, 429 count, latency, rows received, cursor age, and rejected records. Store a manifest that connects each source page or partner file to its S3 object and downstream run. The best ingestion design is restartable: after a failure, it can resume from the last committed cursor without omitting or duplicating data.