Study guide
5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design
Key Takeaways Explain This Topic AI
Apache Parquet and ORC are columnar, row-group-oriented storage formats optimized for read-heavy analytical workloads, offering high compression ratios, predicate pushdown, and projection pushdown.
Apache Avro is a row-based format utilizing JSON schema definitions and binary encoding, making it ideal for high-throughput streaming ingestion, row-level mutations, and schema evolution in Kafka/Kinesis pipelines.
Predicate pushdown allows query engines (Athena, Spark, Presto) to evaluate filter criteria against block-level statistics (min/max values, dictionary indexes) inside Parquet/ORC file footers, skipping irrelevant row groups without reading raw data.
Projection pushdown drastically reduces I/O by fetching only the specific column byte ranges requested in SELECT column_a, column_b queries, ignoring unreferenced columns in multi-hundred-column tables.
Last updated: August 2026
- 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design
- Data Storage Paradigms: Row-Oriented vs. Columnar
Choosing the optimal file format is one of the most impactful architectural decisions in AWS data engineering. Storage formats dictate file serialization efficiency, network I/O, CPU utilization, schema evolution capabilities, and query cost.
Storage formats fall into two broad paradigms: Row-Oriented Layout (Avro, CSV, JSON):
[Row 1: ColA, ColB, ColC] -> [Row 2: ColA, ColB, ColC] -> [Row 3: ColA, ColB, ColC]
Columnar Layout (Parquet, ORC):
[Row Group 1: ColA values] -> [Row Group 1: ColB values] -> [Row Group 1: ColC values]
Row-Oriented Formats (JSON, CSV, Apache Avro) : Store all column values for a single record sequentially on disk. Highly efficient for write-heavy transactional operations, row-by-row streaming ingestion (e.g., Kafka, Kinesis), and queries that read full object payloads ( SELECT * ).
Columnar Formats (Apache Parquet, Apache ORC) : Store all values for a single column contiguously on disk within blocks called Row Groups or Stripes. Optimized for read-heavy analytical queries ( OLAP ), where queries select specific column subsets across millions of records.
- Deep Dive: Apache Parquet & Apache ORC
- Apache Parquet Architecture
Apache Parquet is an open-source, columnar storage format designed for complex nested data structures. A Parquet file contains:
Header : Contains magic bytes identifying the format.
Row Groups : Logical horizontal slices of data containing 128 MB to 1 GB of records.
Column Chunks : Columnar data stored contiguously within a Row Group.
Pages : Individual 1 MB data pages inside column chunks containing compressed values and encoding dictionaries.
File Footer : Crucial metadata section stored at the end of the file. Contains table schemas, Row Group metadata, column statistics (min/max values, null counts), and byte offsets for column chunks.
Apache ORC Architecture
Apache ORC (Optimized Row Columnar) is widely used in Apache Hive and Presto workloads. ORC divides files into Stripes (typically 64 MB to 256 MB). Each stripe contains:
Index Data : Min, max, sum, and count statistics for every 10,000 rows.
Row Data : Columnar data streams.
Stripe Footer : Contains encoding types and column stream locations.
PostScript : Embedded footer containing file metadata, compression buffer size, and schema definitions.
Mechanics of Projection and Predicate Pushdown
Columnar formats achieve massive query acceleration in Amazon Athena, AWS Glue, EMR Spark, and Redshift Spectrum through two key optimization techniques:
1. Projection Pushdown
When an analytical query runs against a 100-column table but selects only two columns ( SELECT user_id, order_total FROM orders ), projection pushdown allows the query engine to:
Fetch and parse the Parquet file footer metadata.
Calculate the exact S3 byte ranges containing only user_id and order_total column chunks.
Issue targeted S3 Byte-Range GET requests to retrieve only those specific column byte streams, ignoring the other 98 columns entirely. This reduces network I/O and data scanning volume by up to 98%.
2. Predicate Pushdown (Row Group Skipping)
When a query contains filter predicates (e.g., WHERE order_total > 5000 ), predicate pushdown evaluates the filter criteria against the min/max statistics embedded inside file footers and row group metadata before reading raw data bytes:
If a Row Group's metadata indicates min_order_total = 10 and max_order_total = 4200 , the engine instantly skips that entire 128 MB Row Group without reading its data pages.
Entire files or row groups are bypassed at zero scan cost.
Deep Dive: Apache Avro & Schema Evolution
Apache Avro is a row-oriented, binary serialization format driven by JSON-defined schemas. Unlike CSV or JSON, Avro stores data in a compact binary payload while enforcing strict schema definitions ( .avsc ).
Avro Schema Evolution Rules
Avro is the preferred storage format for landing raw event streams (e.g., in Kafka/Kinesis pipelines) because of its robust support for Schema Evolution :
Backward Compatibility : A new schema can read data written by an older schema. (Adding optional fields with default values).
Forward Compatibility : An older schema can read data written by a newer schema. (Removing fields that have default values).
Full Compatibility : Schemas are both backward and forward compatible.
Because Avro binary payloads embed or reference schema definitions in the AWS Glue Schema Registry, applications can evolve payload fields over time without breaking downstream ETL ingest pipelines.
Compression Codecs: Snappy vs. ZSTD vs. GZIP
Codec Compression Ratio Compression Speed Decompression Speed Splittable in Hadoop/Spark Recommended Use Case
Snappy Moderate Fast Extremely Fast Yes (when wrapped in Parquet/ORC) Default standard for Parquet data lakes & Spark ETL
- ZSTD (Zstandard) High Fast Fast Yes Modern replacement for GZIP; high compression & speed
- GZIP High Slow Moderate No (Unsplittable as raw file) Cold archiving where file size is absolute priority
- LZO Low Fast Fast Yes (requires index file) Legacy MapReduce workloads
- Splittability Impact on Spark Parallelism
A file format or compression codec is splittable if a distributed framework (Spark/EMR) can divide a single large file across multiple map task threads. Unsplittable raw GZIP files force a single Spark worker node to process the entire file sequentially, creating CPU bottlenecks. Parquet and ORC internal row groups restore splittability even when using compressed blocks.
Comprehensive Storage Format Selection Matrix
Capability / Attribute Apache Parquet Apache ORC Apache Avro JSON / CSV
Primary Design Target Read-heavy OLAP analytics Read-heavy OLAP (Hive/Presto) High-speed row ingestion Human-readable inspection
Storage Structure Columnar (Row Groups) Columnar (Stripes) Row-based (Binary) Row-based (Text)
Query Engine Support Athena, Redshift, Spark, EMR Hive, Presto, Athena, Spark Kinesis, Kafka, Glue, Spark Universal
- Pushdown Support Projection & Predicate Projection & Predicate None (Row scan required) None
- Schema Evolution Supported (Append columns) Supported Exceptional (Glue Schema Reg) Weak / Fragile
- Code & Schema Example: Avro Schema & PySpark Parquet Conversion {
- "type": "record",
- "name": "TelemetryEvent",
- "namespace": "com.aws.datalake",
- "fields": [
- {"name": "device_id", "type": "string"},
- {"name": "event_timestamp", "type": "long"},
- {"name": "temperature", "type": ["null", "double"], "default": null},
- {"name": "status_code", "type": "string", "default": "OK"}
- ]
- } # PySpark ETL: Read Avro/JSON Ingest Stream -> Convert to Compressed Parquet
- from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("AvroToParquetETL").getOrCreate()
- # Read incoming streaming landing data
- raw_df = spark.read.format("avro").load("s3://landing-zone-bucket/avro-events/2026/08/13/")
- # Transform & write to analytics data lake as ZSTD-compressed Parquet
- raw_df.write \
- .mode("append") \
- .format("parquet") \
- .option("compression", "zstd") \
- .option("parquet.block.size", 134217728) # 128 MB Row Group size \
- .save("s3://analytics-data-lake-bucket/fact_telemetry/")