ExamFinch.com My Study

Study guide

12.1 Medallion Architecture Design Patterns

Practice this exam

Databricks Certified Data Engineer Professional Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Developing Code for Data Processing using Python and SQL - Core Spark SQL & PySpark

1.1 Complex PySpark Transformations & DataFrame API 1.2 Advanced Spark SQL Patterns 1.3 Streaming Workloads with Structured Streaming 1.4 Lakeflow & Delta Live Tables (DLT) Declarative Pipelines

2 Chapter 2: Developing Code for Data Processing using Python and SQL - Streaming & Advanced Patterns

2.1 Advanced Delta Lake Features 2.2 Custom & Stateful Streaming Operations 2.3 Python & PySpark Code Modularization 2.4 User-Defined Functions (UDFs) & Performance Impact

3 Chapter 3: Data Ingestion & Acquisition

3.1 Auto Loader (cloudFiles) Production Ingestion 3.2 Change Data Capture (CDC) Ingestion Patterns 3.3 Batch & Streaming Connectors

4 Chapter 4: Data Transformation, Cleansing, and Quality

4.1 Data Validation & Quality Frameworks 4.2 Data Cleansing, Deduplication & Standardization 4.3 Silver and Gold Medallion Layer Refinement 4.4 Handling Schema Drift and Evolution

5 Chapter 5: Data Sharing and Federation

5.1 Delta Sharing Architecture & Governance 5.2 Lakehouse Federation

6 Chapter 6: Monitoring and Alerting & Observability

6.1 System Tables & Observability 6.2 Databricks Workflows & Job Monitoring 6.3 Spark UI & Query Execution Diagnostics 6.4 Data Quality & Pipeline Alerting

7 Chapter 7: Cost & Performance Optimisation - Data Layout & Compute

7.1 Data Storage & Layout Optimization 7.2 Compute Choice & Cluster Sizing 7.3 Photon Engine & Query Acceleration

8 Chapter 8: Cost & Performance Optimisation - Skew & Cost Governance

8.1 Partitioning & Data Skew Remediation 8.2 Cost Governance & DBU Optimization

9 Chapter 9: Ensuring Data Security and Compliance

9.1 Unity Catalog Access Control & Least Privilege 9.2 Dynamic Data Masking & Row/Column-Level Security 9.3 Secrets Management & Network Security 9.4 Compliance, Audit & Data Privacy

10 Chapter 10: Data Governance in Unity Catalog

10.1 Unity Catalog Governance Architecture 10.2 Data Lineage & Tagging in Unity Catalog 10.3 Managed vs External Tables & Storage Governance

11 Chapter 11: Debugging and Deploying - Asset Bundles & CI/CD

11.1 Databricks Asset Bundles (DABs) & Infrastructure as Code 11.2 CI/CD Automation & Databricks CLI / REST API 11.3 Troubleshooting Pipeline & Job Failures 11.4 Code Versioning & Databricks Git Integration

12 Chapter 12: Data Modelling & Lakehouse Analytics

12.1 Medallion Architecture Design Patterns 12.2 Dimensional Modeling in the Lakehouse 12.3 Analytics & BI Optimization

12.1 Medallion Architecture Design Patterns

Key Takeaways Explain This Topic AI

Bronze layer acts as a raw data repository (append-only) retaining the original format and history for auditing and replay.

Silver layer acts as a cleansed, deduplicated, and conformed data foundation representing enterprise entities.

Gold layer is designed specifically for business consumption, utilizing dimensional modeling or wide denormalized tables.

Structured Streaming provides exactly-once processing guarantees and is ideal for moving data from Bronze to Silver.

Denormalized wide tables often outperform traditional dimensional models in the Gold layer due to reduced join overhead in distributed computing.

Last updated: July 2026

Medallion Architecture Principles

The Medallion Architecture is a data design pattern used to logically organize data in a Lakehouse, with the goal of incrementally and progressively improving the structure and quality of data as it flows through each layer of the architecture (from Bronze to Silver to Gold).

The Bronze Layer (Raw Data)

The Bronze layer is the landing zone for all raw data. Its primary purpose is to capture data from external source systems in its original format without any modifications. This ensures that no data is lost and provides a historical archive that can be used for auditing or reprocessing if business logic changes.

Key Characteristics of Bronze

Append-Only : Data is continuously appended. Updates and deletes from source systems are typically recorded as new events rather than mutating existing records.

Schema Evolution : The schema is often flexible, capturing raw JSON, CSV, or Parquet files as they arrive.

SLA Expectations : High throughput and low latency for ingestion. Data availability is the priority over query performance. # Example: Ingesting raw JSON data into Bronze

  • (spark.readStream
  • .format("cloudFiles")
  • .option("cloudFiles.format", "json")
  • .load("/mnt/source/telemetry/")
  • .writeStream
  • .format("delta")
  • .option("checkpointLocation", "/mnt/bronze/_checkpoints/telemetry")
  • .start("/mnt/bronze/telemetry"))
  • The Silver Layer (Cleansed and Conformed Data)

The Silver layer represents a cleansed, validated, and enriched version of the data. This layer acts as the single source of truth for the enterprise, providing a normalized view of core business entities (e.g., Customers, Orders, Products).

Key Characteristics of Silver

Data Quality Rules : Missing values are handled, data types are cast, and invalid records are quarantined or dropped.

Deduplication : Records are deduplicated, and upserts (MERGE operations) are used to handle late-arriving updates.

SLA Expectations : Moderate latency. The focus is on data accuracy and consistency. -- Example: Upserting data into Silver using Delta Lake MERGE

  • MERGE INTO silver.customers target
  • USING bronze.customers_updates source
  • ON target.customer_id = source.customer_id
  • WHEN MATCHED AND source.update_timestamp > target.update_timestamp THEN
  • UPDATE SET *
  • WHEN NOT MATCHED THEN
  • INSERT *
  • The Gold Layer (Business-Level Aggregations)

The Gold layer is highly refined and aggregated, designed specifically for analytics, reporting, and machine learning. This layer is organized around business use cases rather than source systems.

Key Characteristics of Gold

Read-Optimized : Data is structured in dimensional models (Star Schemas) or wide denormalized tables to maximize query performance for BI tools.

Aggregations : Pre-calculated metrics (e.g., daily sales, monthly active users) are stored here to reduce compute costs at query time.

SLA Expectations : High availability and extremely low latency for read queries. This layer must support concurrent BI user workloads.

Batch vs. Streaming Processing Boundaries

Designing the processing boundaries between layers requires balancing latency requirements with compute costs. Databricks Structured Streaming and Delta Live Tables (DLT) provide unified APIs for both paradigms.

Bronze to Silver (Streaming)

Moving data from Bronze to Silver is typically handled using streaming pipelines. This ensures that data is cleansed and available as quickly as possible. The low latency of streaming is beneficial for identifying data quality issues early and supporting operational reporting.

Silver to Gold (Batch)

Moving data from Silver to Gold is often done using scheduled batch jobs. Gold layer tables frequently involve complex aggregations, window functions, or large-scale joins that are computationally expensive to evaluate continuously in a streaming context. Running these on a schedule (e.g., hourly or daily) balances freshness with cost efficiency.

Layer Transition Typical Paradigm Rationale

Source → Bronze Streaming Capture raw events immediately to prevent data loss.

Bronze → Silver Streaming Low-latency deduplication and cleansing for operational use.

Silver → Gold Batch Complex aggregations and large joins are more cost-effective on a schedule.

Dimensional Models vs. Wide Denormalized Tables

When designing the Gold layer in a distributed environment like Databricks, data engineers must choose between traditional dimensional models and wide denormalized tables.

Dimensional Modeling (Star Schema)

A Star Schema organizes data into central Fact tables (events/metrics) surrounded by Dimension tables (context). This is familiar to BI developers and efficiently manages updates to descriptive attributes (e.g., changing a customer's address).

However, in a distributed Lakehouse, joining massive Fact tables with large Dimension tables requires shuffling data across the cluster network, which can degrade query performance.

Wide Denormalized Tables

To optimize read performance in Databricks, engineers often create Wide Denormalized Tables. This approach pre-joins Fact and Dimension data into a single, massive table.

Advantages of Wide Tables:

Zero Joins at Query Time : BI queries execute significantly faster because they only scan a single table, eliminating network shuffle.

Optimized for Columnar Storage : Parquet/Delta formats are highly efficient at scanning wide tables, only reading the specific columns requested by the query.

Trade-offs:

Data Duplication : The same dimensional attribute (e.g., store name) is repeated millions of times.

Complex Updates : If a store changes its name, millions of records in the wide table must be updated (an expensive operation in Delta Lake) compared to a single row update in a Star Schema dimension table.

In modern Lakehouse architectures, a hybrid approach is common: maintaining a Star Schema for flexibility, while using Materialized Views to generate wide denormalized tables for specific, high-performance BI dashboards.