Study guide

9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design

Microsoft Certified: Azure Databricks Data Engineer Associate DP-750

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Azure Databricks Architecture & Compute Selection

1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute

2 Chapter 2: Compute Configuration, Performance & Cost Management

2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions

3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup

3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown

4 Chapter 4: Unity Catalog Privilege Model & Access Control

4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation

5 Chapter 5: Advanced Data Security, Masking & ABAC

5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks

6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing

6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions

7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming

7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory

8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance

8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg

9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture

9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment

10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables

10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs

11 Chapter 11: Workflow Orchestration with Lakeflow Jobs

11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling

12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles

12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines

13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning

13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics

9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design

Key Takeaways Explain This Topic AI

The Medallion Lakehouse architecture describes a multi-hop data design pattern that progressively refines, enriches, and structures data across Bronze (raw ingest), Silver (cleansed/conformed), and Gold (business-aggregated) layers.

The Bronze layer serves as the raw, append-only historical archive of data from source systems, retaining source formats, raw payloads, and ingestion metadata (timestamps, input filenames) for replayability and audit compliance.

The Silver layer enforces enterprise conformance by applying schema validation, deduplication, null handling, type casting, domain lookups, and data quality constraints to establish a clean, queryable single source of truth.

The Gold layer delivers project-specific, business-level aggregated data models, dimensional star/snowflake schemas, and feature stores optimized for BI dashboards (Power BI), executive reporting, and machine learning.

Compute selection aligns with each layer's operational profile: Bronze and Silver pipelines leverage automated Job Compute or Serverless Lakeflow pipelines, while Gold consumption layers leverage Photon-vectorized SQL Warehouses.

Last updated: August 2026

9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design

DP-750 Exam Focus: Master the design principles, data fidelity standards, schema evolution rules, and compute mapping for the Medallion (Multi-Hop) Lakehouse architecture. Understand the specific operational boundaries of Bronze (raw append-only), Silver (cleansed, deduplicated, conformed), and Gold (curated dimensional models and aggregates) layers, including replayability mechanics, audit compliance, and compute tier selection.

1. Architectural Foundations of the Medallion Architecture

In modern data engineering, building robust data platforms requires balancing competing requirements: low-latency ingestion, regulatory auditability, historical reprocessing capabilities, high-performance analytics, and data quality guarantees. Traditional single-tier architectures (such as monolithic data warehouses or uncurated data swamps) force tradeoffs between raw fidelity and downstream query usability.

The Medallion Lakehouse Pattern (also known as the Multi-Hop Architecture ) is the reference design pattern for building scalable, reliable lakehouses on Azure Databricks. By organizing data into distinct logical layers— Bronze , Silver , and Gold —organizations establish a stepwise quality refinement pipeline that decouples source-aligned ingestion from consumer-aligned data delivery. +---------------------------------------------------------------------------------------------------------+

  • | MEDALLION LAKEHOUSE MULTI-HOP DATA FLOW |
  • +---------------------------------------------------------------------------------------------------------+
  • | |
  • | +-------------------+ +-------------------+ +-------------------+ +------------+ |
  • | | SOURCE DATA | | BRONZE LAYER | | SILVER LAYER | | GOLD LAYER | |
  • | | | | (Raw Ingest) | | (Cleansed/Conform)| | (Curated) | |
  • | | - IoT Telemetry | ===> | - Append-only | ===> | - Deduplicated | ===> | - Star | |
  • | | - Azure EventHubs | Ingest | - Raw schema/JSON | Enrich | - Schema-enforced | Model | Schemas | |
  • | | - OLTP Databases | | - Rescued data | & Clean| - Type-casted | & Agg | - BI Hubs | |
  • | | - Files (ADLS Gen2| | - Audit metadata | | - Domain Lookups | | - ML Feats | |
  • | +-------------------+ +-------------------+ +-------------------+ +------------+ |
  • | | | | |
  • | COMPUTE PROFILE: Automated Job / Automated Job / Serverless SQL |
  • | Serverless Lakeflow Serverless Lakeflow Warehouse |
  • +---------------------------------------------------------------------------------------------------------+
  • Why Multi-Hop Design Matters

Replayability & Resiliency : If downstream business logic changes or a bug is identified in business rules, engineers can recompute Silver and Gold layers directly from the immutable Bronze historical record without re-extracting data from production source systems.

Granular Quality Gates : Data quality checks, deduplication routines, and type validations occur progressively. Bronze never fails due to malformed payloads, Silver isolates bad data, and Gold remains pristine for executive consumption.

Performance Optimization : Storage formats, clustering strategies, and indexing can be optimized specifically for each layer's query patterns rather than compromising across ingestion and analytical workloads.

Unified Governance : Unity Catalog applies fine-grained access control across all three layers, ensuring data scientists, BI developers, and compliance auditors only access appropriate tiers.

2. Bronze Layer: Raw Append-Only Ingestion

The Bronze layer (often referred to as the raw table or landing zone ) acts as the initial landing repository for data arriving from upstream operational systems, cloud object stores, message queues, and external APIs.

Core Bronze Design Principles

Append-Only Ingestion : Bronze tables are strictly append-only. Incoming records are inserted as-is without in-place updates ( UPDATE or DELETE operations are avoided, with the exception of regulatory compliance requests such as GDPR Right-to-be-Forgotten).

High Data Fidelity : Data is preserved in its rawest possible state. If a source provides a semi-structured JSON payload or unstructured text, the payload is captured verbatim to prevent information loss.

Minimal Transformation : Only technical metadata columns are appended during ingestion. Business transformations, type conversions, and filtering must never be performed in the Bronze layer.

Schema Preservation & Rescued Data : Ingestion engines (such as Databricks Auto Loader) capture new or unexpected columns automatically, routing schema mismatches into the _rescued_data column rather than failing the stream. -- Example: Bronze Ingestion Table Definition

  • CREATE TABLE bronze.crm.customer_events (
  • raw_payload STRING,
  • source_file_name STRING,
  • source_file_modification_time TIMESTAMP,
  • ingest_timestamp TIMESTAMP,
  • _rescued_data STRING
  • )
  • USING DELTA
  • PARTITIONED BY (DATE(ingest_timestamp));
  • Essential Bronze Metadata Columns

To guarantee full traceability and auditability, enterprise Bronze tables should always append the following metadata attributes:

Metadata Column Derivation Function Architectural Purpose

_rescued_data Built-in Auto Loader feature Stores malformed fields or undeclared schema columns for quarantine and schema evolution.

_input_file_name input_file_name() / _metadata.file_name Tracks the specific cloud storage object or blob URI that generated the record.

_ingest_timestamp current_timestamp() Records the exact UTC timestamp when Azure Databricks processed the record into Delta Lake.

_source_system Literal string identifier Distinguishes data origin in multi-tenant or multi-region ingestion pipelines.

Exam Tip: On the DP-750 exam, any question asking where raw, unparsed JSON strings or historical audit copies of raw event streams belong should immediately point to the Bronze layer .

3. Silver Layer: Cleansed, Conformed, & Enterprise-Standardized

The Silver layer (also called the curated , cleaned , or conformed layer) represents the enterprise single source of truth. It takes raw records from Bronze and applies systematic data cleansing, enrichment, structural normalization, and domain validation. SILVER REFINEMENT STAGES

  • +------------------------+ +------------------------+ +------------------------+
  • | 1. Structural Parsing | ===> | 2. Quality Validation | ===> | 3. Conformance & SCD |
  • | - Parse JSON payloads | | - Deduplicate by PK | | - Resolve Master IDs |
  • | - Extract nested fields| | - Handle NULL values | | - Apply SCD Type 1 / 2 |
  • | - Enforce strict schema| | - Cast verified types | | - Join Reference Dictionaries
  • +------------------------+ +------------------------+ +------------------------+
  • Core Silver Design Principles

Strict Schema Enforcement & Typing : Raw strings are parsed into strongly-typed primitives ( INT , BIGINT , DECIMAL(18,2) , TIMESTAMP , BOOLEAN ).

Deduplication : Ingested records are deduplicated based on enterprise business keys (e.g., order_id , transaction_id ) using deterministic window ranking ( ROW_NUMBER() ) or dropDuplicates() .

Null Imputation & Handling : Missing values are replaced with standardized defaults, coalesced from secondary sources, or quarantined.

Data Conformance & Normalization : Categorical values are normalized to enterprise standards (e.g., converting ['NY', 'New York', 'ny', 'N.Y.'] to 'NY' ), and foreign key references are resolved against master data dimensions.

Change Data Capture (CDC) & History : Source updates are merged into Silver tables using Delta MERGE INTO or Lakeflow APPLY CHANGES INTO , maintaining current state (SCD Type 1) or full versioned history (SCD Type 2). -- Example: Silver Table Curation via MERGE INTO

  • MERGE INTO silver.sales.orders AS target
  • USING (
  • SELECT
  • CAST(get_json_object(raw_payload, '$.order_id') AS BIGINT) AS order_id,
  • CAST(get_json_object(raw_payload, '$.customer_id') AS BIGINT) AS customer_id,
  • TO_TIMESTAMP(get_json_object(raw_payload, '$.order_time'), 'yyyy-MM-dd HH:mm:ss') AS order_timestamp,
  • CAST(get_json_object(raw_payload, '$.amount') AS DECIMAL(10,2)) AS order_amount,
  • UPPER(TRIM(get_json_object(raw_payload, '$.status'))) AS order_status,
  • ingest_timestamp
  • FROM bronze.sales.raw_orders
  • WHERE raw_payload IS NOT NULL
  • ) AS source
  • ON target.order_id = source.order_id
  • WHEN MATCHED AND source.order_timestamp > target.order_timestamp THEN
  • UPDATE SET
  • target.customer_id = source.customer_id,
  • target.order_timestamp = source.order_timestamp,
  • target.order_amount = source.order_amount,
  • target.order_status = source.order_status,
  • target.updated_at = current_timestamp()
  • WHEN NOT MATCHED THEN
  • INSERT (order_id, customer_id, order_timestamp, order_amount, order_status, created_at, updated_at)

VALUES (source.order_id, source.customer_id, source.order_timestamp, source.order_amount, source.order_status, current_timestamp(), current_timestamp());

4. Gold Layer: Business Aggregation, Dimensional Models, & Feature Stores

The Gold layer (often called the consumption , curated reporting , or presentation layer) provides project-specific, business-aligned data structures optimized for read-heavy consumption, executive reporting, business intelligence (Power BI), and machine learning.

Core Gold Design Principles

Business-Level Aggregations : Pre-calculates Key Performance Indicators (KPIs), daily sales summaries, customer lifetime value (LTV), churn scores, and rollups across common reporting hierarchies.

Dimensional Modeling (Kimball Star & Snowflake Schemas) : Structures data into Fact tables (e.g., fact_daily_financial_transactions ) and Dimension tables (e.g., dim_customer , dim_product , dim_date ).

Optimized for High-Concurrency BI : Uses Delta Lake performance features such as Liquid Clustering ( CLUSTER BY ), file compaction ( OPTIMIZE ), and predictive caching to ensure sub-second dashboard query latency.

Strict Access Boundaries : Business users, financial analysts, and Power BI service principals are typically granted SELECT access exclusively on Gold tables, preventing ad-hoc queries against raw or partially cleansed data. -- Example: Gold Aggregated Dimensional Fact Table

  • CREATE OR REPLACE TABLE gold.analytics.fact_daily_regional_sales
  • CLUSTER BY (sale_date, region_code)
  • AS
  • SELECT
  • CAST(o.order_timestamp AS DATE) AS sale_date,
  • c.region_code,
  • c.customer_segment,
  • COUNT(DISTINCT o.order_id) AS total_orders,
  • SUM(o.order_amount) AS gross_revenue,
  • AVG(o.order_amount) AS average_order_value
  • FROM silver.sales.orders o
  • INNER JOIN silver.crm.dim_customer c
  • ON o.customer_id = c.customer_id
  • WHERE o.order_status = 'COMPLETED'
  • GROUP BY 1, 2, 3;
  • 5. Medallion Layer Comparison Matrix

The following matrix summarizes the technical differences across the three layers for the DP-750 exam:

Architectural Dimension Bronze Layer (Raw) Silver Layer (Cleansed) Gold Layer (Aggregated)

Primary Objective Append-only raw ingestion & archive Deduplication, cleansing, conformance Business KPIs, star schemas, BI serving

Data Fidelity 100% raw (including errors/nulls) Cleaned, typed, normalized, conformed Highly summarized, aggregated, or modeled

Data Structure Raw JSON, strings, varied formats Relational 3NF / conformed entities Star schema (Facts/Dims), Flat wide tables

Schema Enforcement Schema evolution, _rescued_data Strict schema, type validation Strict schema, business constraint locked

Write Operations INSERT , COPY INTO , Streaming Append MERGE INTO , APPLY CHANGES , UPDATE MERGE , CTAS, Overwrite partitions

Target Consumers Data Engineers, Pipeline Jobs Data Engineers, Data Scientists Business Analysts, BI (Power BI), Execs

Recommended Retention Long-term / Infinite (Raw audit) Operational retention (1–5+ years) Business analytics window

6. Compute Selection Across Medallion Layers

Matching the correct Azure Databricks compute engine to each medallion layer is critical for cost efficiency, scalability, and performance optimization. +-----------------------------------------------------------------------------------------+

  • | COMPUTE MAPPING ACROSS MEDALLION TIERS |
  • +-----------------------------------------------------------------------------------------+
  • | |
  • | 1. BRONZE INGESTION LAYER |
  • | - Workload: Continuous streaming or micro-batch ingestion (Auto Loader, Event Hubs) |
  • | - Optimal Compute: Automated Job Clusters (Single-Node or Multi-Node) or |
  • | Serverless Lakeflow Pipelines. Billed at lowest DBU tier. |
  • | |
  • | 2. SILVER TRANSFORMATION LAYER |
  • | - Workload: Heavy distributed joins, CDC merges, window deduplication, cleansing |
  • | - Optimal Compute: Automated Multi-Node Job Clusters or Lakeflow Declarative |
  • | Pipelines with Autoscaling and Photon enabled. |
  • | |
  • | 3. GOLD CONSUMPTION LAYER |
  • | - Workload: High-concurrency SQL queries, Power BI DirectQuery, ad-hoc BI analytics |
  • | - Optimal Compute: Serverless SQL Warehouses (or Pro SQL Warehouses) with Photon, |
  • | Predictive I/O, and multi-cluster horizontal auto-scaling. |
  • +-----------------------------------------------------------------------------------------+

Loading diagram... Medallion Architecture End-to-End Data Pipeline Flow