Study guide
7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column
Key Takeaways Explain This Topic AI
Auto Loader automatically infers schemas across semi-structured and structured formats (JSON, CSV, Parquet, Avro, XML) and persists the inferred contract as JSON in 'cloudFiles.schemaLocation'.
Schema evolution is controlled via 'cloudFiles.schemaEvolutionMode', offering four distinct behaviors: 'addNewColumns' (default), 'failOnNewColumns', 'rescue', and 'none'.
The '_rescued_data' column automatically captures unparseable data, type mismatches, and undeclared columns in a JSON string/struct, guaranteeing that no source data is silently lost during ingestion.
Schema hints ('cloudFiles.schemaHints') allow engineers to enforce strict data types (e.g., parsing strings as TIMESTAMP or DECIMAL) while allowing Auto Loader to infer the remainder of the schema dynamically.
When combined with Delta Lake's 'mergeSchema = true' option, Auto Loader seamlessly updates downstream Delta table schemas when new fields appear in source files.
Last updated: August 2026
7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column
DP-750 Exam Focus: Master Auto Loader schema management. Understand how cloudFiles.schemaLocation persists inferred schemas, evaluate the four cloudFiles.schemaEvolutionMode settings ( addNewColumns , failOnNewColumns , rescue , none ), enforce specific data types with cloudFiles.schemaHints , and explain the behavior and utility of the _rescued_data column.
1. Schema Inference & cloudFiles.schemaLocation
In raw Bronze ingestion, source systems frequently emit files with dynamic, unannounced schema variations—such as newly added attributes, omitted fields, or slight data type variations.
Standard Spark structured streaming file readers require a rigidly predefined static schema ( .schema(my_schema) ). If a file arrives with an unexpected schema change, standard Spark readers either fail immediately or parse new columns as null , causing silent data loss.
Auto Loader eliminates manual schema definitions through automatic schema inference and persistent schema storage. +-----------------------------------------------------------------------------------+
- | SCHEMA INFERENCE & EVOLUTION FLOW |
- +-----------------------------------------------------------------------------------+
- | |
- | [ Ingest Files (JSON / CSV / Parquet) ] |
- | | |
- | v |
- | [ Auto Loader Schema Engine ] <====> [ cloudFiles.schemaLocation ] |
- | - Samples initial files (Persists schema contract as JSON) |
- | - Applies schemaHints overrides /_schemas/0/schema.json |
- | | |
- | v |
- | [ Schema Drift Detection ] |
- | | |
- | +---------------------------------------+ |
- | | (New column detected) | (Type mismatch / corrupt data) |
- | v v |
- | [ schemaEvolutionMode ] [ _rescued_data Column ] |
- | - addNewColumns (Default) - Stores invalid record payload |
- | - failOnNewColumns - Captures unmapped columns |
- | - rescue - Prevents silent data corruption |
- | - none | |
- | | | |
- | +-------------------+-------------------+ |
- | | |
- | v |
- | [ Target Bronze Delta Table ] |
- | (Auto-evolved schema with rescued records) |
- +-----------------------------------------------------------------------------------+
- The cloudFiles.schemaLocation Option
To enable schema inference and evolution, data engineers must provide cloudFiles.schemaLocation . This directory stores the learned schema contract in cloud storage (ADLS Gen2): # Auto Loader with Schema Inference
- df_inferred = (spark.readStream
- .format("cloudFiles")
- .option("cloudFiles.format", "json")
- .option("cloudFiles.schemaLocation", "abfss://checkpoints@myadls.dfs.core.windows.net/iot_schema")
- .option("cloudFiles.inferColumnTypes", "true") # Informs Auto Loader to infer types beyond STRING for JSON/CSV
- .load("abfss://raw@myadls.dfs.core.windows.net/iot-telemetry/"))
- Key Operational Behaviors
Initial Sampling: When the stream launches for the first time, Auto Loader samples the first 1,000 files (or up to 50 GB of data by default) to establish the initial schema definition.
Schema Persistence: The inferred schema is written to cloudFiles.schemaLocation as an immutable JSON schema file. On subsequent stream restarts, Auto Loader reads the schema directly from this location, skipping redundant sampling.
cloudFiles.inferColumnTypes : By default for JSON and CSV, Auto Loader infers all columns as StringType to avoid premature type casting errors. Setting cloudFiles.inferColumnTypes to true instructs Auto Loader to infer numeric, boolean, timestamp, and complex nested types.
2. Schema Evolution Modes
When a newly ingested file contains columns that are not present in the persisted schema contract, Auto Loader's behavior is dictated by cloudFiles.schemaEvolutionMode .
Schema Evolution Mode Behavior on New Column Detection Schema Location Updated? Pipeline Behavior
addNewColumns (Default) Adds new columns to the schema contract. Yes Stream continues (or auto-restarts in DLT/workflows) and appends new columns to the downstream Delta table.
failOnNewColumns Throws an UnknownFieldException and fails the micro-batch immediately. No Stream fails. Prevents accidental schema modifications until an engineer validates the new fields.
rescue Does not modify the schema contract. New unmapped columns are captured inside the _rescued_data column. No Stream continues running without altering table structure.
none Ignores new columns completely; does not add them to schema or rescue column. New columns are dropped. No Stream continues running; new columns are silently discarded.
- Mode Comparison & Production Scenarios
- 1. addNewColumns (Default)
Ideal for Bronze lakehouse ingestion where all landing data must be preserved. When a new column (e.g., battery_temp ) appears, Auto Loader updates the schema file in schemaLocation and restarts the stream with the updated schema. # Ingesting with addNewColumns and Delta schema merging
- (spark.readStream
- .format("cloudFiles")
- .option("cloudFiles.format", "json")
- .option("cloudFiles.schemaLocation", "abfss://checkpoints@myadls/schema_sales")
- .option("cloudFiles.schemaEvolutionMode", "addNewColumns")
- .load("abfss://raw@myadls/sales_json/")
- .writeStream
- .format("delta")
- .option("checkpointLocation", "abfss://checkpoints@myadls/cp_sales")
- .option("mergeSchema", "true") # Required for target Delta table to accept evolved columns
- .toTable("bronze.sales.raw_transactions"))
- 2. failOnNewColumns
Used in strict governance environments where upstream source schema changes require manual approval or schema registry validation before entering the lakehouse.
3. rescue
Recommended when the downstream Delta table schema must remain strictly static, but unmapped attributes must be preserved for forensic analysis in _rescued_data rather than discarded.
3. The _rescued_data Column: Deep Dive
To prevent data loss and pipeline downtime from schema mismatches or corrupt records, Auto Loader automatically provisions a hidden column named _rescued_data by default. RESCUED DATA INGESTION MATRIX
Source Record in ADLS Gen2:
{"device_id": 101, "reading": "INVALID_FLOAT", "new_sensor_val": 42.8}
Target Schema:
- device_id: BIGINT
- reading: DOUBLE
- +-----------------------------------------------------------------------------------+
- | INGESTED DELTA ROW |
- +-----------+---------+-------------------------------------------------------------+
- | device_id | reading | _rescued_data |
- +-----------+---------+-------------------------------------------------------------+
- | 101 | null | {"reading":"INVALID_FLOAT", "new_sensor_val":42.8, |
- | | | "_file_path":"abfss://raw@storage/data_01.json"} |
- +-----------+---------+-------------------------------------------------------------+
What Is Captured in _rescued_data ?
Data Type Mismatches: If a column is defined as DOUBLE (e.g., reading ), but a source record contains a non-numeric string ( "INVALID_FLOAT" ), the column value in the main table is set to null , and the raw unparsed value is preserved inside _rescued_data .
Undeclared Columns: Under schemaEvolutionMode = 'rescue' (or if schema evolution is disabled), any column in the source file not declared in the schema is routed into _rescued_data .
Malformed Records: Unparseable JSON or CSV lines that violate standard formatting syntax.
Case Sensitivity Clashes: When case sensitivity is enabled, differences in column casing (e.g., CustomerId vs customerId ) are preserved in _rescued_data to prevent accidental overwrites.
Querying Rescued Data with Databricks SQL
Data engineers can easily query, inspect, and extract rescued fields using standard SQL JSON extraction functions: -- Query: Identify rows with data rescue events and extract original values
- SELECT
- device_id,
- reading,
- _rescued_data,
- get_json_object(_rescued_data, '$.reading') AS original_reading_str,
- get_json_object(_rescued_data, '$._file_path') AS source_file
- FROM bronze.telemetry.raw_events
- WHERE _rescued_data IS NOT NULL;
Exam Tip: To disable the rescued data column, set .option("cloudFiles.schemaEvolutionMode", "none") and .option("cloudFiles.schemaHints", "") or set .option("cloudFiles.rescuedDataColumn", "") (empty string). In production, leaving _rescued_data enabled is the recommended practice for Bronze ingestion.
4. Schema Hints & Explicit Overrides
While Auto Loader infers schemas effectively, automated inference may choose overly permissive or generic data types (for example, inferring an ISO timestamp string as StringType or an integer ID as LongType ).
Schema Hints ( cloudFiles.schemaHints ) allow data engineers to explicitly enforce exact data types for specific columns while allowing Auto Loader to automatically infer the rest of the schema. # Applying Schema Hints for precise datatype enforcement
- df_hints = (spark.readStream
- .format("cloudFiles")
- .option("cloudFiles.format", "csv")
- .option("cloudFiles.schemaLocation", "abfss://checkpoints@myadls/csv_schema")
- .option("header", "true")
- .option("cloudFiles.inferColumnTypes", "true")
- .option("cloudFiles.schemaHints", """
- transaction_id BIGINT,
- amount DECIMAL(10, 2),
- transaction_timestamp TIMESTAMP,
- is_fraudulent BOOLEAN
- """)
- .load("abfss://raw@myadls/finance_csv/"))
- Rules Governing Schema Hints
Precedence: Explicit definitions in cloudFiles.schemaHints override both inferred types and types loaded from cloudFiles.schemaLocation .
Partial Specification: Engineers only need to specify the subset of columns requiring strict typing; unmentioned columns are dynamically inferred.
Type Casting Resilience: If incoming data cannot be cast to the type specified in schemaHints (e.g., "N/A" in a DECIMAL column), the column receives null and the raw value is safely captured in _rescued_data .