Study guide
9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing
Key Takeaways Explain This Topic AI
Apache Spark Higher-Order Functions (transform, filter, exists, aggregate, forall, zip_with) manipulate nested array elements inline using lambda expressions without costly explode() and groupBy() shuffle operations.
JSON parsing in Spark SQL and PySpark relies on from_json(), to_json(), and schema_of_json() to convert semi-structured JSON string payloads into strongly-typed StructTypes and arrays with schema evolution support.
Struct manipulation enables hierarchical data modeling using dot-notation (struct_col.field), star-expansion (col('struct_col.*')), and constructor functions like struct(), named_struct(), and withField().
Array flattening and expansion functions like explode(), explode_outer(), posexplode(), flatten(), and array_distinct() allow normalizing nested repeating data into relational rows when dimensional flattening is required.
Higher-Order Functions execute within a single Spark task execution stage in-memory, providing 2x–5x performance gains over traditional explode-shuffle-aggregate patterns.
Last updated: August 2026
9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing
DP-750 Exam Focus: Master nested and complex data transformations in Azure Databricks. Understand JSON serialization and deserialization using from_json() , to_json() , and schema_of_json() ; struct navigation and manipulation ( withField , dropFields ); array expansions ( explode vs explode_outer , posexplode ); and inline array processing with Higher-Order Functions ( transform , filter , exists , forall , aggregate , zip_with ).
1. JSON Parsing & Serialization in Spark SQL / PySpark
In modern cloud architectures, events from IoT hubs, webhooks, and REST APIs arrive as JSON string payloads stored in Bronze Delta tables. Transforming these strings into queryable relational columns requires robust deserialization functions. +-----------------------------------------------------------------------------------+
- | JSON PARSING & SCHEMA ENGINE |
- +-----------------------------------------------------------------------------------+
- | |
- | 1. schema_of_json(sample_string) |
- | - Dynamically infers DDL schema string from a representative JSON sample. |
- | |
- | 2. from_json(json_column, schema, [options]) |
- | - Parses JSON string into a structured Spark StructType or ArrayType. |
- | |
- | 3. to_json(struct_or_map_column) |
- | - Serializes complex Struct/Map/Array columns back into compact JSON strings. |
- | |
- | 4. get_json_object(json_column, '$.path.to.field') |
- | - Extracts a single scalar value via JSONPath (ideal for lightweight queries).|
- +-----------------------------------------------------------------------------------+
- from_json() with Schema Definition
from_json() deserializes a JSON string column into a structured StructType or ArrayType . It requires an explicit schema supplied as a DDL string or StructType definition: -- SQL: Parsing JSON with DDL schema
- SELECT
- event_id,
from_json(raw_json, 'customer_id BIGINT, items ARRAY<STRUCT<item_id: STRING, price: DOUBLE, qty: INT>>, purchase_time TIMESTAMP') AS parsed_data
- FROM bronze.web_events; # PySpark: Parsing JSON with StructType schema
- from pyspark.sql.functions import from_json, col
- from pyspark.sql.types import StructType, StructField, StringType, DoubleType, IntegerType, ArrayType
- item_schema = StructType([
- StructField("item_id", StringType(), True),
- StructField("price", DoubleType(), True),
- StructField("qty", IntegerType(), True)
- ])
- payload_schema = StructType([
- StructField("customer_id", StringType(), True),
- StructField("items", ArrayType(item_schema), True)
- ])
- df_parsed = df_bronze.withColumn("data", from_json(col("raw_payload"), payload_schema))
- Dynamic Schema Derivation with schema_of_json()
When the exact schema of a complex JSON string is unknown during development, schema_of_json() can inspect a sample string literal and generate the exact DDL schema definition: -- Dynamically derive the schema DDL string
- SELECT schema_of_json('{"order_id": 101, "total": 99.50, "tags": ["sale", "online"]}');
- - Result: STRUCT<order_id: BIGINT, tags: ARRAY<STRING>, total: DECIMAL(3,1)>
- 2. Struct Operations & Hierarchical Data Modeling
StructType columns represent complex hierarchical objects containing nested fields. Databricks provides powerful operators to construct, access, and modify structs without flattening tables.
Accessing and Expanding Struct Fields
Dot Notation : Access child attributes directly via parent_struct.child_field .
- Star Expansion ( .* ) : Expand all subfields of a struct into top-level columns. -- Access nested attributes
- SELECT
- parsed_data.customer_id,
- parsed_data.purchase_time,
- parsed_data.items[0].price AS first_item_price
- FROM silver.parsed_events;
- - Expand all fields of a struct into top-level columns
- SELECT parsed_data.* FROM silver.parsed_events;
Struct Construction & Modification ( withField , dropFields ) # PySpark: Struct manipulation with withField and dropFields
from pyspark.sql.functions import col, struct, lit
- # 1. Create a new struct from existing columns
- df_nested = df.withColumn("address_info", struct(col("street"), col("city"), col("zip_code")))
- # 2. Add or update a nested field inside an existing struct
- df_updated = df_nested.withColumn("address_info", col("address_info").withField("country", lit("USA")))
- # 3. Drop a nested field from an existing struct
- df_dropped = df_updated.withColumn("address_info", col("address_info").dropFields("zip_code"))
- 3. Array Operations: Expansion & Normalization
When nested arrays need to be unpacked into relational rows for traditional BI joins or aggregation, Spark provides array expansion functions. ARRAY EXPANSION FUNCTION COMPARISON
- Original Row: [ ID: 1, Items: ['Apple', 'Banana'] ]
- Original Row: [ ID: 2, Items: [] / NULL ]
- 1. explode(items)
- > ID: 1, Item: Apple
- > ID: 1, Item: Banana
- (Row 2 is DROPPED entirely because array was empty/null)
- 2. explode_outer(items)
- > ID: 1, Item: Apple
- > ID: 1, Item: Banana
- > ID: 2, Item: NULL <-- Preserves parent row
- 3. posexplode(items)
- > ID: 1, Pos: 0, Item: Apple
- > ID: 1, Pos: 1, Item: Banana
- Function Null / Empty Array Behavior Index Tracking Common Use Case
- explode(arr) Drops parent row if array is NULL or empty No Standard one-to-many normalization
explode_outer(arr) Preserves parent row, emitting NULL for array element No Preserving parent entities with zero children
posexplode(arr) Drops parent row if array is NULL or empty Emits 0-based pos Maintaining ordinal sequence of elements
flatten(arr_of_arr) Merges 2D nested arrays ( ARRAY<ARRAY<T>> ) into 1D ( ARRAY<T> ) No Consolidating multi-level nested collections
4. Higher-Order Functions: In-Memory Array Processing
Historically, modifying or filtering elements inside an array required a 3-step anti-pattern:
explode() the array into multiple individual rows.
Apply the transformation or filter expression.
groupBy() on the original primary key and re-collect using collect_list() .
The Problem with Explode-GroupBy
The explode-groupBy pattern is computationally catastrophic for big data: it duplicates parent columns, multiplies row volume, forces an expensive distributed shuffle across network boundaries during groupBy() , and risks Out-Of-Memory (OOM) errors during collect_list() .
The Higher-Order Functions Solution
Higher-Order Functions operate directly on arrays inline using anonymous lambda expressions ( x -> expression ). They execute within a single Spark task in-memory , eliminating distributed shuffles entirely and delivering a 2x to 5x performance acceleration . +---------------------------------------------------------------------------------------+
- | HIGHER-ORDER FUNCTIONS VS. EXPLODE-GROUPBY ARCHITECTURE |
- +---------------------------------------------------------------------------------------+
- | |
- | TRADITIONAL PATTERN (EXPENSIVE): |
- | [ Array Row ] ---> explode() ---> [ Multi-Rows ] ---> groupBy() [SHUFFLE] ---> Array |
- | |
- | HIGHER-ORDER PATTERN (ZERO SHUFFLE / IN-MEMORY): |
- | [ Array Row ] ===> transform(items, x -> x * 1.1) ===> [ Modified Array Row ] |
- +---------------------------------------------------------------------------------------+
- 5. Comprehensive Higher-Order Function Reference
- 1. transform() : Element-wise Mapping
Applies a transformation expression to every element in an array, returning a new array with modified elements. -- Apply 10% tax to all item prices in an array
- SELECT
- order_id,
- transform(item_prices, price -> ROUND(price * 1.10, 2)) AS prices_with_tax
- FROM silver.orders;
- 2. filter() : Array Element Filtering
Evaluates a boolean predicate on each element, returning an array containing only elements that evaluate to TRUE . -- Retain only items where price > 50.00
- SELECT
- order_id,
- filter(items, item -> item.price > 50.0) AS premium_items
- FROM silver.orders;
- 3. exists() : Existential Quantifier
Returns TRUE if at least one element in the array satisfies the boolean predicate; otherwise returns FALSE . -- Identify orders containing any discounted item
- SELECT
- order_id,
- exists(items, item -> item.is_discounted = TRUE) AS has_discount
- FROM silver.orders;
- 4. forall() : Universal Quantifier
Returns TRUE if every single element in the array satisfies the boolean predicate. -- Check if all items in the order are in stock
- SELECT
- order_id,
- forall(items, item -> item.in_stock = TRUE) AS all_in_stock
- FROM silver.orders;
- 5. aggregate() : Array Reduction to Scalar
Reduces all elements in an array to a single scalar value using an accumulator and an optional finisher expression. -- Syntax: aggregate(array, initial_value, (acc, x) -> merge_expr, acc -> finisher_expr)
- - Calculate total order amount across all items (price * quantity)
- SELECT
- order_id,
- aggregate(
- items,
- 0.0,
- (acc, item) -> acc + (item.price * item.qty),
- acc -> ROUND(acc, 2)
- ) AS total_order_amount
- FROM silver.orders;
- 6. zip_with() : Pairwise Array Merging
Combines two equal-sized arrays into a single array by applying a binary lambda function across corresponding element indices. -- Multiply quantities by unit costs across two arrays
- SELECT
- order_id,
- zip_with(quantities, unit_prices, (qty, price) -> qty * price) AS line_item_totals
- FROM silver.inventory;