ExamFinch.com My Study

Study guide

1.1 Databricks Architecture & Compute Concepts

Practice this exam

Databricks Certified Data Analyst Associate Exam

Track your progress as you read

Finished sections get a check mark.

1 Platform Overview & Unity Catalog Foundations

1.1 Databricks Architecture & Compute Concepts 1.2 Databricks Marketplace & Data Sharing Basics 1.3 Unity Catalog Namespace & Object Hierarchy 1.4 Governed Asset Discovery & System Tables

2 Managing & Cleaning Governed Data

2.1 Dataset Discovery & Asset Tagging 2.2 Lineage Inspection & Data Traceability 2.3 SQL Data Cleaning & Transformation Techniques

3 Data Ingestion & Import Patterns

3.1 Workspace Uploads & Cloud File Ingestion 3.2 Incremental File Ingestion with Auto Loader 3.3 Ingesting Data via Delta Sharing & External Connections

4 Databricks SQL Warehouses & Core Querying

4.1 Databricks SQL Interface & SQL Assistant 4.2 SQL Warehouses Sizing & Management 4.3 Table Creation & Data Modification in Databricks SQL

5 Advanced SQL Authoring & Time Travel

5.1 Advanced Aggregations, Window Functions & Analytics 5.2 Complex Joins, CTEs & Set Operations 5.3 Views, Materialized Views & Delta Time Travel

6 Query Performance Analysis & Photon Engine

6.1 Photon Engine Architecture & Acceleration 6.2 Query History & Execution Profile Inspection 6.3 Delta Lake Audit History & Commit Logs

7 Liquid Clustering, Caching & Performance Troubleshooting

7.1 Caching Hierarchy in Databricks SQL 7.2 Data Layout Optimization: Liquid Clustering vs Partitioning & Z-Ordering 7.3 Troubleshooting Slow & Failing Queries

8 AI/BI Dashboards & Visualization Design

8.1 AI/BI Dashboards Overview & Canvas Layout 8.2 Chart Types & Visualization Selection 8.3 Notebook & SQL Editor Visualizations

9 Dashboard Interactivity, Publishing & Alerts

9.1 Dashboard Parameters & Interactive Filtering 9.2 Publishing, Sharing & Access Control 9.3 Scheduled Refresh, Email Subscriptions & Alerts

10 Developing & Maintaining AI/BI Genie Spaces

10.1 AI/BI Genie Architecture & Space Setup 10.2 Curation via Instructions, Descriptions & Sample Questions 10.3 Trusted Assets & Parameterized Functions 10.4 Monitoring, Benchmarks, Feedback & Maintenance

11 Data Modeling & Medallion Architecture

11.1 Dimensional Modeling Fundamentals 11.2 Medallion Architecture for Analytics 11.3 Constraint Enforcement & Relationships in Unity Catalog

12 Securing Data & Unity Catalog Governance

12.1 Unity Catalog Privilege Model & Grants 12.2 Ownership, Inheritance & Access Control Patterns 12.3 Fine-Grained Security: Row Filters & Column Masks

1.1 Databricks Architecture & Compute Concepts

Key Takeaways Explain This Topic AI

Databricks separates architecture into a Control Plane managed by Databricks and a Compute Plane that processes data either in the customer cloud account (classic) or in a Databricks-managed serverless compute plane.

Serverless SQL Warehouses reduce cold-start startup latency from approximately 4 minutes down to typically 2–6 seconds by utilizing pre-warmed compute pools managed directly by Databricks.

Compute resources in Databricks are categorized into All-Purpose Clusters for interactive development, Job Clusters for scheduled automated workflows, and SQL Warehouses for BI analytical workloads.

Auto-scaling SQL Warehouses scale out across up to 10 cluster instances automatically when query queueing is detected and scale down after 10-15 minutes of idle time.

Last updated: July 2026

Databricks Architecture & Compute Concepts

The Databricks Data Intelligence Platform is built upon a lakehouse architecture that combines the performance, structure, and governance of traditional data warehouses with the flexibility, open formats, and cost efficiency of cloud object storage. To effectively analyze data and build high-performance reporting solutions, a Databricks Data Analyst must understand how compute resources operate, how workloads are processed across different architectural tiers, and how to select the right compute engine for specific analytical tasks.

The Two-Tiered Architecture: Control Plane vs. Data Plane

Databricks operates using a fundamental separation of concerns divided into two primary planes: the Control Plane and the Data Plane . This decoupled structure ensures high operational reliability, centralized administration, and robust data security.

1. Control Plane

The Control Plane consists of backend services managed entirely by Databricks within its own cloud account infrastructure. It houses the services responsible for platform management and orchestration, including:

Web UI & Workspace Application: The web interface for notebooks, SQL editors, dashboards, and Catalog Explorer.

Notebook & Job Management: Storage and versioning of notebook code, job schedules, and workflow definitions.

Query Orchestration & Parsing: Initial query receiving, user authentication, authorization checks, and command routing.

Cluster Management Service: Launching, monitoring, and terminating compute clusters in the Data Plane.

Control Plane Security: Management of user tokens, single sign-on (SSO) integrations, and audit logging.

Key Concept: Customer data at rest never resides in the Control Plane. The Control Plane only stores control metadata (such as notebook source code, workspace configurations, and user identity credentials).

2. Data Plane

The Data Plane is where your actual data resides and where compute engines execute code. Depending on your deployment architecture, the Data Plane operates in one of two environments:

Classic Data Plane: Compute resources (virtual machines) are launched inside the customer's cloud account (AWS VPC, Azure VNet, or GCP VPC). Storage services such as AWS S3, Azure ADLS Gen2, or Google Cloud Storage remain in the customer account.

Serverless Data Plane: Compute resources run in secure, isolated cloud accounts managed directly by Databricks. Databricks handles instance provisioning, scaling, networking, and software updates while connecting securely to the customer's data storage.

  • Architectural Layer Control Plane Classic Data Plane Serverless Data Plane
  • Account Management Databricks Cloud Account Customer Cloud Account Databricks Cloud Account

Hosted Components Web UI, Notebooks, Jobs, Workspace DB EC2/VM instances, Driver/Worker nodes Pre-warmed VM pools, Serverless compute

  • Data Storage Location Metadata & Notebook Code only Customer S3 / ADLS / GCS Customer S3 / ADLS / GCS
  • Network Boundary Managed by Databricks Customer VPC / VNet Databricks Managed Secure VPC
  • Startup Latency Instant 5 – 7 Minutes Under 5 Seconds
  • Classic Compute vs. Serverless Compute Models

Databricks provides two distinct compute provisioning models: Classic Compute and Serverless Compute .

Classic Compute

In the classic model, when a user provisions a cluster or SQL warehouse, Databricks requests virtual machines from the cloud provider within the customer's cloud subscription. The user selects specific VM node types (e.g., memory-optimized or compute-optimized), configures minimum and maximum node limits for auto-scaling, and sets auto-stop thresholds.

Pros: Full network isolation within customer VPCs; custom cloud security configurations.

Cons: Cold-start latency (typically 5 to 7 minutes while cloud instances initialize); potential cloud quota limits; manual cluster management overhead.

Serverless Compute

In the serverless model, compute resources run in a pool of pre-warmed infrastructure managed by Databricks. When a data analyst executes a query in Databricks SQL or runs a notebook attached to a serverless compute engine, compute capacity is assigned instantly.

Pros: Ultra-fast start time (typically 2–6 seconds); zero infrastructure management; dynamic auto-scaling tailored precisely to query complexity; automatic patching and optimization.

Cons: Requires cloud network configuration setup for private storage connectivity (e.g., Network Connectivity Configurations / NCC).

Compute Types & Workload Suitability

Databricks organizes compute resources into three primary types based on user personas and workload profiles:

1. All-Purpose Clusters

All-Purpose Clusters are interactive compute resources designed for collaborative development. Multiple data analysts and data scientists can attach interactive notebooks to the same shared cluster.

Supported Languages: Python, SQL, R, Scala.

Typical Use Cases: Exploratory data analysis, ad-hoc data transformations, machine learning prototyping, feature engineering.

Lifecycle: Manually started by users or admins, auto-terminated after a configurable period of inactivity (e.g., 30 minutes).

2. Job Clusters

Job Clusters (also known as Automated Clusters) are created specifically to execute scheduled automated workflows and data pipelines.

Lifecycle: The Databricks Workflow service provisions the job cluster automatically when a scheduled job starts, executes the assigned tasks, and immediately terminates the cluster upon job completion.

Cost Efficiency: Job clusters are billed at a significantly lower Databricks Unit (DBU) rate compared to interactive All-Purpose Clusters.

3. SQL Warehouses

SQL Warehouses (formerly SQL Endpoints) are compute resources specifically optimized for SQL queries, BI dashboards, ad-hoc analytical queries, and external BI tool connections (such as Power BI, Tableau, and ThoughtSpot).

SQL Warehouse Architecture & Sizing

Databricks SQL Warehouses are available in three channels/types: Classic , Pro , and Serverless . +-----------------------------------------------------------------------+

  • | DATABRICKS SQL WAREHOUSE |
  • | |
  • | +---------------------------------------------------------------+ |
  • | | Photon Engine Acceleration | |
  • | +---------------------------------------------------------------+ |
  • | | Cluster Instance Sizing (2X-Small -> 4X-Large T-Shirt Sizes) | |
  • | +---------------------------------------------------------------+ |
  • | | Multi-Cluster Auto-Scaling (Scaling out for Concurrency) | |
  • | +---------------------------------------------------------------+ |
  • +-----------------------------------------------------------------------+
  • T-Shirt Sizing

SQL Warehouses use T-shirt sizing to simplify resource allocation:

  • Size Driver Node / Worker VM Capacity Typical Analytical Workload
  • 2X-Small 1 Small Worker Lightweight queries, dashboard development, low concurrency
  • X-Small 2 Workers Ad-hoc SQL exploration, single-user dashboard building
  • Small 4 Workers Standard team dashboard refresh, moderate aggregations
  • Medium 8 Workers Complex analytical joins, high-volume reporting
  • Large to 4X-Large 16 to 128+ Workers Enterprise data warehouse migration, massive ETL/ELT aggregations
  • Concurrency Scaling & Multi-Cluster Warehouses

When multiple analysts or BI dashboards query a SQL Warehouse simultaneously, query queueing can occur. To handle concurrent execution without performance degradation, SQL Warehouses support multi-cluster auto-scaling :

Min Clusters: The minimum number of cluster instances running continuously (e.g., 1).

Max Clusters: The maximum number of cluster instances Databricks can launch to handle peak concurrency (e.g., 10).

Scaling Behavior: As query volume increases, Databricks automatically launches additional cluster instances of the selected T-shirt size. When concurrency drops, idle clusters are automatically spun down after 15 minutes.

Real-World Scenario: Designing Compute for an Analytics Team

Consider a retail organization with two distinct workload demands:

Morning Executive Dashboard Peak: 150 business analysts open interactive sales dashboards between 8:00 AM and 9:00 AM daily.

Overnight Data Mart Refresh: A scheduled workflow executes at 2:00 AM to process millions of transactions and update analytical summary tables.

Optimal Solution:

For the Executive Dashboard Peak , configure a Serverless SQL Warehouse with T-shirt size Medium and multi-cluster auto-scaling set to Min: 1 and Max: 6 . Serverless compute ensures instantaneous startup at 8:00 AM without keeping idle servers running overnight.

For the Overnight Data Mart Refresh , configure a scheduled Databricks Workflow running on a Job Cluster (or Serverless Job Compute). This ensures lowest DBU billing costs and automatic cluster destruction immediately upon completion.