Study guide
5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags
Key Takeaways Explain This Topic AI
Attribute-Based Access Control (ABAC) replaces complex, brittle Role-Based Access Control (RBAC) matrices by dynamically evaluating user attributes against metadata classification tags.
Unity Catalog tags can be applied at the catalog, schema, table, volume, and column levels using Catalog Explorer or ANSI SQL DDL statements.
Tag-based masking policies bind masking functions directly to column classification tags (e.g., pii_type = 'ssn'), automatically enforcing protection on newly created or tagged columns.
Tag inheritance allows schemas and tables to inherit governance classifications from parent securables, simplifying compliance audits across large lakehouses.
The system.information_schema views (table_tags, column_tags, schema_tags) provide SQL-queryable auditability of all classification metadata across the metastore.
Last updated: August 2026
5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags
DP-750 Exam Focus: Understand how Attribute-Based Access Control (ABAC) scales data governance in Azure Databricks. Master the DDL syntax for applying metadata tags ( SET TAGS , UNSET TAGS ) across catalogs, schemas, tables, and columns. Learn how tag-based masking policies automatically protect sensitive columns across the entire metastore, how policy inheritance operates, and how to query system.information_schema to audit classification compliance.
1. RBAC vs. ABAC: The Governance Scalability Challenge
As enterprise lakehouses expand to house thousands of tables and petabytes of data, traditional Role-Based Access Control (RBAC) encounters severe operational bottlenecks:
Role Explosion: Organizations end up creating hundreds of bespoke roles (e.g., us_finance_analyst_masked , eu_marketing_lead_unmasked ) to handle subtle combinations of access needs.
Manual Schema Maintenance: Whenever a data engineer creates a new table containing Personally Identifiable Information (PII), security administrators must manually write and apply individual ALTER COLUMN ... SET MASK statements on that specific table.
High Risk of Human Error: If a developer forgets to apply a column mask to a newly created table in a bronze or silver schema, sensitive data remains exposed until the next compliance audit.
Attribute-Based Access Control (ABAC) solves these challenges by decoupling access policies from individual table schemas. In ABAC, security decisions are determined dynamically at query runtime by comparing subject attributes (such as the user's group, department, or location) against object attributes (such as metadata classification tags applied to catalogs, tables, or columns). +-----------------------------------------------------------------------------------------+
- | RBAC VS. ABAC ARCHITECTURE COMPARISON |
- +-----------------------------------------------------------------------------------------+
- | |
- | TRADITIONAL RBAC (Role-Based): |
- | [ User ] ---> ( Assigned Role ) ---> [ Explicit GRANT on Specific Table/Column ] |
- | * Bottleneck: Must explicitly apply masks and permissions to thousands of columns. |
- | |
- | MODERN ABAC (Attribute-Based): |
- | [ User Attributes ] <--- Evaluated at Query Runtime ---> [ Securable Object Tags ] |
- | (Group: 'compliance') (Column Tag: pii='ssn') |
- | | |
- | v |
- | [ Automated Tag Masking Policy ] |
- | - Applied globally across entire metastore |
- | - Protects any column tagged 'ssn' automatically |
- +-----------------------------------------------------------------------------------------+
- 2. Unity Catalog Tagging Taxonomy & Mechanics
Unity Catalog supports applying metadata tags to any securable object within the 3-level namespace hierarchy: Catalogs , Schemas , Tables , Views , Volumes , and Columns .
Tag Structure: Key-Value Tags and Tag-Only Labels
Key-Value Tags: Composed of a key and a value string (e.g., 'environment' = 'production' , 'pii_classification' = 'restricted' ). Used for granular policy evaluation and detailed metadata categorization.
Tag-Only Labels (Simple Tags): Contain only a key with no value or an empty string (e.g., 'deprecated' , 'gdpr_article_9' ).
- ANSI SQL DDL Syntax for Tag Management
- 1. Tagging Catalogs and Schemas -- Apply classification and ownership tags to a catalog
- ALTER CATALOG prod_finance
- SET TAGS (
- 'data_tier' = 'enterprise_curated',
- 'compliance_scope' = 'sox_pci',
- 'owner_team' = 'finance_data_eng'
- );
- - Apply confidentiality tags to a schema
- ALTER SCHEMA prod_finance.payroll
- SET TAGS (
- 'data_classification' = 'highly_confidential',
- 'retention_days' = '2555'
- );
- 2. Tagging Tables and Views -- Apply multiple tags to an existing Delta table
- ALTER TABLE prod_finance.payroll.employee_salaries
- SET TAGS (
- 'contains_pii' = 'true',
- 'data_domain' = 'human_resources',
- 'governance_review' = 'passed_2026_q3'
- );
- - Remove a tag from a table
- ALTER TABLE prod_finance.payroll.employee_salaries
- UNSET TAGS ('governance_review');
- 3. Tagging Individual Columns -- Tagging sensitive columns with specific PII classification types
- ALTER TABLE prod_finance.payroll.employee_salaries
- ALTER COLUMN national_insurance_number
- SET TAGS ('pii_type' = 'ssn', 'sensitivity' = 'critical');
- ALTER TABLE prod_finance.payroll.employee_salaries
- ALTER COLUMN work_email
- SET TAGS ('pii_type' = 'email', 'sensitivity' = 'moderate');
- ALTER TABLE prod_finance.payroll.employee_salaries
- ALTER COLUMN bank_iban
- SET TAGS ('financial_data' = 'iban', 'sensitivity' = 'critical');
- 4. Applying Tags during Table Creation CREATE TABLE prod_finance.payroll.contractor_payments (
- contractor_id STRING NOT NULL,
- tax_id STRING NOT NULL COMMENT 'National Tax Identifier' WITH TAGS ('pii_type' = 'tax_id'),
- routing_number STRING WITH TAGS ('financial_data' = 'routing_number'),
- payment_amount DECIMAL(18,2)
- )
- SET TAGS ('business_domain' = 'procurement');
- 3. Data Classification & PII Governance Frameworks
Organizations implementing Unity Catalog ABAC establish a standardized Classification Taxonomy . This taxonomy guarantees consistent terminology across data engineering, security, and legal teams.
Enterprise Data Classification Matrix
Classification Tier Sensitivity Description Example Data Elements Mandatory Governance Controls
Tier 1: Public Data intended for public consumption; zero confidentiality impact. Marketing collateral, published product catalogs, public press releases. Standard SELECT grants; no masking required.
Tier 2: Internal Non-sensitive internal operational data; low confidentiality risk. Department codes, internal warehouse IDs, aggregated operational KPIs. Restricted to authenticated enterprise users; no column masks.
Tier 3: Confidential Sensitive business data; unauthorized disclosure could cause financial or reputational harm. Sales pipeline forecasts, supplier cost models, contractor billing rates. Restrictive RBAC; table-level auditing enabled.
Tier 4: Restricted / PII / PHI Highly sensitive personal, medical, or financial information subject to regulatory compliance (GDPR, HIPAA, PCI-DSS, CPRA). Social Security Numbers (SSN), credit card numbers, IBANs, medical diagnoses, personal passwords. Mandatory ABAC Tagging ( pii_type ), automated column masking, row filtering, and access logging.
4. Automated Tag-Based Masking Policies
The most powerful capability of Unity Catalog ABAC is Tag-Based Column Masking . Instead of binding a masking function to a specific table column, security administrators bind the masking function to a Metadata Tag Key or Key-Value Pair . TAG-BASED MASKING ARCHITECTURE
1. Define Central Mask Function:
CREATE FUNCTION ssn_tag_mask(val STRING) RETURNS STRING ...
- |
- v
2. Create Tag-Based Masking Policy:
- Bind ssn_tag_mask() to Tag: [ pii_type = 'ssn' ]
- |
- v
- +-------------------------------+-------------------------------+
- | | |
- v v v
- [ Table: customers ] [ Table: employees ] [ Table: applicants ]
- Column: ssn Column: tax_num Column: ssn_identifier
- Tag: pii_type = 'ssn' Tag: pii_type = 'ssn' Tag: pii_type = 'ssn'
- | | |
- +-------------------------------+-------------------------------+
- |
- v
- [ AUTOMATIC ENFORCEMENT: All three columns masked across all queries without table DDL ]
- Step-by-Step Implementation of Tag-Based Masking
Step 1: Author the Governance Mask Function -- Create the generic mask function in the central governance schema
- CREATE OR REPLACE FUNCTION prod.governance.generic_ssn_mask(val STRING)
- RETURNS STRING
- LANGUAGE SQL
- DETERMINISTIC
- RETURN
- CASE
- WHEN is_account_group_member('data_security_officers') THEN val
- WHEN is_account_group_member('payroll_auditors') THEN val
- WHEN val IS NULL THEN NULL
- ELSE concat('***-**-', right(regexp_replace(val, '[^0-9]', ''), 4))
- END;
- Step 2: Bind the Mask Function to the Classification Tag (Catalog Explorer or Policy API)
In Unity Catalog, tag-based policies are registered such that any column possessing the tag 'pii_type' = 'ssn' automatically inherits the generic_ssn_mask policy.
Step 3: Automated Enforcement on Schema Evolution
When data engineers ingest new datasets or add columns to existing tables, they simply apply the classification tag: -- Data engineer creates a brand new table
- CREATE TABLE prod.marketing.lead_capture (
- lead_id BIGINT,
- full_name STRING,
- tax_id STRING WITH TAGS ('pii_type' = 'ssn'),
- email_address STRING WITH TAGS ('pii_type' = 'email')
- );
-- Result: The tax_id column is INSTANTLY masked for unauthorized users.
-- The security team never had to write an ALTER TABLE statement.
5. Policy Conflict Resolution & Evaluation Precedence
When multiple security policies and tags intersect on a single table or column, Unity Catalog evaluates access rules according to strict, deterministic Precedence Rules : +-----------------------------------------------------------------------------------------+
- | POLICY RESOLUTION PRECEDENCE HIERARCHY |
- +-----------------------------------------------------------------------------------------+
- | |
- | HIGHEST PRECEDENCE |
- | [ 1. Direct Column Mask (ALTER TABLE ... ALTER COLUMN ... SET MASK) ] |
- | - If a column has an explicit, direct mask assigned via DDL, it completely |
- | overrides any tag-based masking policies. |
- | |
- | [ 2. Tag-Based Masking Policy on Column Tag ] |
- | - Evaluates policies attached to tags explicitly placed on the column. |
- | |
- | [ 3. Tag-Based Policy on Inherited Parent Tags (Table -> Schema -> Catalog) ] |
- | - Evaluates policies matching tags inherited from parent securable hierarchy. |
- | |
- | LOWEST PRECEDENCE (No Policy: Full Unmasked Value Returned if user has SELECT) |
- +-----------------------------------------------------------------------------------------+
Exam Trap: If a column has both an explicit direct column mask (applied via ALTER COLUMN SET MASK ) AND an ABAC tag-based policy (applied via pii_type = 'ssn' ), Unity Catalog always executes the direct column mask and ignores the tag-based policy. Always remove explicit column masks if you intend to standardize on tag-based policies.
6. Auditing & Compliance Monitoring with System Information Schema
Unity Catalog exposes all tagging metadata through SQL-queryable system views located in system.information_schema . Security officers and data engineers can write automated compliance queries to discover unmasked PII, verify tag coverage, and monitor data governance health. INFORMATION SCHEMA TELEMETRY
- +-------------------------------------------------------------------------+
- | system.information_schema Catalog |
- +-------------------------------------------------------------------------+
- | |
- | +--------------------+ +--------------------+ +--------------------+ |
- | | column_tags | | table_tags | | schema_tags | |
- | | (Column-level tags)| | (Table-level tags) | | (Schema-level tags)| |
- | +--------------------+ +--------------------+ +--------------------+ |
- +-------------------------------------------------------------------------+
- Essential Governance Audit Queries
- 1. Identify All Columns Tagged as PII Across the Entire Metastore SELECT
- catalog_name,
- schema_name,
- table_name,
- column_name,
- tag_name,
- tag_value
- FROM system.information_schema.column_tags
- WHERE tag_name = 'pii_type'
- ORDER BY catalog_name, schema_name, table_name;
- 2. Detect Untagged Tables in Production Schemas (Compliance Gap Analysis) SELECT
- t.table_catalog,
- t.table_schema,
- t.table_name
- FROM system.information_schema.tables t
- LEFT JOIN system.information_schema.table_tags tg
- ON t.table_catalog = tg.catalog_name
- AND t.table_schema = tg.schema_name
- AND t.table_name = tg.table_name
- WHERE t.table_catalog = 'prod_enterprise'
- AND t.table_type = 'MANAGED'
- AND tg.tag_name IS NULL;
3. Audit Tables Tagged as Containing PII but Lacking Column Classification -- Find tables labeled as contains_pii = 'true' where no individual columns are tagged
- SELECT
- tt.catalog_name,
- tt.schema_name,
- tt.table_name
- FROM system.information_schema.table_tags tt
- WHERE tt.tag_name = 'contains_pii' AND tt.tag_value = 'true'
- AND NOT EXISTS (
- SELECT 1
- FROM system.information_schema.column_tags ct
- WHERE ct.catalog_name = tt.catalog_name
- AND ct.schema_name = tt.schema_name
- AND ct.table_name = tt.table_name
- );