Study guide
6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
Key Takeaways Explain This Topic AI
Table and column descriptions are set with COMMENT ON TABLE and ALTER TABLE ... ALTER COLUMN ... COMMENT, and they are the single highest-leverage input to Catalog Explorer search, lineage readability, and AI/BI Genie accuracy.
A Genie Agent (formerly a Genie Space) draws on Unity Catalog table and column metadata, primary and foreign key relationships, general instructions, example SQL queries, SQL functions, and knowledge store context.
Trusted assets are parameterized example SQL queries and Unity Catalog SQL functions whose logic an author has verified; when Genie answers from one, the response is marked as verified.
Each Genie Agent enforces two separate limits: 100 instructions, where each example query, each SQL function, and the entire general instructions block counts as one, and 200 knowledge store snippets covering table descriptions, join relationships, and SQL expressions.
Genie respects primary and foreign keys declared in Unity Catalog, but any join it cannot infer must be defined manually with an explicit cardinality so queries do not double-count or explode row counts.
Last updated: August 2026
6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
DP-750 Exam Focus: Two blueprint bullets converge here - "Create, implement, and preserve table and column definitions and descriptions for data discovery" from skill area 2, and "Configure AI/BI Genie instructions for data discovery" from skill area 1. They are the same problem viewed twice: a human searching Catalog Explorer and a natural-language agent generating SQL both depend on the metadata you write.
1. Descriptions Are Governance Metadata, Not Documentation
A table with no description is invisible to search, ambiguous in lineage graphs, and unusable by Genie. Unity Catalog exposes descriptions at three levels: -- Catalog and schema level
- COMMENT ON CATALOG prod_retail IS 'Production retail lakehouse. Owner: retail-data-platform.';
- COMMENT ON SCHEMA prod_retail.gold IS 'Curated, business-ready aggregates. SLA: 06:00 UTC daily.';
- - Table level: state the grain, the SLA, and the caveats
- COMMENT ON TABLE prod_retail.gold.store_sales_daily IS
'Grain: one row per store_id per business_date. Revenue is net of returns.
Restated for 7 days after business_date to absorb late-arriving sales.';
- - Column level
- ALTER TABLE prod_retail.gold.store_sales_daily
- ALTER COLUMN net_revenue_usd COMMENT 'Gross sales minus returns and discounts, USD, DECIMAL(18,2).';
- ALTER TABLE prod_retail.gold.store_sales_daily
- ALTER COLUMN business_date COMMENT 'Store local calendar date. Not the UTC ingest date.';
- - Descriptions can also be set inline at creation time
- CREATE TABLE prod_retail.silver.customers (
- customer_id BIGINT COMMENT 'Surrogate key. Stable across source system migrations.',
- email_address STRING COMMENT 'Lowercased. Masked for non-privileged readers.',
- created_at TIMESTAMP COMMENT 'UTC. First appearance in the CRM source.'
- ) COMMENT 'Grain: one row per current customer. SCD Type 1 overwrite.';
- What a Good Description Contains
- Include Example
- The grain "one row per store per business_date"
- Units and precision "USD, DECIMAL(18,2)", "milliseconds since epoch"
- Time semantics "store local date, not UTC ingest date"
- Restatement window "restated for 7 days after business_date"
- Known exclusions "excludes cancelled and test orders"
Avoid descriptions that restate the column name ( customer_id -> "the customer id"). They consume the reader's attention and teach Genie nothing.
Preserving Descriptions Across Rebuilds
The word preserve in the blueprint bullet matters. A CREATE OR REPLACE TABLE rewrites the table definition and drops comments that were applied afterwards with ALTER . Two defensive patterns:
Declare comments in the DDL itself (as in the CREATE TABLE above) so they are part of the table definition that the pipeline recreates.
Declare them in Lakeflow Spark Declarative Pipelines with the comment argument on the table decorator or the COMMENT clause in SQL, so the pipeline is the source of truth and every full refresh restores them.
Tags Complement, but Do Not Replace, Descriptions
Section 6.3 covered tag governance and Section 5.2 covered ABAC. The division of labor:
Descriptions are free text for humans and for Genie. They are not enforceable.
Tags are structured key-value pairs that ABAC policies, row filters, and column masks can act on programmatically. ALTER TABLE prod_retail.silver.customers
SET TAGS ('data_domain' = 'customer', 'contains_pii' = 'true', 'certified' = 'gold');
Catalog Explorer surfaces both: descriptions power keyword search and the object detail pane, tags power faceted filtering and drive ABAC enforcement.
2. AI/BI Genie: What It Reads
An AI/BI Genie Agent (formerly called a Genie Space) is a natural-language interface over a curated set of Unity Catalog tables. It converts a business question into SQL, runs it, and returns the result. Its accuracy is a direct function of the context you give it:
- Context source What Genie uses it for
- Unity Catalog table metadata Table names, descriptions, and declared primary key / foreign key relationships
- Column names and descriptions Genie filters for relevant columns and reads their descriptions
Knowledge store context Agent-local metadata the author adds; does not alter the underlying Unity Catalog metadata
- Example SQL queries Reference answers Genie selects from when a prompt resembles one
- SQL functions Unity Catalog functions added to the agent, used as verified logic
- General instructions Plain-text notes providing global context
Benchmarks Evaluation only - test questions used to measure accuracy. Genie does not learn from benchmarks.
- 3. Configuring Genie Instructions
- General Instructions
A single plain-text block giving global context: what the business does, what an "active customer" means, which fiscal calendar applies. Databricks guidance is to use them sparingly and to prefer more precise mechanisms:
Write specific instructions. "When users ask about sales metrics without specifying product name or sales channel, ask them to specify both" beats "ask clarifying questions about sales."
Describe the business narrative - entities, lifecycles, relationships - rather than dictating SQL behavior. Do not force table selection, hardcode filters, or specify output formatting here.
Never use general instructions to paper over missing table descriptions, join definitions, or example queries.
Keep them consistent with every other instruction type. If the text says round to two decimals, the example queries must round to two decimals too.
Example SQL Queries
The highest-value instruction type. Provide complete, correct queries for prompts that are hard to interpret, multi-part, or involve intricate joins - "break down my team's performance", "for recently joined customers, which products perform best". Genie matches a user prompt to a relevant example and learns the pattern for related questions. You can attach usage guidance explaining when a given example is especially relevant.
SQL Functions and Trusted Assets
Trusted assets are the subset of instructions whose logic has been verified by an author, so that when Genie answers from one the response is presented as a verified answer :
Parameterized example SQL queries - when the exact text of a parameterized query generates the response, users can edit the parameter value and rerun.
SQL functions registered in Unity Catalog - the right home for a calculation that must always be performed the same way (a churn definition, a margin formula).
Permissions: a user needs at least CAN EDIT on the agent to add or remove trusted assets, and agent users need EXECUTE on any SQL function used as one. -- Codify a metric once, then register it as a trusted asset in the Genie Agent
- CREATE OR REPLACE FUNCTION prod_retail.gold.net_margin_pct(
- revenue DECIMAL(18,2), cogs DECIMAL(18,2))
- RETURNS DECIMAL(9,4)
- COMMENT 'Net margin percent = (revenue - cogs) / revenue. Returns NULL when revenue is zero.'
- RETURN CASE WHEN revenue = 0 THEN NULL ELSE (revenue - cogs) / revenue END;
- GRANT EXECUTE ON FUNCTION prod_retail.gold.net_margin_pct TO analytics_readers;
- Knowledge Store: Descriptions, Joins, and SQL Expressions
The knowledge store holds agent-local semantic context that does not modify Unity Catalog:
Table and column descriptions scoped to this agent, for when the shared catalog description is too generic.
Join relationships - Genie respects primary and foreign keys declared in Unity Catalog, but any missing link must be defined manually. Always state the cardinality (one-to-one, one-to-many, many-to-many); without it Genie can generate joins that explode row counts and double-count metrics.
SQL expressions - reusable measures, filters, and dimensions that encode business semantics.
Value dictionaries / example values for categorical columns, so Genie can match "Australia" to the stored code AUS instead of guessing.
The Two Limits
Each Genie Agent enforces two separate caps:
Limit Count What consumes it
Instructions 100 per agent Each example SQL query, each SQL function, and the entire general instructions text block each count as one
Knowledge store snippets 200 per agent Table descriptions, join relationships, and SQL expressions (measures, filters, dimensions) share this pool
More instructions is not better. Databricks warns that too many instructions reduce effectiveness, especially in longer conversations, because Genie struggles to prioritize.
Benchmarks
A benchmark is a set of test questions with expected SQL, used only to evaluate accuracy. Genie does not use benchmark questions or their SQL as context, so adding a benchmark never improves an answer - it only measures whether your instructions did.
4. A Working Order of Operations
Point the agent at gold-layer tables. Bronze and silver usually lack curated comments and dimensional joins, so accuracy drops.
Make sure every included table and column has a real Unity Catalog description .
Declare primary and foreign keys in Unity Catalog; add any remaining joins in the knowledge store with cardinality .
Add value dictionaries for categorical columns.
Add example SQL queries for the hard questions, and SQL functions for calculations that must never vary.
Add a short, specific general instructions block last, only for what nothing above covers.
Build a benchmark and measure.
5. Exam Traps
Benchmarks do not train Genie. They evaluate it.
Knowledge store context is agent-local. It does not rewrite Unity Catalog metadata, so two agents can describe the same table differently.
A missing join is not fixed by a general instruction. Define the relationship and its cardinality.
Tags are not descriptions. ABAC policies act on tags; Genie and Catalog Explorer search read descriptions.
CREATE OR REPLACE TABLE silently drops comments applied later with ALTER . Declare them in the DDL or the pipeline definition.