Study guide
10.4 Monitoring, Benchmarks, Feedback & Maintenance
Databricks Certified Data Analyst Associate Exam
Track your progress as you read
Finished sections get a check mark.
1 Platform Overview & Unity Catalog Foundations
1.1 Databricks Architecture & Compute Concepts 1.2 Databricks Marketplace & Data Sharing Basics 1.3 Unity Catalog Namespace & Object Hierarchy 1.4 Governed Asset Discovery & System Tables
2 Managing & Cleaning Governed Data
2.1 Dataset Discovery & Asset Tagging 2.2 Lineage Inspection & Data Traceability 2.3 SQL Data Cleaning & Transformation Techniques
3 Data Ingestion & Import Patterns
3.1 Workspace Uploads & Cloud File Ingestion 3.2 Incremental File Ingestion with Auto Loader 3.3 Ingesting Data via Delta Sharing & External Connections
4 Databricks SQL Warehouses & Core Querying
4.1 Databricks SQL Interface & SQL Assistant 4.2 SQL Warehouses Sizing & Management 4.3 Table Creation & Data Modification in Databricks SQL
5 Advanced SQL Authoring & Time Travel
5.1 Advanced Aggregations, Window Functions & Analytics 5.2 Complex Joins, CTEs & Set Operations 5.3 Views, Materialized Views & Delta Time Travel
6 Query Performance Analysis & Photon Engine
6.1 Photon Engine Architecture & Acceleration 6.2 Query History & Execution Profile Inspection 6.3 Delta Lake Audit History & Commit Logs
7 Liquid Clustering, Caching & Performance Troubleshooting
7.1 Caching Hierarchy in Databricks SQL 7.2 Data Layout Optimization: Liquid Clustering vs Partitioning & Z-Ordering 7.3 Troubleshooting Slow & Failing Queries
8 AI/BI Dashboards & Visualization Design
8.1 AI/BI Dashboards Overview & Canvas Layout 8.2 Chart Types & Visualization Selection 8.3 Notebook & SQL Editor Visualizations
9 Dashboard Interactivity, Publishing & Alerts
9.1 Dashboard Parameters & Interactive Filtering 9.2 Publishing, Sharing & Access Control 9.3 Scheduled Refresh, Email Subscriptions & Alerts
10 Developing & Maintaining AI/BI Genie Spaces
10.1 AI/BI Genie Architecture & Space Setup 10.2 Curation via Instructions, Descriptions & Sample Questions 10.3 Trusted Assets & Parameterized Functions 10.4 Monitoring, Benchmarks, Feedback & Maintenance
11 Data Modeling & Medallion Architecture
11.1 Dimensional Modeling Fundamentals 11.2 Medallion Architecture for Analytics 11.3 Constraint Enforcement & Relationships in Unity Catalog
12 Securing Data & Unity Catalog Governance
12.1 Unity Catalog Privilege Model & Grants 12.2 Ownership, Inheritance & Access Control Patterns 12.3 Fine-Grained Security: Row Filters & Column Masks
10.4 Monitoring, Benchmarks, Feedback & Maintenance
Key Takeaways Explain This Topic AI
Genie Benchmarks enable analysts to create test suites of natural language questions and gold-standard SQL to evaluate accuracy across space updates.
After Unity Catalog schema or comment changes, refresh Genie metadata, then retune instructions/trusted assets and re-run benchmarks.
User feedback mechanisms—such as thumbs up/down ratings and SQL edits—provide direct diagnostic signals for identifying unhandled business terms.
Unity Catalog system tables (system.access.audit) retain 365 days of audit telemetry to track Genie space usage, user sessions, and warehouse compute costs.
Continuous space maintenance follows an iterative loop: review unhandled queries -> update instructions/descriptions -> run benchmarks -> publish updates.
Last updated: July 2026
10.4 Monitoring, Benchmarks, Feedback & Maintenance
Deploying an AI/BI Genie space is not a one-time event; it is an ongoing lifecycle that requires continuous monitoring, evaluation, and refinement. As business definitions evolve, new datasets are ingested, and user query patterns shift, data analysts must actively maintain Genie spaces to preserve answer quality, detect regression, and optimize warehouse compute costs. This section details the tools and procedures required for operational excellence, including Genie Benchmarks , User Feedback Loops , and System Table Telemetry .
The AI/BI Genie Maintenance Lifecycle
A healthy Genie space relies on a structured, four-stage feedback and maintenance cycle: +-------------------------------------------------+
- | 1. Deploy & Monitor |
- | - Track query logs via System Tables |
- | - Monitor SQL Warehouse usage and costs |
- +-------------------------------------------------+
- |
- v
- +-------------------------------------------------+
- | 2. Collect Feedback |
- | - Review Thumbs Up / Thumbs Down ratings |
- | - Analyze user-edited SQL queries & comments |
- +-------------------------------------------------+
- |
- v
- +-------------------------------------------------+
- | 3. Refine & Benchmark |
- | - Update Instructions, Descriptions & Assets |
- | - Execute automated Benchmark Evaluation suites|
- +-------------------------------------------------+
- |
- v
- +-------------------------------------------------+
- | 4. Publish & Iterate |
- | - Release space updates to end users |
- | - Verify resolution of unhandled questions |
- +-------------------------------------------------+
- User Feedback Collection & Query Audit Logs
Genie incorporates native feedback tools directly into the conversational chat interface:
Micro-Feedback Signals
Thumbs Up / Thumbs Down : Users click feedback icons on response cards. A thumbs-down rating alerts space editors that an answer was incomplete, incorrect, or poorly formatted.
User SQL Edits : Business users or analysts with edit privileges can inspect generated SQL and modify it directly in the chat interface. When a user manually corrects generated SQL, Genie records the edit as a high-value signal for space refinement.
Diagnostic Analysis of Unhandled Queries
Space managers should regularly inspect the History Log tab within Genie Space Management to review unhandled or failed queries. Common failure patterns include:
Missing Business Vocabulary : User asked for a metric using jargon not documented in Space Instructions.
Missing Schema Join Path : Genie attempted to join two tables that lacked declared primary/foreign keys or explicit relationship instructions.
Complex Aggregation Failure : The query required advanced analytics (e.g., cohort retention) better suited for a Trusted Asset .
Building and Running Genie Benchmarks
To prevent accuracy regression when editing space instructions or adding tables, analysts construct Genie Benchmarks . A benchmark is an automated test suite consisting of pairs of natural language questions and verified Gold-Standard SQL statements.
Benchmark Evaluation Workflow
Define Test Suite : Assemble 20 to 50 representative natural language questions covering core business metrics.
Attach Gold-Standard SQL : For each question, author and verify the exact ANSI SQL query expected from the system.
Run Evaluation : Execute the benchmark suite against the Genie space. Genie generates SQL for each question and compares the output tables against the gold-standard results.
Review Accuracy Score : The evaluation returns an overall accuracy percentage and highlights specific queries where generated SQL diverged from the gold standard. -- Benchmark Test Case Example
-- Question: "What is our average order value (AOV) for Q1 2026?"
-- Gold-Standard SQL:
- SELECT
- AVG(net_amount) AS average_order_value
- FROM main.sales_gold.fact_orders
- WHERE order_date >= '2026-01-01' AND order_date < '2026-04-01'
- AND order_status = 'COMPLETED';
Running benchmarks before publishing space changes ensures that modifying an instruction to fix one query does not inadvertently break three others.
Monitoring Space Usage with System Tables
Databricks captures full operational telemetry for Genie interactions in Unity Catalog System Tables . Analysts can query system.access.audit and operational views to monitor usage trends, active users, and execution performance. -- Querying system audit logs to monitor AI/BI Genie space interactions
- SELECT
- event_time,
- user_identity.email AS user_email,
- request_params.space_id AS genie_space_id,
- action_name,
- service_name,
- response.status_code
- FROM system.access.audit
- WHERE service_name = 'genie'
- AND event_date >= CURRENT_DATE() - INTERVAL 30 DAYS
- ORDER BY event_time DESC;
- Key Metrics to Track via System Tables
Active Monthly Users (MAU) : Volume of distinct business users interacting with the space.
Query Latency : Execution duration on the underlying SQL Warehouse.
Cost Allocation : Compute DBUs consumed by Genie queries on assigned SQL Warehouses.
Refreshing Unity Catalog Metadata After Schema Changes
When underlying Unity Catalog tables gain columns, updated comments, renamed fields, or revised descriptions, Genie’s understanding can drift until metadata is refreshed. The exam guide calls out refreshing Unity Catalog metadata as part of maintaining Genie spaces.
Why refresh matters
Genie relies on table/column names, comments, synonyms, and curated instructions to generate SQL.
Stale metadata produces wrong joins, missing filters, or hallucinated columns after upstream schema evolution.
Certification and tagging changes on source tables also affect which assets should remain in the space.
Maintenance practice
After upstream schema or comment changes, open the Genie space dataset configuration and refresh metadata so Genie re-reads Unity Catalog definitions.
Re-run benchmarks and spot-check popular questions after the refresh.
Update instructions, sample questions, and trusted assets when business definitions changed — not only when physical columns changed.
Remove deprecated tables (for example, system.certification_status=deprecated ) from the space curation list. UC schema/comment change → Refresh Genie metadata → Retune instructions/trusted assets → Re-run benchmarks → Monitor feedback
Iterative Curation & Continuous Improvement Workflow
Space maintainers should establish a weekly maintenance schedule to process feedback and publish space updates:
Maintenance Task Frequency Target Objective Action Items
Review Thumbs-Down Queries Weekly Identify instruction gaps Add acronym definitions or metric formulas to Space Instructions.
Inspect SQL Edits Weekly Capture user corrections Convert recurring manual SQL edits into Sample Questions.
Run Benchmark Suite Bi-Weekly Verify space accuracy Run full evaluation suite; resolve any accuracy score regressions.
Audit Table Documentation Monthly Maintain catalog alignment Ensure newly added Unity Catalog columns have explicit descriptions.
Optimize Compute Monthly Control DBU expenditure Adjust SQL Warehouse auto-stop timeouts and cluster scaling limits.