13.1 Big Data V's & Technologies

DAMA Certified Data Management Professional CDMP

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview & DMBoK Framework

1.1 CDMP Exam Details & Strategy 1.2 The DMBoK Framework 1.3 Data Management Organization & Roles

2 Chapter 2: Data Governance (Part A)

2.1 Data Governance Principles & Drivers 2.2 Operating Models & Structures 2.3 Policies, Standards, and Procedures

3 Chapter 2: Data Governance (Part B)

2.4 Data Stewardship 2.5 Governance Metrics & Implementation

4 Chapter 3: Data Modeling and Design (Part A)

Section 3.1: Conceptual, Logical, and Physical Models Section 3.2: ERD Notations & Relational Modeling Section 3.3: Normalization

5 Chapter 3: Data Modeling and Design (Part B)

3.4 Dimensional Modeling 3.5 Naming Standards & Modeling Best Practices

6 Chapter 4: Data Quality (Part A)

Section 4.1: Data Quality Dimensions Section 4.2: Data Quality Policy & Rules Section 4.3: DMAIC & DQ Improvement Lifecycle

7 Chapter 4: Data Quality (Part B)

4.4 Root-Cause Analysis & Remediation 4.5 DQ Metrics, Tools, and Reporting

8 Chapter 5: Metadata Management (Part A)

5.1 Business, Technical, and Operational Metadata 5.2 Metadata Lineage & Impact Analysis 5.3 Repository Architectures

9 Chapter 5: Metadata Management (Part B)

5.4 Standards and Semantics 5.5 Metadata Governance & Integration

10 Chapter 6: Master and Reference Data Management

6.1 Master vs Reference Data 6.2 MDM Repository Architectures 6.3 Match-Merge & Identity Resolution 6.4 Reference Data Governance & Standards

11 Chapter 7: Data Warehousing and Business Intelligence

7.1 DW Architectures: Inmon vs Kimball 7.2 Data Warehousing Storage & Load Patterns 7.3 BI Delivery & OLAP 7.4 Data Marts & Enterprise Data Warehouses

12 Chapter 8: Data Architecture

8.1 Enterprise Data Architecture & Zachman Framework 8.2 Data Flow Diagrams & Value Chains 8.3 Architecture Frameworks & Standards

13 Chapter 9: Document and Content Management

9.1 Records Management & ECM 9.2 Retention, Disposal, and E-Discovery 9.3 ARMA GARP Principles & Taxonomies

14 Chapter 10: Data Integration and Interoperability

Section 10.1: Integration Patterns & Technologies Section 10.2: Replication & Virtualization Section 10.3: Canonical Models & Integration Standards

15 Chapter 11: Data Security

Section 11.1: Security Principles: CIA Triad & Privacy Section 11.2: Access Control & Encryption Section 11.3: Security Regulations & Standards

16 Chapter 12: Data Storage and Operations

Section 12.1: Database Technologies & DBA Roles Section 12.2: Backup, Recovery, and High Availability Section 12.3: Performance Tuning & Operations

17 Chapter 13: Big Data and Data Science

13.1 Big Data V's & Technologies

18 Chapter 14: Data Ethics

14.1 DMBoK Data Ethics Framework

19 Chapter 15: Data Management Process and Maturity

15.1 Maturity Assessment Frameworks

13.1 Big Data V's & Technologies

Key Takeaways Explain This Topic AI

Big Data is defined by the 5 V's (Volume, Velocity, Variety, Veracity, and Value), which describe datasets that exceed the storage and processing limits of traditional relational database management systems (RDBMS).

Hadoop Distributed File System (HDFS) is a distributed master/slave storage architecture where the NameNode manages metadata and file mappings, while DataNodes store data blocks with a default replication factor of 3 for fault tolerance.

Apache Spark achieves up to 100x faster performance than Hadoop MapReduce for iterative workloads by utilizing in-memory computing and Resilient Distributed Datasets (RDDs) that track lineage to recover from failures.

Data Lakehouse architectures layer transactional frameworks (like Delta Lake or Apache Iceberg) over object storage, providing ACID compliance and schema enforcement to solve the metadata and governance issues of Data Swamps.

The standard Machine Learning (ML) pipeline spans Ingestion, Preparation/Feature Engineering (using feature stores), Model Training, Validation (preventing overfitting), and Deployment/Monitoring (preventing model and concept drift).

Last updated: July 2026

• 13.1 Big Data V's & Technologies
• The Foundations of Big Data: The 5 V's

The concept of Big Data describes datasets whose size, complexity, and rate of generation exceed the storage capacity and processing capabilities of traditional relational database management systems ( RDBMS ). The DAMA Guide to the Data Management Body of Knowledge ( DMBoK2 ) defines Big Data through the lens of five core dimensions, commonly referred to as the 5 V's :

Volume : The physical scale of data, typically ranging from terabytes to petabytes and exabytes. As organizations capture millions of transactions, Internet of Things ( IoT ) sensor readings, and web logs daily, the physical limit of single-node storage is quickly exceeded, necessitating distributed architecture.

Velocity : The speed at which data is generated, ingested, and processed. Rather than relying on traditional nightly batch extraction, transformation, and loading ( ETL ) cycles, modern systems must handle continuous streams of data in real-time or near-real-time (e.g., Kafka messaging feeds).

Variety : The structural range of incoming data formats. Unlike highly structured relational tables, big data encompasses structured data , semi-structured data (e.g., XML, JSON, and YAML configurations), and unstructured data (e.g., emails, PDFs, audio files, video streams, and social media posts).

Veracity : The trustworthiness, quality, and cleanliness of the data. Big data often originates from external, uncontrolled sources (e.g., public weather data, social media sentiment, GPS logs), which introduces noise, anomalies, duplicate entries, and missing values. Establishing veracity requires robust data quality profiling.

Value : The business utility and insights derived from data assets. Collecting large volumes of data is a liability unless the organization can transform it into actionable decisions, predictive analytics, or operational efficiencies.

• Dimension Primary Challenge DMBoK Governance Focus
• Volume Scalability of storage and compute resources Storage tiering, archiving, and lifecycle policies
• Velocity Latency of ingestion and processing pipelines Real-time stream governance and event auditing

Variety Lack of standardized schemas across source feeds Semantic harmonization, metadata tagging, and catalogs

Veracity Noise, bias, anomalies, and source untrustworthiness Data quality profiling, lineage tracing, and source validation

• Value Cost-to-value ratio and alignment with strategy Business case alignment and ROI tracking
• Distributed Storage and Compute: Hadoop vs. Apache Spark

Traditional databases process queries using single-node engines. When data exceeds single-node capacity, organizations must scale horizontally using distributed computing frameworks.

The Hadoop Ecosystem

The Apache Hadoop project introduced the foundational framework for distributed storage and parallel processing. It consists of two primary components:

Hadoop Distributed File System (HDFS) : A distributed file system designed to run on commodity hardware. HDFS splits files into large blocks (typically 128MB or 256MB) and distributes them across a cluster. It employs a master/slave architecture:

NameNode : The master node that maintains the directory tree, file-to-block mappings, and metadata. It represents a single point of failure ( SPOF ) in early Hadoop deployments, though modern setups utilize high-availability active/passive pairs.

DataNodes : Slave nodes that store the actual blocks of data and periodically send heartbeats and block reports to the NameNode. HDFS replicates blocks (default replication factor of 3 ) across different racks to ensure fault tolerance.

MapReduce : A software framework and programming model for processing large datasets in parallel. It splits tasks into two main phases:

Map Phase : Processes raw input data and emits intermediate key-value pairs.

Reduce Phase : Aggregates and consolidates the intermediate data based on keys to produce the final output.

Exam Trap : MapReduce writes intermediate state data directly to local disk drives after the Map phase. This disk write introduces significant Input/Output ( I/O ) latency, making MapReduce inefficient for iterative processes like machine learning.

Apache Spark: In-Memory Processing

Apache Spark was developed to overcome MapReduce's disk-bound limitations by introducing in-memory computing . Spark processes data in random-access memory ( RAM ) rather than writing intermediate outputs to disk.

Resilient Distributed Datasets (RDDs) : Spark's core abstraction. RDDs are read-only, fault-tolerant collections of objects distributed across cluster nodes. They are 'resilient' because they track their lineage (the history of transformations applied to the data), allowing them to automatically reconstruct lost partitions if a node fails.

DataFrames and Spark SQL : Higher-level abstractions that provide a schema to distributed data, allowing developers to query data using SQL syntax and optimized engine plans (via the Catalyst Optimizer).

Compared to MapReduce, Spark can perform iterative machine learning and graph algorithms up to 100 times faster because it eliminates the physical disk I/O bottleneck between pipeline stages.

Architectural Evolutions: Data Lakes and Lakehouses

Data architectures have evolved to accommodate the 5 V's, moving from rigid warehouses to flexible hybrid environments.

Data Lakes

A Data Lake is a centralized repository that stores raw, unstructured, semi-structured, and structured data in its native format.

Schema-on-Read : Unlike traditional Data Warehouses that require schema-on-write (transforming data to fit a strict relational schema before loading), Data Lakes ingest data as-is. Schema structures are only applied when the data is read or queried by an application. This accelerates ingestion speed but transfers the burden of validation to downstream consumers.

Data Swamp Risk : Without strict metadata management, data catalogs, and data governance controls, a Data Lake quickly degenerates into a Data Swamp —an unorganized repository where users cannot find reliable data assets, leading to compliance risks and wasted storage costs.

Lakehouse Architecture

A Lakehouse is a modern data design pattern that combines the cost-effective scalability of a Data Lake with the transactional reliability of a Data Warehouse. It implements a metadata and transaction layer (e.g., Delta Lake, Apache Iceberg, or Apache Hudi) on top of standard cloud object storage (like AWS S3 or Azure ADLS). Key characteristics include:

ACID Transactions : Ensures Atomicity, Consistency, Isolation, and Durability for concurrent read and write operations, preventing partial updates and data corruption.

Schema Enforcement and Evolution : Prevents bad data from corrupting the table by validating inserts against a target schema, while allowing controlled schema updates over time.

Time Travel : Enables users to query historical snapshots of data using transaction logs, which is vital for compliance audits and model retraining.

The Machine Learning Pipeline

Data Science relies on structured workflows to build, train, deploy, and monitor predictive models. The standard Machine Learning (ML) Pipeline comprises five critical stages:

Data Ingestion and Sourcing : Extracting raw structured, semi-structured, and unstructured data from source systems (APIs, logs, databases) and loading it into the landing zone of the data lake.

Data Preparation and Feature Engineering : The most time-consuming stage. Data scientists clean missing values, normalize numerical scales, and transform raw variables into features (inputs optimized for model algorithms). Features are cataloged in a feature store to promote reuse and consistency across training and serving environments.

Model Training : Feeding historical feature data into machine learning algorithms (e.g., random forests, neural networks) to allow the model to learn mathematical relationships and coefficients.

Model Evaluation and Validation : Testing model performance using unseen test datasets to verify accuracy, precision, recall, and F1-score. This prevents overfitting (where a model performs well on training data but poorly on real-world inputs).

Model Deployment and Monitoring (MLOps) : Serving the trained model as an API endpoint for real-time predictions or integrating it into batch systems. Continuous monitoring is essential to detect model drift (performance degradation due to changing real-world patterns) and concept drift (changes in the underlying definition of target variables).

Important

CDMP Exam Tip : Understand the difference between Schema-on-Write (traditional Data Warehouses) and Schema-on-Read (Data Lakes). The CDMP exam often tests this distinction, particularly how it affects storage costs, ingest speed, and the necessity of data catalogs to prevent Data Swamps.