Study guide
5.2 Healthcare Data Collection Methods & Architectures
AHIMA Certified Health Data Analyst CHDA
Track your progress as you read
Finished sections get a check mark.
1 Introduction & CHDA Exam Overview
0.1 About the AHIMA CHDA Exam 0.2 How to Use This Study Guide
2 Chapter 1: Healthcare Delivery Systems & Revenue Cycle
1.1 Healthcare Delivery Settings & Organizational Structures 1.2 Healthcare Revenue Cycle Management & Reimbursement Models 1.3 Regulatory Reporting & Accreditation Agencies
3 Chapter 2: Health Informatics, Datasets & AI
2.1 Electronic Health Records (EHR) & Health Informatics Foundations 2.2 Healthcare Datasets, Benchmarks & Population Health 2.3 Analytics Hierarchy & Artificial Intelligence in Healthcare
4 Chapter 3: Clinical Terminologies, Classifications & Standards
3.1 Clinical Classification Systems: ICD-10-CM/PCS & CPT/HCPCS 3.2 Clinical Terminologies & Ontologies: SNOMED CT, LOINC & RxNorm 3.3 Interoperability & Data Exchange Standards: HL7 v2, CDA & FHIR
5 Chapter 4: Business Needs Assessment & Scoping
4.1 Stakeholder Engagement & Analytics Needs Elicitation 4.2 Defining Metrics, KPIs & Healthcare Value 4.3 Analytics Project Management & Analysis Planning
6 Chapter 5: Data Sourcing, Lineage & Collection Methods
5.1 Healthcare Data Sourcing & Lineage 5.2 Healthcare Data Collection Methods & Architectures 5.3 Clinical & Financial Data Extraction
7 Chapter 6: Data Cleansing, Transformation & Validation
6.1 Healthcare Data Quality Assessment 6.2 Data Cleansing & Anomaly Remediation 6.3 Data Transformation, Mapping & Validation
8 Chapter 7: Exploratory Data Analysis & Descriptive Statistics
7.1 Healthcare Data Types & Scales of Measurement 7.2 Measures of Central Tendency & Dispersion 7.3 Exploratory Data Analysis in Clinical Populations
9 Chapter 8: Inferential Statistics & Hypothesis Testing
8.1 Sampling Techniques & Probability in Healthcare 8.2 Hypothesis Testing Fundamentals 8.3 Parametric vs. Non-Parametric Tests
10 Chapter 9: Advanced Statistical Modeling & Risk Adjustment
9.1 Correlation & Regression Analysis 9.2 Risk Adjustment & Healthcare Severity Modeling 9.3 Survival Analysis & Model Evaluation
11 Chapter 10: Healthcare Epidemiology & Study Methodologies
10.1 Epidemiological Measures of Disease Frequency & Association 10.2 Healthcare Study Designs & Methodologies 10.3 Confounding, Bias & Health Equity Analytics
12 Chapter 11: Healthcare Data Visualization & Dashboard Design
11.1 Principles of Effective Healthcare Data Visualization 11.2 Selecting Appropriate Healthcare Visualizations 11.3 Executive & Clinical Dashboard Architecture
13 Chapter 12: Healthcare Performance Measurement & CMS Quality Programs
12.1 CMS Quality Programs & Value-Based Purchasing 12.2 Clinical Quality Measures & Benchmarking 12.3 Financial Analytics & Case Mix Index Analysis
14 Chapter 13: Communicating Insights & Narrative Analytics
13.1 Translating Complex Analytics for Diverse Audiences 13.2 Contextualizing Limitations, Assumptions & Uncertainty 13.3 Narrative Reporting & Data-Driven Decision Support
15 Chapter 14: Healthcare Data Governance Frameworks & Stewardship
14.1 Enterprise Data Governance Frameworks 14.2 Data Stewardship & Master Patient Index (MPI) Management 14.3 Metadata Management & Business Glossaries
16 Chapter 15: Healthcare Privacy, Security & Data Ethics
15.1 HIPAA Privacy & Security Rules for Analysts 15.2 De-Identification Methodologies 15.3 Data Ethics, Audit Trails & Secondary Data Use
5.2 Healthcare Data Collection Methods & Architectures
Key Takeaways Explain This Topic AI
Healthcare data collection encompasses manual abstracting (clinical registries evaluated via Cohen's Kappa for inter-rater reliability), barcode scanning (BCMA eMAR five rights and specimen PPID), continuous biomedical device integration, and electronic patient-reported outcomes (ePROs / FHIR Questionnaire).
Cohen's Kappa (κ = (P_o - P_e) / (1 - P_e)) adjusts observed agreement for chance agreement, providing the gold standard metric for evaluating clinical chart abstractor consistency across national registries (ACS NSQIP, STS, trauma, cancer).
Enterprise data storage tiers serve distinct operational needs: Operational Data Stores (ODS) deliver low-latency near-real-time operational integration; Enterprise Data Warehouses (EDW) provide historical, non-volatile, curated dimensional models; Data Lakes store raw multi-structured data (JSON, DICOM, HL7) with schema-on-read; and Data Lakehouses unify lake elasticity with warehouse ACID transactions.
The traditional ETL (Extract, Transform, Load) paradigm cleanses and normalizes data prior to loading (schema-on-write), whereas the modern ELT (Extract, Load, Transform) paradigm loads raw data directly into scalable cloud warehouses and transforms data in-engine (schema-on-read).
Data ingestion pipelines span real-time event streaming (HL7 v2 / Kafka for immediate clinical alerts), micro-batching (1–15 minute intervals for operational tracking), and scheduled batch ETL (nightly runs utilizing Change Data Capture [CDC] log-scraping or watermark timestamps).
Last updated: August 2026
Healthcare Data Collection Methods & Architectures
Data acquisition in healthcare encompasses a wide spectrum of collection methodologies—ranging from rigorous manual chart abstraction by clinical specialists to automated, high-frequency telemetry streams emitted by bedside biomedical devices. Once captured, this data flows through complex architectural tiers designed to balance real-time operational needs with massive, longitudinal analytical processing. For a Certified Health Data Analyst (CHDA), mastering data collection protocols, inter-rater reliability statistics, data storage tiers (ODS, EDW, Data Lake, Lakehouse), transformation paradigms (ETL vs. ELT), and data pipeline ingestion patterns is vital to architecting robust, trustworthy analytics solutions.
1. Healthcare Data Collection Methodologies
Data enters healthcare information systems through four primary operational collection mechanisms, each presenting distinct data quality profiles, latency characteristics, and validation requirements: +---------------------------------------------------------------------------------------------------+
- | HEALTHCARE DATA COLLECTION METHODOLOGIES |
- +-------------------+--------------------+--------------------+--------------------+----------------+
- | MANUAL ABSTRACT | BARCODE SCANNING | BIOMEDICAL DEVICE | ePROs & PORTALS |
- | - Registry review | - BCMA (5 Rights) | - Telemetry / ECG | - PROMIS / PHQ-9 |
- | - NSQIP, STS, NCDR| - Specimen PPID | - Vent / Infusion | - VR-12 / SF-36 |
- | - Cohen's Kappa | - 2D Datamatrix | - Waveform vs Num | - Patient Portals |
- | - Dual-audit check| - Medication NDC | - Edge filtering | - FHIR Q / QR Res |
- +-------------------+--------------------+--------------------+--------------------+----------------+
- Manual Clinical Chart Abstraction
Manual abstraction involves trained health information professionals, certified tumor registrars (CTRs), or clinical nurse specialists systematically reviewing structured and unstructured electronic medical records to extract discrete variables for clinical quality registries and epidemiological databases.
Prominent Clinical Registries:
American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP): Collects 30-day risk-adjusted surgical outcomes and complication variables.
Society of Thoracic Surgeons (STS) National Database: Captures detailed clinical, operative, and outcome variables for adult cardiac, congenital heart, and general thoracic surgery.
National Cardiovascular Data Registry (NCDR): Tracks cardiology catheterization, percutaneous coronary intervention (PCI), and electrophysiology procedures.
National Trauma Data Bank (NTDB) & State Cancer Registries: Standardized surveillance of traumatic injuries and neoplastic disease staging (AJCC TNM staging).
Abstraction Protocols & Data Dictionaries: Registries enforce rigorous, version-controlled abstraction manuals defining exact inclusion/exclusion criteria, variable definitions, qualifying laboratory thresholds, and acceptable clinical data sources.
Inter-Rater Reliability (IRR) & Cohen's Kappa
Because manual abstraction relies on human clinical interpretation of unstructured text, healthcare organizations must routinely measure Inter-Rater Reliability (IRR) through blinded dual-abstraction audits (where two independent abstractors review identical patient charts).
Limitation of Simple Percent Agreement: Simple percent agreement ( P o = Agreed Items / Total Items P_o = \text{Agreed Items} / \text{Total Items} P o = Agreed Items / Total Items ) fails to account for agreement that occurs purely by random chance—especially for rare clinical events where abstractors would agree on negative findings most of the time simply by guessing.
Cohen's Kappa Statistic ( κ \kappa κ ): The standard statistical metric used in health data analytics to quantify inter-rater agreement for categorical variables while correcting for chance agreement: κ = P o − P e 1 − P e \kappa = \frac{P_o - P_e}{1 - P_e} κ = 1 − P e P o − P e
Where:
P o P_o P o = Relative observed proportion of agreement between raters (sum of diagonal agreement cells divided by total sample size N N N ).
P e P_e P e = Hypothetical expected proportion of agreement occurring by random chance, calculated from the marginal row and column totals: P e = ∑ ( Row Margin i N × Column Margin i N ) P_e = \sum \left( \frac{\text{Row Margin}_i}{N} \times \frac{\text{Column Margin}_i}{N} \right) P e = ∑ ( N Row Margin i × N Column Margin i )
Landis & Koch Benchmark Interpretation Scale for Cohen's Kappa:
- Kappa Statistic ( κ \kappa κ ) Strength of Agreement
- < 0.00 < 0.00 < 0.00 Poor (Less than chance agreement)
- 0.01 − 0.20 0.01 - 0.20 0.01 − 0.20 Slight Agreement
- 0.21 − 0.40 0.21 - 0.40 0.21 − 0.40 Fair Agreement
- 0.41 − 0.60 0.41 - 0.60 0.41 − 0.60 Moderate Agreement
- 0.61 − 0.80 0.61 - 0.80 0.61 − 0.80 Substantial Agreement
- 0.81 − 1.00 0.81 - 1.00 0.81 − 1.00 Almost Perfect / Excellent Agreement
Analytical Standard: National registry data validation protocols typically mandate a minimum Cohen's kappa of κ ≥ 0.80 \kappa \ge 0.80 κ ≥ 0.80 across abstractors to ensure data validity for published quality metrics.
Barcode Scanning & Auto-Identification (AIDC)
Automatic Identification and Data Capture (AIDC) technologies eliminate manual transcription errors at the point of care.
Barcode Medication Administration (BCMA): Nurses use optical barcode scanners at the patient bedside to scan the patient's identification wristband (verifying identity against the EHR ADT record) and the unit-dose pharmaceutical package (verifying the NDC code against the active CPOE order). This enforces the Five Rights of Medication Administration : (1) Right Patient, (2) Right Medication, (3) Right Dose, (4) Right Route, and (5) Right Time. System logs record the exact scan timestamp, user credential, and NDC-to-RxNorm crosswalk.
Specimen Tracking & Positive Patient Identification (PPID): Phlebotomists scan patient wristbands and specimen collection tubes at the bedside. Barcode formats (GS1-128, 2D DataMatrix) encode patient MRN, order number, and specimen type, drastically reducing pre-analytical mislabeled specimen errors and wrong-blood-in-tube (WBIT) incidents.
Direct Biomedical Device Integration
Bedside medical devices interface directly with clinical networks to transmit physiological parameters without manual nursing transcription.
Integrated Modalities: Continuous physiological monitors (electrocardiography [ECG], pulse oximetry [ S p O 2 SpO_2 Sp O 2 ], invasive arterial blood pressure, intracranial pressure [ICP]), mechanical ventilators (tidal volume, positive end-expiratory pressure [PEEP], fraction of inspired oxygen [ F i O 2 FiO_2 F i O 2 ]), smart intravenous infusion pumps, and anesthesia delivery workstations.
Data Granularity & Edge Filtering: Raw telemetry waveforms generate high-frequency time-series data (e.g., 250–500 Hz). Storing every raw microsecond waveform in an enterprise data warehouse creates massive storage bloat. Consequently, healthcare architectures implement edge filtering and downsampling : bedside device gateways calculate running median or average values over 1-minute or 5-minute epochs (e.g., capturing mean minute heart rate and oxygen saturation for permanent EHR storage) while archiving raw waveform streams into specialized clinical data lakes for retrospective AI waveform analysis.
Alarm Fatigue Mitigation: Device integration engines apply threshold algorithms to suppress false-positive artifact alarms (e.g., patient movement causing transient lead disconnects).
Electronic Patient-Reported Outcomes (ePROs) & Patient Portals
ePRO systems capture subjective health status, functional independence, symptom burden, and quality of life directly from patients via smartphone apps, web-based patient portals (e.g., Epic MyChart), or waiting-room tablets.
Standardized Survey Instruments:
PROMIS (Patient-Reported Outcomes Measurement Information System): NIH-developed validated item banks measuring physical function, fatigue, pain interference, depression, and social participation using Item Response Theory (IRT).
PHQ-9 (Patient Health Questionnaire-9): Standardized 9-item screening tool scoring depression severity (0–27).
VR-12 / SF-36: Veteran RAND 12-Item and Short Form 36 health surveys calculating Physical Component Summary (PCS) and Mental Component Summary (MCS) scores.
CAHPS (Consumer Assessment of Healthcare Providers and Systems): Standardized CMS patient experience surveys.
Data Standards & Interoperability: Transmitted using HL7 FHIR Questionnaire (defining survey structure and question logic) and QuestionnaireResponse (capturing discrete patient-submitted answers) resources.
Analytical Considerations: Analysts must account for response bias and the digital divide (systematic non-response among elderly, low-income, or non-English speaking patient cohorts), applying statistical weighting when evaluating population-level ePRO metrics.
2. Enterprise Healthcare Data Architecture Tiers
To accommodate diverse operational, clinical, and strategic requirements, enterprise healthcare information architecture is structured across four distinct data storage tiers: +---------------------------------------------------------------------------------------------------+
- | ENTERPRISE HEALTHCARE DATA ARCHITECTURE TIERS |
- +-------------------+--------------------+--------------------+-------------------------------------+
- | OPERATIONAL DATA | ENTERPRISE DATA | DATA LAKE | DATA LAKEHOUSE |
- | STORE (ODS) | WAREHOUSE (EDW) | (RAW STORAGE) | (HYBRID ARCHITECTURE) |
- | - Near-real-time | - Historical depth | - Raw multi-format | - ACID transactions on data lake |
- | - Subject-oriented| - Star / Snowflake | - Parquet / Cloud | - Unified BI + ML analytics engine |
- | - Low latency | - Curated & clean | - Schema-on-Read | - Schema enforcement & governance |
- | - Offloads OLTP | - Non-volatile | - Big Data / AI | - Delta Lake / Apache Iceberg |
- +-------------------+--------------------+--------------------+-------------------------------------+
- 1. Operational Data Store (ODS)
Architectural Role: An intermediate relational database that ingests near-real-time transactional feeds from primary operational systems (EHR, LIS, Bed Management) to support operational decision-making.
Characteristics: Subject-oriented, integrated across immediate operational sources, low latency (seconds to minutes), and volatile (contains current or recent operational data, typically covering the past 30 to 90 days, with minimal multi-year history).
Primary Use Case: Offloads high-frequency operational reporting (e.g., active emergency department wait times, current bed occupancy, today's operating room schedule utilization) from production OLTP EHR databases without impacting clinical response times.
2. Enterprise Data Warehouse (EDW)
Architectural Role: The centralized, enterprise-wide analytical repository that consolidates structured historical data from all clinical, financial, supply chain, and administrative systems.
Characteristics: Subject-oriented, highly integrated, non-volatile for analytical use (history is managed through versioning or snapshots under the organization’s approved retention schedule), cleansed, standardized, and structured into dimensional star or snowflake schemas (Kimball methodology) or normalized enterprise schemas (Inmon methodology).
Primary Use Case: Serves as the single authoritative single source of truth for executive key performance indicators, Case Mix Index (CMI) financial tracking, regulatory quality reporting (HEDIS, CMS MIPS, Hospital Inpatient Quality Reporting [IQR]), and longitudinal clinical research.
3. Data Lake
Architectural Role: A massive, highly scalable distributed storage repository that ingests and retains structured, semi-structured, and unstructured data in its native, raw format without requiring upfront schema modeling.
Characteristics: Stores structured relational tables, semi-structured files (JSON, XML, HL7 v2 messages, FHIR bundles), and unstructured assets (clinical free-text notes, diagnostic imaging DICOM files, pathology slide images, continuous ICU telemetry waveforms, genomic FASTQ/BAM files). Built on distributed cloud object storage (Amazon S3, Google Cloud Storage, Azure Data Lake Storage) and distributed file systems (Hadoop HDFS), utilizing open columnar formats like Apache Parquet and Apache ORC .
Schema-on-Read Paradigm: Unlike traditional data warehouses that enforce strict schema-on-write (where data must fit a predefined schema before being loaded), data lakes apply schema-on-read (the schema and structure are applied by queries or analytical engines only when the data is read).
Data Governance Risk (Data Swamp): Without strict metadata cataloging, data dictionaries, automated lineage tracking, and role-based access controls, a data lake rapidly degenerates into an unsearchable, ungoverned data swamp where data assets cannot be validated or discovered.
4. Data Lakehouse
Architectural Role: A modern hybrid architecture that combines the cost-effective scalability and multi-format storage flexibility of a data lake with the transactional reliability, ACID guarantees, data governance, and high-performance SQL query capabilities of an enterprise data warehouse.
Underlying Technology: Implemented using open table formats such as Delta Lake , Apache Iceberg , or Apache Hudi . These storage frameworks introduce a metadata transaction log layer on top of raw Parquet files, enabling:
ACID Transactional Guarantees: Ensures reliable concurrent reads and writes without data corruption.
Schema Enforcement & Evolution: Prevents bad or malformed records from polluting analytical tables while allowing controlled schema modifications.
Time-Travel Queries: Enables analysts to query data exactly as it existed at a historical timestamp for point-in-time regulatory reproducibility.
Unified Analytics Engine: Directly supports both high-speed SQL BI reporting and advanced machine learning / deep learning pipelines on the same storage foundation.
3. Architecture Comparison Table
Dimension Operational Data Store (ODS) Enterprise Data Warehouse (EDW) Data Lake Data Lakehouse
Primary Purpose Operational reporting & real-time tactical workflows Strategic analytics, BI reporting & regulatory filing Raw multi-format storage & big data / AI experimentation Unified BI, SQL analytics & machine learning on single platform
Data Latency Near-real-time (Seconds to minutes) Batch (Nightly, weekly, monthly) Near-real-time to Batch (Raw ingest) Real-time streaming to Batch
Data Structures Structured relational Structured dimensional (Star / Snowflake) Structured, semi-structured (JSON/HL7), unstructured (Images/Notes) Structured & semi-structured (Parquet with ACID metadata)
Schema Philosophy Schema-on-Write Schema-on-Write (Strict, curated) Schema-on-Read (Flexible, ungoverned) Schema-on-Write + Schema Evolution (Enforced)
Historical Depth Transient / Short (30–90 days) Deep longitudinal (7–10+ years) Indefinite / Full raw retention Deep longitudinal (With time-travel capability)
Primary End Users Unit managers, bed coordinators, operational staff Executives, financial analysts, quality reporting teams Data scientists, ML engineers, bioinformaticians Business analysts, data scientists, executive leadership
Typical Technology PostgreSQL, SQL Server, Oracle (OLTP) Snowflake, Amazon Redshift, Google BigQuery, Teradata AWS S3, Azure ADLS, Google Cloud Storage, Hadoop HDFS Delta Lake (Databricks), Apache Iceberg, Apache Hudi
4. ETL vs. ELT Paradigms in Healthcare
Transforming operational data into analytical data assets follows one of two foundational integration paradigms: traditional ETL or modern ELT. +---------------------------------------------------------------------------------------------------+
- | ETL VS. ELT DATA PIPELINES |
- +---------------------------------------------------------------------------------------------------+
- TRADITIONAL ETL (Extract -> Transform -> Load)
- [Source Systems] ===> [Dedicated ETL Server] ===(Cleansed / Transformed)===> [Target EDW]
- Cleanses, standardizes
- Applies business logic
- Validates schema-on-write
- MODERN ELT (Extract -> Load -> Transform)
- [Source Systems] ===(Raw Data Ingest)===> [Cloud Data Warehouse / Lake] ===> [In-Engine SQL/dbt]
- Raw storage preserved - Transform on demand
- Massively Parallel Processing - Dynamic models
- +---------------------------------------------------------------------------------------------------+
- Traditional ETL (Extract, Transform, Load)
Process Flow: Data is extracted from source systems, transferred to a dedicated intermediate processing server where transformation logic (cleansing, deduplication, code crosswalking, aggregation, PHI de-identification) is executed in memory, and the finalized cleansed data is loaded into the target data warehouse.
Advantages in Healthcare:
Data Quality Gatekeeping: Strict schema-on-write prevents corrupt, invalid, or malformed data from ever reaching analytical tables.
Privacy & Compliance: Sensitive ePHI can be masked, tokenized, or de-identified before data lands in persistent analytical storage, satisfying the HIPAA Minimum Necessary rule.
Disadvantages: The dedicated ETL compute server becomes an enterprise processing bottleneck as data volumes grow. If business logic changes, historical raw data cannot be reprocessed unless it was explicitly archived in a secondary staging area.
Modern ELT (Extract, Load, Transform)
Process Flow: Data is extracted from source systems and loaded directly into a high-performance cloud data platform (Snowflake, BigQuery, Databricks) in its raw format. Transformations are subsequently executed directly inside the target database engine utilizing Massively Parallel Processing (MPP) SQL engines and transformation frameworks (e.g., dbt - data build tool).
Advantages in Healthcare:
Raw Data Preservation: The immutable raw data is permanently retained, allowing analysts to re-run historical transformations or develop new clinical metrics retroactively.
High Scalability & Speed: Leverages the virtually unlimited compute power of cloud data platforms, dramatically accelerating data ingestion pipelines.
Flexibility: Supports schema-on-read modeling for emerging semi-structured healthcare data formats (JSON FHIR resources, genomics).
Disadvantages: Storing raw data in the target environment requires robust column-level and row-level access controls to ensure unmasked PHI is accessible only to authorized personnel.
5. Data Ingestion Pipelines & Ingestion Patterns
Healthcare data pipelines ingest data across three primary architectural patterns based on business and clinical latency requirements:
1. Real-Time Streaming Ingestion
Mechanics: Ingests and processes individual event messages continuously with sub-second to low-minute latency.
Technologies: Distributed event streaming platforms such as Apache Kafka , AWS Kinesis , or Azure Event Hubs interfacing with interface engines (e.g., Mirth Connect / NextGen Connect, Cloverleaf, Rhapsody).
Healthcare Applications: Ingesting continuous HL7 v2 ADT registration events to trigger automated patient communication, processing continuous ICU physiological monitor telemetry, and powering real-time clinical decision support algorithms (e.g., real-time automated Sepsis Early Warning Score [NEWS2 / MEWS] alerts that notify rapid response teams).
2. Micro-Batching
Mechanics: Collects incoming data streams and processes them in small, discrete temporal batches at frequent intervals (e.g., every 1, 5, or 15 minutes).
Technologies: Stream processing engines like Apache Spark Streaming , Apache Flink, or cloud continuous ingestion utilities like Snowflake Snowpipe .
Healthcare Applications: Near-real-time operational tracking dashboards (e.g., monitoring ED boarding times, post-anesthesia care unit [PACU] bed availability, urgent care center patient flow).
3. Scheduled Batch Ingestion & Change Data Capture (CDC)
Mechanics: Ingests large volumes of transactional data on a scheduled periodic cadence (e.g., nightly at 02:00, weekly, or monthly) during off-peak processing windows.
Workflow Orchestration: Managed using workflow orchestrators such as Apache Airflow , Prefect , or enterprise schedulers (Control-M), defining Directed Acyclic Graphs (DAGs) that manage job dependencies, retries, and alerting.
Change Data Capture (CDC) Mechanisms: To avoid inefficient full-table extraction of millions of clinical records every night, pipelines employ CDC techniques:
Log-Based CDC: The ingestion engine reads the transaction log of the source database (e.g., Oracle redo logs, SQL Server transaction logs via tools like Debezium), capturing exact INSERT , UPDATE , and DELETE operations with zero performance impact on application tables.
Timestamp / High-Watermark CDC: The pipeline queries the source table for records where last_modified_timestamp > :last_successful_pipeline_run_timestamp . While simple to implement, this method cannot detect hard-deleted rows unless soft-deletion flags ( is_deleted = 1 ) are enforced.