Study guide
7.4 Vector Stores, Indexes & LLM Data Processing
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
7.4 Vector Stores, Indexes & LLM Data Processing
Key Takeaways Explain This Topic AI
An embedding maps content into a fixed-dimensional numeric vector; similarity search retrieves nearby vectors but does not by itself verify truth, authorization, or document freshness.
HNSW builds a navigable graph for low-latency approximate search with memory and build-cost tradeoffs, while IVF narrows search to selected clusters and depends strongly on training and probe settings.
AWS vector choices include OpenSearch, Aurora PostgreSQL with pgvector, and other Bedrock Knowledge Bases-supported stores; select from access pattern, scale, filtering, operations, and consistency needs.
A governed LLM processing pipeline preserves raw input, chunks deterministically, records model and prompt versions, protects sensitive data, evaluates outputs, and can re-embed when source or model versions change.
Vectorization converts source content into numeric embeddings with a selected model; changing the model or dimension requires a deliberate re-embedding and index migration plan.
Last updated: August 2026
7.4 Vector Stores, Indexes & LLM Data Processing
Vector workloads add a new data type and retrieval pattern to the data engineer's toolkit. The pipeline still needs ingestion, quality, security, lineage, monitoring, and lifecycle controls; a foundation model does not remove those responsibilities.
Embeddings and similarity
An embedding model converts text, images, or other supported content into a numeric vector with a fixed number of dimensions. Vectors that represent semantically related content tend to be close under the metric used by the model and index, such as cosine similarity, inner product, or Euclidean distance.
Model, dimension, and metric must agree with the index. Changing the embedding model can change both vector meaning and dimension, so store an embedding-version field and rebuild or isolate the index during migration. Never mix incompatible vectors in one field and expect meaningful distances.
Exact nearest-neighbor search compares a query with every vector and becomes expensive at scale. Approximate nearest-neighbor (ANN) indexes trade a small amount of recall for much lower latency.
HNSW and IVF
Hierarchical Navigable Small World (HNSW) builds a multilayer proximity graph. Queries enter at a sparse upper layer and navigate toward nearby vectors at denser layers. HNSW usually provides strong low-latency recall but consumes memory and takes time to build. Construction and search breadth parameters trade resources for recall.
Inverted File (IVF) clustering assigns vectors to coarse lists. A query first selects nearby cluster centroids and searches a configured number of lists. Searching more lists improves recall and increases work. IVF needs representative training and can perform poorly when cluster distribution shifts.
- Requirement Likely direction Tradeoff to evaluate
- Very low-latency ANN with enough memory HNSW Memory, build time, update behavior
- Very large batch-oriented index with tunable probes IVF family Training quality and recall sensitivity
- Small dataset or strict recall Exact or flat search Linear scan cost
Do not claim one index is always better. Benchmark recall at the required latency with production-like vectors and filters.
AWS store selection
Amazon OpenSearch Service combines text search, filtering, aggregations, and vector search. It fits hybrid keyword-plus-semantic retrieval and log/search platforms.
Amazon Aurora PostgreSQL with pgvector fits applications that need vectors near relational rows and transactional metadata. Relational filtering, joins, and operational consistency can outweigh the scale advantages of a dedicated search engine.
Amazon MemoryDB can serve fast key/value and supported vector use cases when in-memory latency and Redis-compatible access are central. Other stores supported by Amazon Bedrock Knowledge Bases can be appropriate. Choose from vector count, update rate, metadata filters, latency, durability, regional availability, and operational ownership.
Bedrock Knowledge Bases ingestion
For unstructured sources, a typical Knowledge Bases flow is:
Connect a supported source such as S3.
Parse documents and split them into chunks.
Use an embedding model to vectorize chunks.
Write vectors plus source metadata to a vector store.
At query time, embed the question, retrieve similar chunks, and provide them as context to a model.
Chunking affects retrieval quality. Chunks that are too small lose context; chunks that are too large combine unrelated topics and consume the model's context window. Preserve document ID, version, page or section, access classification, and chunk offsets so a response can be traced back to authorized source material.
Structured sources can use managed natural-language-to-SQL paths rather than vectorizing every row. The data engineer must still restrict schemas, validate generated queries, cap cost, and prevent a request from reading unauthorized tables.
LLMs as data processors
Version 1.1 explicitly includes integrating large language models for data processing. Useful cases include classification, metadata enrichment, extraction from irregular text, and summarization. LLM output is probabilistic. Do not silently replace deterministic validation with a model response.
A production pattern writes model output to a candidate layer with:
- source object and checksum;
- model, prompt, parser, and guardrail versions;
- invocation time and token or cost metrics;
- confidence or validation result;
human-review status where impact is high.
Validate required fields with deterministic rules, constrain output to a schema, quarantine parsing failures, and sample accepted outputs for drift. Remove or mask sensitive data before invocation when policy requires it, and ensure the selected service and Region meet residency rules.
Retrieval security and freshness
The nearest chunk is not necessarily authorized. Apply tenant and classification filters at retrieval time, and ensure the caller cannot manipulate filters to cross boundaries. When a source document is deleted or access is revoked, remove or tombstone its chunks and verify the vector store no longer returns them.
Track ingestion lag from source version to searchable vector version. A successful model response based on a stale policy document is a data freshness failure, even if the prose sounds convincing.
Vectorization Lifecycle
Vectorization is a versioned data transformation, not a one-time storage setting. Record the embedding model, model version, output dimension, normalization choice, source chunk, and preprocessing version with each vector. A new model can change both meaning and dimension, so mixing old and new embeddings in one index can make distance scores incomparable or invalid. Re-embed into a parallel index, validate retrieval quality and latency, then shift consumers with a rollback path.