Study guide
11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store
Key Takeaways Explain This Topic AI
Secrets Manager can automate supported credential rotation with managed strategies or Lambda functions; clients still need bounded caching, refresh-on-authentication-failure, and connection-pool handling to avoid downtime.
Systems Manager Parameter Store provides low-cost parameter management with standard (free up to 10k parameters, 4KB size) and advanced (up to 8KB size) parameter tiers, using KMS for SecureString encryption.
AWS Parameters and Secrets Lambda Extension caches secrets in Lambda memory, drastically reducing API calls, latency, and cost for high-throughput serverless ETL workloads.
AWS Glue ETL jobs and Amazon EMR utilize Secrets Manager integration to retrieve database connection parameters dynamically at runtime, avoiding hardcoded plaintext credentials in job scripts.
Secrets Manager staging labels (AWSCURRENT, AWSPENDING, AWSPREVIOUS) manage credential state transitions during Lambda-executed secret rotation to prevent breaking active client connections.
Last updated: August 2026
Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store
Data pipelines frequently interact with relational databases (Amazon RDS, Amazon Redshift, PostgreSQL, MySQL), third-party SaaS APIs, and external storage engines. Storing hardcoded database credentials, API keys, or private tokens inside code repositories, Glue scripts, or Airflow DAGs creates severe security vulnerabilities. AWS provides two key management services for securing configurations: AWS Secrets Manager and AWS Systems Manager Parameter Store .
Secrets Manager vs Parameter Store: Architectural Comparison
Selecting the right credentials management tool is a core topic on the DEA-C01 exam. The following table contrasts their capabilities:
Feature AWS Secrets Manager Systems Manager Parameter Store
Primary Purpose Storing confidential credentials requiring automatic rotation. Storing configuration parameters, license keys, & simple secrets.
Max Payload Size 64 KB per secret Standard: 4 KB | Advanced: 8 KB
Automatic Secret Rotation Built-in native integration via AWS Lambda (RDS, Redshift, DocDB). Manual or custom EventBridge + Lambda workflow.
Cross-Account Sharing Built-in via Secret Resource Policies. Requires IAM role assumption or AWS RAM (Advanced parameters).
Cost Model Per-secret and API charges; check current Regional pricing. Standard and advanced tiers differ in quotas and charges; check current pricing.
KMS Encryption Always encrypted at rest (default key or CMK). Optional encryption ( String / StringList vs SecureString ).
AWS Secrets Manager Deep Dive
Secrets Manager stores secrets as JSON structures containing database usernames, passwords, hostnames, ports, and engine types.
Automated Secret Rotation Mechanics
Secrets Manager coordinates credential rotation; downtime avoidance depends on the rotation strategy and client refresh behavior using an AWS Lambda function executing a four-step staging process: [ CreateSecret ] --> 1. createSecret (Generates new password -> AWSPENDING)
- |
- v
- 2. setSecret (Updates RDS/Redshift user password)
- |
- v
- 3. testSecret (Verifies connection using AWSPENDING)
- |
- v
- 4. finishSecret (Swaps AWSCURRENT label to new password)
The Staging Label Workflow:
AWSCURRENT : The active secret version currently used by applications.
AWSPENDING : The candidate secret version currently being configured and tested against the database.
AWSPREVIOUS : The prior secret version maintained for rollback or legacy connection cleanup.
During rotation, Secrets Manager updates the target database first using a master credential, verifies connection with AWSPENDING , and then atomically promotes AWSPENDING to AWSCURRENT .
- Data Pipeline Integration Patterns
- 1. AWS Glue ETL Jobs Integration
Instead of passing JDBC passwords into Glue job arguments, AWS Glue native connections integrate directly with Secrets Manager: import boto3
import json
def get_secret(secret_name, region_name="us-east-1"):
- client = boto3.client('secretsmanager', region_name=region_name)
- response = client.get_secret_value(SecretId=secret_name)
- return json.loads(response['SecretString'])
- # Retrieve DB credentials securely inside Glue Executor
- db_credentials = get_secret("prod/redshift/analytics_user")
- db_url = f"jdbc:redshift://{db_credentials['host']}:{db_credentials['port']}/{db_credentials['dbname']}"
- 2. High-Throughput Lambda Caching (Parameters & Secrets Extension)
If an AWS Lambda function processes millions of Kinesis streaming records and invokes GetSecretValue on every execution, it will incur high API costs and hit Secrets Manager API throttling limits.
Best Practice Solution : Use the AWS Parameters and Secrets Lambda Extension as a Lambda layer. The extension runs a local HTTP cache inside the Lambda execution environment. Lambda code fetches secrets over local HTTP ( http://localhost:2773/secretsmanager/get?secretId=... ), serving requests directly from local memory cache.
AWS Systems Manager Parameter Store Deep Dive
SSM Parameter Store is ideal for centralizing non-sensitive environment variables (e.g., S3 bucket names, database endpoint URLs, max batch sizes) or low-cost secrets using SecureString .
Parameter Types & Encryption
String : Plaintext text values (e.g., /config/prod/s3_landing_bucket ).
StringList : Comma-separated lists (e.g., /config/prod/subnets ).
SecureString : Encrypted at rest using a customer managed or AWS managed KMS key; callers request decryption with WithDecryption and need kms:Decrypt permission when applicable.
Parameter Policies
Parameter Store supports automated lifecycle management policies:
Expiration : Automatically deletes parameters after a specified date/time.
ExpirationNotification : Triggers an EventBridge notification prior to parameter expiration.
NoChangeNotification : Alerts administrators if a parameter has not been updated for a set number of days.
Security & IAM Best Practices for Secrets
Restrict Secret Retrieval : Grant secretsmanager:GetSecretValue and ssm:GetParameter strictly to designated execution roles.
KMS Key Access : Ensure the principal has kms:Decrypt permissions on the KMS key used to encrypt the secret.
Enforce Secret Naming Hierarchy : Use path hierarchies (e.g., /prod/etl/redshift/app_user ) to grant scoped access via IAM wildcards ( arn:aws:ssm:us-east-1:123456789012:parameter/prod/etl/* ).
Summary Checklist for DEA-C01 Exam
Automatic Rotation : Choose Secrets Manager over Parameter Store when automatic database credential rotation is required.
Lambda Caching : Use the AWS Parameters and Secrets Lambda Extension to prevent API throttling.
Parameter Store Cost : Use SSM Standard Parameters for free, lightweight, non-rotating configuration storage.
Rotation-safe consumers
Automatic rotation only helps when clients can adopt the new value. Applications should fetch by secret identifier rather than embedding a version ID, cache for a bounded time, and retry authentication once after refreshing the cache. During rotation, Secrets Manager staging labels identify the current and pending versions. The rotation function creates, applies, tests, and then promotes the pending credential; a failed test must not promote it. Database connection pools need a strategy to retire connections authenticated with the former credential without interrupting in-flight work.
Use resource policies and KMS key policies for cross-account secret access, and restrict both GetSecretValue and decrypt permission. Parameter Store SecureString can protect sensitive configuration, but it does not provide the same managed rotation workflow as Secrets Manager.