Study guide
14.4 Automated Secrets Management & Secret Rotation with Secrets Manager
AWS Certified DevOps Engineer – Professional Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: CI/CD Pipeline Architecture & Source Integration
1.1 Multi-Stage & Multi-Account CodePipeline Design 1.2 CodeBuild Projects, Build Environments & Pipeline Secrets 1.3 Source Control Workflows, Webhooks & Branching Models
2 Chapter 2: Automated Testing, Test Gates & Quality Verification
2.1 Automated Unit, Integration & Acceptance Testing in Pipelines 2.2 Load Testing, Performance Benchmarking & Synthetic Canaries 2.3 Security Scanning, SAST/DAST & Approval Gates
3 Chapter 3: Artifact Management & Multi-Platform Deployment Strategies
3.1 Artifact Lifecycle Management with CodeArtifact, ECR & S3 3.2 Automated Image Baking with EC2 Image Builder 3.3 In-Place, Rolling & Immutable Deployments on EC2 3.4 Blue/Green & Canary Deployments for ECS, EKS & Lambda
4 Chapter 4: CloudFormation, CDK & Reusable IaC Components
4.1 Advanced CloudFormation: Nested Stacks, Change Sets & Custom Resources 4.2 Cross-Account & Multi-Region StackSets & Drift Detection 4.3 Modern IaC: AWS CDK, SAM & Service Catalog Governance
5 Chapter 5: Multi-Account Architecture, Organizations & Control Tower
5.1 AWS Organizations Hierarchy, OUs & Account Provisioning 5.2 Service Control Policies (SCPs) & Multi-Account Guardrails 5.3 AWS Control Tower, Account Factory & Landing Zone Automation
6 Chapter 6: Systems Manager & Large-Scale Fleet Configuration
6.1 SSM Agent, Hybrid Activation & Session Manager 6.2 Systems Manager Run Command, State Manager & Associations 6.3 Patch Manager, Parameter Store vs Secrets Manager & AppConfig
7 Chapter 7: High Availability & Multi-Region Resiliency
7.1 Designing Resilient Multi-AZ & Multi-Region Architectures 7.2 Route 53 Traffic Flow, Health Checks & DNS Failover 7.3 Data Replication: Aurora Global DB, DynamoDB Global Tables & S3 CRR
8 Chapter 8: Scalability, Auto Scaling & Disaster Recovery Automation
8.1 EC2 Auto Scaling Policies, Lifecycle Hooks & Warm Pools 8.2 Container & Serverless Scalability: ECS Capacity Providers, EKS & API Gateway 8.3 Disaster Recovery Strategies: RTO, RPO & Architecture Tiers 8.4 Automated Backup & Recovery with AWS Backup & Elastic Disaster Recovery
9 Chapter 9: CloudWatch Metrics, Agent Collection & Log Aggregation
9.1 CloudWatch Metrics, Dimensions, Metric Math & High Resolution 9.2 Unified CloudWatch Agent & Custom Metric Collection 9.3 Centralized Log Aggregation, Retention & Cross-Account Subscriptions
10 Chapter 10: Log Analysis, Dashboards & Distributed Tracing with X-Ray
10.1 CloudWatch Logs Insights, Filter Syntax & Query Optimization 10.2 CloudWatch Dashboards, Composite Alarms & Anomaly Detection 10.3 Distributed Tracing with AWS X-Ray & Service Maps
11 Chapter 11: Automated Observability & Event-Driven Monitoring
11.1 Application & Infrastructure Health Checks with Route 53 & ALB 11.2 Centralized Observability with Amazon OpenSearch & Athena 11.3 Synthetic Monitoring with CloudWatch Synthetics Canaries
12 Chapter 12: Event-Driven Incident Detection & Notification Pipelines
12.1 AWS Health, CloudTrail Events & Operational Notification 12.2 EventBridge Event Routing, Patterns & Multi-Account Buses 12.3 Asynchronous Event Processing: SNS, SQS Queues & Dead-Letter Handling
13 Chapter 13: Automated Remediation & Deployment Troubleshooting
13.1 AWS Config Managed Rules & Automated Remediation with SSM 13.2 Event-Driven Infrastructure Self-Healing with Lambda & Step Functions 13.3 Troubleshooting CI/CD Pipeline & Deployment Failures 13.4 Root Cause Analysis for Auto Scaling & Container Failures
14 Chapter 14: Identity, Access Management & Secrets at Scale
14.1 IAM Policy Evaluation Logic, NotAction & Explicit Denies 14.2 IAM Permissions Boundaries & Delegated Administration 14.3 IAM Identity Center, Cross-Account Roles & STS Session Policies 14.4 Automated Secrets Management & Secret Rotation with Secrets Manager
15 Chapter 15: Security Automation, Data Protection & Compliance Auditing
15.1 Network Security Automation: Security Groups, NACLs, Network Firewall & WAF 15.2 Cryptographic Controls & Key Management with AWS KMS & ACM 15.3 Automated Threat Detection: GuardDuty, Inspector & Security Hub 15.4 Continuous Compliance Auditing & Log Integrity with CloudTrail & Config
14.4 Automated Secrets Management & Secret Rotation with Secrets Manager
Key Takeaways Explain This Topic AI
AWS Secrets Manager provides encrypted secret storage using AWS KMS, supporting automated rotation via AWS Lambda and fine-grained version staging labels (AWSCURRENT, AWSPREVIOUS, AWSPENDING).
The 4-step rotation protocol (createSecret, setSecret, testSecret, finishSecret) guarantees that new credentials are fully written to the target database and verified before being promoted to AWSCURRENT.
When rotating database credentials in private VPC subnets, the rotation Lambda function must be attached to the private VPC with security group access to the database, while reaching Secrets Manager and KMS via VPC Interface Endpoints (PrivateLink).
Single-user rotation incurs a potential race condition and transient connection failures during password update, whereas alternating dual-user rotation achieves zero-downtime database rotation by alternating between two active application accounts.
High-throughput microservices must utilize AWS Secrets Manager client-side caching libraries with configurable TTLs to eliminate latency, prevent API rate throttling (ThrottlingException), and eliminate unnecessary API costs.
Last updated: September 2026
AWS Secrets Manager Architecture & Core Capabilities
Hardcoded database credentials, API tokens, and private encryption keys inside source code repositories or configuration files represent one of the most severe security vulnerabilities in modern DevOps pipelines. AWS Secrets Manager eliminates this risk by providing centralized, encrypted secret storage with native automated lifecycle rotation.
Secret Storage and Encryption Mechanics
Payload Format : Secrets Manager stores secrets as JSON key-value pairs (ideal for database credentials: username , password , host , port , dbname ) or arbitrary binary/plaintext strings.
AWS KMS Envelope Encryption : All secrets are encrypted at rest using AWS Key Management Service (KMS). Organizations can utilize the default AWS-managed key ( aws/secretsmanager ) or a Customer Managed Key (CMK). A CMK is mandatory when secrets must be shared across AWS accounts or when strict regulatory compliance mandates customer-controlled key rotation and auditing.
Resource-Based Policies : Secrets Manager secrets support resource-based policies. This allows a secret in a centralized Security or Shared Services account to be accessed directly by IAM execution roles in spoke application accounts without requiring cross-account role assumption.
Secret Versioning & Staging Labels
Secrets Manager versions are immutable. Each modification or rotation creates a new version containing a unique VersionId (UUID). Versions are tracked and manipulated using Staging Labels :
Staging Label Lifecycle State & Functional Role
AWSCURRENT The active version of the secret. Active applications query this version by default when calling GetSecretValue without specifying a VersionId or VersionStage . Exactly one version can hold AWSCURRENT at any time.
AWSPREVIOUS The version that immediately preceded the current version. Maintained for instantaneous rollback if a recently rotated credential causes unforeseen application faults. Exactly one version holds AWSPREVIOUS .
AWSPENDING The newly generated staging version undergoing active rotation and verification. Staged during the rotation workflow and promoted to AWSCURRENT only after rigorous connectivity testing succeeds.
Custom Labels Arbitrary labels attached by operators or automated CI/CD deployment scripts (e.g., staging-v2 , green-deployment ) to support blue/green application deployment models.
The 4-Step Rotation Protocol
When automated rotation is enabled (e.g., every 30 days) or manually triggered via the RotateSecret API, Secrets Manager initiates an execution workflow that invokes an AWS Lambda rotation function four consecutive times. Each invocation passes a specific Step parameter in the event payload: Secrets Manager Rotation Engine
- │
- ├─ 1. Step: "createSecret" ──> Lambda creates AWSPENDING version with new random password
- │
- ├─ 2. Step: "setSecret" ────> Lambda connects to DB (as AWSCURRENT or Master) and updates DB password
- │
- ├─ 3. Step: "testSecret" ───> Lambda tests DB connection using AWSPENDING password
- │
- └─ 4. Step: "finishSecret" ─> Lambda moves AWSCURRENT label to AWSPENDING; marks old version AWSPREVIOUS
- The Four Steps in Detail
createSecret :
The Lambda function queries Secrets Manager to determine if a version with the AWSPENDING staging label already exists. If not, it generates a cryptographically secure random password (e.g., using secrets.token_urlsafe or GetRandomPassword ) and writes the new credentials to Secrets Manager under a new VersionId labeled AWSPENDING . If AWSPENDING already exists, it leaves it intact to ensure idempotency across automated retries.
setSecret :
The Lambda function retrieves the current credentials (from AWSCURRENT or master credentials) and establishes an administrative connection to the target database engine (e.g., Amazon RDS MySQL, PostgreSQL, or Aurora). It executes an ALTER USER or SET PASSWORD statement to update the database user's password in the database engine to match the password stored in the AWSPENDING secret version.
testSecret :
The Lambda function attempts to establish a brand-new database connection using the AWSPENDING credentials. It executes a lightweight validation query (such as SELECT 1; ). If authentication fails, the function raises an exception, the rotation process halts, and AWSCURRENT remains completely unchanged.
finishSecret :
Once testSecret succeeds, the Lambda function calls the Secrets Manager UpdateSecretVersionStage API. It moves the AWSCURRENT label from the old version to the AWSPENDING version, automatically applies AWSPREVIOUS to the older version, and strips the AWSPENDING label. Applications querying GetSecretValue now receive the newly validated credentials.
Important
Idempotency Requirement : Secrets Manager can retry any rotation step if a transient network failure occurs. The Lambda rotation function must be written to be strictly idempotent . For instance, in setSecret , if the database password was already updated in an earlier attempt that timed out during response delivery, the Lambda function must verify connectivity with AWSPENDING before failing.
Private VPC Networking for Secrets Manager Rotation
In enterprise production environments, databases (Amazon RDS, Amazon Aurora) are deployed inside isolated private VPC subnets without public internet access or public IP addresses. This creates a critical networking requirement for the rotation Lambda function. ┌────────────────────────────────────────────────────────────────────────┐
- │ Amazon VPC (10.0.0.0/16) │
- │ │
- │ ┌────────────────────────────────┐ ┌──────────────────────────┐ │
- │ │ Private Application Subnet │ │ Private Database Subnet │ │
- │ │ │ │ │ │
- │ │ [ Lambda Rotation Function ] │───>│ [ Amazon RDS Database ] │ │
- │ │ Attached to VPC Subnets │TCP │ Port 3306 / 5432 │ │
- │ │ Security Group: Lambda-SG │ │ Security Group: DB-SG │ │
- │ └───────────────┬────────────────┘ └──────────────────────────┘ │
- │ │ │
- │ │ HTTPS (Port 443) via AWS PrivateLink │
- │ ▼ │
- │ ┌────────────────────────────────┐ ┌──────────────────────────┐ │
- │ │ VPC Interface Endpoint: │ │ VPC Interface Endpoint: │ │
- │ │ com.amazonaws.<reg>.secretsmanager │ │ com.amazonaws.<reg>.kms │ │
- │ └────────────────────────────────┘ └──────────────────────────┘ │
- └────────────────────────────────────────────────────────────────────────┘
- Mandatory Networking Components
VPC Subnet Attachment : The Lambda rotation function must be configured with VPC attachment, placing its Elastic Network Interfaces (ENIs) inside the private subnets where the database is reachable.
Security Group Rules :
Database Security Group : Must have an inbound rule allowing TCP traffic on the database port (e.g., 3306 for MySQL, 5432 for PostgreSQL) originating from the Lambda function's security group.
Lambda Security Group : Outbound rule allowing traffic to the database security group on the database port, and outbound rule allowing HTTPS (port 443) to the VPC Interface Endpoints.
AWS PrivateLink (VPC Interface Endpoints) :
Because the Lambda function is inside a private VPC without internet access (no NAT Gateway or Internet Gateway), it cannot reach the public Secrets Manager or AWS KMS API endpoints. You must provision VPC Interface Endpoints for:
- com.amazonaws.<region>.secretsmanager
- com.amazonaws.<region>.kms (if using customer-managed KMS keys)
Both endpoints must have Private DNS enabled and attach a security group allowing inbound HTTPS (port 443) from the Lambda rotation function.
Rotation Strategies: Single-User vs. Alternating Dual-User
Architectural Attribute Single-User Rotation Strategy Alternating Dual-User Rotation Strategy
Account Management Rotates credentials for a single database user Manages two distinct database users ( UserA and UserB ) plus a Master secret
Downtime & Race Conditions Potential Downtime : In setSecret , the password changes in the DB immediately. Any application connection pool using old credentials will fail until the app updates its cache. Zero Downtime : While UserA is active in AWSCURRENT , UserB is rotated and updated in the DB. Existing application connections continue using UserA without interruption.
Target Database Users Non-critical internal tools, background ETL scripts Production databases, high-availability microservices, 24/7 web applications
Prerequisites User has permissions to alter its own password Requires a separate Master Secret containing administrative credentials to alter user passwords [ Dual-User Zero-Downtime Rotation Cycle ]
Phase 1 (Normal Operations):
- Database Users: User_A (Active), User_B (Standby)
- Secrets Manager: AWSCURRENT points to User_A
- Applications connect using User_A
Phase 2 (Rotation Triggered):
- Master Secret used to connect to database
- Lambda alters User_B's password in the database to AWSPENDING password
- Lambda tests connection using User_B with AWSPENDING password
- Applications continue processing transactions using User_A uninterrupted!
Phase 3 (Promotion):
- finishSecret promotes User_B to AWSCURRENT; User_A becomes AWSPREVIOUS
- Applications refresh credentials and transition seamlessly to User_B
- Client-Side Caching & Scalability
In high-throughput microservices architectures (e.g., an Amazon EKS cluster processing 20,000 requests per second), applications must never call GetSecretValue on every incoming transaction. Secrets Manager API calls are rate-limited per region and incur costs ($0.05 per 10,000 API calls). Querying Secrets Manager per request triggers ThrottlingException: Rate exceeded and generates massive AWS bills.
AWS Secrets Manager Client-Side Caching Libraries
AWS provides native open-source caching libraries (for Java, Python, Go, Node.js, and .NET):
In-Memory Caching : Credentials are cached in application process memory with a configurable Time-to-Live (TTL, e.g., default of 300 seconds / 5 minutes to 1 hour).
Asynchronous Background Refresh : Caching libraries refresh secrets in the background before the TTL expires, ensuring zero added latency for user requests.
Authentication Failure Hook : If the application experiences an authentication failure ( AccessDenied or SQLException ), the caching library automatically invalidates the cached secret immediately and performs a synchronous fresh GetSecretValue call to fetch the newly rotated AWSCURRENT credentials.