9.2 Unified CloudWatch Agent & Custom Metric Collection

AWS Certified DevOps Engineer – Professional Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: CI/CD Pipeline Architecture & Source Integration

1.1 Multi-Stage & Multi-Account CodePipeline Design 1.2 CodeBuild Projects, Build Environments & Pipeline Secrets 1.3 Source Control Workflows, Webhooks & Branching Models

2 Chapter 2: Automated Testing, Test Gates & Quality Verification

2.1 Automated Unit, Integration & Acceptance Testing in Pipelines 2.2 Load Testing, Performance Benchmarking & Synthetic Canaries 2.3 Security Scanning, SAST/DAST & Approval Gates

3 Chapter 3: Artifact Management & Multi-Platform Deployment Strategies

3.1 Artifact Lifecycle Management with CodeArtifact, ECR & S3 3.2 Automated Image Baking with EC2 Image Builder 3.3 In-Place, Rolling & Immutable Deployments on EC2 3.4 Blue/Green & Canary Deployments for ECS, EKS & Lambda

4 Chapter 4: CloudFormation, CDK & Reusable IaC Components

4.1 Advanced CloudFormation: Nested Stacks, Change Sets & Custom Resources 4.2 Cross-Account & Multi-Region StackSets & Drift Detection 4.3 Modern IaC: AWS CDK, SAM & Service Catalog Governance

5 Chapter 5: Multi-Account Architecture, Organizations & Control Tower

5.1 AWS Organizations Hierarchy, OUs & Account Provisioning 5.2 Service Control Policies (SCPs) & Multi-Account Guardrails 5.3 AWS Control Tower, Account Factory & Landing Zone Automation

6 Chapter 6: Systems Manager & Large-Scale Fleet Configuration

6.1 SSM Agent, Hybrid Activation & Session Manager 6.2 Systems Manager Run Command, State Manager & Associations 6.3 Patch Manager, Parameter Store vs Secrets Manager & AppConfig

7 Chapter 7: High Availability & Multi-Region Resiliency

7.1 Designing Resilient Multi-AZ & Multi-Region Architectures 7.2 Route 53 Traffic Flow, Health Checks & DNS Failover 7.3 Data Replication: Aurora Global DB, DynamoDB Global Tables & S3 CRR

8 Chapter 8: Scalability, Auto Scaling & Disaster Recovery Automation

8.1 EC2 Auto Scaling Policies, Lifecycle Hooks & Warm Pools 8.2 Container & Serverless Scalability: ECS Capacity Providers, EKS & API Gateway 8.3 Disaster Recovery Strategies: RTO, RPO & Architecture Tiers 8.4 Automated Backup & Recovery with AWS Backup & Elastic Disaster Recovery

9 Chapter 9: CloudWatch Metrics, Agent Collection & Log Aggregation

9.1 CloudWatch Metrics, Dimensions, Metric Math & High Resolution 9.2 Unified CloudWatch Agent & Custom Metric Collection 9.3 Centralized Log Aggregation, Retention & Cross-Account Subscriptions

10 Chapter 10: Log Analysis, Dashboards & Distributed Tracing with X-Ray

10.1 CloudWatch Logs Insights, Filter Syntax & Query Optimization 10.2 CloudWatch Dashboards, Composite Alarms & Anomaly Detection 10.3 Distributed Tracing with AWS X-Ray & Service Maps

11 Chapter 11: Automated Observability & Event-Driven Monitoring

11.1 Application & Infrastructure Health Checks with Route 53 & ALB 11.2 Centralized Observability with Amazon OpenSearch & Athena 11.3 Synthetic Monitoring with CloudWatch Synthetics Canaries

12 Chapter 12: Event-Driven Incident Detection & Notification Pipelines

12.1 AWS Health, CloudTrail Events & Operational Notification 12.2 EventBridge Event Routing, Patterns & Multi-Account Buses 12.3 Asynchronous Event Processing: SNS, SQS Queues & Dead-Letter Handling

13 Chapter 13: Automated Remediation & Deployment Troubleshooting

13.1 AWS Config Managed Rules & Automated Remediation with SSM 13.2 Event-Driven Infrastructure Self-Healing with Lambda & Step Functions 13.3 Troubleshooting CI/CD Pipeline & Deployment Failures 13.4 Root Cause Analysis for Auto Scaling & Container Failures

14 Chapter 14: Identity, Access Management & Secrets at Scale

14.1 IAM Policy Evaluation Logic, NotAction & Explicit Denies 14.2 IAM Permissions Boundaries & Delegated Administration 14.3 IAM Identity Center, Cross-Account Roles & STS Session Policies 14.4 Automated Secrets Management & Secret Rotation with Secrets Manager

15 Chapter 15: Security Automation, Data Protection & Compliance Auditing

15.1 Network Security Automation: Security Groups, NACLs, Network Firewall & WAF 15.2 Cryptographic Controls & Key Management with AWS KMS & ACM 15.3 Automated Threat Detection: GuardDuty, Inspector & Security Hub 15.4 Continuous Compliance Auditing & Log Integrity with CloudTrail & Config

9.2 Unified CloudWatch Agent & Custom Metric Collection

Key Takeaways Explain This Topic AI

The Unified CloudWatch Agent collects guest OS-level telemetry—including memory utilization, disk space, swap usage, network socket statistics, and per-process metrics (procstat)—which are invisible to the default EC2 hypervisor.

The agent is declaratively configured via amazon-cloudwatch-agent.json, divided into agent, metrics (with metrics_collected and procstat), and logs sections (including Windows Event Logs and multi-line application logs).

Fleet-wide deployment is managed without SSH/RDP access using AWS Systems Manager (SSM) Run Command or State Manager associations using documents AWS-ConfigureAWSPackage and AmazonCloudWatch-ManageAgent.

Agent configuration files should be stored centrally in AWS Systems Manager Parameter Store as String or SecureString parameters, allowing dynamic fleet configuration updates with zero instance downtime.

The Unified Agent natively embeds StatsD and collectd daemons, enabling applications to push custom application metrics locally over UDP/TCP port 8125, which the agent aggregates and flushes via PutMetricData.

Last updated: September 2026

Unified CloudWatch Agent Architecture and Capabilities

When Amazon EC2 instances run workloads in AWS, the native CloudWatch hypervisor metrics ( AWS/EC2 namespace) provide visibility into the host boundary: CPUUtilization , NetworkIn , NetworkOut , DiskReadOps , and DiskWriteOps (for instance store volumes).

However, the hypervisor cannot inspect guest operating system internals due to virtualization isolation boundaries. The hypervisor has zero visibility into:

• RAM utilization (active, buffered, cached, free memory)
• Filesystem disk space utilization and inode consumption
• Swap file and pagefile utilization
• Operating system process performance (CPU and memory per process)
• Local application log files and Windows Event Viewer logs

The Unified CloudWatch Agent is a high-performance Go-based daemon that installs inside the guest OS on both Linux and Windows Server instances, as well as on-premises physical and virtual servers. It replaces legacy, deprecated tools (such as the legacy Python-based awslogsd agent and Perl monitoring scripts) with a single, consolidated telemetry collector.

Declarative Configuration: amazon-cloudwatch-agent.json

The behavior of the Unified CloudWatch Agent is controlled entirely by a JSON configuration file. The configuration is organized into three primary top-level blocks:

agent : Defines global runtime properties (collection intervals, execution user, debug logging).

metrics : Configures OS metric plugins, collection frequencies, custom dimensions, procstat , and embedded telemetry listeners ( statsd and collectd ).

logs : Configures log file harvesting, Windows Event Log channels, timestamp extraction, and multi-line parsing rules.

• Production-Grade Configuration Example {
• "agent": {
• "metrics_collection_interval": 60,
• "run_as_user": "cwagent"
• },
• "metrics": {
• "namespace": "CWAgent",
• "metrics_collected": {
• "mem": {
• "measurement": [
• "mem_used_percent",
• "mem_available_percent",
• "mem_used",
• "mem_available"
• ],
• "metrics_collection_interval": 60
• },
• "disk": {
• "measurement": [
• "used_percent",
• "inodes_free"
• ],
• "metrics_collection_interval": 60,
• "resources": [
• "/",
• "/var/log",
• "/data"
• ]
• },
• "swap": {
• "measurement": [
• "swap_used_percent"
• ]
• },
• "netstat": {
• "measurement": [
• "tcp_established",
• "tcp_time_wait"
• ],
• "metrics_collection_interval": 60
• },
• "procstat": [
• {
• "pattern": "nginx",
• "measurement": [
• "cpu_usage",
• "memory_rss"
• ]
• },
• {
• "exe": "payment-service",
• "measurement": [
• "cpu_usage",
• "memory_vms"
• ]
• }
• ],
• "statsd": {
• "service_address": ":8125",
• "metrics_collection_interval": 10,
• "metrics_aggregation_interval": 60
• }
• },
• "aggregation_dimensions": [
• ["AutoScalingGroupName"],
• ["InstanceId", "InstanceType"]
• ],
• "append_dimensions": {
• "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
• "ImageId": "${aws:ImageId}",
• "InstanceId": "${aws:InstanceId}",
• "InstanceType": "${aws:InstanceType}"
• }
• },
• "logs": {
• "logs_collected": {
• "files": {
• "collect_list": [
• {
• "file_path": "/var/log/application/*.log",
• "log_group_name": "/aws/ec2/production/application",
• "log_stream_name": "{instance_id}",
• "multi_line_start_pattern": "{datetime_format}",
• "timestamp_format": "%Y-%m-%d %H:%M:%S"
• }
• ]
• }
• }
• }
• }
• Advanced Metric Capabilities: procstat and statsd

Process Monitoring with procstat : Monitors individual operating system processes. Processes can be matched using exe (executable name), pattern (command line regex matching), or pid_file (reading process ID from /var/run/*.pid ). Metrics emitted include procstat_cpu_usage , procstat_memory_rss (resident set size), and procstat_num_threads .

Embedded StatsD Listener : The agent can expose a local UDP or TCP socket (default port 8125). Applications running on the instance can send custom metrics using standard StatsD formatting ( statsd.increment("checkout.success") ). The agent buffers and aggregates these counters locally before publishing to CloudWatch, drastically reducing PutMetricData API calls.

Fleet-Wide Deployment and Automation with AWS Systems Manager (SSM)

In enterprise DevOps environments, manually connecting to instances via SSH or RDP to install and configure software is strictly prohibited. Fleet-wide agent lifecycle management is orchestrated entirely through AWS Systems Manager .

Centralized Configuration via SSM Parameter Store

Instead of baking configuration files into Amazon Machine Images (AMIs) or hardcoding them on local disks, store the agent configuration centrally in AWS Systems Manager Parameter Store :

• Parameter Name : /configuration/cloudwatch-agent/linux-production
• Type : String

Value : The raw JSON configuration content.

• Step-by-Step Automated Deployment Workflow 1. Attach IAM Instance Profile
• ├── CloudWatchAgentServerPolicy
• └── AmazonSSMManagedInstanceCore
• │
• 2. Store Agent JSON in SSM Parameter Store
• └── /enterprise/cwagent/linux-config
• │
• 3. Install Agent via SSM Run Command / State Manager
• └── Document: AWS-ConfigureAWSPackage (Package: AmazonCloudWatchAgent)
• │
• 4. Configure & Start Agent via SSM Run Command
• └── Document: AmazonCloudWatch-ManageAgent (Action: configure, Mode: ec2)

IAM Permissions : The EC2 instance profile must include:

CloudWatchAgentServerPolicy : Grants permissions to publish metrics ( cloudwatch:PutMetricData ), create log streams, and put log events.

AmazonSSMManagedInstanceCore : Grants permissions for the SSM Agent to communicate with Systems Manager.

Inline or custom policy allowing ssm:GetParameter on the specific Parameter Store configuration ARN.

Package Installation : Use SSM Run Command or SSM State Manager with document AWS-ConfigureAWSPackage :

• action : Install
• name : AmazonCloudWatchAgent

Agent Configuration & Startup : Use SSM document AmazonCloudWatch-ManageAgent :

• action : configure
• mode : ec2
• optional_configuration_source : ssm
• optional_configuration_location : /configuration/cloudwatch-agent/linux-production
• optional_restart : yes

Continuous Drift Management with State Manager : Create an SSM State Manager Association targeting instances by tag (e.g., Environment=Production ). The association periodically enforces that the agent package is installed and running with the designated Parameter Store configuration, eliminating configuration drift across dynamic Auto Scaling groups.

Windows Telemetry: Performance Counters and Event Logs

On Windows Server instances, the Unified CloudWatch Agent hooks directly into the Windows Management Instrumentation (WMI) subsystem and the Windows Event Log service.

Windows Performance Counters

In the metrics_collected block under windows_events and Windows performance counter sections, administrators collect standard OS counters:

Memory : % Committed Bytes In Use , Available MBytes

LogicalDisk : % Free Space , Free Megabytes for volumes C: , D: , etc.

• Paging File : % Usage
• System : Processor Queue Length
• Windows Event Log Ingestion
• The logs_collected section captures critical Windows Event channels: {
• "logs": {
• "logs_collected": {
• "windows_events": {
• "collect_list": [
• {
• "event_name": "System",
• "event_levels": ["ERROR", "CRITICAL"],
• "log_group_name": "/aws/ec2/windows/system",
• "log_stream_name": "{instance_id}"
• },
• {
• "event_name": "Security",
• "event_levels": ["INFORMATION", "WARNING", "ERROR"],
• "log_group_name": "/aws/ec2/windows/security",
• "log_stream_name": "{instance_id}"
• }
• ]
• }
• }
• }
• }
• Troubleshooting Agent Permissions and Network Connectivity

Symptom Root Cause Remediation

Agent status is running , but no metrics appear in CWAgent namespace EC2 IAM role missing cloudwatch:PutMetricData permission Attach CloudWatchAgentServerPolicy managed policy to instance profile

Instances in private subnets fail to send metrics or logs; timeout errors in agent log Private subnet lacks internet egress and has no VPC Endpoints Provision VPC Interface Endpoints for monitoring ( com.amazonaws.<region>.monitoring ), logs ( com.amazonaws.<region>.logs ), and ssm ( ssm , ssmmessages , ec2messages )

SSM Run Command AmazonCloudWatch-ManageAgent fails with InvalidParameterException Parameter Store path is incorrect or IAM role lacks ssm:GetParameter permission Verify the parameter name matches the SSM document input and verify the instance profile allows ssm:GetParameter on the parameter ARN

Log files stop uploading after log rotation Agent cannot track newly created files because file_path regex does not match the rotated suffix Use wildcards in file_path (e.g., /var/log/app/*.log ) and configure multi_line_start_pattern appropriately

• Local Verification Commands
• On a Linux host, check agent runtime status using the bundled CLI utility: # Check agent status
• /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status

• # Inspect local agent log
• sudo tail -n 100 /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

• # Manually apply configuration from local file for testing
• sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
• a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

Loading diagram... Unified CloudWatch Agent Fleet Deployment & Telemetry Flow