Instance Performance Monitoring in AWS

AWS CloudWatch is Amazon’s built-in monitoring service. It collects metrics (including custom ones you define) and can trigger alarms when a metric crosses a threshold. It is the main tool for performance monitoring inside AWS.

This guide covers a typical scenario: you’ve deployed an application to the cloud and need to work out whether performance problems come from the application code or from limits that Amazon enforces.

You’ve just started with AWS and are running applications on the free tier or on general-purpose T2 instances. These instances use a CPU credit system: credits let the instance burst through short load spikes, but once they run out, performance drops back to a low, fixed baseline. The details of the credit mechanics may seem obscure, but you still need to know whether your application can meet its service targets with this setup.

If you’re responsible for meeting service targets, you need control over the cloud environment. Treat the following as red flags:

• You don’t have full access to the AWS Console, so you can’t experiment with instance and storage volume types.
• You don’t have access to the monitoring engine (CloudWatch).

If you’re only handed a bare cloud server and told to deploy the application and hit service targets, walk away. You’re being set up for failure through no fault of your own.

AWS doesn’t expose hypervisor metrics from inside the instance. This is standard industry practice: monitoring should be external. From within the instance, enforced resource caps show up only as general slowness or timeouts.

You can, however, retrieve static instance metadata from inside the instance:

[root@ec2instance ~]# curl http://169.254.169.254/latest/meta-data/
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
...

Every instance and volume has a console tab with graphs of hypervisor-generated metrics: CPU utilization, network traffic and I/O statistics. By default, nothing OS-level (memory, disk usage) is included, so you must define custom metrics for those. Amazon provides helper tools, such as the CloudWatch Monitoring Scripts for Linux.

For an application on a performance-capped general-purpose instance, watch these metrics:

• Credit usage is a prime indicator of compute spikes.
• An exhausted balance means the hypervisor throttled your instance’s CPU.
• A constantly zero balance means a compute-intensive application is on the wrong instance type for the expected load. It can also point to a problem in the application itself. (The same behavior can be reproduced easily with a cryptocurrency miner or a fork bomb.)

• If the value doesn’t return to 0 over extended periods, read or write operations routinely exceed the IOPS limits Amazon enforces.
• Read Throughput and Write Throughput give additional detail.

Ways to address I/O pressure:

• Excessive reads: use instances with more memory so caching absorbs them (common for web servers).
• Excessive writes: check for misconfigured logging and reduce the data written to the volume.
• Otherwise: use larger volumes (for gp2, the IOPS quota scales with volume size) or switch to provisioned IOPS (io1) volumes.

Note 1: The CPU credit system applies only to “T” (general-purpose) instance types. Other types deliver their nominal performance at all times.

Note 2: IOPS limits apply only to EBS (SSD) volumes. Some instance types include local instance storage with performance closer to raw hardware limits. Newer HDD types (st1, sc1) are throughput-limited rather than operation-limited.

You can pull CloudWatch data with the AWS CLI, which is useful for deeper analysis such as examining specific time periods. Installing the CLI is out of scope here; you’ll need a key pair to use it.

You can also get these from the AWS Console, e.g. by instance name (if the Name tag is set):

aws ec2 describe-instances --filter Name=tag:Name,Values=myinstance.local

Or by instance ID:

aws ec2 describe-instances --instance-ids i-abcd1234

From the large JSON response, identify:

1. The instance ID
2. The volume ID(s), if there is more than one

aws cloudwatch get-metric-statistics \
  --metric-name CPUCreditBalance \
  --start-time 2016-06-05T23:18:00 \
  --end-time 2016-06-06T23:18:00 \
  --period 3600 \
  --namespace AWS/EC2 \
  --statistics Maximum \
  --dimensions Name=InstanceId,Value=i-abcd1234

Example output:

{
  "Datapoints": [
    {
      "Timestamp": "2016-06-06T21:18:00Z",
      "Maximum": 287.74,
      "Unit": "Count"
    },
    {
      "Timestamp": "2016-06-06T09:18:00Z",
      "Maximum": 287.81,
      "Unit": "Count"
    }
  ]
}

This example requests the maximum, but minimum may give better insight. Experiment with parameters to find the best combination. Datapoints are returned unsorted, so import the JSON into another tool for analysis.

aws cloudwatch list-metrics \
  --namespace AWS/EBS \
  --dimensions Name=VolumeId,Value=vol-abcd1234

Example output:

{
  "Metrics": [
    {
      "Namespace": "AWS/EBS",
      "Dimensions": [
        { "Name": "VolumeId", "Value": "vol-abcd1234" }
      ],
      "MetricName": "VolumeQueueLength"
    },
    {
      "Namespace": "AWS/EBS",
      "Dimensions": [
        { "Name": "VolumeId", "Value": "vol-abcd1234" }
      ],
      "MetricName": "VolumeReadOps"
    }
  ]
}

Note that the metric names differ from the graph titles in the AWS Console.

aws cloudwatch get-metric-statistics \
  --metric-name VolumeQueueLength \
  --start-time 2016-06-05T23:18:00 \
  --end-time 2016-06-06T23:18:00 \
  --period 3600 \
  --namespace AWS/EBS \
  --statistics Sum \
  --dimensions Name=VolumeId,Value=vol-abcd1234

Example output:

{
  "Datapoints": [
    {
      "Timestamp": "2016-06-06T21:18:00Z",
      "Sum": 0.0643,
      "Unit": "Count"
    },
    {
      "Timestamp": "2016-06-06T09:18:00Z",
      "Sum": 0.0066,
      "Unit": "Count"
    }
  ]
}

Every operation passes through the queue on its way to or from storage, so the sum is never exactly 0. What matters is how long requests wait in the queue: small values indicate healthy performance. As before, datapoints are unsorted, so you may need to import the JSON elsewhere to analyze it.

AWS instances don’t behave like physical servers: the hypervisor enforces various resource limits. Always monitor instance behavior through CloudWatch.

Note: This article was written by an AWS Certified Solutions Architect (Associate). Always work with an expert when setting up production environments.