I have mentioned before that regular EBS storage in AWS works with a system of credits that are accrued when the volume is idle and are spent when the operation rate requested on the volume grows over the predefined baseline. One can find the detailed explanation in the AWS documentation. This limitation came as an unexpected surprise during the failed project I was part of 18 months back.
A few days ago while syncing data from a large volume to another, larger volume, I ran into the same problem as well and I decided to capture the state to make things clear for everybody (well, people that are reading this blog).
The CloudWatch read throughput (metric name: VolumeReadOps) graph looked like this during the data sync:
I have marked 3 areas in need of explanation:
In this area the volume was in “Credit Consumption” mode. As long as there were credits available, the performance went close to the maximum provided by Amazon given the instance and the volume type.
Credits were exhausted, the volume was reverted to the low baseline (3 times size in Gb – so 150 for this particular volume).
A short idle period caused some credit accrual that consequently allowed a small burst before the performance was again reverted to the baseline. You can see it here in a bit more detail:
Conclusion? Be careful when planning load in AWS. Do always monitor CloudWatch metrics and be prepared to switch to Provisioned IOPS volumes.
Note: This text was written by an AWS Certified Solutions Architect (Associate). Please do always work with an expert when setting up production environments.