# EC2 Autoscaling with Custom Metrics

Today you will learn how to `define/create/publish` the custom metrics and configure target tracking Autoscaling for your EC2 Autoscaling groups based on the metrics. In this blog we will:

* Create ASG, SQS Queue, Target Tracking Policy
    
* Use bash scripts to publish/consume messages to/from the SQS Queue
    
* Use bash scripts to publish the custom metrics
    
* Finally, autoscale based on the message count on the queue
    

## Scenario

Let's suppose a scenario where users publish a data processing request in an AWS SQS. The SLA for the end user is that they should get the results within 2 seconds of the request being sent. Let's suppose each request takes `0.1 seconds` to be processed by an EC2 Instance.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1686042437197/c1aa0628-a9dd-4153-9298-9eaba71fb188.png align="center")

**Calculation backlog per Instance**:

Now let's calculate the **message backlog count per instance** to determine when to run the autoscaling.

Backlog Per Instance = Time User Can Wait/Time it takes to serve a Request  
Backlog Per Instance = 2/0.1 = 20

This means the number of messages in the `SQS queue/Instance count` shouldn't be greater than 20.

Now we have found **Backlog Per Instance=20** as our target for autoscaling. Enough theory now let's get started with creating the necessary setup in the AWS Account.

**NOTE:** <mark>Please be careful with this tutorial as it can add extra costs to your AWS bills</mark>

## Pre-requisite

* EC2 Autoscaling Group
    
* Amazon SQS
    
* Basic Knowledge of AWS CLI and CloudWatch Metrics
    

### **Create Autoscaling Group**

I already have an Autoscaling group with the name `CustomMetricsASG` and please verify your `min/max/desired` capacity is properly set to allow the autoscaling to happen.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1686039139890/ec52a831-ee57-404e-8794-a00f459c1457.png align="center")

### **Create Queue**

Create a queue using the below command and verify

```bash
➜ aws sqs create-queue --queue-name myasgqueue
{
    "QueueUrl": "https://sqs.us-east-1.amazonaws.com/922726392568/myasgqueue"
}
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1686039295302/66c72acb-e519-46f7-b845-0f561573309f.png align="center")

### **Publish Messages to SQS**

Now you can use the below commands to publish the messages in the SQS queue which acts like the requests sent to our system in real-world scenario.

```bash
##Producer

##Replace with your queue endpoint
QUEUE_URL=https://sqs.us-east-1.amazonaws.com/922726392568/myasgqueue
while true
do
  echo "Publishing messages..."
  sleep 3
  aws sqs send-message --queue-url ${QUEUE_URL} \
   --message-body "aws asg mock messages to increase the load..." \
   --no-cli-pager
done
```

If the above script ran successfully you will see outputs like below:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1686039916183/765280c6-39ca-46aa-9e08-80fa3d0b0424.png align="center")

### **Consume Messages from SQS(Optional...)**

This is optional but if you want to make it more realistic what you can do is increase the sleep time below such that publishing is done fast and the consuming is slower which increases the backlog in the Queue.

```bash
## Conumer
##Replace with your queue endpoint
QUEUE_URL=https://sqs.us-east-1.amazonaws.com/922726392568/myasgqueue
while true
do
  echo "consuming messages..."
  sleep 3
  aws sqs receive-message --queue-url ${${QUEUE_URL}} \
  --no-cli-pager
done
```

### **Publishing Custom Metrics**

The below script does the following things:

* Fetches the number of available messages in the SQS Queue
    
* Fetches the number of the EC2 Instances in `InService` state
    
* Divides `Message Count/InstanceCount` to get the message `BacklogPerInstance`
    
* Then publish the custom metrics every 60 seconds setting:
    
    * MetricName: `MyBacklogPerInstance`
        
    * Namespace: `MyCustomASGMetrics`
        
    * Dimension as: `QueueName=${QUEUE_NAME}`
        

**NOTE:** Before running the script, please replace the values of the below variables.

```bash
#!/bin/bash
##Replace these values, before running the script
QUEUE_URL=https://sqs.us-east-1.amazonaws.com/922726392568/myasgqueue
ASG_NAME=CustomMetricsASG
QUEUE_NAME=myasgqueue

while true
do
  echo "....starting...."
  sleep 60
  printf "[INFO] Querying.. Available Queue Message\n"
  APPROX_AVAILABLE_MESSAGES=$(aws sqs get-queue-attributes --queue-url ${QUEUE_URL} --attribute-names ApproximateNumberOfMessages --query Attributes.ApproximateNumberOfMessages --output text | tr -d '[:space:]')
  STATUS_CODE=$?
  if [[ ${STATUS_CODE} -ne 0 ]]
  then
    printf "[WARN] APPROX_AVAILABLE_MESSAGES retrival failed with status code: %s ...\n" ${STATUS_CODE}
    continue
  fi
  printf "[INFO] The Number of Available Message: %s\n" "${APPROX_AVAILABLE_MESSAGES}"

  printf "[INFO] Querying.. Number of Instance in Service\n"
  IN_SERVICE_COUNT=$(aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names ${ASG_NAME} --query "AutoScalingGroups[].Instances[?LifecycleState=='InService'].[InstanceId]"\
    --output text | wc -l | tr -d '[:space:]')
  STATUS_CODE=$?
  if [[ ${STATUS_CODE} -ne 0 ]]
  then
    printf "[WARN] IN_SERVICE_COUNT retrival failed with status code: %s ...\n" ${STATUS_CODE}
    continue
  fi
  printf "[INFO] The Number of Instance in Service: %s\n" "${IN_SERVICE_COUNT}"

  BACKLOG_PER_INSTANCE=$((APPROX_AVAILABLE_MESSAGES / IN_SERVICE_COUNT))
  STATUS_CODE=$?
  if [[ ${STATUS_CODE} -ne 0 ]]
  then
    printf "[WARN] BACKLOG_PER_INSTANCE calculation failed with status code: %s ...\n" ${STATUS_CODE}
    continue
  fi
  printf "[INFO] Calculated Backlog per instance %s/%s: %s\n" "${APPROX_AVAILABLE_MESSAGES}" "${IN_SERVICE_COUNT}" "${BACKLOG_PER_INSTANCE}"

  aws cloudwatch put-metric-data --metric-name MyBacklogPerInstance --namespace MyCustomASGMetrics \
  --unit None --value ${BACKLOG_PER_INSTANCE} --dimensions QueueName=${QUEUE_NAME}
  STATUS_CODE=$?
  if [[ ${STATUS_CODE} -ne 0 ]]
  then
    printf "[WARN] put-metrics-data failed with status code: %s ...\n" ${STATUS_CODE}
    continue
  fi
  printf "[INFO] Successfully published custom metrics with value: %s ...\n" ${BACKLOG_PER_INSTANCE}
done
```

Once you run the above script, if everything goes fine you should be able to see the custom metrics in the AWS CloudWatch Metrics

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1686042805428/b7e83710-a0d8-4f40-805e-1a8008953730.png align="center")

### **Creating TargetTracking Autoscaling**

Until now you can't create target tracking autoscaling Policy with custom metrics from AWS Console so, you need to create it from AWS CLI.

**Policy Config**

Please copy the below config and update with your relevant values:

```json
{
   "TargetValue":20,
   "CustomizedMetricSpecification":{
      "MetricName":"MyBacklogPerInstance",
      "Namespace":"MyCustomASGMetrics",
      "Dimensions":[
         {
            "Name":"QueueName",
            "Value":"myasgqueue"
         }
      ],
      "Statistic":"Average",
      "Unit":"None"
   }
}
```

**Create the Policy**

Replace the values in the below variables and run the command

```bash
##Replace the below ASG_NAME with your values
ASG_NAME=CustomMetricsASG
aws autoscaling put-scaling-policy --policy-name sqs20-target-tracking-scaling-policy \
  --auto-scaling-group-name ${ASG_NAME} --policy-type TargetTrackingScaling \
  --target-tracking-configuration file://config.json
```

Once created successfully you will be able to see it in the Autoscaling section of the ASG

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1686042765011/8c81b527-42f6-4c5b-a830-17321572adc0.png align="center")

### Scaling in Action

If there are enough points in the custom metrics and the value is greater than the defined threshold you will quickly see the scaling happening in the action.

**Checking Activity in ASG**

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1686042992797/3d1ff355-b86c-4278-b5cb-b081bf708cd1.png align="center")

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1686043042479/5cc0a421-0684-4597-9905-18741d1c7e69.png align="center")

### Conclusion

This is how you can setup autoscaling for any custom metrics and with any other autoscaling policies. You can even mix it with other autoscaling policies like: simple, step autoscaling.

Thanks for reading! Catch you next time!