Faster autoscaling on Amazon SageMaker realtime endpoints (Application Autoscaling)

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

In this notebook we show how the new faster autoscaling feature helps scale sagemaker inference endpoints by almost 6x faster than earlier.

We deploy Meta's Llama3-8B-Instruct model to an Amazon SageMaker realtime endpoint using Text Generation Inference (TGI) Deep Learning Container (DLC) and apply Application Autoscaling scaling policies to the endpoint.

Prerequisites

Install packages using uv, an extremely fast python package installer
Read more about uv here https://astral.sh/blog/uv

Initiate sagemaker session

Deploy model

Create and deploy model using Amazon SageMaker HuggingFace TGI DLC

https://sagemaker.readthedocs.io/en/stable/api/inference/model.html#sagemaker.model.Model.deploy

Inference

Invoke and test endpoint using messages API. Refer to HF Messages API for more info.

Baseline average latency at various concurrency levels (Optional)

By capturing average latency across various concurrency levels, we can get a fair idea on after how many concurrent request does endpoint performance would degrade significantly.

Having this information can help define values for scaling policy accordingly.

Apply Autoscaling policies to the endpoint

Apply Application Autoscaling Policy to endpoint

1. Register Scalable Target

Use the latest high-resolution Metrics to trigger auto-scaling

• New feature introduces a new PredefinedMetricType for scaling policy configuration i.e. SageMakerVariantConcurrentRequestsPerModelHighResolution to trigger scaling actions.
• Creating a scaling policy with this metric type will create cloudwatch alarms that track a new metric called ConcurrentRequestsPerModel.
• These high-resolution metrics are published at sub-minute intervals (10s intervals to CW + any additional jitter + delays)
• We should observe significant improvement in scale out times with this new metric

Steps to create Application autoscaling policy

• Create scaling policy
  • Set PolicyType to TargetTrackingScaling
  • Set TargetValue to 5.0. i.e., Scaling triggers when endpoint receives 5 ConcurrentRequestsPerModel
  • Set PredefinedMetricType to SageMakerVariantConcurrentRequestsPerModelHighResolution
  • Set ScaleInCoolDown and ScaleOutCoolDown values to 300 seconds

Trigger autoscaling action

LLMPerf to generate traffic to the endpoint

Refer to https://github.com/philschmid/llmperf for more details on LLMPerf.

Run the LLMPerf traffic generation script in the background using subprocess.Popen

Monitor Scale-Out Alarm Trigger times and scaling event times

As llmperf generates traffic to the endpoint continuously this trigger auto-scaling.

The monitor_scaling_events function does the following:

• Calculates time taken for alarm to go into InAlarm state.
• checks if alarm is InAlarm state. If yes, then starts the scaling timer
• continuously monitors the DesiredInstanceCount property of the endpoint
  • waits till CurrentInstanceCount == DesiredInstanceCount and EndpointStatus is InService
• Calculates time taken to scale out instances prints the times in a table

The below cell triggers auto scaling action and calls the monitor_scaling_events immediately on the AlarmHigh

Monitor if the background process (llmperf) is completed.

Print LLMPerf results

LLMPerf writes the results to "results/" directory. summary.json file has the endpoint benchmarking data.

Monitor Scale-in Alarm Trigger times and scaling event times

Cleanup

• Deregister scalable target. This automatically deletes associated cloudwatch alarms.
• Delete model
• Delete endpoint

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.