Optimize SageMaker Endpoint Auto scaling using Inference recommender

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

1. Introduction

Amazon SageMaker real-time endpoints allow you to host ML applications at scale. SageMaker Hosting offers various scaling options for real-time endpoint and they are detailed in this blog. In summary, SageMaker endpoint supports three scaling options. The first and the recommended option is TragetTracking. In this option, a target value that represents the average utilization or/and throughput of single host is set as a scaling threshold. Secondly, you can define StepScaling, which is an advanced method for scaling based on the size of alarm breach. The final one is scheduled scaling, which lets you specify an one-time schedule or a recurring schedule. It is always recommended to combine the scaling options to better resilience.

In this notebook, we provide a design pattern for arriving right deployment configuration for your application. In addition, we provide a list of steps to follow, so even if your application has an unique behavior, such as a different system characteristics or traffic pattern, these systemic approach can be applied to arrive the scaling policies. The procedure is further simplified with the use of Inference recommender, a right sizing and benchmarking tool, built inside SageMaker.

2. Setup

Note that we are using the conda_python3 kernel in SageMaker Notebook Instances. This is running Python 3.6. If you'd like to use the same setup, in the AWS Management Console, go to the Amazon SageMaker console. Choose Notebook Instances, and click create a new notebook instance. Upload the current notebook and set the kernel. You can also run this in SageMaker Studio Notebooks with the Python 3 (Data Science) kernel.

In the next steps, you import standard methods and libraries as well as set variables that will be used in this notebook. The get_execution_role function retrieves the AWS Identity and Access Management (IAM) role you created at the time of creating your notebook instance.

2.1 Permission update

In addition to the Sagemaker permission, you need application auto-scaling and cloudwatch permission to execute this notebook. Please add the below policy to your notebook execution role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
            	"application-autoscaling:RegisterScalableTarget",
                "application-autoscaling:DescribeScalableTargets",
                "application-autoscaling:DescribeScalingPolicies",
                "application-autoscaling:PutScalingPolicy",
                "application-autoscaling:DeleteScalingPolicy",
                "application-autoscaling:DeregisterScalableTarget",
                "cloudwatch:GetMetricStatistics",
                "cloudwatch:GetMetricWidgetImage"
            ],
            "Resource": "*"
        }
    ]
}

Optional: Train an XGBoost model

Let's quickly train an XGBoost model. If you already have a model, you can skip this step and proceed to the next section.

For the purposes of this notebook, we are training an XGBoost model on random data. As a first step, install scikit-learn and XGBoost

Generate Model and payload

As mentioned, you should skip this step and proceed to the next section if you already have the model. In addition to the model, you should also have a valid payload which can be used for testing your model.

In this case, we are training the XGBoost model with random data and also generating a sample payload for the exercise. However this procedure is independent of the deployment model or configuration. So you can adopt the same approach for your application and for your deployment choice.

3. Create a tarball

If you are bringing your own XGBoost model, SageMaker requires that it is in .tar.gz format containing a model file and, optionally, inference code.

Similarly for the payload, SageMaker Inference recommender service expects a single archive file in .tar.gz format, which contains a list of valid samples.

4. Upload to S3

We now have a model and the payload archive ready. We need to upload it to S3 before we can use with Inference Recommender. Furthermore, we use the SageMaker Python SDK to handle the upload.

5. Container image URL

If you don’t have an inference container image, you can use one of the open source AWS Deep Learning Containers (DLCs) provided by AWS to serve your ML model. The code below retrieves a SageMaker managed container for XGBoost.

6a. Identify Application characteristics

To dertermine to the correct scaling property, the first step in the plan is to find application behavior on the chosen hardware. This can be achieved by running the application on a single host and increasing the request load to the endpoint gradually until it saturates.

For benchmarking, we use the Inference recommender Default job. In Default recommendation job, we can limit the search to a single instance by passing the single instance type in the supported_instance_types. The service then provisioning the endpoint. It then gradually increases the request and stops when the benchmark reaches saturation or when 1% of invocations call (invoke_endpoint) fails.

6b. Analyzing the result:

We then analyze the results using hosting endpoint metrics. Below is a visualization of the invocation metrics, and from that, it follows that the hardware utilization. By layering invocations and utilization graphs, we are able to easily set limits for invocations per instance.

In this step we run various scaling percentage to find the right scaling limit. As a general scaling rule, the utilization percentage should be around 40% if you are optimizing for availability, around 70% if you are optimizing for cost, and around 50% if you want to balance availability and cost. The above guidance gives an overview of the two dimensions, availability and cost. The lower the threshold, the better the availability, and the higher the threshold, the better the cost.

7. Set Scaling expectation:

The next step in the plan is to set the scaling expectation and develop scaling policies based on that expectation. This step involves defining the maximum and minimum requests to be served.

For our application, the expectations are maximum requests per second (max) = 500, and minimum request per second (min) = 70.

8. Create endpoint and set scaling policy

The final step in the plan is to define a scaling policy and evaluate its impact. The evaluation step is essential to validate the results of the calculations made so far. In addition, it helps us adjust the scaling setting if it does not meet the need.

The first step in this procedure is to provision an Endpoint.

8a. Apply scaling policies to SageMaker endpoint

In order to define the scaling policies, you should register scaling and then add policies. The following section defines functions for registering scaling, setting target tracking policies on CPU utilization and InvocationsPerInstance.

We use the defined functions to apply scaling configuration to provision SageMaker endpoint

8c. Evaluate the scaling policy

The evaluation is done using the Inference recommender Advanced job, where we specify the traffic pattern, MaxInvocations and Endpoint to benchmark against. In this case, we provision the endpoint and set the scaling policies, then run the inference recommender Advanced job to validate the policy.

Now with the scaling policy, we could achieve 500 TPS at a request rate where we increase one user per minute. But does the same policy work when we increase the request rate twice? Let's rerun the same evaluation set with a higher request rate.

To execute the step again, clear the auto scaling policies we set before and reset to the initial instance count.

We use the same defined functions to apply scaling configuration to provision SageMaker endpoint, again.

8d. Repeat the process if the anything changes.

We then increase the request rate (aka spawn rate) to 4 user per minute, meaning we increase the request rate twice as fast.

Note, the evaluation fails as the max invocation reached is less than 30000 invocations / 500 TPS. So to improve availability, we now need to alter our scaling policy.

Using step scaling configuration, we try to set scaling adjustment to provision n of instances when a threshold is met. To try Step scaling, we now reset to the initial instance count and attempt setting the revised scaling policies.

Repeat the process of evaluation with Inference recommender advanced job.

Step scaling now allows us to reach the expected TPS at twice the rate of the original test.

Therefore, defining the scaling policies and evaluating the results using the Inference recommender is a necessary part of validation.

9. Clean up the resources

If you don't intend on trying out inference or to do anything else with the endpoint, you should delete it.

10. Conclusion:

The process of defining the scaling policy for your application can be challenging. You must understand the characteristics of the application, determine your scaling needs, and iterate scaling policies to meet those needs. This notebook has reviewed each of these steps carefully and explained the approach you should take at each step. Furthermore, the notebook simplified the process with the SageMaker Inference recommender. You can find your application characteristics and evaluate scaling policies by using the Inference recommender benchmarking system. The proposed design pattern can help you create a scalable application that takes into account the availability and cost of your application in hours rather than days.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.