Leverage deployment guardrails to update a SageMaker Inference endpoint using rolling deployment

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

SageMaker Studio Kernel: Data Science

Contents

• Introduction
• Setup
• Step 1: Create and deploy the pre-trained models
• Step 2: Invoke Endpoint
• Step 3: Create CloudWatch alarms to monitor Endpoint performance
• Step 4: Update Endpoint with rolling deployment configurations
• Cleanup

Introduction

Deployment guardrails are a set of model deployment options in Amazon SageMaker Inference to update your machine learning models in production. Using the fully managed deployment guardrails options, you can control the switch from the current model in production to a new one.

When you update your endpoint, you can specify a rolling deployment to gradually shift traffic from your old fleet to a new fleet. You can control the size of the traffic shifting steps, as well as specify an evaluation period to monitor the new instances for issues before terminating instances from the old fleet. With rolling deployments, instances on the old fleet are cleaned up after each traffic shift to the new fleet, reducing the amount of additional instances needed to update your endpoint. This is useful especially for accelerated instances that are in high demand.

Rolling deployments work similarly to the linear traffic shifting mode in blue/green deployments, but rolling deployments provide you with the added benefit fewer capacity needs. Compared to blue/green deployments with canary or linear traffic shifting modes, rolling deployments can also have lower costs. With rolling deployments, fewer instances are active at a time, and you have more granular control over how many instances you want to update in the new fleet. You should consider using a rolling deployment instead of a blue/green deployment if you have large models or a large endpoint with many instances.

The following list describes the key features of rolling deployments in SageMaker:

• Baking period. The baking period is a set amount of time to monitor the new fleet before proceeding to the next deployment stage. If any of the pre-specified alarms trip during any baking period, then all endpoint traffic rolls back to the old fleet. The baking period helps you to build confidence in your update before making the traffic shift permanent.
• Rolling batch size. You have granular control over the size of each batch for traffic shifting, or the number of instances you want to update in each batch. This number can range for 5-50% of the size of your fleet. You can specify the batch size as a number of instances or as the overall percentage of your fleet.
• Auto-rollbacks. You can specify Amazon CloudWatch alarms that SageMaker uses to monitor the new fleet. If an issue with the updated code trips any of the alarms, SageMaker initiates an auto-rollback to the old fleet in order to maintain availability, thereby minimizing risk.

In this notebook we'll update endpoint with following deployment configurations:

• Rolling update policy
• Configure CloudWatch alarms to monitor model performance and trigger auto-rollback action.

To demonstrate rolling deployments and the auto-rollback feature, we will update an Endpoint with an incompatible model version and deploy it as a Rolling fleet, taking a small percentage of the traffic. Requests sent to this Rolling fleet will result in errors, which will be used to trigger a rollback using pre-specified CloudWatch alarms. Finally, we will also demonstrate a success scenario where no alarms are tripped and the update succeeds.

This notebook is organized in 4 steps -

• Step 1 creates the models and Endpoint Configurations required for the 3 scenarios - the baseline, the update containing the incompatible model version and the update containing the correct model version.
• Step 2 invokes the baseline Endpoint prior to the update.
• Step 3 specifies the CloudWatch alarms used to trigger the rollbacks.
• Finally in step 4, we update the endpoint to trigger a rollback and demonstrate a successful update.

Setup

First we ensure we have an updated version of boto3, which includes the latest SageMaker features:

Let's set up some required imports and basic initial variables:

Download the Input files and pre-trained model from S3 bucket

Step 1: Create and deploy the models

First, we upload our pre-trained models to Amazon S3

This code uploads two pre-trained XGBoost models that are ready for you to deploy. These models were trained using the XGB Churn Prediction Notebook in SageMaker. You can also use your own pre-trained models in this step. If you already have a pretrained model in Amazon S3, you can add it by specifying the s3_key.

The models in this example are used to predict the probability of a mobile customer leaving their current mobile operator. The dataset we use is publicly available and was mentioned in the book Discovering Knowledge in Data by Daniel T. Larose. It is attributed by the author to the University of California Irvine Repository of Machine Learning Datasets.

Next, we create our model definitions

Start with deploying the pre-trained churn prediction models. Here, you create the model objects with the image and model data.

Create Endpoint Configs

We now create three EndpointConfigs, each with its own different model (these could also have different instance types).

Create Endpoint

Let's go ahead and deploy the model to a SageMaker endpoint:

Wait for the endpoint creation to complete.

Step 2: Invoke Endpoint

You can now send data to this endpoint to get inferences in real time.

This step invokes the endpoint with included sample data with maximum invocations count and waiting intervals.

Invocations Metrics

Amazon SageMaker emits metrics such as Latency and Invocations (full list of metrics here) per variant and endpoint configuration in Amazon CloudWatch.

Let’s query CloudWatch to get number of Invocations and latency metrics per variant and endpoint configuration.

Plot endpoint invocation metrics:

Below, we are going to plot graphs to show the Invocations,Invocation4XXErrors,Invocation5XXErrors,ModelLatency and OverheadLatency against the Endpoint.

You will observe that there should be a flat line for Invocation4XXErrors and Invocation5XXErrors as we are using the correct model version and configs. Additionally, ModelLatency and OverheadLatency will start decreasing over time.

Step 3: Create CloudWatch alarms to monitor Endpoint performance

In this step we're going to create CloudWatch alarms to monitor Endpoint performance with following metrics:

• Invocation5XXErrors
• ModelLatency

Following metric dimensions are used to select the metric per Endpoint config and variant:

• EndpointName
• VariantName

Step 4: Update Endpoint with rolling deployment configurations

Now we try to update the endpoint with rolling deployment configurations and monitor the performance from CloudWatch metrics.

Rolling update policy

We define the following deployment configuration to perform Rolling deployment strategy. The rolling deployment provisions capacity and shifts traffic to a new fleet in steps of a batch size that you specify. Instances on the new fleet are updated with the new deployment configuration, and if no alarms trip during the baking period, then SageMaker cleans up instances on the old fleet.

Rollback Case

Update the Endpoint with an incompatible model version to simulate errors and trigger a rollback.

We invoke the endpoint during the update operation is in progress.

Note : Invoke endpoint in this notebook is in single thread mode, to stop the invoke requests please stop the cell execution

The E's denote the errors generated from the incompatible model version in the rolling fleet.

The purpose of the below cell is to simulate errors in the rolling fleet. Since the nature of traffic shifting to the rolling fleet is probabilistic, you should wait until you start seeing errors. Then, you may proceed to stop the execution of the below cell. If not aborted, cell will run for 600 invocations.

Wait for the update operation to complete and verify the automatic rollback.

Collect the endpoint metrics during the deployment:

Below, we are going to plot graphs to show the Invocations,Invocation5XXErrors and ModelLatency against the Endpoint.

You can expect to see as the new endpoint config-2 (erroneous due to model version) starts getting deployed, it encounters failure and leads to the rollback to endpoint config-1. This can be seen in the graphs below as the Invocation5XXErrors and ModelLatency increases during this rollback phase

Let's take a look at the Success case where we use the same Rolling deployment configuration but a valid endpoint configuration.

Success Case

Now let's update the endpoint to a valid endpoint configuration version with the same Rolling deployment configuration:

We invoke the endpoint during the update operation is in progress:

Wait for the update operation to complete:

Collect the endpoint metrics during the deployment:

Below, we are going to plot graphs to show the Invocations,Invocation5XXErrors and ModelLatency against the Endpoint.

You can expect to see that, as the new endpoint config-3 (correct model version) starts getting deployed, it takes over endpoint config-2 (erroneous due to model version) without any errors. This can be seen in the graphs below as the Invocation5XXErrors and ModelLatency decreases during this transition phase

The Amazon CloudWatch metrics for the total invocations for each endpoint config shows how invocation requests are shifted from the old version to the new version during deployment.

You can now safely update your endpoint and monitor model regressions during deployment and trigger auto-rollback action.

Cleanup

If you do not plan to use this endpoint further, you should delete the endpoint to avoid incurring additional charges.

You should also clean up the other resources created in this notebook: endpoint configurations, models, and CloudWatch alarms.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.