Bring your own container

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

This notebook shows an example of bring your own container. This example leverages the MultiModelServer to host MME and this example can be further modified and adapted to fit your needs

Multi-Model Endpoint - CatBoost

This example notebook also showcases how to use a custom container to host multiple CatBoost models on a SageMaker Multi Model Endpoint. The model this notebook deploys is taken from this CatBoost tutorial.

We are using this framework as an example to demonstrate deployment and serving using MultiModel Endpoint and showcase the capability. This notebook can be extended to any framework.

Catboost is gaining in popularity and is not yet supported as a framework for SageMaker MultiModelEndpoint. Further this example serves to demostrate how to bring your own container to a MultiModelEndpoint

In this Notebook we will use identical model to simulate multiple models for loading and inference

Prerequisites

Packages and Permissions

The SageMaker SDK uses the SageMaker default S3 bucket when needed. If the get_execution_role does not return a role with the appropriate permissions, you'll need to specify an IAM role ARN that does. Please make sure the SageMakerFullAccess policy is attached to the execution role you are using.

Load model and test local inference

Here, install catboost to test we can load up the model locally and make inference.

We load up the model locally using CatBoostClassifier(). test_data.csv contains a single row of test inference data.

Upload tar ball to s3

Create a model tar ball

SageMaker requires our model to be packaged in a tar.gz file.

Upload 5 copies of the model to S3

Multi-Model Endpoints require all our models to be in a specific S3 prefix. Here we upload 100 of them to our default bucket.

This is a simulation of having different models which we need to use to predict. In reality you would probably have each of these models trained separately

Upload the model tar balls using boto3 with a unique name

List all models in s3 prefix we will use for our Multi-Model Endpoint

Building the custom container

The container folder in this example contains 3 files:

├── container
│   ├── dockerd-entrypoint.py
│   ├── Dockerfile
│   └── model_handler.py

• dockerd-entrypoint.py is the entry point script that will start the multi model server.
• Dockerfile contains the container definition that will be used to assemble the image. This includes the packages that need to be installed.
• model_handler.py is the script that will contain the logic to load up the model and make inference.

Take a look through the files to see if there is any customization that you would like to do. Below cells highlight the main part of the files.

Install catboost in the Dockerfile

Update initialize function in model_handler.py with logic to load up the model

In this case we are using CatBoostClassifier(). Feel free to update the loading logic in this function to your needs.

Update handle function in model_handler.py with logic to load up the model

Build and Push the custom image to ECR

**This steps takes atleast 5-6 minutes so please be patient and ignore any "warnings" **

Deploy Multi Model Endpoint

Create the SageMaker Multi-Model

Create the SageMaker Endpoint Configuration

Create the SageMaker Multi-Model Endpoint

This step will take a couple of minutes

Invoke each of the 5 models

We have identical models here to simulate multiple models belonging to the same framework

Invoke just one of models 1000 times

Since the models are in memory and loaded, these invocations should not have any latency

Optional Clean up

Clean up and delete the end point

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.