Multi-model SageMaker Pipeline with Hyperparamater Tuning and Experiments

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

Before proceeding, please see the context of this notebook in README.md. This notebook has been tested in a SageMaker notebook that is using a kernel with at least Python 3.7 installed, e.g. conda_mxnet_latest_p37, conda_python3. Make sure you have created a SageMaker project outside of this notebook with the name restate. Recommendation is to create a SageMaker project using SageMaker-provide MLOps template for model building, training, and deployment template. Note that this notebook will not create the SageMaker project for you.

Pre-requisities

We create an S3 bucket and with encryption enabled for additional security.

We create IAM role AWSGlueServiceRole-restate.

We create IAM role AmazonSageMakerServiceCatalogProductsUseRole-restate.

Prepare Athena table

At this point, it is assumed that S3 bucket sagemaker-restate-<AWS ACCOUNT ID> and the necessary IAM roles are created. For the complete list of prerequisites, please see README.md.

We move the raw data to S3 bucket sagemaker-restate-<AWS ACCOUNT ID>.

The step below creates a Glue database and table containing the raw data by running a Glue crawler. It is recommended to configure Glue encryption for additional security.

Once crawler is done crawling, table california in database restate should be visible in Glue catalog. We rename the Glue table columns for readability.

Table california in database restate should be visible in Athena. We filter only the data where housingmedianage > 10.

Make sure Athena query result location and encryption settings are updated accordingly before proceeding to the next step.

Prepare Decision Tree custom Docker image

We make a Docker image containing a custom algorithm using Scikit-learn Decision Tree Regressor. Note that the Docker image has been modified to support hyperparameter tuning and validation data.

Once Docker image is pushed to ECR repository, we make the image accessible from SageMaker.

Start the SageMaker pipeline

Manually update restate-athena-california.flow with the queryString and s3OutputLocation of your choice. This has to be done outside of this Jupyter notebook. Once done, proceed to create your pipeline.

Verify that you can successfully run get-pipeline-definition.

At this point, it is assumed that you have already created a SageMaker project with a name restate and a pipeline with a name sagemaker-restate.

If you inspect the pipeline, you will see that the XGBoost model performs better than the decision tree model. Therefore, the XGBoost model is registered in the model registry.

You can experiment on the data, e.g. use data for housingmedianage > 50, by changing the Athena query in restate-athena-california.flow. You can check if XGBoost would still be the winning model after these changes.

Deploy the winning model

Make sure to update your desired MODEL_VERSION. We assume we approve the model version 1.

At this point, you can deploy the approved model version by going through the steps below, or using MLOps template for model deployment.

Inference

Use the following data for inference:

-117.18,32.75,52.0,1504.0,208.0,518.0,196.0

This is a census block group with longitude -117.18, latitude 32.75, housing median age of 52.0, total rooms of 1504, total bedrooms of 208, population of 518, and households count of 196.

Let's see its predicted value using our generated model.

Now you try:

-117.17,32.76,45.0,3149.0,639.0,1160.0,661.0

This is a census block group with longitude -117.17, latitude 32.76, housing median age of 45.0, total rooms of 3149, total bedrooms of 639, population of 1160, and households count of 661.

Cleanup

Cleanup the Glue database, table, crawler, and S3 buckets used.

Cleanup the ECR and SageMaker images created.

Cleanup the SageMaker model and endpoint resources.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.