Train, Deploy, and Monitor the Music Recommender Model using SageMaker Pipelines

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

Background

In this notebook, we'll build an end-to-end pipeline to create a music recommender using SageMaker Pipelines, which will automate the entire modeling process from the beginning of data ingestion to monitoring the model. SageMaker Pipelines is a tool for building machine learning pipelines that take advantage of direct SageMaker integration. Because of this integration, you can create a pipeline and set up SageMaker Projects for orchestration using a tool that handles much of the step creation and management for you.

If you want to learn more about each step of the pipeline, feel free to look at the series of notebooks listed below. It basically implements the same process in this notebook in a manual way with more detailed descriptions of what each step does. Please see the README.md for more information about the use case implemented by this sequence of notebooks.

1. Music Recommender Data Exploration
2. Music Recommender Data Preparation with SageMaker Feature Store and SageMaker Data Wrangler
3. Train, Deploy, and Monitor the Music Recommender Model using SageMaker SDK

Contents

1. Architecture: Create a SageMaker Pipeline to Automate All the Steps from Data Prep to Model Deployment
2. SageMaker Pipeline Overview
3. Clean Up

Install required and/or update third-party libraries

Import libraries

Set region and boto3 config

Architecture: Create a SageMaker Pipeline to Automate All the Steps from Data Prep to Model Deployment

Prereqs: Get Data

Here we will download the music data from a public S3 bucket that we'll be using for this demo and uploads it to your default S3 bucket that was created for you when you initially created a SageMaker Studio workspace.

For this example, we will provide the processed data you need to complete this task. But you are free to take a look at how we processed the data:

• If you are curious as to how tracks_new.csv and ratings_new.csv are generated, see Music Recommender Data Exploration
• If you are curious as to how the rest of the files are generated, see Music Recommender Data Preparation with SageMaker Feature Store and SageMaker Data Wrangler

SageMaker Pipeline Overview

List of Steps

1. Step 1: Data Wrangler Preprocessing Step
2. Step 2: Create Dataset and Train/Test Split
3. Step 3: Train XGBoost Model
4. Step 4: Model Pre-Deployment Step
5. Step 5: Register Model
6. Step 6: Deploy Model
7. Step 7: Monitor Model Deployed to SageMaker Hosted Endpoint
8. Combine Steps and Run Pipeline

Now that you've manually done each step in our machine learning workflow, you can certain steps to allow for faster model experimentation without sacrificing transparncy and model tracking. In this section you will create a pipeline which trains a new model, persists the model in SageMaker and then adds the model to the registry.

Pipeline parameters

An important feature of SageMaker Pipelines is the ability to define the steps ahead of time, but be able to change the parameters to those steps at execution without having to re-define the pipeline. This can be achieved by using ParameterInteger, ParameterFloat or ParameterString to define a value upfront which can be modified when you call pipeline.start(parameters=parameters) later. Only certain parameters can be defined this way.

Step 1: Data Wrangler Preprocessing Step

Update the data source in the .flow file

The 01_music_datapred.flow file is a JSON file containing instructions for where to find your data sources and how to transform the data. We'll be updating the object telling Data Wrangler where to find the input data on S3. We will set this to your default S3 bucket. With this update to the .flow file it now points to your new S3 bucket as the data source used by SageMaker Data Wrangler.

Make sure the .flow file is closed before running this next step or it won't update the new s3 file locations in the file

Upload flow to S3

This will become an input to the first step and, as such, needs to be in S3.

Define Data Wrangler step's inputs

In this step, new data from source= will be transformed according to the SageMaker Data Wrangler .flow file and later added to the existing feature groups we created in the 02 notebooks.

Define outputs for the Data Wranger step

Define processor and processing step

Step 2: Create Dataset and Train/Test Split

Step 3: Train XGBoost Model

In this step we use the ParameterString train_instance_param defined at the beginning of the pipeline.

Step 4: Model Pre-Deployment Step

Step 5: Register Model

In this step you will use the ParameterString model_approval_status defined at the outset of the pipeline code.

Step 6: Deploy Model

Step 7: Monitor Model Deployed to SageMaker Hosted Endpoint

Combine Steps and Run Pipeline

Once all of our steps are defined, we can put them together using the SageMaker Pipeline object. While we pass the steps in order so that it is easier to read, technically the order that we pass them does not matter since the pipeline DAG will parse it out properly based on any dependencies between steps. If the input of one step is the output of another step, the Pipelines understands which must come first.

Option 1: The Entire Pipeline End to end

Note: You may want to refrain from running an entire data prep and train deploy pipeline as one long pipeline and rather consider breaking the pipeline into two parts:

Data Prep Pipeline

Sometimes we may want to run a number of data prep steps and split the data, getting it ready for training and beyond. This may require multiple iterations. We can separate this process from the rest of the pipeline by including only these data prep steps in their own smaller data prep pipeline.

Train Deploy Monitor Pipeline

This allows you to have separation of concerns around the preparation of data distinct from that of training, tuning, deployment, inference and monitoring until you want to kick off a retraining only, data prep only, or the complete pipeline. With SageMaker Pipelines you have the flexibility of doing any one of these in a modular and iterative manner.

Option 2: Data Prep Pipeline

Option 3: Train Deploy Monitor Pipeline

Submit the pipeline definition to the SageMaker Pipeline service

Note: If an existing pipeline has the same name it will be overwritten.

Let's choose the pipeline we want to run

View the entire pipeline definition

Viewing the pipeline definition will all the string variables interpolated may help debug pipeline bugs. It is commented out here due to length.

Run the pipeline

Note this will take about 1 hour to complete. You can watch the progress of the Pipeline Job on your SageMaker Studio Components panel

Earlier in the notebook, we defines several ProcessingStep()s and a TrainingStep() which our Pipeline() instance here will reference and kick off.

After completion we can use Sagemaker Studio's Components and Registries tab to see our Pipeline graph and any further error or log messages.

Clean Up

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.