SageMaker Data Quality Model Monitor for Batch Transform with SageMaker Pipelines On-demand

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

In this notebook, we use SageMaker Pipelines and SageMaker Model Monitor to monitor the data quality of a batch transform job.

Data quality monitoring automatically monitors machine learning (ML) models in production and notifies you when data quality issues arise. ML models in production have to make predictions on real-life data that is not carefully curated like most training datasets. If the statistical nature of the data that your model receives while in production drifts away from the nature of the baseline data it was trained on, the model begins to lose accuracy in its predictions.

We introduce a new step type called MonitorBatchTransformStep to do this.

If you run this notebook in SageMaker Studio, you need to make sure latest python SDK is installed and restart the kernel, so please uncomment the code in the next cell, and run it.

Construct a SageMaker Pipeline

Amazon SageMaker Model Building Pipelines is a tool for building machine learning pipelines that take advantage of direct SageMaker integration. We can leverage it to run batch transform job with monitoring on-demand.

In this notebook, we showcase how to use SageMaker Pipeline to orchestrate the on-demand batch inference monitoring. In summary, we create and execute a pipeline to:

• Create a model
• Run a batch inference with the model
• Run a model monitoring job to evaluate the inference inputs/outputs.

Create a model

Here we take a pretrained model and upload it to S3. We use this model in our batch transform step.

Configure a transformer

We must first upload the dataset used to generate predictions to S3. We then define a transformer object to be used in the MonitorBatchTransformStep.

Configure data quality monitoring

In this section, we first run a baseline job, and use the suggested constraints and statistics as the baseline for running the data quality monitoring job during pipeline execution.

Generate a baseline for Model Monitor

We use the training dataset called training-dataset-with-header.csv to generate a baseline that will be used by the Data Quality Monitor. To do this, we use the suggest_baseline method. The purpose of this is to generate a set of statistics and constraints file. These files will be used by Model Monitor to compare the data passed to the Transform job and report any violations that are detected.

The suggest_baseline method has an argument called baseline_dataset. This is typically the dataset used during training.

We upload the dataset used for baselining and the data used for inference to S3.

Configure the Data Quality Check

There are two configurations we create here, one is CheckJobConfig and the other is DataQualityCheckConfig. The CheckJobConfig is used to configure the underlying processing job used by Model Monitor. This is where users can specify the role, instance type, etc.

The DataQualityCheckConfig is used to configure how Model Monitor runs the data quality check. It accepts an argument called baseline_dataset. This is the dataset that is passed to the transform job. The dataset passed here is compared against the baseline and statistics file generated by the suggest_baseline method.

Use the MonitorBatchTransformStep to monitor the transform job

This step runs a batch transform job using the transformer object configured above and monitors the data passed to the transformer before executing the job.

The baselines calculated above must be passed to this step so that the incoming data can be compared against them to detect violations.

You can configure the step to fail if a violation to Data Quality is found by toggling the fail_on_violation flag.

Create and run the pipeline

Start a pipeline execution

Read the model monitor reports

You must wait for the pipeline to finish executing before you can read the violation reports.

This pipeline succeeds even though violations are found by model monitor because fail_on_violation is set to False.

Other commands

We can also start and stop the monitoring schedules.

Delete the resources

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.