SageMaker Model Monitor with Batch Transform - Data Quality Monitoring On-Schedule

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

In this notebook, we use SageMaker Model Monitor to monitor the data quality of a batch transform job.

Data quality monitoring automatically monitors machine learning (ML) models in production and notifies you when data quality issues arise. ML models in production have to make predictions on real-life data that is not carefully curated like most training datasets. If the statistical nature of the data that your model receives while in production drifts away from the nature of the baseline data it was trained on, the model begins to lose accuracy in its predictions.

Setup

If you run this notebook in SageMaker Studio, you need to make sure latest python SDK is installed and restart the kernel, so please uncomment the code in the next cell, and run it.

1) Create model in Amazon SageMaker

Create a SageMaker Model from pre-trained churn prediction model.

2) Upload test data for batch inference that will be used as input for a Batch Transform Job

3) Create the Batch Transform Job

4) Examine the Batch Transform Captured Data

There are two directory under s3_capture_upload_path, one is the /input, another is the /output. Under the /input is the captured data file for transform input, whereas, the under the /output is the captured data file for transform output. Note that, batch transform data capture is unlike Endpoint data capture, it does not capture the data and log to s3 as this will create tremendous amount of duplications. Instead, batch transform captures data in manifests. The manifests contain the source transform input or output s3 locations.

Lets take a look at the captured data.

Like, we have output here.

5) Create a Baseline that will be used by Model Monitor

In general this can be done parallel to the Transform Job

6) Monitoring Schedule

Create a schedule

You can create a model monitoring schedule. Use the baseline resources (constraints and statistics) to compare against the batch transform inference inputs and outputs.

7) Describe and inspect the schedule

Once you describe, observe that the MonitoringScheduleStatus changes to Scheduled.

List executions

The schedule starts jobs at the previously specified intervals. Here, you list the latest five executions. Note that if you are kicking this off after creating the hourly schedule, you might find the executions empty. You might have to wait until you cross the hour boundary (in UTC) to see executions kick off. The code below has the logic for waiting.

Note: Even for an hourly schedule, Amazon SageMaker has a buffer period of 20 minutes to schedule your execution. You might see your execution start in anywhere from zero to ~20 minutes from the hour boundary. This is expected and done for load balancing in the backend.

Inspect a specific execution (latest execution)

In the previous cell, you picked up the latest completed or failed scheduled execution. Here are the possible terminal states and what each of them mean:

• Completed - This means the monitoring execution completed and no issues were found in the violations report.
• CompletedWithViolations - This means the execution completed, but constraint violations were detected.
• Failed - The monitoring execution failed, maybe due to client error (perhaps incorrect role permissions) or infrastructure issues. Further examination of FailureReason and ExitMessage is necessary to identify what exactly happened.
• Stopped - job exceeded max runtime or was manually stopped.

List the generated reports

Violations report

If there are any violations compared to the baseline, they will be listed here.

Other commands

We can also start and stop the monitoring schedules.

8) Delete the resources

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.