Sm Pipelines Clarify Model Monitor Integration
SageMaker Pipelines integration with Model Monitor and Clarify
This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.
This notebook showcases how Model Monitor and Clarify steps can be integrated with SageMaker Pipelines. This allows users to calculate baselines for data quality and model quality checks by running the underlying Model Monitor and Clarify containers.
Data/Model Quality, Bias, and Model Explainability Checks in SageMaker Pipelines
This notebook introduces two new step types in SageMaker Pipelines -
QualityCheckStepClarifyCheckStep
With these two steps, the pipeline is able to perform baseline calculations that are needed as a standard against which data/model quality issues can be detected (including bias and explainability).
These steps leverage SageMaker pre-built containers:
QualityCheckStep(for Data/Model Quality): sagemaker-model-monitor-analyzerClarifyCheckStep(for Data/Model Bias and Model Explainability): sagemaker-clarify-processing
The training dataset that you used to train the model is usually a good baseline dataset. The training dataset data schema and the inference dataset schema should exactly match (the number and order of the features). Note that the prediction/output columns are assumed to be the first columns in the training dataset. From the training dataset, you can ask SageMaker to suggest a set of baseline constraints and generate descriptive statistics to explore the data.
These two new steps will always calculate new baselines using the dataset provided.
Drift Check Baselines in the Model Registry
The RegisterStep has a new parameter called drift_check_baselines. This refers to the baseline files associated with the model. When deployed, these baseline files are used by Model Monitor for Model Quality/Data Quality checks. In addition, these baselines can be used in QualityCheckStep and ClarifyCheckStep to compare newly trained models against models that have already been registered in the Model Registry.
Step Properties
The QualityCheckStep has the following properties -
CalculatedBaselineStatistics: The baseline statistics file calculated by the underlying Model Monitor container.CalculatedBaselineConstraints: The baseline constraints file calculated by the underlying Model Monitor container.BaselineUsedForDriftCheckStatisticsandBaselineUsedForDriftCheckConstraints: These are the two properties used to setdrift_check_baselinein the Model Registry. The values set in these properties vary depending on the parameters passed to the step. The different behaviors are described in the table below.
The ClarifyCheckStep has the following properties -
CalculatedBaselineConstraints: The baseline constraints file calculated by the underlying Clarify container.BaselineUsedForDriftCheckConstraints: This property is used to setdrift_check_baselinein the Model Registry. The values set in this property will vary depending on the parameters passed to the step. The different behaviors are described in the table below.
Notebook Overview
This notebook should be run with Python 3.9 using the SageMaker Studio Python3 (Data Science) kernel.
Let's start by installing the SageMaker Python SDK, boto, and AWS CLI.
Create the SageMaker Session
Define variables and parameters needed for the Pipeline steps
Define pipeline parameters
Both QualityCheckStep and ClarifyCheckStep use two boolean flags skip_check and register_new_baseline to control their behavior.
skip_check: This determines if a drift check is executed or not.register_new_baseline: This determines if the newly calculated baselines (in the step propertyCalculatedBaselines) should be set in the step propertyBaselineUsedForDriftCheck.supplied_baseline_statisticsandsupplied_baseline_constraints: Ifskip_checkis set to False, baselines can be provided to this step through this parameter. If provided, the step will compare the newly calculated baselines (CalculatedBaselines) against those provided here instead of finding the latest baselines from the Model Registry. In the case ofClarifyCheckStep, onlysupplied_baseline_constraintsis a valid parameter, forQualityCheckStep, both parameters are used.model_package_group_name: The step will use thedrift_check_baselinesfrom the latest approved model in the model package group for the drift check. Ifsupplied_baseline_*is provided, this field will be ignored.
The first time the pipeline is run, the skip_check value should be set to True using the pipeline execution parameters so that new baselines are registered and no drift check is executed.
Combining Pipeline parameters
This table summarizes how the pipeline parameters work when combined.
The parameter drift_check_baselines is used to supply baselines to the RegisterStep that will be used for all drift checks involving the model.
Newly calculated baselines can be reference by the properties CalculatedBaselineStatistics and CalculatedBaselineConstraints on the QualityCheckStep and CalculatedBaselineConstraints on the ClarifyCheckStep.
For example, data_quality_check_step.properties.CalculatedBaselineStatistics and data_quality_check_step.properties.CalculatedBaselineConstraints. This property refers to the baseline that is calculated when the data quality check step is executed.
skip_check / register_new_baseline | Does step do a drift check? | Value of step property CalculatedBaseline | Value of step property BaselineUsedForDriftCheck | Possible Circumstances for this parameter combination |
|---|---|---|---|---|
| F / F | Drift Check executed against existing baselines. | New baselines calculated by step execution | Baseline from latest approved model in Model Registry or baseline supplied as step parameter | Regular re-training with checks enabled to get a new model version, but carry over previous baselines as DriftCheckBaselines in Registry for new model version. |
| F / T | Drift Check executed against existing baselines. | New baselines calculated by step execution | Newly calculated baseline by step execution (value of property CalculatedBaseline) | Regular re-training with checks enabled to get a new model version, but refresh DriftCheckBaselines in Registry with newly calculated baselines for the new model version. |
| T / F | No Drift Check. | New baselines calculated by step execution | Baseline from latest approved model in Model Registry or baseline supplied as step parameter | Violation detected by the model monitor on endpoint for a particular type of check and the pipeline is triggered for retraining a new model. Skip the check against previous baselines, but carry over previous baselines as DriftCheckBaselines in Registry for new model version. |
| T / T | No Drift Check. | New baselines calculated by step execution | Newly calculated baseline by step execution (value of property CalculatedBaseline) | a. Initial run of the pipeline, building the first model version and generate initial baselines. b. Violation detected by the model monitor on endpoint for a particular type of check and the pipeline is triggered for retraining a new model. Skip the check against previous baselines and refresh DriftCheckBaselines with newly calculated baselines in Registry directly. |
Processing step for feature engineering
Calculating the Data Quality
CheckJobConfig is a helper function that's used to define the job configurations used by the QualityCheckStep. By separating the job configuration from the step parameters, the same CheckJobConfig can be used across multiple steps for quality checks.
The DataQualityCheckConfig is used to define the Quality Check job by specifying the dataset used to calculate the baseline, in this case, the training dataset from the data processing step, the dataset format, in this case, a csv file with no headers, and the output path for the results of the data quality check.
Calculating the Data Bias
The job configuration from the previous step is used here and the DataConfig class is used to define how the ClarifyCheckStep should compute the data bias. The training dataset is used again for the bias evaluation, the column representing the label is specified through the label parameter, and a BiasConfig is provided.
In the BiasConfig, we specify a facet name (the column that is the focal point of the bias calculation), the value of the facet that determines the range of values it can hold, and the threshold value for the label.
More details on BiasConfig can be found here.
Train an XGBoost Model
Create the model
The model is created so that a batch transform job can be used to get predictions from the model on a test dataset. These predictions are used when calculating model quality, model bias, and model explainability.
Transform Output
The output of the transform step combines the prediction and the input label. The output format is
prediction, original label
Check the Model Quality
In this QualityCheckStep we calculate the baselines for statistics and constraints using the predictions that the model generates from the test dataset (output from the TransformStep). We define the problem type as 'Regression' in the ModelQualityCheckConfig along with specifying the columns which represent the input and output. Since the dataset has no headers, _c0, _c1 are auto-generated header names that should be used in the ModelQualityCheckConfig.
Check for Model Bias
Similar to the Data Bias check step, a BiasConfig is defined and Clarify is used to calculate the model bias using the training dataset and the model.
Check Model Explainability
SageMaker Clarify uses a model-agnostic feature attribution approach, which you can use to understand why a model made a prediction after training and to provide per-instance explanation during inference. The implementation includes a scalable and efficient implementation of SHAP, based on the concept of a Shapley value from the field of cooperative game theory that assigns each feature an importance value for a particular prediction.
For Model Explainability, Clarify requires an explainability configuration to be provided. In this example, we use SHAPConfig. For more information of explainability_config, visit the Clarify documentation.
Evaluate the performance of the model
Using a processing job, evaluate the performance of the model. The performance is used in the Condition Step to determine if the model should be registered or not.
Define the metrics to be registered with the model in the Model Registry
Register the model
The two parameters in RegisterModel that hold the metrics calculated by the ClarifyCheckStep and QualityCheckStep are model_metrics and drift_check_baselines.
drift_check_baselines - these are the baseline files that will be used for drift checks in QualityCheckStep or ClarifyCheckStep and model monitoring jobs that are set up on endpoints hosting this model.
model_metrics - these should be the latest baselines calculated in the pipeline run. This can be set using the step property CalculatedBaseline
The intention behind these parameters is to give users a way to configure the baselines associated with a model so they can be used in drift checks or model monitoring jobs. Each time a pipeline is executed, users can choose to update the drift_check_baselines with newly calculated baselines. The model_metrics can be used to register the newly calculated baselines or any other metrics associated with the model.
Every time a baseline is calculated, it is not necessary that the baselines used for drift checks are updated to the newly calculated baselines. In some cases, users may retain an older version of the baseline file to be used for drift checks and not register new baselines that are calculated in the Pipeline run.
Create the Pipeline
Get Pipeline definition
First time executing
The first time the pipeline is run the parameters need to be overridden so that the checks are skipped and newly calculated baselines are registered
Wait for the pipeline execution to complete
Cleaning up resources
Users are responsible for cleaning up resources created when running this notebook. Specify the ModelName, ModelPackageName, and ModelPackageGroupName that need to be deleted. The model names are generated by the CreateModel step of the Pipeline and the property values are available only in the Pipeline context. To delete the models created by this pipeline, navigate to the Model Registry and Console to find the models to delete.
Notebook CI Test Results
This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.