SageMaker Pipelines Tuning Step

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

This notebook illustrates how a Hyperparameter Tuning Job can be run as a step in a SageMaker Pipeline.

The steps in this pipeline include -

• Preprocessing the abalone dataset
• Running a Hyperparameter Tuning job
• Creating the 2 best models
• Evaluating the performance of the top performing model of the HPO step
• Registering the top model in the model registry using a conditional step based on evaluation metrics

Data Preparation

An SKLearn processor is used to prepare the dataset for the Hyperparameter Tuning job. Using the script preprocess.py, the dataset is featurized and split into train, test, and validation datasets.

The output of this step is used as the input to the TuningStep

Hyperparameter Tuning

Amazon SageMaker automatic model tuning, also known as hyperparameter tuning, finds the best version of a model by running many training jobs on your dataset using the algorithm and ranges of hyperparameters that you specify. It then chooses the hyperparameter values that result in a model that performs the best, as measured by a metric that you choose.

Valid metrics for XGBoost Tuning Job

You can learn more about Hyperparameter Tuning in the SageMaker docs.

Warm start for Hyperparameter Tuning Job

Use warm start to start a hyperparameter tuning job using one or more previous tuning jobs as a starting point. The results of previous tuning jobs are used to inform which combinations of hyperparameters to search over in the new tuning job. Hyperparameter tuning uses either Bayesian or random search to choose combinations of hyperparameter values from ranges that you specify.

Find more information on Warm Starts in the SageMaker docs.

In a training pipeline, the parent tuning job name can be provided as a pipeline parameter if there is an already complete Hyperparameter tuning job that should be used as the basis for the warm start.

This step is left out of the pipeline steps in this notebook. It can be added into the steps while defining the pipeline and the appropriate parent tuning job should be specified.

Creating and Registering the best models

After successfully completing the Hyperparameter Tuning job. You can either create SageMaker models from the model artifacts created by the training jobs from the TuningStep or register the models into the Model Registry.

When using the Model Registry, if you register multiple models from the TuningStep, they will be registered as versions within the same model package group unless unique model package groups are specified for each ModelStep that is part of the pipeline.

In this example, the two best models from the TuningStep are added to the same model package group in the Model Registry as v0 and v1.

You use the get_top_model_s3_uri method of the TuningStep class to get the model artifact from one of the top performing model versions

Evaluate the top model

Use a processing job to evaluate the top model from the tuning step

Execute the Pipeline

Cleaning up resources

Users are responsible for cleaning up resources created when running this notebook. Specify the ModelName, ModelPackageName, and ModelPackageGroupName that need to be deleted. The model names are generated by the CreateModel step of the Pipeline and the property values are available only in the Pipeline context. To delete the models created by this pipeline, navigate to the Model Registry and Console to find the models to delete.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.