SageMaker Pipelines EMR Step With Cluster Lifecycle Management

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

This notebook illustrates how an EMR step can be run from a SageMaker Pipeline. This example requires a cluster config. The EMR step uses the cluster config to create an EMR cluster, performs the required job and finally closes the cluster.

The steps in this pipeline include:

• Preprocess the Abalone dataset with PySpark on EMR
• Train an XGBoost Model
• Evaluate the model performance
• Create a model

A step to register the model into a Model Registry can be added to the pipeline using the RegisterModel step.

Contents

1. Prerequisites
2. Configuration Setup
3. Parameters
4. Data Preparation
5. Model Training and Evaluation
6. Model Registry
7. Execute the Pipeline
8. Cleanup

Prerequisites

To run this notebook you will need:

• EMR roles
• IAM policies which enable the notebook to run a step on an Amazon EMR cluster

EMR roles

You'll need:

• Service role for Amazon EMR (EMR role) - this is passed as the ServiceRole parameter
• Service role for cluster EC2 instances (EC2 instance profile) - this is passed as the JobFlowRole parameter

See 'EMR IAM roles' for more details.

IAM policy

The notebook execution role should have policies which enable the notebook to run a step on an Amazon EMR cluster. The Amazon managed policy AmazonSageMakerPipelinesIntegrations should be added to the notebook execution role.

Setup

Setup Dependencies

Configuration Setup

Let's now configure the setup we need, which includes the session object from the SageMaker Python SDK, and necessary configurations for the pipelines, such as object types, input and output buckets and so on.

Parameters

Data Preparation

A PySpark job on EMR is used to prepare the for the training job. Using the script preprocess.py, the dataset is featurized and split into train, test, and validation datasets.

The output of this step is used as the input to the TrainingStep.

The dataset you use is the UCI Machine Learning Abalone Dataset [1]. The aim for this task is to determine the age of an abalone snail from its physical measurements. At the core, this is a regression problem.

The dataset contains several features: length (the longest shell measurement), diameter (the diameter perpendicular to length), height (the height with meat in the shell), whole_weight (the weight of whole abalone), shucked_weight (the weight of meat), viscera_weight (the gut weight after bleeding), shell_weight (the weight after being dried), sex ('M', 'F', 'I' where 'I' is Infant), and rings (integer).

The number of rings turns out to be a good approximation for age (age is rings + 1.5). However, to obtain this number requires cutting the shell through the cone, staining the section, and counting the number of rings through a microscope, which is a time-consuming task. However, the other physical measurements are easier to determine. You use the dataset to build a predictive model of the variable rings through these other physical measurements.

Before you upload the data to an S3 bucket, install the SageMaker Python SDK and gather some constants you can use later in this notebook.

[1] Dua, D. and Graff, C. (2019). UCI Machine Learning Repository. Irvine, CA: University of California, School of Information and Computer Science.

Model Training and Evaluation

We will now train an XGBoost model using the SageMaker Python SDK and the output of the EMR Step.

Evaluating the model

Use a processing job to evaluate the model from the TrainingStep. If the output of the evaluation is True, a model is created.

Model registry

Execute the pipeline

Cleanup

Running the following cell will delete the following resources created in this notebook

• SageMaker Pipeline

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.