SageMaker Pipeline - Local Mode

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

This notebook demonstrates how to orchestrate SageMaker jobs locally using SageMaker Pipelines.

Using a LocalPipelineSession object, you can now run your pipelines on your local machine before running them in the cloud.

The LocalPipelineSession object is used while defining each pipeline step and when defining the complete Pipeline object. To run this pipeline in the cloud, each step along with the Pipeline object must be redefined using PipelineSession.

Note: This notebook will not run in SageMaker Studio. You can run this on SageMaker Classic Notebook instances OR your local IDE.

In this notebook, we will execute in our local environment a pipeline that will perform the following steps:

• ProcessingStep by using FrameworkProcessor
• TrainingStep by using Estimator with a custom PyTorch container

Dataset

We are using a subset of ~20000 records of synthetic transactions, each of which is labeled as fraudulent or not fraudulent. We'd like to train a model based on the features of these transactions so that we can predict risky or fraudulent transactions in the future.

This is a binary classification problem:

• 1 - Fraud
• 0 - No Fraud

Prerequisites

Install the latest version of the SageMaker Python SDK

Build Container

In order to use Amazon SageMaker Training Job with a custom image, the first step is to build it and push in a private Amazon ECR Repository.

The Dockerfile defined is creating starting from the public torch 1.12.1 image, and by the usage of sagemaker-training-toolkit we are making our container compatible with Amazon SageMaker for providing our training script during the definition of the Estimator.

For facilitating the steps of building the Docker Image and push it in the Amazon ECR Repository, we are providing a utility script build_image.sh.

For more information on the usage, please read the README

Part 1/3 - Setup

Here we'll import some libraries and define some variables.

Create LocalPipelineSession object so that each pipeline step will run locally.

To run this pipeline in the cloud, you must change LocalPipelineSession() to PipelineSession()

Please Note: Provide SageMaker Execution Role ARN if not running on SageMaker Notebook environment

Upload Dataset in the Default Amazon S3 Bucket

In order to make the data available, we are uploading the downloaded dataset into the default S3 bucket

Part 2/3 - Create Amazon SageMaker Pipeline

In this section, we are creating the Amazon SageMaker Pipeline and define the proper Input Parameters for making it usable for both local mode and for cloud executions

Compress source code for installing additional python modules

By using sagemaker-training-toolkit, we can provide the execution scripts and the requirements.txt for installing additional dependencies to the Estimator that we will define some steps below.

In order to make sure that Amazon SageMaker will install our additional Python modules by reading requirements.txt, we are compressing the content of the training folder and uploading it in the default S3 Bucket.

By using FrameworkProcessor, we can provide to the Amazon SageMaker Job the execution scripts and the requirements.txt for installing additional Python modules. Look at the documentation for additional info.

In order to make sure that Amazon SageMaker will install our additional Python modules by reading requirements.txt, we are compressing the content of the processing folder and uploading it in the default S3 Bucket.

Global Parameters

Pipeline Parameters

In order to make the Amazon SageMaker Pipeline available for executing it both in local mode and in the cloud, we are defining the following ParameterString for providing the execution type at runtime

SageMaker Processing Step

Define the FrameworkProcessor object

SageMaker Training Step

Utility methods

For providing the compressed sourcedir to Estimator, we are defining a utility method for encoding the job hyperparameters

Get ECR image uri

Let's take the image_uri related to our custom image we want to use for our training job

Define the Estimator object

Pipeline definition

Let's create the pipeline object, which contains as parameters the inputs defined in the previous sections, and as steps the ProcessingStep and TrainingStep defined few cells above

Part 3/3 - Run SageMaker Pipeline

For executing the Amazon SageMaker Pipeline in our local environment, we are providing for both the ProcessingStep and TrainingStep the parameter local for the instance_type to use.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.