Copyright (c) Microsoft Corporation. All rights reserved.

Licensed under the MIT License.

Automated Machine Learning - AutoFeaturization (Part 1)

Autofeaturization of credit card fraudulent transactions dataset on remote compute

Contents

1. Introduction
2. Setup
3. Data
4. Autofeaturization

Introduction

Autofeaturization is a new feature to let you as the user run an AutoML experiment to only featurize the datasets. These datasets along with the transformer will be stored in the experiment which can later be retrieved and used to train models, either via AutoML or custom training.

To run Autofeaturization, pass in zero iterations and featurization as auto. This will featurize the datasets and terminate the experiment. Training will not occur.

Limitations - Sparse data cannot be supported at the moment. Any dataset that has extensive categorical data might be featurized into sparse data which will not be allowed as input to AutoML. Efforts are underway to support sparse data and will be updated soon.

In this example we use the credit card fraudulent transactions dataset to showcase how you can use AutoML for autofeaturization. The goal is to clean and featurize the training dataset.

This notebook is using remote compute to complete the featurization.

If you are using an Azure Machine Learning Compute Instance, you are all set. Otherwise, go through the configuration notebook first if you haven't already, to establish your connection to the AzureML Workspace.

In the below steps, you will learn how to:

1. Create an autofeaturization experiment using an existing workspace.
2. View the featurized datasets and transformer

Setup

As part of the setup you have already created an Azure ML Workspace object. For Automated ML you will need to create an Experiment object, which is a named object in a Workspace used to run experiments.

This sample notebook may use features that are not available in previous versions of the Azure ML SDK.

Create or Attach existing AmlCompute

A compute target is required to execute the Automated ML run. In this tutorial, you create AmlCompute as your training compute resource.

Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.

Creation of AmlCompute takes approximately 5 minutes.

If the AmlCompute with that name is already in your workspace this code will skip the creation process. As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read this article on the default limits and how to request more quota.

Data

Load Data

Load the credit card fraudulent transactions dataset from a CSV file, containing both training features and labels. The features are inputs to the model, while the training labels represent the expected output of the model.

Here the autofeaturization run will featurize the training data passed in.

Training Dataset

AutoFeaturization

Instantiate an AutoMLConfig object. This defines the settings and data used to run the autofeaturization experiment.

You can find more information about primary metrics here

Call the submit method on the experiment object and pass the run configuration. Depending on the data this can run for a while. Validation errors and current status will be shown when setting show_output=True and the execution will be synchronous.

Transformer and Featurized Datasets

The given datasets have been featurized and stored under Outputs + logs from the details page of the remote run. The structure is shown below. The featurized dataset is stored under /outputs/featurization/data and the transformer is saved under /outputs/featurization/pipeline

Below you will learn how to refer to the data saved in your run and retrieve the same.

Results

Widget for Monitoring Runs

The widget will first report a "loading" status while running the first iteration. After completing the first iteration, an auto-updating graph and table will be shown. The widget will refresh once per minute, so you should see the graph update as child runs complete.

Note: The widget displays a link at the bottom. Use this link to open a web interface to explore the individual run details

Automated Machine Learning - AutoFeaturization (Part 2)

Training using a custom model with the featurized data from Autofeaturization run of credit card fraudulent transactions dataset

Contents

1. Introduction
2. Data Setup
3. Autofeaturization Data
4. Train
5. Results
6. Test

Introduction

Here we use the featurized dataset saved in the above run to showcase how you can perform custom training by using the transformer from an autofeaturization run to transform validation / test datasets.

The goal is to use autofeaturized run data and transformer to transform and run a custom training experiment independently

In the below steps, you will learn how to:

1. Read transformer from a completed autofeaturization run and transform data
2. Pull featurized data from a completed autofeaturization run
3. Run a custom training experiment with the above data
4. Check results

Data Setup

We will load the featurized training data and also load the transformer from the above autofeaturized run. This transformer can then be used to transform the test data to check the accuracy of the custom model after training.

Load Test Data

load test dataset from CSV and split into X and y columns to featurize with the transformer going forward.

Load data_transformer from the above remote run artifact

(Method 1)

Method 1 allows you to read the transformer from the remote storage.

(Method 2)

Method 2 downloads the transformer to the local directory and then can be used to transform the data. Uncomment to use.

Transform Data

Note: Not all datasets produce a y_transformer. The dataset used in the current notebook requires a transformer as the y column data is categorical.

We will go ahead and download the mlflow transformer model and use it to transform test data that can be used for further experimentation below. To run the commented code, make sure the environment requirement is satisfied. You can go ahead and create the environment from the conda.yaml file under /outputs/featurization/pipeline/ and run the given code in it.

Run the following cell to see the featurization summary of X and y transformers. Uncomment to use.

Load Datastore

The below data store holds the featurized datasets, hence we load and access the data. Check the path and file names according to the saved structure in your experiment Outputs + logs as seen in Autofeaturization Part 1

Autofeaturization Data

We will load the training data from the previously completed Autofeaturization experiment. The resulting featurized dataframe can be passed into the custom model for training. Here we are saving the file to local from the experiment storage and reading the data.

Another way to load the data is to go to the above autofeaturization experiment and check for the featurized dataset ids under Output datasets. Uncomment and replace them accordingly below, to use.

Training Data

We are dropping the y column and weights column from the featurized training dataset.

Train

Here we are passing our training data to the lightgbm classifier, any custom model can be used with your data. Let us first install lightgbm.

Once training is done, the test data obtained after transforming from the above downloaded transformer can be used to calculate the accuracy

Analyze results

Retrieve the Model

Test the fitted model

Now that the model is trained, split the data in the same way the data was split for training (The difference here is the data is being split locally) and then run the test data through the trained model to get the predicted values.

Experiment Complete!