Auto Ml Regression Explanation Featurization
Copyright (c) Microsoft Corporation. All rights reserved.
Licensed under the MIT License.
Introduction
In this example we use the Hardware Performance Dataset to showcase how you can use AutoML for a simple regression problem. The Regression goal is to predict the performance of certain combinations of hardware parts. After training AutoML models for this regression data set, we show how you can compute model explanations on your remote compute using a sample explainer script.
If you are using an Azure Machine Learning Compute Instance, you are all set. Otherwise, go through the configuration notebook first if you haven't already to establish your connection to the AzureML Workspace.
In this notebook you will learn how to:
- Create an
Experimentin an existingWorkspace. - Instantiating AutoMLConfig with FeaturizationConfig for customization
- Train the model using remote compute.
- Explore the results and featurization transparency options
- Setup remote compute for computing the model explanations for a given AutoML model.
- Start an AzureML experiment on your remote compute to compute explanations for an AutoML model.
- Download the feature importance for engineered features and visualize the explanations for engineered features on azure portal.
- Download the feature importance for raw features and visualize the explanations for raw features on azure portal.
Setup
As part of the setup you have already created an Azure ML Workspace object. For Automated ML you will need to create an Experiment object, which is a named object in a Workspace used to run experiments.
This sample notebook may use features that are not available in previous versions of the Azure ML SDK.
Create or Attach existing AmlCompute
You will need to create a compute target for your AutoML run. In this tutorial, you create AmlCompute as your training compute resource.
Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.
Creation of AmlCompute takes approximately 5 minutes. If the AmlCompute with that name is already in your workspace this code will skip the creation process.
As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read this article on the default limits and how to request more quota.
Setup Training and Test Data for AutoML experiment
Load the hardware dataset from a csv file containing both training features and labels. The features are inputs to the model, while the training labels represent the expected output of the model. Next, we'll split the data using random_split and extract the training data for the model. We also register the datasets in your workspace using a name so that these datasets may be accessed from the remote compute.
Train
Instantiate an AutoMLConfig object to specify the settings and data used to run the experiment.
| Property | Description |
|---|---|
| task | classification, regression or forecasting |
| primary_metric | This is the metric that you want to optimize. Regression supports the following primary metrics: spearman_correlation normalized_root_mean_squared_error r2_score normalized_mean_absolute_error |
| experiment_timeout_hours | Maximum amount of time in hours that all iterations combined can take before the experiment terminates. |
| enable_early_stopping | Flag to enble early termination if the score is not improving in the short term. |
| featurization | 'auto' / 'off' / FeaturizationConfig Indicator for whether featurization step should be done automatically or not, or whether customized featurization should be used. Setting this enables AutoML to perform featurization on the input to handle missing data, and to perform some common feature extraction. Note: If the input data is sparse, featurization cannot be turned on. |
| n_cross_validations | Number of cross validation splits. |
| training_data | (sparse) array-like, shape = [n_samples, n_features] |
| label_column_name | (sparse) array-like, shape = [n_samples, ], targets values. |
Customization
Supported customization includes:
- Column purpose update: Override feature type for the specified column.
- Transformer parameter update: Update parameters for the specified transformer. Currently supports Imputer and HashOneHotEncoder.
- Drop columns: Columns to drop from being featurized.
- Block transformers: Allow/Block transformers to be used on featurization process.
Create FeaturizationConfig object using API calls
Call the submit method on the experiment object and pass the run configuration. Execution of local runs is synchronous. Depending on the data and the number of iterations this can run for a while.
In this example, we specify show_output = True to print currently running iterations to the console.
Run the following cell to access previous runs. Uncomment the cell below and update the run_id.
Transparency
View featurization summary for the best model - to study how different features were transformed. This is stored as a JSON file in the outputs directory for the run.
Results
Widget for Monitoring Runs
The widget will first report a "loading" status while running the first iteration. After completing the first iteration, an auto-updating graph and table will be shown. The widget will refresh once per minute, so you should see the graph update as child runs complete.
Note: The widget displays a link at the bottom. Use this link to open a web interface to explore the individual run details
Explanations
This section will walk you through the workflow to compute model explanations for an AutoML model on your remote compute.
Retrieve any AutoML Model for explanations
Below we select an AutoML pipeline from our iterations. The get_output method returns the a AutoML run and the fitted model for the last invocation. Overloads on get_output allow you to retrieve the best run and fitted model for any logged metric or for a particular iteration.
Setup model explanation run on the remote compute
The following section provides details on how to setup an AzureML experiment to run model explanations for an AutoML model on your remote compute.
Sample script used for computing explanations
View the sample script for computing the model explanations for your AutoML model on remote compute.
Substitute values in your sample script
The following cell shows how you change the values in the sample script so that you can change the sample script according to your experiment and dataset.
Create conda configuration for model explanations experiment from automl_run object
Submit the experiment for model explanations
Submit the experiment with the above run_config and the sample script for computing explanations.
Feature importance and visualizing explanation dashboard
In this section we describe how you can download the explanation results from the explanations experiment and visualize the feature importance for your AutoML model on the azure portal.
Download engineered feature importance from artifact store
You can use ExplanationClient to download the engineered feature explanations from the artifact store of the automl_run. You can also use azure portal url to view the dash board visualization of the feature importance values of the engineered features.
Download raw feature importance from artifact store
You can use ExplanationClient to download the raw feature explanations from the artifact store of the automl_run. You can also use azure portal url to view the dash board visualization of the feature importance values of the raw features.
Operationalize
In this section we will show how you can operationalize an AutoML model and the explainer which was used to compute the explanations in the previous section.
Register the AutoML model and the scoring explainer
We use the TreeScoringExplainer from azureml-interpret package to create the scoring explainer which will be used to compute the raw and engineered feature importances at the inference time. In the cell below, we register the AutoML model and the scoring explainer with the Model Management Service.
View your scoring file
Deploy the service
In the cell below, we deploy the service using the automl training environment and the scoring file from the previous steps.
View the service logs
Inference using some test data
Inference using some test data to see the predicted value from autml model, view the engineered feature importance for the predicted value and raw feature importance for the predicted value.
Delete the service
Delete the service once you have finished inferencing.