Auto Ml Forecasting Backtest Single Model
Copyright (c) Microsoft Corporation. All rights reserved.
Licensed under the MIT License.
Automated MachineLearning
The model backtesting
Contents
Introduction
Model backtesting is used to evaluate its performance on historical data. To do that we step back on the backtesting period by the data set several times and split the data to train and test sets. Then these data sets are used for training and evaluation of model.
This notebook is intended to demonstrate backtesting on a single model, this is the best solution for small data sets with a few or one time series in it. For scenarios where we would like to choose the best AutoML model for every backtest iteration, please see AutoML Forecasting Backtest Many Models Example notebook.
This notebook demonstrates two ways of backtesting:
- AutoML backtesting: we will train separate AutoML models for historical data
- Model backtesting: from the first run we will select the best model trained on the most recent data, retrain it on the past data and evaluate.
Setup
This notebook is compatible with Azure ML SDK version 1.35.1 or later.
As part of the setup you have already created a Workspace.
Data
For the demonstration purposes we will simulate one year of daily data. To do this we need to specify the following parameters: time column name, time series ID column names and label column name. Our intention is to forecast for two weeks ahead.
Let's see what the training data looks like.
Prepare remote compute and data.
The Machine Learning service workspace, is paired with the storage account, which contains the default data store. We will use it to upload the artificial data and create tabular dataset for training. A tabular dataset defines a series of lazily-evaluated, immutable operations to load data from the data source into tabular representation.
You will need to create a compute target for backtesting. In this tutorial, you create AmlCompute as your training compute resource.
Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.
Create the configuration for AutoML backtesting
This dictionary defines the AutoML and many models settings. For this forecasting task we need to define several settings including the name of the time column, the maximum forecast horizon, and the partition column name definition.
| Property | Description |
|---|---|
| task | forecasting |
| primary_metric | This is the metric that you want to optimize. Forecasting supports the following primary metrics normalized_root_mean_squared_error normalized_mean_absolute_error |
| iteration_timeout_minutes | Maximum amount of time in minutes that the model can train. This is optional but provides customers with greater control on exit criteria. |
| iterations | Number of models to train. This is optional but provides customers with greater control on exit criteria. |
| experiment_timeout_hours | Maximum amount of time in hours that the experiment can take before it terminates. This is optional but provides customers with greater control on exit criteria. |
| label_column_name | The name of the label column. |
| max_horizon | The forecast horizon is how many periods forward you would like to forecast. This integer horizon is in units of the timeseries frequency (e.g. daily, weekly). Periods are inferred from your data. |
| n_cross_validations | Number of cross validation splits. The default value is "auto", in which case AutoMl determines the number of cross-validations automatically, if a validation set is not provided. Or users could specify an integer value. Rolling Origin Validation is used to split time-series in a temporally consistent way. |
| cv_step_size | Number of periods between two consecutive cross-validation folds. The default value is "auto", in which case AutoMl determines the cross-validation step size automatically, if a validation set is not provided. Or users could specify an integer value. |
| time_column_name | The name of your time column. |
| grain_column_names | The column names used to uniquely identify timeseries in data that has multiple rows with the same timestamp. |
To train AutoML on backtesting folds we will use the Azure Machine Learning pipeline. It will generate backtest folds, then train model for each of them and calculate the accuracy metrics. To run pipeline, you also need to create an Experiment. An Experiment corresponds to a prediction problem you are trying to solve (here, it is a forecasting), while a Run corresponds to a specific approach to the problem.
Run the pipeline and wait for results.
After the run is complete, we can download the results.
Move the directory and list its contents.
The directory contains a set of files with results:
- forecast.csv contains forecasts for all backtest iterations. The backtest_iteration column contains iteration identifier with the last training date as a suffix
- scores.csv contains all metrics. If data set contains several time series, the metrics are given for all combinations of time series id and iterations, as well as scores for all iterations and time series id are marked as "all_sets"
- plots_fcst_vs_actual.pdf contains the predictions vs forecast plots for each iteration and time series.
For demonstration purposes we will display the table of metrics for one of the time series with ID "ts0". Again, we will create the utility function, which will be re used in model backtesting.
Forecast vs actuals plots.
We will backtest the model trained on the most recet data.
Retrain the models.
Assemble the pipeline, which will retrain the best model from AutoML run on historical data.
Launch the backtesting pipeline.
The metrics are stored in the pipeline output named "score". The next code will download the table with metrics.
Again, we will copy the data files from the downloaded directory, but in this case we will call the folder "model_backtest"; it will contain the same files as the one for AutoML backtesting.
Finally, we will display the metrics.
Forecast vs actuals plots.