02 Export Feature Groups
Music Recommender Data Preparation with SageMaker Feature Store and SageMaker Data Wrangler
This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.
Background
This notebook is part of a notebook series that goes through the ML lifecycle and shows how we can build a Music Recommender System using a combination of SageMaker services and features. This notebook uses Amazon SageMaker Feature Store (Feature Store) to create a feature group,
executes your Data Wrangler Flow 01_music_dataprep.flow on the entire dataset using a SageMaker
Processing Job and ingest processed data to Feature Store. It is the second notebook in the series. You can choose to run this notebook by itself or in sequence with the other notebooks listed below. Please see the README.md for more information about this use case implement of this sequence of notebooks.
- Music Recommender Data Exploration
- Music Recommender Data Preparation with SageMaker Feature Store and SageMaker Data Wrangler (current notebook)
- Train, Deploy, and Monitor the Music Recommender Model using SageMaker SDK
Contents
- Prereqs: Get Data
- Update the Data Source in the .flow File
- Create Feature Group
- Configure Feature Group
- Initialize & Create Feature Group
- Inputs and Outputs
- Upload Flow to S3
- Run Processing Job
- Fetch Data from Offline Feature Store
Prereqs: Get Data
Here we will download the music data from a public S3 bucket that we'll be using for this demo and uploads it to your default S3 bucket that was created for you when you initially created a SageMaker Studio workspace.
Update the Data Source in the .flow File
The 01_music_datapred.flow file is a JSON file containing instructions for where to find your data sources and how to transform the data. We'll be updating the object telling Data Wrangler where to find the input data on S3. We will set this to your default S3 bucket. With this update to the .flow file it now points to your new S3 bucket as the data source used by SageMaker Data Wrangler.
Make sure the .flow file is closed before running this next step or it won't update the new s3 file locations in the file
Create Feature Group
Amazon SageMaker Feature Store is a fully managed, purpose-built repository to store, update, retrieve, and share machine learning (ML) features. Features are the attributes or properties models use during training and inference to make predictions. For example, in a ML application that recommends a music playlist, features could include song ratings, which songs were listened to previously, and how long songs were listened to. The accuracy of a ML model is based on a precise set and composition of features. Often, these features are used repeatedly by multiple teams training multiple models. And whichever feature set was used to train the model needs to be available to make real-time predictions (inference). Keeping a single source of features that is consistent and up-to-date across these different access patterns is a challenge as most organizations keep two different feature stores, one for training and one for inference.
Amazon SageMaker Feature Store is a purpose-built repository where you can store and access features so it’s much easier to name, organize, and reuse them across teams. SageMaker Feature Store provides a unified store for features during training and real-time inference without the need to write additional code or create manual processes to keep features consistent. SageMaker Feature Store keeps track of the metadata of stored features (e.g. feature name or version number) so that you can query the features for the right attributes in batches or in real time using Amazon Athena, an interactive query service. SageMaker Feature Store also keeps features updated, because as new data is generated during inference, the single repository is updated so new features are always available for models to use during training and inference.
What is a feature group
A single feature corresponds to a column in your dataset. A feature group is a predefined schema for a collection of features - each feature in the feature group has a specified data type and name. A single record in a feature group corresponds to a row in your dataframe. A feature store is a collection of feature groups. To learn more about SageMaker Feature Store, see Amazon Feature Store Documentation.
Define Feature Group
Select Record identifier and Event time feature name. These are required parameters for feature group creation.
- Record identifier name is the name of the feature defined in the feature group's feature definitions whose value uniquely identifies a Record defined in the feature group's feature definitions.
- Event time feature name is the name of the EventTime feature of a Record in FeatureGroup. An EventTime is a timestamp that represents the point in time when a new event occurs that corresponds to the creation or update of a Record in the FeatureGroup. All Records in the FeatureGroup must have a corresponding EventTime.
Feature Definitions
The following is a list of the feature names and feature types of the final dataset that will be produced
when your data flow is used to process your input dataset. These are automatically generated from the
step Custom Pyspark from Source: Answers.Csv. To save from a different step, go to Data Wrangler to
select a new step to export.
- You can select a subset of the features. By default all columns of the result dataframe will be used as features.
- You can change the Data Wrangler data type to one of the Feature Store supported types (Integral, Fractional, or String). The default type is set to String. This means that, if a column in your dataset is not a float or long type, it will default to String in your Feature Store.
For Event Time features, make sure the format follows the feature store Event Time feature format
The following is a list of the feature names and data types of the final dataset that will be produced when your data flow is used to process your input dataset.
Below we create the SDK input for those feature definitions. Some schema types in Data Wrangler are not supported by Feature Store. The following will create a default_FG_type set to String for these types.
Configure Feature Group
- feature_group_name: name of the feature group.
- feature_store_offline_s3_uri: SageMaker FeatureStore writes the data in the OfflineStore of a FeatureGroup to a S3 location owned by you.
- enable_online_store: controls if online store is enabled. Enabling the online store allows quick access to the latest value for a Record via the GetRecord API.
- iam_role: IAM role for executing the processing job.
Initialize & Create Feature Group
Feature group is initialized and created below
Now that the feature group is created, You will use a processing job to process your data at scale and ingest the transformed data into this feature group.
Inputs and Outputs
The below settings configure the inputs and outputs for the flow export.
In Input - Source you can configure the data sources that will be used as input by Data Wrangler
- For S3 sources, configure the source attribute that points to the input S3 prefixes
- For all other sources, configure attributes like query_string, database in the source's DatasetDefinition object.
If you modify the inputs the provided data must have the same schema and format as the data used in the Flow. You should also re-execute the cells in this section if you have modified the settings in any data sources.
Input - S3 Source: tracks.csv
Input - S3 Source: ratings.csv
Output: Feature Store
Below are the inputs required by the SageMaker Python SDK to launch a processing job with feature store as an output. Notice the output_name variable below; this ID is found within the .flow file at the node point you want to capture transformations up to. The .flow file contains instructions for SageMaker Data Wrangler to know where to look for data and how to transform it. Each data transformation action is associated with a node and therefore a node ID. Using the associated node ID + output name tells SageMaker up to what point in the transformation process you want to export to a feature store.
Upload Flow to S3
To use the Data Wrangler as an input to the processing job, first upload your flow file to Amazon S3.
The Data Wrangler Flow is also provided to the Processing Job as an input source which we configure below.
Run Processing Job
Job Configurations
You can configure the following settings for Processing Jobs. If you change any configurations you will need to re-execute this and all cells below it by selecting the Run menu above and click Run Selected Cells and All Below
- IAM role for executing the processing job.
- A unique name of the processing job. Give a unique name every time you re-execute processing jobs
- Data Wrangler Container URL.
- Instance count, instance type and storage volume size in GB.
- Content type for each output. Data Wrangler supports CSV as default and Parquet.
- Network Isolation settings
Create Processing Job
To launch a Processing Job, you will use the SageMaker Python SDK to create a Processor function.
Job Status & S3 Output Location
Below you wait for processing job to finish. If it finishes successfully, your feature group should be populated with transformed feature values. In addition the raw parameters used by the Processing Job will be printed.
You can view newly created feature group in Studio, refer to Use Amazon SageMaker Feature Store with Amazon SageMaker Studio for detailed guide. Learn more about SageMaker Feature Store
Fetch Data from Offline Feature Store
There are 3 feature stores for the ratings, tracks, and user preferences data. We retrieve data from all 3 before joining them.
Check if the Athena queries have been done and the data sets exist, then just do train test split or just proceed to training
Notebook CI Test Results
This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.