Amazon SageMaker Lineage

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

Amazon SageMaker Lineage enables events that happen within SageMaker to be traced via a graph structure. The data simplifies generating reports, making comparisons, or discovering relationships between events. For example easily trace both how a model was generated and where the model was deployed.

The lineage graph is created automatically by SageMaker and you can directly create or modify your own graphs.

Key Concepts

• Lineage Graph - A connected graph tracing your machine learning workflow end to end.
• Artifacts - Represents a URI addressable object or data. Artifacts are typically inputs or outputs to Actions.
• Actions - Represents an action taken such as a computation, transformation, or job.
• Contexts - Provides a method to logically group other entities.
• Associations - A directed edge in the lineage graph that links two entities.
• Lineage Traversal - Starting from an arbitrary point trace the lineage graph to discover and analyze relationships between steps in your workflow.
• Experiments - Experiment entites (Experiments, Trials, and Trial Components) are also part of the lineage graph and can be associated wtih Artifacts, Actions, or Contexts.

Notebook Overview

This notebook demonstrates how to:

• Understand the basics of lineage entities.
• Create and associate lineage entities to track your workflow.
• Traverse the associations between lineage entities.

Prerequisites

Select the Python 3 (Data Science) kernel in SageMaker Studio.

Cleanup

Caveats

• Associations cannot be created between two experiment entities. For example between an Experiment and Trial.
• Associations can only be created between the following resources: Action, Artifact, or Context.
• The maximum number of manually created lineage entities are:
  • Artifacts: 6000
  • Contexts: 500
  • Actions: 3000
  • Associations: 6000
• There is no limit on the number of lineage entities created automatically by SageMaker.

Contact

Submit any questions or issues to https://github.com/aws/sagemaker-experiments/issues or mention @aws/sagemakerexperimentsadmin

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.