Amazon SageMaker Using AWS Large Model Inference (LMI) Deep Learning Container

Recommended Kernel(s): You can run this notebook with any Amazon SageMaker Studio kernel. We recommend using the Data Science 3.0 kernel.

This notebook demonstrates how to deploy a meta-llama/Meta-Llama-3-8B HuggingFace model to a SageMaker Endpoint for text generation. In this example, the SageMaker-managed LMI (Large Model Inference) Docker image will serve as the inference image. LMI images feature a DJL serving stack powered by the Deep Java Library.

1. Dependency Installation

1.1. Python Dependencies & Imports

This notebook requires the following Python dependencies:

• AWS sagemaker_core

Let's install or upgrade these dependencies using the following command:

Download custom draft model

2. Deploy Speculative Decoding-Enabled Endpoint

2.1. Endpoint Deployment

We will configure what models to use and server startup parameters and via environment variables.

• OPTION_MODEL_ID points to the HF Model ID or base S3 prefix of the model artifacts You can read about the other config paramters in LMI documentation.

Start-up of LLM inference containers can last longer than smaller models, mainly due to longer model downloading and loading times. Timeout values need to be increased accordingly from their default values. Each endpoint deployment takes a few minutes. We also set routing_strategy which would benefit us if we were to back our endpoint with multiple instances.

This cells will block until the endpoint is deployed, which is necessary for the following steps.

Endpoint invocation

Let's invoke our endpoint and get a sample response.

3. Clean Up Endpoint