BYOC LLM Monitoring: Bring Your Own Container Llama2 Monitoring with SageMaker Model Monitor

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

In this demo notebook, we demonstrate how to use the SageMaker Python SDK to deploy and monitor a JumpStart Llama 2 fine-tuned model for Toxicity levels. The container associated with this notebook employs the FMEval open-source library for LLM evaluation.

To perform inference on these models, you need to pass custom_attributes='accept_eula=true' as part of header. This means you have read and accept the end-user-license-agreement (EULA) of the model. EULA can be found in model card description or from https://ai.meta.com/resources/models-and-libraries/llama-downloads/. By default, this notebook sets custom_attributes='accept_eula=false', so all inference requests will fail until you explicitly change this custom attribute.

Note: Custom_attributes used to pass EULA are key/value pairs. The key and value are separated by '=' and pairs are separated by ';'. If the user passes the same key more than once, the last value is kept and passed to the script handler (i.e., in this case, used for conditional logic). For example, if 'accept_eula=false; accept_eula=true' is passed to the server, then 'accept_eula=true' is kept and passed to the script handler.

Background

SageMaker Model Monitor allows users to provide images of their own custom-built containers to be run at each monitoring job. This notebook leverages the BYOC feature to monitor the Llama2-7b model for 7 different Toxicity levels.

Prerequisites

• IF RUNNING LOCALLY (not SageMaker Studio/Classic): An IAM role that gives SageMakerFullAccess. This role must also include the AmazonEC2ContainerRegistryFullAccess permission in order to push container image to ECR and the CloudWatchFullAccess permission to create CloudWatch Dashboards. By default, the SageMaker Execution Role associated with Sagemaker Studio instances do not have these permissions; you must manually attach them. For information on how to complete this, see this documentation

• IF RUNNING ON SAGEMAKER STUDIO/STUDIO CLASSIC (not locally): Please ensure that Docker access is enabled in your domain and that you have downloaded Docker for this notebook instance. Please follow the guide at the end of this notebook to complete Docker setup.

Setup

This notebook is best suited for a kernel of python verion >= 3.11

Retreive your SageMaker Session and Configure Execution Role

You can continue with the default model or choose a different model: this notebook will run with the following model IDs :

• meta-textgeneration-llama-2-7b-f
• meta-textgeneration-llama-2-13b-f
• meta-textgeneration-llama-2-70b-f

Deploy model

You can now deploy the model using SageMaker JumpStart.

Set up DataCapture

Deploy JumpStart Model

Note: This will take roughly 10 mins

Invoke the endpoint

Supported Parameters

This model supports the following inference payload parameters:

• max_new_tokens: Model generates text until the output length (excluding the input context length) reaches max_new_tokens. If specified, it must be a positive integer.
• temperature: Controls the randomness in the output. Higher temperature results in output sequence with low-probability words and lower temperature results in output sequence with high-probability words. If temperature -> 0, it results in greedy decoding. If specified, it must be a positive float.
• top_p: In each step of text generation, sample from the smallest possible set of words with cumulative probability top_p. If specified, it must be a float between 0 and 1.

You may specify any subset of the parameters mentioned above while invoking an endpoint.

Notes

• If max_new_tokens is not defined, the model may generate up to the maximum total tokens allowed, which is 4K for these models. This may result in endpoint query timeout errors, so it is recommended to set max_new_tokens when possible. For 7B, 13B, and 70B models, we recommend to set max_new_tokens no greater than 1500, 1000, and 500 respectively, while keeping the total number of tokens less than 4K.
• In order to support a 4k context length, this model has restricted query payloads to only utilize a batch size of 1. Payloads with larger batch sizes will receive an endpoint error prior to inference.
• This model only supports 'system', 'user' and 'assistant' roles, starting with 'system', then 'user' and alternating (u/a/u/a/u...).

Single invocation

NOTE: Read the end-user-license-agreement here https://ai.meta.com/resources/models-and-libraries/llama-downloads/ and accept by setting accept_eula to true, otherwise an error will be raised.

Send artificial traffic to the endpoint.

The following cell will send 10 queries to the endpoint. Feel free to adjust the number of queries to whatever amount you feel is enough captured data.

NOTE: Read the end-user-license-agreement here https://ai.meta.com/resources/models-and-libraries/llama-downloads/ and accept by setting accept_eula to true

Build and Push the Image to ECR

IMPORTANT: If running locally (not on SageMaker Studio), delete ' --network sagemaker'

Build the image. This will take some time.

Create the repository. Ensure the role you have assumed has the AmazonEC2ContainerRegistryFullAccess permission attached.

Push the image to ECR. This will take some time, as we are pushing a ~9GB image. Ensure that your AWS credentials are fresh.

Set a Monitoring Schedule

Note: The following cell sets a one-time monitoring schedule for demonstration purposes. A one-time monitoring schedule will execute immediately. If you would like to set an hourly schedule, swap out the commented line. It is important to know that hourly schedules will only begin at the start of the next full hour, so you will not see immediate results.

View Results

The following cell prints the output report stored in Amazon S3. It includes evaluations for at most 100 samples of the captured data.

NOTE: The report will show up once the job is finished. Please try again in a few minutes.

View Cloudwatch Dashboard Graph

The following cell will generate a CloudWatch Dashboard for viewing the evaluation results from the monitoring schedule you ran. For more information on dashboard formatting, see here

Click the link from the following cell output to view the created CloudWatch Dashboard

Clean up resources

SageMaker Studio Docker Guide

To set up docker in your SageMaker studio environment, follow these steps:

1. Run the following command in the AWS CLI, inputting your region and SageMaker domain ID:

aws --region <region> \
    sagemaker update-domain --domain-id <domain-id> \
    --domain-settings-for-update '{"DockerSettings": {"EnableDockerAccess": "ENABLED"}}'

2. Open a new notebook instance. Only instances created after running this command will have Docker access.
3. Open the terminal in this new instance and follow the installation directions

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.