Text completion: Run Llama 3 models in SageMaker JumpStart

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

In this demo notebook, we demonstrate how to use the SageMaker Python SDK to deploy a JumpStart model for Text Generation using the Llama 3 pretrained model.

To perform inference on these models, you need to pass custom_attributes='accept_eula=true' as part of header. This means you have read and accept the end-user-license-agreement (EULA) of the model. EULA can be found in model card description or from https://ai.meta.com/resources/models-and-libraries/llama-downloads/. By default, this notebook sets custom_attributes='accept_eula=false', so all inference requests will fail until you explicitly change this custom attribute.

Note: Custom_attributes used to pass EULA are key/value pairs. The key and value are separated by '=' and pairs are separated by ';'. If the user passes the same key more than once, the last value is kept and passed to the script handler (i.e., in this case, used for conditional logic). For example, if 'accept_eula=false; accept_eula=true' is passed to the server, then 'accept_eula=true' is kept and passed to the script handler.

Setup

You can continue with the default model or choose a different model: this notebook will run with the following model IDs :

• meta-textgeneration-llama-3-8b
• meta-textgeneration-llama-3-70b

Deploy model

You can now deploy the model using SageMaker JumpStart.

Changing instance type

Inferencing Llama 3 Models are supported on the following instance types:

By default, the JumpStartModel class selects a default instance type available in your region. If you would like to use a different instance type, you can do so by specifying instance type in the JumpStartModel class.

my_model = JumpStartModel(model_id=model_id, instance_type="ml.g5.12xlarge")

Invoke the endpoint

Supported Parameters

This model supports the following inference payload parameters:

• max_new_tokens: Model generates text until the output length (excluding the input context length) reaches max_new_tokens. If specified, it must be a positive integer.
• temperature: Controls the randomness in the output. Higher temperature results in output sequence with low-probability words and lower temperature results in output sequence with high-probability words. If temperature -> 0, it results in greedy decoding. If specified, it must be a positive float.
• top_p: In each step of text generation, sample from the smallest possible set of words with cumulative probability top_p. If specified, it must be a float between 0 and 1.
• return_full_text: If True, input text will be part of the output generated text. If specified, it must be boolean. The default value for it is False.

You may specify any subset of the parameters mentioned above while invoking an endpoint.

Notes

• If max_new_tokens is not defined, the model may generate up to the maximum total tokens allowed, which is 8K for these models. This may result in endpoint query timeout errors, so it is recommended to set max_new_tokens when possible. For 8B and 70B models, we recommend to set max_new_tokens no greater than 1500, and 500 respectively, while keeping the total number of tokens less than 8K.
• In order to support a 8k context length, this model has restricted query payloads to only utilize a batch size of 1. Payloads with larger batch sizes will receive an endpoint error prior to inference.

Example 1

Example 2

Example 3

Example 4

Clean up the endpoint

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.