SageMaker JumpStart Foundation Models - HuggingFace Text2Text Generation Batch Transform and Real-Time Batch Inference

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

Welcome to Amazon SageMaker JumpStart! You can use Sagemaker JumpStart to solve many Machine Learning tasks through one-click in SageMaker Studio, or through SageMaker Python SDK.

In this demo notebook, we demonstrate how to use the SageMaker Python SDK for doing the Batch Transform and Real-Time Batch Inference for various NLP tasks. Batch transform allows you to perform inference on large datasets without the need for real-time interaction with the model. In batch transform, you start by creating a batch transform job, which takes as input a dataset and a pre-trained model, and outputs predictions for each data point in the dataset. Batch transform can be useful in situations where you have a large dataset that needs to be processed in a time-efficient manner.

Real-time batch inference is a technique that combines the benefits of real-time and batch inference. In this approach, data is processed in batches, but the batches are small enough to enable real-time processing. Real-time batch inference is useful in situations where you need to process a continuous stream of data in near real-time, but it is not feasible to process each data point individually due to time or resource constraints. Instead, you process the data in small batches, which enables you to take advantage of parallel processing while still maintaining low latency. By using real-time batch inference, you can achieve a balance between low latency and high throughput, enabling you to process large volumes of data in a timely and efficient manner.

Here, we show how to use the state-of-the-art pre-trained text2text FLAN T5, FLAN UL2, T0pp models from Hugging Face for batch transform and real-Time batch inference on the HuggingFace cnn_dailymail dataset for Text summarization task.

Note: This notebook was tested on ml.t3.medium instance in Amazon SageMaker Studio with Python 3 (Data Science) kernel and in Amazon SageMaker Notebook instance with conda_python3 kernel with Python 3.10.10.

1. Set Up
2. Select a pre-trained model
3. Retrieve Artifacts for Model
4. Specify Batch Transform Job HyperParameters
5. Prepare Data for Batch Transform
6. Run Batch Transform Job
7. Computing Rouge Score
8. Real-Time Batch Inference
9. Conclusion

1. Set Up

Before executing the notebook, there are some initial steps required for set up. This notebook requires additional packages. If you are running the following cell for the first time, it is recommended to restart your kernel after installing packages.

Permissions and environment variables

To host on Amazon SageMaker, we need to set up and authenticate the use of AWS services. Here, we use the execution role associated with the current notebook as the AWS account role with SageMaker access.

2. Select a pre-trained model

You can continue with the default model, or can choose a different model from the dropdown generated upon running the next cell. A complete list of SageMaker pre-trained models can also be accessed at Sagemaker pre-trained Models.

[Optional] Select a different Sagemaker pre-trained model. Here, we download the model_manifest file from the Built-In Algorithms s3 bucket, filter-out all the Text2Text Generation models and select a model for inference.

Choose a model for Inference

3. Retrieve Artifacts for Model

Using SageMaker, we can perform inference on the pre-trained model, even without fine-tuning it first on a new dataset. We start by retrieving the deploy_image_uri, and model_uri for the pre-trained model.

4. Specify Batch Transform Job HyperParameters

Batch transform jobs support many advanced parameters while performing inference. They include:

• max_length: Model generates text until the output length (which includes the input context length) reaches max_length. If specified, it must be a positive integer.
• num_return_sequences: Number of output sequences returned. If specified, it must be a positive integer.
• num_beams: Number of beams used in the greedy search. If specified, it must be integer greater than or equal to num_return_sequences.
• no_repeat_ngram_size: Model ensures that a sequence of words of no_repeat_ngram_size is not repeated in the output sequence. If specified, it must be a positive integer greater than 1.
• temperature: Controls the randomness in the output. Higher temperature results in output sequence with low-probability words and lower temperature results in output sequence with high-probability words. If temperature -> 0, it results in greedy decoding. If specified, it must be a positive float.
• early_stopping: If True, text generation is finished when all beam hypotheses reach the end of stence token. If specified, it must be boolean.
• do_sample: If True, sample the next word as per the likelyhood. If specified, it must be boolean.
• top_k: In each step of text generation, sample from only the top_k most likely words. If specified, it must be a positive integer.
• top_p: In each step of text generation, sample from the smallest possible set of words with cumulative probability top_p. If specified, it must be a float between 0 and 1.
• seed: Fix the randomized state for reproducibility. If specified, it must be an integer.
• batch_size: Batching could speed things up, it may be useful to try tuning the batch_size parameter. In some cases can actually be quite slower as explained here. We will use a batch_size of 4. But If you get Cuda Out of Memory Error please decrease it accordingly. If specified, it must be a positive integer.

We may specify any subset of the hyper parameters mentioned above for a batch transform job. These hyperparameters will be passed as an env variables to the batch transform job. If any of these are passed, then the advanced hyper-parameters from the individual examples in the JSONLINES payload will not be used. If you don't want to pass it, set it to None, and in that case advanced hyper-parameters from the individual examples in the JSONLINES payload will be used and inference will be run on each example at a time.

5. Prepare Data for Batch Transform

If you want to specify different parameters for each test input rather than same for whole batch please do so here while creating the dataset.

• The input format for the batch transform job is a jsonl file with entries as ->
  {"id":1,"text_inputs":"Translate to German: My name is Arthur", "max_length":50,.....}
{"id":2,"text_inputs":"Tell me the steps to make a pizza", "max_length":50,.....}
• While the output format is ->
  {"id":1, generated_texts':['Ich bin Arthur']}
{"id":2, 'generated_texts':['Preheat oven to 400 degrees F. Spread pizza sauce on a pizza pan. Bake for']}

6. Run Batch Transform Job

When you start a batch transform job, Amazon SageMaker launches the necessary compute resources to process the data, including CPU or GPU instances depending on the selected instance type. During the batch transform job, Amazon SageMaker automatically provisions and manages the compute resources required to process the data, including instances, storage, and networking resources. Once the batch transform job has completed, the compute resources are automatically cleaned up by Amazon SageMaker. This means that the instances and storage used during the job are terminated and removed, freeing up resources and minimizing costs.

7. Computing Rouge Score

ROUGE, or Recall-Oriented Understudy for Gisting Evaluation, is a set of metrics and a software package used for evaluating automatic summarization and machine translation in natural language processing. The metrics compare an automatically produced summary or translation against a reference or a set of references (human-produced) summary or translation.

8. Real-Time Batch Inference

We can also run the real-time batch inference on an endpoint by providing the inputs as a list. Real-time batch inference is useful in situations where you need to process a continuous stream of data in near real-time, but it is not feasible to process each data point individually due to time or resource constraints. Instead, you process the data in small batches, which enables you to take advantage of parallel processing while still maintaining low latency.

8.1 Deploying the Model

8.2 Running Inference on the Model

9. Conclusion

In this notebook, we conducted batch transform and real-time batch inference. Additionally, we used the Rouge score to compare the test data summarization with the model-generated summarization. We found that batch transform is advantageous in obtaining inferences from large datasets without requiring a persistent endpoint. Furthermore, we linked input records with inferences to aid in result interpretation. On the other hand, teal-time batch inference is beneficial in achieving high throughput.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.