SageMaker JumpStart Foundation Models - HuggingFace Text2Text Generation

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

Welcome to Amazon SageMaker JumpStart! You can use SageMaker JumpStart to solve many Machine Learning tasks through one-click in SageMaker Studio, or through SageMaker Python SDK.

In this demo notebook, we demonstrate how to use the SageMaker Python SDK for deploying Foundation Models as an endpoint and use them for various NLP tasks. The Foundation models perform Text2Text Generation. It takes a prompting text as an input, and returns the text generated by the model according to the prompt.

Here, we show how to use the state-of-the-art pre-trained FLAN T5 models and FLAN UL2 for Text2Text Generation in the following tasks. You can directly use FLAN-T5 model for many NLP tasks, without fine-tuning the model.

• Text summarization
• Common sense reasoning / natural language inference
• Question and answering
• Sentence / sentiment classification
• Translation
• Pronoun resolution

1. Set Up
2. Select a pre-trained model & deploy
3. Query endpoint and parse response
4. Advanced features: How to use prompts engineering to solve different tasks
5. Clean up the endpoint

Note: This notebook was tested on ml.t3.medium instance in Amazon SageMaker Studio with Python 3 (Data Science) kernel and in Amazon SageMaker Notebook instance with conda_python3 kernel.

1. Set Up

Before executing the notebook, there are some initial steps required for set up.

2. Select a pre-trained model & deploy

You can continue with the default model, or can choose a different model from the list. A complete list of SageMaker pre-trained models can also be accessed at SageMaker pre-trained Models.

3. Query endpoint and parse response

Supported parameters

This model supports many parameters while performing inference. They include:

• max_length: Model generates text until the output length (which includes the input context length) reaches max_length. If specified, it must be a positive integer.
• max_new_tokens: Model generates text until the output length (excluding the input context length) reaches max_new_tokens. If specified, it must be a positive integer.
• num_beams: Number of beams used in the greedy search. If specified, it must be integer greater than or equal to num_return_sequences.
• no_repeat_ngram_size: Model ensures that a sequence of words of no_repeat_ngram_size is not repeated in the output sequence. If specified, it must be a positive integer greater than 1.
• temperature: Controls the randomness in the output. Higher temperature results in output sequence with low-probability words and lower temperature results in output sequence with high-probability words. If temperature -> 0, it results in greedy decoding. If specified, it must be a positive float.
• early_stopping: If True, text generation is finished when all beam hypotheses reach the end of sentence token. If specified, it must be boolean.
• do_sample: If True, sample the next word as per the likelihood. If specified, it must be boolean.
• top_k: In each step of text generation, sample from only the top_k most likely words. If specified, it must be a positive integer.
• top_p: In each step of text generation, sample from the smallest possible set of words with cumulative probability top_p. If specified, it must be a float between 0 and 1.
• return_full_text: If True, input text will be part of the output generated text. If specified, it must be boolean. The default value for it is False.
• *stop: If specified, it must a list of strings. Text generation stops if any one of the specified strings is generated.

We may specify any subset of the parameters mentioned above while invoking an endpoint. Next, we show an example of how to invoke endpoint with these arguments.

4. Advanced features: How to use prompts engineering to solve different tasks

Below we demonstrate solving 5 key tasks with Flan T5 model. The tasks are: text summarization, common sense reasoning / question answering, sentence classification, translation, pronoun resolution.

Note . The notebook in the following sections are particularly designed for Flan T5 models (small, base, large, xl). There are other models like T5-one-line-summary which are designed for text summarization in particular. In that case, such models cannot perform all the following tasks.

4.1. Summarization

Define the text article you want to summarize.

4.2. Common sense reasoning / natural language inference

In the common sense reasoning, you can design a prompt and combine it with the premise, hypothesis, and options, send the combined text into the endpoint to get an answer. Examples are demonstrated as below.

Define the premise, hypothesis, and options that you hope the model to reason.

4.3. Question and Answering

Now, let's try another reasoning task with a different type of prompt template. You can simply provide context and question as shown below.

4.4. Sentence / Sentiment Classification

Define the sentence you want to classify and the corresponding options.

4.5. Translation

Define the sentence and the language you want to translate the sentence to.

4.6. Pronoun resolution

Define the sentence, pronoun, and options you want to reason.

4.7. Imaginary article generation based on a title

4.8 Summarize a title based on the article

5. Clean up the endpoint

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.