Hugging Face LLM Chatbot application using Gradio

---

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

---

AWS Deep Learning Containers (DLC) quickly deploy deep learning environments with optimized, prepackaged container images. This sample notebook is a quick start to deploy open source LLMs to Amazon SageMaker for inference using the Hugging Face LLM Inference Container which is powered by Text Generation Inference (TGI). TGI is a toolkit for deploying and serving Large Language Models (LLMs).

This sample notebook uses open source chat LLM trained using the Open assistant initiative with Gradio as demo Chat app to validate inference.

Enviroment

• Amazon SageMaker Studio JupyterLab (JupyterLab 3.0, 5GB storage and ml.t3.medium instance) with public internet access
• Amazon Sagemaker python SDK version 2.163.0 +
• Gradio python version 4.16.0 +

Step-1: Uninstall packages

This is an optional step included to minimize chances of package dependency conflicts while running the Gradio chat apps later in this notebook

Step-2: Install packages and environment

Install Amazon SageMaker Python SDK and Gradio

Step-3: Retrieve LLM image URI

Using Amazon SageMaker SDK get_huggingface_llm_image_uri() helper function to retrieve appropriate image URI from Amazon ECR for the Hugging Face Large Language Model (LLM). This method allows to retrieve the URI for the desired Hugging Face LLM DLC based on the specified backend, session, region, and version. Refer Amazon SageMaker SDK for details.

• backend : Valid values include "huggingface" and "lmi". The "lmi" stands for SageMaker LMI inference backend, and "huggingface" refers to using Hugging Face TGI inference backend
• session : The SageMaker Session to use. (Default: None)
• region : The AWS region to use for image URI. (default: None)
• version : The framework version for which to retrieve an image URI

Step-4: Configure and create SageMaker Endpoint

In this section,

• Configure the instance type for the SageMaker Deployment that will be used in the deploy() function in Step-5
• Configure the env variables for the HuggingFaceModel class. Environment variables include
  • HF_MODEL_ID which corresponds to the model from the HuggingFace Hub that will be deployed
  • SM_NUM_GPUS to the number of available GPUs on the selected instance type
  • HF_MODEL_QUANTIZE environment variable to reduce the memory footprint of the model
• Configure the Hugging Face model object image_uri, Amazon SageMaker execution role and env variables specified

Step-5: Deploy model for real-time inference

After creating the HuggingFaceModel, deploy the model by invoking the deploy() function. Here the model will be deployed to ml.g4dn.12xlarge instance type. TGI will automatically distribute and shard the model across all GPUs. This may take up to 15 mins -20 mins to complete.

Step-6: Evaluation

Once the endpoint is InService status, execute the following cell for validation.

We will use the predict method from the predictor to evaluate the inference endpoint. We can evaluate with different parameters to impact the generation. Parameters can be defined as in the parameters attribute of the payload. Please refer open api specification of the TGI in the swagger documentation for parameters supported by TGI.

The OpenAssistant/pythia-12b-sft-v8-7k-steps is a conversational chat model meaning we can chat with it using the following prompt:

<|prompter|>[Instruction]<|endoftext|>
<|assistant|>

Step-7: Create a chatbot dummy app using Gradio

Gradio helps to build and share demo applications quickly. Here we will use its rich functions to develop chat applications to demonstrate the chatbot. In this section, a dummy chatbot will be created to validate that the Gradio dependencies are imported and working fine.

If the result shows a successful dummy chatbot app with local URL and public URL, proceed to Step-8.

Step-8: Chatbot application

Once the dummy chatbot application is working. Let's create a customized Chat App that will invoke the Hugging face chat LLM using the SageMaker inference end point that we deployed and validated using prompts in previous steps.

In this chatbot application, Gradio’s low-level Blocks APIs are used.

Successful Result will show and output with a responsive chatbot with local URL and Public URL

Step-9: Cleaning Up

As a best practice and to avoid incurring costs, delete Amazon Sagemaker endpoints

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.