[Gemma 3]Gradio LlamaCpp Chatbot
Copyright 2025 Google LLC.
Build Chatbot with Gradio & Llama.cpp
Author: Sitam Meur
- GitHub: github.com/sitammeur
- X: @sitammeur
Description: Google recently released Gemma 3 QAT—the Quantization Aware Trained (QAT) Gemma 3 checkpoints. These models maintain similar quality to half precision while using three times less memory. This notebook demonstrates creating a user-friendly chat interface for the gemma-3-1b-it-qat text model using Llama.cpp (for inference) and Gradio (for user interface).
Setup
Select the Colab runtime
To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the Gemma 3 QAT model. In this case, you can use a CPU:
- In the upper-right of the Colab window, select ▾ (Additional connection options).
- Select Change runtime type.
- Under Hardware accelerator, select CPU.
Gemma Setup
Before we dive into the tutorial, let's get you set up:
- Hugging Face Account: If you don't already have one, you can create a free Hugging Face account by clicking here.
- Model Access: Head over to the model page and accept the usage conditions.
- Hugging Face Token: Generate a Hugging Face access (preferably
writepermission) token by clicking here. You'll need this token later in the tutorial.
Once you've completed these steps, you're ready to move on to the next section where we'll set up environment variables in your Colab environment.
Configure your HF token
Add your Hugging Face token to the Colab Secrets manager to securely store it.
- Open your Google Colab notebook and click on the 🔑 Secrets tab in the left panel.

- Create a new secret with the name
HF_TOKEN. - Copy/paste your token key into the Value input box of
HF_TOKEN. - Toggle the button on the left to allow notebook access to the secret.
Install dependencies
Run the cell below to install all the required dependencies.
Log into Hugging Face Hub
Instantiate the Gemma 3 QAT model
We’ll use the Hugging Face Hub to download the 4-bit (Q4_0) quantized Gemma 3 1B instruction-tuned text-only model. New quantized Gemma 3 models, created using Quantization-Aware Training (QAT), offer improved accessibility through reduced memory usage (3x less than bfloat16) without significant accuracy loss. QAT simulates low-precision operations during training for smaller, faster models.
Let's get started by downloading the model from Hugging Face Hub.
Loading the model from HF Hub
Prompt Formatting
Gemma 3 model requires specific formatting to understand the roles of different participants in a conversation. The prompt format is as follows:
<bos><start_of_turn>user
{system_prompt}
{prompt}<end_of_turn>
<start_of_turn>model
Chat with Gemma 3
This function handles loading the model and generating responses. Streaming provides real-time generation rather than waiting for the complete response.
Gradio UI
This offers a user interface with customizable buttons, editable messages, and options for advanced users to tune model parameters like temperature and top-p.
Finally, we’ll launch our chat interface with Gradio
What's Next?
That's it! Here are some ideas to explore further:
-
Explore the Gemma family models: Visit Gemma Open Models to learn about the latest updates regarding the Gemma family models, new capabilities, versions, and more.
-
Gradio Customization: Explore the Gradio documentation to learn about customizing your chat interface, adding new options and features.
-
Share Your Gradio Dashboard: Check out the Sharing your Gradio app page to learn how to safely share your Gradio dashboard with others!
Run in Google Colab