Convert Gemma 3 270M To LiteRT For MediaPipe LLM Inference API
Copyright 2025 Google LLC.
Convert Gemma 3 270M to LiteRT for use with MediaPipe LLM Inference API
This notebook converts a Gemma 3 270M for use with the MediaPipe LLM Inference API, a library that enables inference on mobile devices or in web browsers. The entire process takes about 15 minutes:
- Set up the Colab environment
- Load the model from Hugging Face
- Convert the model with the AI Edge Torch converter
- Package the model with the MediaPipe Task bundler
- Download the model
Gemma 3 270M is designed for task-specific fine-tuning and engineered for efficient performance on mobile, web, and edge devices. You can fine-tune your own model using this notebook and run it in a demo web app once converted.
Set up development environment
The first step is to install packages using pip.
Restart the session runtime to ensure you're using the newly installed packages.
Load the model
To access models on Hugging Face, provide your Access Token. You can store it as a Colab secret in the left toolbar by specifying HF_TOKEN as the 'Name' and adding your unique token as the 'Value'.
Specify the Hugging Face repo ID of the model to convert. It'll be saved to your Colab files for conversion.
Convert the model
Convert and quantize the model using the AI Edge Torch converter. You can adjust the conversions parameters based on your task's requirements:
prefill_seq_len: maximum length of supported inputkv_cache_max_len: maximum of prefill + decode context lengthquantize: the quantization scheme. 8-bit integer quantization (INT8) is good for web environments
This takes about 10 minutes. The .tflite model will be saved temporarily to your Colab files.
Create a MediaPipe Task Bundle
A MediaPipe Task file (.task) bundles the original model tokenizer, the LiteRT model (.tflite), and additional metadata needed to run end-to-end inference with the MediaPipe LLM Inference API.
To use the bundler, install the MediaPipe PyPI package (>0.10.14) in this step as it comes with its own set of dependencies.
Do a fresh install of protobuf and tensorflow and restart the Colab runtime to get the latest versions.
Now, you'll configure and create the Task bundle:
- Update
tflite_modelto point to the newly converted .tflite model in your Colab files - Update
tokenizer_modelto point to the tokenizer.model that was downloaded from Hugging Face Hub. - Name your .task file in the
output_filename.
Download & run your model on-device
Your model is now ready for on-device inference using the MediaPipe LLM Inference API!
Download the .task file from your Colab environment to use it in your projects.
Try it in the emoji generation web app which runs the model directly in the browser. You can also explore documentation for building cross-platform mobile and web apps.
Run in Google Colab