Notebooks
G
Google Gemini
Convert Gemma 3 270M To LiteRT For MediaPipe LLM Inference API

Convert Gemma 3 270M To LiteRT For MediaPipe LLM Inference API

recurrentgemmagemmapaligemmaresourcesgemma-cookbookEmoji-Gemma-on-WebDemoscodegemma

Copyright 2025 Google LLC.

[ ]

Convert Gemma 3 270M to LiteRT for use with MediaPipe LLM Inference API

This notebook converts a Gemma 3 270M for use with the MediaPipe LLM Inference API, a library that enables inference on mobile devices or in web browsers. The entire process takes about 15 minutes:

  1. Set up the Colab environment
  2. Load the model from Hugging Face
  3. Convert the model with the AI Edge Torch converter
  4. Package the model with the MediaPipe Task bundler
  5. Download the model

Gemma 3 270M is designed for task-specific fine-tuning and engineered for efficient performance on mobile, web, and edge devices. You can fine-tune your own model using this notebook and run it in a demo web app once converted.

Set up development environment

The first step is to install packages using pip.

[ ]

Restart the session runtime to ensure you're using the newly installed packages.

Load the model

To access models on Hugging Face, provide your Access Token. You can store it as a Colab secret in the left toolbar by specifying HF_TOKEN as the 'Name' and adding your unique token as the 'Value'.

[ ]

Specify the Hugging Face repo ID of the model to convert. It'll be saved to your Colab files for conversion.

[ ]
model_author
model_name

Convert the model

Convert and quantize the model using the AI Edge Torch converter. You can adjust the conversions parameters based on your task's requirements:

  • prefill_seq_len: maximum length of supported input
  • kv_cache_max_len: maximum of prefill + decode context length
  • quantize: the quantization scheme. 8-bit integer quantization (INT8) is good for web environments

This takes about 10 minutes. The .tflite model will be saved temporarily to your Colab files.

[ ]

Create a MediaPipe Task Bundle

A MediaPipe Task file (.task) bundles the original model tokenizer, the LiteRT model (.tflite), and additional metadata needed to run end-to-end inference with the MediaPipe LLM Inference API.

To use the bundler, install the MediaPipe PyPI package (>0.10.14) in this step as it comes with its own set of dependencies.

[ ]

Do a fresh install of protobuf and tensorflow and restart the Colab runtime to get the latest versions.

[ ]

Now, you'll configure and create the Task bundle:

  1. Update tflite_model to point to the newly converted .tflite model in your Colab files
  2. Update tokenizer_model to point to the tokenizer.model that was downloaded from Hugging Face Hub.
  3. Name your .task file in the output_filename.
[ ]

Download & run your model on-device

Your model is now ready for on-device inference using the MediaPipe LLM Inference API!

Download the .task file from your Colab environment to use it in your projects.

[ ]

Try it in the emoji generation web app which runs the model directly in the browser. You can also explore documentation for building cross-platform mobile and web apps.