Notebooks
G
Google Gemini
[PaliGemma 2]Using With Transformersjs

[PaliGemma 2]Using With Transformersjs

recurrentgemmagemmapaligemmagemma-cookbookcodegemma
Copyright 2024 Google LLC.
[ ]

PaliGemma2 - Run with Transformers.js

Author: Sitam Meur

Description: This notebook demonstrates how you can run inference on PaliGemma2 model using Node.js and Transformers.js. Transformers.js lets you run Hugging Face's transformer models directly in browser, offering a JavaScript API similar to Python's. It supports NLP, computer vision, audio, and multimodal tasks using ONNX Runtime and allows easy conversion of PyTorch, TensorFlow, and JAX models.

Setup

Select the Colab runtime

To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the PaliGemma 2 model. In this case, you can use CPU runtime:

  1. In the upper-right of the Colab window, select ▾ (Additional connection options).
  2. Select Change runtime type.
  3. Under Hardware accelerator, select CPU.

Installation

Let's get started with installing the dependencies.

[ ]

Create Node.js project

Create a new Node.js project and install the required transformers package via NPM.

[ ]
[ ]

Transformers.js Inference

Now, let's run inference on the PaliGemma2 model using Transformers.js. First, load the model and processor and then prepare inputs (Text query + Image) to run inference and get the output as desired image caption. For reference, you can check the model's page on the Hugging Face model hub under ONNX models section here.

[ ]

It's an image of a cat sitting on a bag, now let's see what the model predicts.

[ ]

Run Application

[ ]

Conclusion

Congratulations! You have successfully run inference on PaliGemma2 model using Transformers.js via Node.js environment. You can now integrate this into your projects.