Notebooks
G
Google Gemini
[PaliGemma 2]Keras Quickstart

[PaliGemma 2]Keras Quickstart

recurrentgemmagemmapaligemmagemma-cookbookcodegemma
Copyright 2024 Google LLC.
[1]

Keras PaliGemma 2 Quickstart

This is a quick demo of PaliGemma 2 running on Keras

We'll explore image captioning and object detection using 3B weights, fine-tuned with 448*448 input images on the DOCCI dataset.

Setup

Select the Colab runtime

To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the Gemma model:

  1. In the upper-right of the Colab window, select ▾ (Additional connection options).
  2. Select Change runtime type.
  3. Under Hardware accelerator, select T4 GPU.

Gemma setup on Kaggle

To complete this tutorial, you'll first need to complete the setup instructions at Gemma setup, as PaliGemma is a Gemma variant.

In brief, you will need to

  • Get access to Gemma on kaggle.com.
  • Generate and configure a Kaggle username and API key.

After you've completed the Gemma setup, move on to the next section, where you'll set your username and API key as environment variables for your Colab environment.

Set environment variables

Set environment variables for KAGGLE_USERNAME and KAGGLE_KEY.

[2]

Install dependencies

Install Keras and KerasNLP.

[ ]

Load Image

[4]
Output

Load Model

[5]
Downloading from https://www.kaggle.com/api/v1/models/keras/paligemma2/keras/pali_gemma_2_ft_docci_3b_448/1/download/config.json...
100%|██████████| 1.06k/1.06k [00:00<00:00, 1.37MB/s]
Downloading from https://www.kaggle.com/api/v1/models/keras/paligemma2/keras/pali_gemma_2_ft_docci_3b_448/1/download/model.weights.h5...
100%|██████████| 5.65G/5.65G [01:56<00:00, 52.3MB/s]
Downloading from https://www.kaggle.com/api/v1/models/keras/paligemma2/keras/pali_gemma_2_ft_docci_3b_448/1/download/tokenizer.json...
100%|██████████| 431/431 [00:00<00:00, 280kB/s]
Downloading from https://www.kaggle.com/api/v1/models/keras/paligemma2/keras/pali_gemma_2_ft_docci_3b_448/1/download/assets/tokenizer/vocabulary.spm...
100%|██████████| 4.07M/4.07M [00:00<00:00, 9.33MB/s]
Downloading from https://www.kaggle.com/api/v1/models/keras/paligemma2/keras/pali_gemma_2_ft_docci_3b_448/1/download/image_converter.json...
100%|██████████| 510/510 [00:00<00:00, 609kB/s]

Inference

[11]
<image>caption en
A medium view of a girl and a cat on a rooftop. The girl is facing the right side and is looking at a brown cat that is standing on a white carpet that is covered in snow. The cat is looking at the girl, and its tail is sticking out to the right. The carpet is green, and the ground is also green. The girl is wearing a pink and gray striped sweater and pants. On the left side of the carpet, there is a yellow box that is filled with snow. Behind the cat and girl, there is a white wall that has a red brick border. On the other side of the wall, there are many buildings that are covered in snow.
Output
[10]
detect  cat ; person
<loc0240><loc0277><loc0990><loc0435> <seg027><seg009><seg060><seg088><seg021><seg116><seg064><seg055><seg000><seg066><seg114><seg089><seg027><seg063><seg057><seg089> person ; <loc0362><loc0644><loc0699><loc0825> <seg114><seg041><seg037><seg121><seg029><seg073><seg060><seg089><seg051><seg117><seg074><seg012><seg044><seg116><seg057><seg012> cat
Output