[PaliGemma 2]Keras Quickstart
recurrentgemmagemmapaligemmagemma-cookbookcodegemma
Export
Copyright 2024 Google LLC.
[1]
Keras PaliGemma 2 Quickstart
This is a quick demo of PaliGemma 2 running on Keras
We'll explore image captioning and object detection using 3B weights, fine-tuned with 448*448 input images on the DOCCI dataset.
Setup
Select the Colab runtime
To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the Gemma model:
- In the upper-right of the Colab window, select ▾ (Additional connection options).
- Select Change runtime type.
- Under Hardware accelerator, select T4 GPU.
Gemma setup on Kaggle
To complete this tutorial, you'll first need to complete the setup instructions at Gemma setup, as PaliGemma is a Gemma variant.
In brief, you will need to
- Get access to Gemma on kaggle.com.
- Generate and configure a Kaggle username and API key.
After you've completed the Gemma setup, move on to the next section, where you'll set your username and API key as environment variables for your Colab environment.
Set environment variables
Set environment variables for KAGGLE_USERNAME and KAGGLE_KEY.
[2]
Install dependencies
Install Keras and KerasNLP.
[ ]
Load Image
[4]
Load Model
[5]
Downloading from https://www.kaggle.com/api/v1/models/keras/paligemma2/keras/pali_gemma_2_ft_docci_3b_448/1/download/config.json...
100%|██████████| 1.06k/1.06k [00:00<00:00, 1.37MB/s]
Downloading from https://www.kaggle.com/api/v1/models/keras/paligemma2/keras/pali_gemma_2_ft_docci_3b_448/1/download/model.weights.h5...
100%|██████████| 5.65G/5.65G [01:56<00:00, 52.3MB/s]
Downloading from https://www.kaggle.com/api/v1/models/keras/paligemma2/keras/pali_gemma_2_ft_docci_3b_448/1/download/tokenizer.json...
100%|██████████| 431/431 [00:00<00:00, 280kB/s]
Downloading from https://www.kaggle.com/api/v1/models/keras/paligemma2/keras/pali_gemma_2_ft_docci_3b_448/1/download/assets/tokenizer/vocabulary.spm...
100%|██████████| 4.07M/4.07M [00:00<00:00, 9.33MB/s]
Downloading from https://www.kaggle.com/api/v1/models/keras/paligemma2/keras/pali_gemma_2_ft_docci_3b_448/1/download/image_converter.json...
100%|██████████| 510/510 [00:00<00:00, 609kB/s]
Inference
[11]
<image>caption en A medium view of a girl and a cat on a rooftop. The girl is facing the right side and is looking at a brown cat that is standing on a white carpet that is covered in snow. The cat is looking at the girl, and its tail is sticking out to the right. The carpet is green, and the ground is also green. The girl is wearing a pink and gray striped sweater and pants. On the left side of the carpet, there is a yellow box that is filled with snow. Behind the cat and girl, there is a white wall that has a red brick border. On the other side of the wall, there are many buildings that are covered in snow.
[10]
detect cat ; person <loc0240><loc0277><loc0990><loc0435> <seg027><seg009><seg060><seg088><seg021><seg116><seg064><seg055><seg000><seg066><seg114><seg089><seg027><seg063><seg057><seg089> person ; <loc0362><loc0644><loc0699><loc0825> <seg114><seg041><seg037><seg121><seg029><seg073><seg060><seg089><seg051><seg117><seg074><seg012><seg044><seg116><seg057><seg012> cat
Run in Google Colab
View source on GitHub