Copyright 2025 Google LLC.

Multi-spectral remote sensing with Gemini

In this notebook, you'll learn how to use Gemini's image understanding capabilities for multi-spectral analysis and remote sensing.

What is remote sensing?

Remote sensing is the process of acquiring information about an object or area without making physical contact with it. It's essentially "seeing" and measuring something from a distance.

The most common form of remote sensing involves using sensors on satellites to collect data about the Earth's surface. Having a record of images of the Earth's surface across time enables us to track changes like: Where are forests disappearing? Where are fires occurring? And answer questions like: Are the crops in this field healthy? What is the solar potential of this roof? All of these questions (and more!) can be answered with the help of satellite data.

Much of the power of these satellite images lies in their ability to capture multi-spectral data. Instead of the 3 RGB bands you're used to, these satellites can capture data across many different bands of the electromagnetic spectrum, including those humans can't see, like Near-Infrared (NIR) and Short-Wave Infrared (SWIR). Certain wavelengths are beneficial for certain use cases, for example:

• Vegetation Health: Healthy plants reflect a lot of NIR light. By looking at the NIR band, you can assess crop health or monitor deforestation far more accurately than with a simple green photo.
• Water Detection: Water absorbs infrared light, making it easy to distinguish from land, map floodplains, or even analyze water quality.
• Burn Scars: SWIR bands are excellent at piercing through smoke and identifying recently burned areas after a wildfire.
• Material Identification: Different minerals and man-made materials have unique spectral "fingerprints," allowing you to identify them from space.

Mulitmodal foundation models like Gemini are trained to understand RGB images and not all of these other possible spectral bands. However, by mapping other parts of the electromagnetic spectrum to the RGB space and providing domain-specific instructions, you can leverage Gemini's understanding of the visual space for zero-shot prompting of multi-spectral images.

In this notebook you'll learn how to leverage these generalist multimodal models for specialized tasks without additional training or fine-tuning.

This notebook is based off of the paper from Google DeepMind and Google Research, Zero-Shot Multi-Spectral Learning: Reimagining a Generalist Multimodal Gemini 2.5 Model for Remote Sensing Applications

Setup

Install SDK

Set up your API key

To run the following cell, your API key must be stored it in a Colab Secret named GEMINI_API_KEY. If you don't already have an API key, or you're not sure how to create a Colab Secret, see the Authentication quickstart for an example.

For this notebook, use Gemini 2.5 Pro.

Remote sensing with RGB images

First, test Gemini's abilitiy to interpret satellite data with an RGB only image.

Here is an image of a forest taken with the satellite Sentinel 2.

Next, prompt Gemini to classify this image.

Gemini incorrectly predicts that this image is a SeaLake, reasoning about the blue/green areas. The model is missing the critical multi-spectral data needed to make an accurate prediction.

Remote sensing with multi-spectral images

To perform zero shot multi-spectral remote sensing with Gemini you need to:

1. Transform multi-spectral inputs to psudo image space.
2. Create a prompt which scientifically describes what the new inputs represent.

To see this process in action, you'll add five additional images to the prompt. These images include information about Sentinel 2's other spectral bands.

Download images

Download the images from a public Cloud Storage bucket for viewing.

image_0 is the RGB image from the previous section.

Images 1-5 represent combinations of different spectral bands. You can see that these band combinations reveal new features that were not visible in the RGB image.

Create multi-spectral prompt

In addition to the multi-spectral images, you'll provide Gemini with domain specific information about how each of these new images was created.

Upload the images with the Files API so they can be processed by Gemini.

The model responds with (2), which corresponded to "Forest" in our prompt.

Equipped with the multi-spectral information, Gemini correctly classifies the image.

Creating multi-spectral composite images

In the previous section you saw that adding multi-spectral data to your prompt can help Gemini to correctly classify satellite images.

In this section you'll learn how to create these multi-spectral images, and how you can create your own images based on the bands relevant to your domain, in two steps:

1. Select three spectral bands that are important for your specific problem.

2. Scale the data from each band to a standard 0-255 integer range and assign them to Red, Green and Blue channels of a new image.

First, download the multi-spectral data for the forest image.

This forest satellite image is represented as a numpy array with shape (64,64,13), meaning there are 13 different spectral bands captured by the satellite. R, G, and B are 3 of these 13 bands. Each band is represented by a 64x64 matrix.

If you wanted to recreate the RGB image you plotted earlier, you would stack the RGB channels and end up with an array of size 64x64x3.

Example multi-spectral image: NDVI

You are going to recreate image_2 which represents the Normalized Difference Vegetation Index (NDVI).

NDVI is a commonly used metric in remote sensing to quantify the amount, density, and health of vegetation in a given area. It works by measuring the unique way plants interact with specific wavelengths of light.

To calculate NDVI, you use the Red Light (Red) and Near-Infrared Light (NIR) bands. The formula is:

(nir_band - red_band) / (nir_band + red_band)

To do this calculation, you need to know the mapping of band names to indices in the numpy array. Then you can extract the Red and NIR bands. To see a full description of Sentinel band names and attributes refer to the Sentinel 2 Earth Engine docs.

You can see that the Red and NIR bands are shape (64x64)

The function below takes in the Red and NIR bands and calculates NDVI.

Let's walk through the steps of this function.

First, perform the NDVI calculation

ndvi_image = (nir_band - red_band) / (nir_band + red_band)

The ndvi_image at this point is a grayscale-like array where each pixel value is a floating-point number representing the NDVI score. While it has numerical meaning, it can't be viewed as a standard color image yet.

To make this information understandable by Gemini, you need to need to scale the data to a standard 0-255 integer range and assign them to Red, Green and Blue channels of a new image. This is known as color mapping.

ndvi_image = plt.get_cmap('RdYlGn')(ndvi_image)

The color mapping used here defines a spectrum where one end (Red) corresponds to the lowest values (e.g., −1.0), and the other end (Green) corresponds to the highest values (e.g., +1.0). When the colormap function is called with the ndvi_image array (...('RdYlGn')(ndvi_image)), it performs an element-wise lookup and conversion.

• Lowest NDVI scores (water, rock, or unhealthy plants) are assigned a Red or deep orange color.

• Mid-range scores are assigned Yellow.

• Highest NDVI scores (dense, healthy vegetation) are assigned a Green color.

In the next cell, you'll plot the colormapped image and you'll see it's mostly green (that makes sense since it's an image of a forest!)

Lastly, to prepare the image for display or saving in standard image formats (like PNG or JPEG), the floating-point values (0.0 to 1.0) are converted to the 0 to 255 integer range.

ndvi_image = np.uint8(ndvi_image * 255)

Now you can run the NDVI function to create the new composite image

By following this process of selecting relevant bands, and then color mapping them, you can inject multi-spectral information into your prompt and use Gemini for zero shot remote sensing.

What's next

To learn more about how to use Gemini for remote sensing, check out the original paper.