Notebooks
G
Google Gemini Animated Story Video Generation Gemini

Animated Story Video Generation Gemini

gemini-cookbookgemini-apiexamplesgemini
alph-notebooks/gemini-cookbook / Animated_Story_Video_Generation_gemini.ipynb
Copyright 2025 Google LLC.
[ ]

Gemini API: Animated Story Video Generation

This Colab Notebook demonstrates how to generate an animated story video by:

  1. Generating a story sequence using structured Google Gemini API (for character consistency).
  2. Generating images for each scene using Google’s Imagen API.
  3. Synthesizing narration audio using Gemini Live.
  4. Creating short video clips (image + audio overlay) for each scene.
  5. Combining all clips into one final video.
  6. Cleaning up temporary files after processing.
[ ]
[ ]

Set up your API key

To run the following cell, your API key must be stored it in a Colab Secret named GOOGLE_API_KEY. If you don't already have an API key, or you're not sure how to create a Colab Secret, see Authentication for an example

[ ]

Animated Story Video Generation Steps

1. Story Generation:

  • Google Gemini API is used to generate a sequence of story scenes. Each scene includes an image prompt, audio narration text, and character descriptions. However, to ensure consistency in storytelling and character representation, Gemini is constrained to respond with JSON, a structured data format.
  • User Input: The user provides a theme and the desired number of scenes for their story. This information guides the story generation process.

2. Image Generation:

  • Google Imagen API: Based on the image prompts generated in the previous step, Imagen generates images for each scene, bringing the story to life visually. This ensures visually appealing animations with a consistent art style.

3. Audio Generation:

  • Google Gemini Live API: The narration or dialogue for each scene, generated in the story sequence, is converted to audio using the Gemini Live API. This adds a voiceover to the animated video, enhancing storytelling.

4. Video Composition:

  • MoviePy: This open-source Python library is used to combine the generated images and audio to create short video clips for each scene. It is also used to stitch these clips together into a final animated video.

5. Final Output:

  • The notebook produces an animated video in MP4 format, bringing together all the elements generated in previous steps.

Installation and Setup Commands

[ ]

Key Library Explanations:

MoviePy: A Python library for video editing, enabling tasks such as:

  • Creating video clips from images (ImageClip).
  • Adding audio to videos (AudioFileClip).
  • Combining multiple clips (CompositeVideoClip, concatenate_videoclips).
  • Exporting final videos with proper encoding.

Wave: A built-in Python library for handling WAV format audio files. It is used in this notebook to:

  • Process the audio stream received from Google's API.
  • Save the audio stream as WAV files.

PIL (Python Imaging Library): A fundamental library for image processing in Python. It's used here to:

  • Open and manipulate images generated by Google's Imagen API.
  • Save images in various formats.

Nest_asyncio: A library that enables the use of asynchronous code (using async and await) within Jupyter notebooks. This is crucial for:

  • Real-time audio generation using Google's Live API, which streams audio data.
[ ]

Google Generative Models Setup

v1alpha because you are using the live api to get audio output

[ ]

SECTION 1: Story Generation

This code uses Gemini's structured output capabilities to generate story sequences.

use structured output (JSON schema) here to:

  1. Ensure consistent formatting across all generated scenes
  2. Make it easier to process the output programmatically
  3. Force the model to maintain specific fields needed for video generation
  4. Prevent inconsistent responses

Each scene includes:

  • image_prompt: short scene description
  • audio_text: dialogue or narration text
  • character_description: detailed character/background hints

Furthermore, the code uses TypedDict for type safety and clear documentation of the expected structure, making it easier to maintain and debug the application.

[ ]

Example Usage: Define a Theme and Generate Scenes Write your own theme and number of scenes to play and experiment with it.

[ ]
theme
num_scenes
3

Generated Story Segments:
[
  {
    "audio_text": "Whiskers the cat pounces playfully, startling Buster the dog in the sunny garden.",
    "character_description": "Whiskers: A playful ginger cat with big green eyes. Art style: Simple, rounded shapes, bright colors, kid-friendly animation style. Buster: A golden retriever puppy, floppy ears, big paws. Art style: Same as Whiskers.",
    "image_prompt": "Kids animation style: Whiskers jumps at Buster in a flower filled garden with a blue fence."
  },
  {
    "audio_text": "Buster wags his tail and chases Whiskers around a colorful mushroom house.",
    "character_description": "Whiskers: A playful ginger cat with big green eyes. Art style: Simple, rounded shapes, bright colors, kid-friendly animation style. Buster: A golden retriever puppy, floppy ears, big paws. Art style: Same as Whiskers.",
    "image_prompt": "Kids animation style: Buster chases Whiskers around a red mushroom house with white spots."
  },
  {
    "audio_text": "Together, Whiskers and Buster slide down a rainbow into a pool filled with toys.",
    "character_description": "Whiskers: A playful ginger cat with big green eyes. Art style: Simple, rounded shapes, bright colors, kid-friendly animation style. Buster: A golden retriever puppy, floppy ears, big paws. Art style: Same as Whiskers.",
    "image_prompt": "Kids animation style: Whiskers and Buster slide down a rainbow into a toy-filled pool."
  }
]

SECTION 2: Image and Audio Generation

[ ]
[ ]
[ ]
Processing scene 0:
Image Prompt: Kids animation style: Whiskers jumps at Buster in a flower filled garden with a blue fence.
Audio Text: Whiskers the cat pounces playfully, startling Buster the dog in the sunny garden.
Character Description: Whiskers: A playful ginger cat with big green eyes. Art style: Simple, rounded shapes, bright colors, kid-friendly animation style. Buster: A golden retriever puppy, floppy ears, big paws. Art style: Same as Whiskers.
--------------------------------
Output
Processing scene 1:
Image Prompt: Kids animation style: Buster chases Whiskers around a red mushroom house with white spots.
Audio Text: Buster wags his tail and chases Whiskers around a colorful mushroom house.
Character Description: Whiskers: A playful ginger cat with big green eyes. Art style: Simple, rounded shapes, bright colors, kid-friendly animation style. Buster: A golden retriever puppy, floppy ears, big paws. Art style: Same as Whiskers.
--------------------------------
Output
Processing scene 2:
Image Prompt: Kids animation style: Whiskers and Buster slide down a rainbow into a toy-filled pool.
Audio Text: Together, Whiskers and Buster slide down a rainbow into a pool filled with toys.
Character Description: Whiskers: A playful ginger cat with big green eyes. Art style: Simple, rounded shapes, bright colors, kid-friendly animation style. Buster: A golden retriever puppy, floppy ears, big paws. Art style: Same as Whiskers.
--------------------------------
Output

SECTION 3: Final Video Assembly and Cleanup

[ ]
Writing final video to 1740512475_output_video.mp4
Moviepy - Building video 1740512475_output_video.mp4.
MoviePy - Writing audio in 1740512475_output_videoTEMP_MPY_wvf_snd.mp3

MoviePy - Done.
Moviepy - Writing video 1740512475_output_video.mp4


Moviepy - Done !
Moviepy - video ready 1740512475_output_video.mp4

Final Notes

This notebook is designed to run without modifications. It generates an animated story video using multiple Google APIs and open source libraries. Make sure to have a valid API key and to install all the necessary dependencies before running the notebook in Google Colab.

If you like to view premade videos using this concept, Checkout this website.

Next Steps

Useful API References:

Related Examples

Continue Your Discovery of the Gemini API

Check out other great Gemini features: