Podcast Notes, Summarization, and Translation using Phi‑4 Multimodal LLM with NVIDIA NIM Microservices

This notebook demonstrates a complete workflow using the Phi‑4 Multimodal LLM model. Below are some key model details (from the internal model card):

• Parameters: 5.6B
• Inputs: Text, Image, Audio
• Context Length: 128K tokens
• Training Data: 5T text tokens, 2.3M speech hours, 1.1T image-text tokens
• Supported Languages: Multilingual text and audio (e.g. English, Chinese, German, French, etc.)

The Phi-4 LLM will be accelerated with NVIDIA NIM Microservices in this tutorial. NVIDIA NIM Microservices that is a set of easy-to-use inference microservices for accelerating the deployment of foundation models on any cloud or data center and helping to keep your data secure.

We will be using preview NIM microservice API for Phi-4 through the NVIDIA API Catalog

This notebook covers:

1. A podcast notes and summarization use-case (with long-audio chunking).
2. Translation of the transcript and summary into another language.

Ensure that you have the required dependencies (e.g. pydub, requests, Pillow) installed.

Utility Functions for Audio Processing

In this section, we define helper functions to process audio files, such as converting an audio segment to a base64-encoded string and splitting long audio into manageable chunks for transcription.

Function Overview

• audio_to_base64: Converts audio segments to base64-encoded strings for API transmission
• generate_notes_chunk: Processes individual audio chunks and generates transcription
• generate_detailed_notes: Handles long audio files by splitting them into manageable chunks
• refine_transcription_to_notes: Transforms raw transcription into well-formatted notes
• summarize_notes: Creates a concise summary from the detailed notes
• translate_text: Translates content to other languages while preserving formatting
• save_text_to_file: Exports results to text files

Workflow for detailed notes generation

Translation

The following cell translates the combined notes and summary into another language (e.g., Spanish).

Troubleshooting

If you encounter issues with the API, here are some common problems and solutions:

1. Payload Too Large: Use the optimization functions with more aggressive parameters
2. Authentication Errors: Verify your API key is correct and has the necessary permissions
3. Model Not Available: Check that you're using a valid model name for your account tier
4. Format Issues: Ensure your media files are in supported formats
5. Rate Limiting: If processing many chunks, add delays between API calls to avoid rate limits