Observability for Portkey LLM Gateway with Langfuse

This guide shows you how to integrate Portkey's AI gateway with Langfuse. Portkey's API endpoints are fully compatible with the OpenAI SDK, allowing you to trace and monitor your AI applications seamlessly.

What is Portkey? Portkey is an AI gateway that provides a unified interface to interact with 250+ AI models, offering advanced tools for control, visibility, and security in your Generative AI apps.

What is Langfuse? Langfuse is an open source LLM engineering platform that helps teams trace LLM calls, monitor performance, and debug issues in their AI applications.

Step 1: Install Dependencies

Step 2: Set Up Environment Variables

Next, set up your Langfuse API keys. You can get these keys by signing up for a free Langfuse Cloud account or by self-hosting Langfuse. These environment variables are essential for the Langfuse client to authenticate and send data to your Langfuse project.

Step 3: Use Langfuse OpenAI Drop-in Replacement

Next, you can use Langfuse’s OpenAI-compatible client (from langfuse.openai import OpenAI) to trace all requests sent through the Portkey gateway. For detailed setup instructions on the LLM gateway and virtual LLM keys, refer to the Portkey documentation.

Step 4: Run an Example

Step 5: See Traces in Langfuse

After running the example, log in to Langfuse to view the detailed traces, including:

• Request parameters
• Response content
• Token usage and latency metrics
• LLM model information through Portkey gateway

Public example trace link in Langfuse