title: Example - Tracing and Evaluation for the OpenAI-Agents SDK description: This guide shows how to evaluate the OpenAI-Agents SDK with Langfuse using online and offline evaluation methods. category: Integrations

Evaluation for the OpenAI-Agents SDK

In this tutorial, we will learn how to monitor the internal steps (traces) of the OpenAI agent SDK and evaluate its performance using Langfuse and Hugging Face Datasets.

This guide covers online and offline evaluation metrics used by teams to bring agents to production fast and reliably. To learn more about evaluation strategies, check out our blog post.

Why AI agent Evaluation is important:

• Debugging issues when tasks fail or produce suboptimal results
• Monitoring costs and performance in real-time
• Improving reliability and safety through continuous feedback

Step 0: Install the Required Libraries

Below we install the openai-agents library (the OpenAI Agents SDK link text), the openinference OpenTelemetry instrumentation, langfuse and the Hugging Face datasets library

Step 1: Instrument Your Agent

In this notebook, we will use Langfuse to trace, debug and evaluate our agent.

Note: If you use another framework (such as LangGraph, LlamaIndex, or CrewAI), you can find documentation on instrumenting them in our integration section.

With the environment variables set, we can now initialize the Langfuse client. get_client() initializes the Langfuse client using the credentials provided in the environment variables.

Now, we initialize the OpenInference OpenAI Agents instrumentation. This third-party instrumentation automatically captures OpenAI Agents operations and exports OpenTelemetry (OTel) spans to Langfuse.

Note: nest_asyncio.apply() is not compatible with uvloop, which is commonly used with FastAPI to manage the event loop. If your application uses uvloop and you require nest_asyncio (e.g., for certain instrumentation or tracing libraries), you'll need to disable uvloop in the affected parts of your codebase and fall back to Python’s standard asyncio event loop.

Step 2: Test Your Instrumentation

Here is a simple Q&A agent. We run it to confirm that the instrumentation is working correctly. If everything is set up correctly, you will see logs/spans in your observability dashboard.

Check your Langfuse Traces Dashboard to confirm that the spans and logs have been recorded.

Example trace in Langfuse:

Link to the trace

Step 3: Observe and Evaluate a More Complex Agent

Now that you have confirmed your instrumentation works, let's try a more complex query so we can see how advanced metrics (token usage, latency, costs, etc.) are tracked.

Trace Structure

Langfuse records a trace that contains spans, which represent each step of your agent’s logic. Here, the trace contains the overall agent run and sub-spans for:

• The tool call (get_weather)
• The LLM calls (Responses API with 'gpt-4o')

You can inspect these to see precisely where time is spent, how many tokens are used, and so on:

Link to the trace

Online Evaluation

Online Evaluation refers to evaluating the agent in a live, real-world environment, i.e. during actual usage in production. This involves monitoring the agent’s performance on real user interactions and analyzing outcomes continuously.

We have written down a guide on different evaluation techniques here.

Common Metrics to Track in Production

1. Costs — The instrumentation captures token usage, which you can transform into approximate costs by assigning a price per token.
2. Latency — Observe the time it takes to complete each step, or the entire run.
3. User Feedback — Users can provide direct feedback (thumbs up/down) to help refine or correct the agent.
4. LLM-as-a-Judge — Use a separate LLM to evaluate your agent’s output in near real-time (e.g., checking for toxicity or correctness).

Below, we show examples of these metrics.

1. Costs

Below is a screenshot showing usage for gpt-4o calls. This is useful to see costly steps and optimize your agent.

Link to the trace

2. Latency

We can also see how long it took to complete each step. In the example below, the entire run took 7 seconds, which you can break down by step. This helps you identify bottlenecks and optimize your agent.

Link to the trace

3. Add Additional Attributes

Langfuse allows you to pass additional attributes to your spans. These can include user_id, tags, session_id, and custom metadata. Enriching traces with these details is important for analysis, debugging, and monitoring of your application's behavior across different users or sessions.

The following code demonstrates how to start a custom span with langfuse.start_as_current_observation and then update the trace associated with this span using propagate_attributes().

→ Learn more about Updating Trace and Span Attributes.

4. User Feedback

If your agent is embedded into a user interface, you can record direct user feedback (like a thumbs-up/down in a chat UI). Below is an example using IPython.display for simple feedback mechanism.

In the code snippet below, when a user sends a chat message, we capture the OpenTelemetry trace ID. If the user likes/dislikes the last answer, we attach a score to the trace.

Note: The following cell is build to work in Google Colab notebooks and can be skipped or adjusted for other environments.

User feedback is then captured in Langfuse:

5. LLM-as-a-Judge

LLM-as-a-Judge is another way to automatically evaluate your agent's output. You can set up a separate LLM call to gauge the output’s correctness, toxicity, style, or any other criteria you care about.

Workflow:

1. You define an Evaluation Template, e.g., "Check if the text is toxic."
2. You set a model that is used as judge-model; in this case gpt-4o-mini.
3. Each time your agent generates output, you pass that output to your "judge" LLM with the template.
4. The judge LLM responds with a rating or label that you log to your observability tool.

Example from Langfuse:

You can see that the answer of this example is judged as "not toxic".

6. Observability Metrics Overview

All of these metrics can be visualized together in dashboards. This enables you to quickly see how your agent performs across many sessions and helps you to track quality metrics over time.

Offline Evaluation

Online evaluation is essential for live feedback, but you also need offline evaluation—systematic checks before or during development. This helps maintain quality and reliability before rolling changes into production.

Dataset Evaluation

In offline evaluation, you typically:

1. Have a benchmark dataset (with prompt and expected output pairs)
2. Run your agent on that dataset
3. Compare outputs to the expected results or use an additional scoring mechanism

Below, we demonstrate this approach with the search-dataset, which contains questions that can be answered via the web search tool and expected answers.

Next, we create a dataset entity in Langfuse to track the runs. Then, we add each item from the dataset to the system.

Running the Agent on the Dataset

We use the experiment runner SDK to run our agent against each dataset item. The experiment runner handles concurrent execution, automatic tracing, and evaluation.

We define a task function that:

1. Starts a Langfuse span for tracing
2. Runs our agent on the prompt
3. Returns the generated answer

You can repeat this process with different:

• Search tools (e.g. different context sized for OpenAI's WebSearchTool)
• Models (gpt-5.2, gpt-5.2-mini, etc.)
• Tools (search vs. no search)

Then compare them side-by-side in Langfuse. In this example, I did run the agent 3 times on the 50 dataset questions. For each run, I used a different setting for the context size of OpenAI's WebSearchTool. You can see that an increased context size also slightly increased the answer correctness from 0.89 to 0.92. The correct_answer score is created by an LLM-as-a-Judge Evaluator that is set up to judge the correctness of the question based on the sample answer given in the dataset.

Resources

Check out the Langfuse docs to learn more ways to evaluate and debug your agent.