Evaluating Multi-Turn Conversations (Simulation)

AI applications with conversational interfaces, such as chatbots, engage in multiple interactions with a user, which are also known as conversation turns. There are multiple ways of evaluating the performance of these apps in a structured way, like N+1 Evaluations.

In this cookbook we'll cover how to use agents that simulate a user having a conversation with your chatbot, and measure the output of these conversations. We'll cover:

• Creating structured datasets of features to test, scenarios, personas...
• Using OpenEvals to simulate a specific use-case.
• Running the simulations against the production application
• Evaluating the output of the simulation with an LLM-as-a-Judge.

Not using Langfuse yet? Get started by capturing LLM events.

Setup

In this example, we’ll build a cooking assistant chatbot and then simulate conversations against it.

Step 1 - Create a chat app and generate traces in Langfuse

First you need to install Langfuse via pip and then set the environment variables.

Now we’ll create a simple cooking assistant chatbot that uses OpenAI as the LLM and trace it with Langfuse.

Now if you run a simple question, you should see traces in Langfuse:

Step 2 - Create a structured Dataset of user scenarios

Define relevant dimensions for your use case, such as:

• Features: Specific functionalities of your AI product.
• Scenarios: Situations or problems the AI may encounter and needs to handle.
• Personas: Representative user profiles with distinct characteristics and needs.

For our cooking assistant chatbot, we'll create pairs of Scenarios and Personas, as follows:

You can add more columns for your specific use-case, like adding data to test against your system. For example, if you're running a customer service chatbot for a logistics company, you might want to simulate the scenario of providing a non-existent order tracking ID to see how your chatbot handles the scenario.

Once you've defined your dataset, create it in Langfuse's Dataset, this way you can assign performance scores to measure how the system improves on the dataset with every new update:

Step 3 - Prepare your application to run against the simulator

The OpenEvals library requires a specific input and output format. It also needs conversation history management, which is why we'll store instances of our chatbot together with a thread_id provided by OpenEvals.

Step 4 - Run an experiment

We'll use Langfuse's Dataset Experiments SDK to fetch the scenario/persona pairs we've created and trace and store the output of the experiment so we can then evaluate.

Once you run the experiment, you should see the results in your Dataset runs tab:

Step 5 - Run evaluations against your Dataset runs

To evaluate the output, you can create LLM-as-a-Judge that runs against your Dataset Runs and scores the output, giving an individual score and reason for the specific scenario/persona pair and an overall average score for the entire dataset run.

In this case, I create a simple "Conciseness" evaluator from the pre-made library of evaluators, but you could create one (or multiple of them!) that evaluate based on your use-case and needs. In the following image, you can see the overall score assigned to multiple Dataset runs, and how over time that score improves given changes to the prompt.

Conclusion

This notebook demonstrated a systematic approach to simulate and evaluate multi-turn conversations, specifically addressing use-cases and scenarios that are relevant to the chatbot.

You can read more about effective ways to create datasets for experiments here.