Model Comparison for an Email Text Extraction Service

Imagine you're deploying a service that condenses emails into concise summaries. One challenge of using LLMs for summarization is that even the best models can miscategorize key details, or miss those details entirely.

In this tutorial, you will construct a dataset and run experiments to engineer a prompt template that produces accurately summarizes your emails. You will:

• Upload a dataset of examples containing emails to Phoenix
• Define an experiment task that extracts and formats the key details from those emails
• Devise an evaluator measuring Jaro-Winkler Similarity
• Run experiments to iterate on your prompt template and to compare the summaries produced by different LLMs

⚠️ This tutorial requires and OpenAI API key.

Let's get started!

Set Up OpenAI API Key

Import Modules

Launch Phoenix

First we have to set up our instance of Phoenix and our instrumentors to capture traces from our agent. We'll use both our Langchain and OpenAI auto instrumentors because while our task uses Langchain, our evaluation function will call OpenAI directly.

Next, instantiate a Phoenix client. This acts as the main entry point for interacting with the Phoenix API and can be installed independently of Phoenix itself.

Notice that we're using the AsyncClient class, allowing us to run experiments asynchronously. If you need to use Phoenix using synchronous code, you can use the Client class instead.

Instrument LangChain and OpenAI

Experiments in Phoenix

Experiments in Phoenix are made up of 3 elements: a dataset, a task, and an evaluator. The dataset is a collection of the inputs and expected outputs that we'll use to evaluate. The task is an operation that should be performed on each input. Finally, the evaluator compares the result against an expected output.

For this example, here's what each looks like:

• Dataset - a dataframe of emails to analyze, and the expected output for our agent
• Task - a langchain agent that extracts key info from our input emails. The result of this task will then be compared against the expected output
• Eval - Jaro-Winkler distance calculation on the task's output and expected output

Download JSON Data

We've prepared some example emails and actual responses that we can use to evaluate our two models. Let's download those and save them to a temporary file.

Upload Dataset to Phoenix

Next, we'll upload our dataset to Phoenix. Once this is present in Phoenix, we can run multiple experiments with different models on this one dataset, and compare their performance.

Set Up LangChain

Now we'll set up our Langchain agent. This is a straightforward agent that makes a call to our specified model and formats the response as JSON.

Define Task Function

Next, we need to define a Task for our experiment to use.

Check that the task is working by running it on at least one Example

Run Experiment

Now we're ready to run our experiment. We'll specify our dataset and task, and generate responses for us to evaluate in the next step.

Define Evaluator

Finally, we need to define our evaluation function. Here we'll use a Jaro-Winkler similarity function that generates a score for how similar the output and expected text are. Jaro-Winkler similarity is technique for measuring edit distance between two strings.

Evaluate Experiment

Now we have scores on how well GPT-4o does at extracting email facts. This is helpful, but doesn't mean much on its own. Let's compare it against another model.

Re-run with GPT 3.5 Turbo and Compare Results

To compare results with another model, we simply need to redefine our task. Our dataset and evaluator can stay the same.

Compare Results

View your Phoenix UI to compare the models side-by-side. Higher Jaro-Winkler scores indicate better extraction accuracy. GPT-4o typically outperforms GPT-3.5-turbo on structured extraction tasks.

From here you could try out different models or iterate on your prompt, then run the same experiment with a modified Task to compare results.