Notebooks
A
Arize AI
Ai Evals Hw5 Solution

Ai Evals Hw5 Solution

hw5_phoenixagentsllmsLlamaIndexarize-phoenixopenaitutorialsevalsllmopsai-monitoringai_evals_courseaiengineeringprompt-engineeringdatasetsllm-evalai-observabilityllm-evaluationsmolagentsanthropiclangchain

Open In Colab

Homework 5 - Evals for Failure Analysis with Phoenix

phoenix logo
Docs | GitHub | Community

Launch Phoenix

First, let's set up Phoenix on our local machine. You should run these commands within your terminal in your chosen environment.

(If you have already done this in a previous HW assignment, you are good to go.)

Install Phoenix

pip install arize-phoenix

Boot up Phoenix on localhost

phoenix serve

Run phoenix serve in your terminal to boot up Phoenix locally.

[ ]

Pipeline States

Below are the states of the recipe agent pipeline we are simulating, in order.

ParseRequest - LLM interprets and analyzes the user's query to understand what they're asking for

PlanToolCalls - LLM decides which tools to invoke and in what order based on the parsed request

GenRecipeArgs - LLM constructs JSON arguments for the recipe database search based on customer profile

GetRecipes - Executes the recipe-search tool to find relevant recipes matching the criteria

GenWebArgs - LLM constructs JSON arguments for web search to find additional cooking tips/information

GetWebInfo - Executes the web-search tool to retrieve supplementary cooking information

ComposeResponse - LLM drafts the final answer combining recipes and web information

DeliverResponse - Agent sends the composed response to the user

[ ]

Generate Phoenix Traces

Here we are making 100 requests to the recipe bot and then collecting traces for those requests in Phoenix.

You can look at the generate_traces_phoenix.py file for more details on how this is implemented.

[ ]

Evals

Here we run evals. We have 7 different evaluators with their own prompts, each designed to evaluate one of the 7 states of the recipe bot application. You can see the evaluator prompts in evaluators directory.

We use the Phoenix method SpanQuery() to load spans for the 7 different states.

We use the Phoenix method llm_generate to run the evals. llm_generate has built in concurrency that makes running your llm calls for your evals much quicker.

Finally we use log_annotations_dataframe to log our evals back to our spans in Phoenix.

[ ]

Attach Evals up to Root Trace

This code propagates our evals back to the root trace, so each root trace contains information on whether the 7 individual states within passed or failed.

[ ]
[ ]

Analysis of System Reliability and Failure Distribution

  1. Overall Reliability Out of approximately 100 traces, about one-third (32–33) completed successfully without any failures. The remaining two-thirds experienced at least one failure along the pipeline. This indicates that while the system is not universally fragile, it is also not yet stable enough for production use.

  2. Where Failures Occur Failures are not evenly distributed across the pipeline — instead, they cluster around a few key stages:

  • GetWebInfo (33 failures): The largest single source of errors, suggesting issues with external dependencies such as API reliability, query quality, or rate limits.
  • ParseRequest (18) and PlanToolCalls (17): Together these account for 35 failures, making input interpretation and tool planning the second biggest weakness.
  • GenRecipeArgs, GetRecipes, GenWebArgs (8–11 each): Moderate failure levels, likely reflecting occasional misalignments when constructing arguments or retrieving results.
  • ComposeResponse (1): Very low failure rate, showing the model can reliably synthesize answers when given valid inputs. Errors here are likely side effects of upstream problems.
  1. Key Insights The system demonstrates a bimodal pattern:
  • Either a trace runs flawlessly end-to-end,
  • Or it fails in predictable spots (primarily GetWebInfo, ParseRequest, or PlanToolCalls).

This clustering of root causes means that targeted improvements in just a few stages could significantly increase the overall success rate and bring the pipeline closer to production stability.

Completing the Feedback Loop

Evals are powerful because they don't only tell us where stuff went wrong, but WHY it did. This information is stored in the eval explanations.

You can use the eval explanations to generate feedback for improvement strategies for your application, using LLMs!

feedback loop

[ ]
[ ]
[ ]

All done!

We learned how to:

  • Hyper focus our evals
  • Build effective eval prompts
  • Judge our application based on our evals
  • Use eval explanations to generate targeted feedback