Judge Prompt Comparison: Simple/Complex × Reasoning/Non-Reasoning Models

This notebook uses Arize Phoenix llm_classify to evaluate tool-calling predictions on the Berkeley Function Calling Leaderboard (BFCL) dataset with:

• a simple binary prompt (Yes/No), and
• a complex multi-class prompt (correct / partially_correct / incorrect).

We keep it minimal and focused on classification-style LLM-as-a-judge using a non-reasoning model and a reasoning model for the judge.

Note: This notebook was last updated on August 20, 2025.

Install & Imports

Configure Judge Models

We will be using gpt-4o-mini as our nonreasoning & o3 for the Reasoning Model. Make sure to set your OPENAI_API_KEY.

Load BFCL (V3 Exec Splits)

The Berkeley function calling leaderboard is a live leaderboard to evaluate the ability of different LLMs to call functions (also ?referred to as tools). We built this dataset from our learnings to be representative of most users' function calling use-cases, for example, in agents, as a part of enterprise workflows, etc. To this end, our evaluation dataset spans diverse categories, and across multiple languages.

The exec_simple dataset is where the 'single function evaluation contains the simplest but most commonly seen format, where the user supplies a single JSON function document, with one and only one function call being invoked.'

The exec_multiple dataset is where the 'multiple function category contains a user question that only invokes one function call out of 2 to 4 JSON function documentations. The model needs to be capable of selecting the best function to invoke according to user-provided context.'

More information about these datasets can be found here: https://huggingface.co/datasets/gorilla-llm/Berkeley-Function-Calling-Leaderboard

Prepare & Format DataFrames

Pull out different parts of the data like, instruction, functions, ground truth, & predictions.

Modify Benchmark Dataset

The BFCL Dataset does not have any negative examples, i.e. only question, available_tools, and ground_truth are present. In order to accurately benchmark our LLM-as-a-Judge, this code implements a data corruption strategy to generate synthetic evaluation datasets for testing LLM-as-a-Judge systems. It's designed to create realistic "negative examples" (incorrect tool calls) from existing ground truth data, enabling comprehensive evaluation of classification models.

We will be using a small subset of our data for testing purposes. Here we are generating our testing dataset

Define your LLM-as-a-Judge Templates & Rails

Run our Simple Evaluation on both Judge Models

Run our Complex Evaluation on both Judge Models

View Results

We will compare the number of times the models disagree on their evaluation labels as well as how many tokens they used to complete their evaluations.

References

• Phoenix Evals Overview: https://arize.com/docs/phoenix/evaluation/llm-evals
• Using llm_classify (Docs): https://arize.com/docs/phoenix/evaluation/how-to-evals/bring-your-own-evaluator
• BFCL dataset: https://huggingface.co/datasets/gorilla-llm/Berkeley-Function-Calling-Leaderboard