Notebooks
O
OpenAI Tools Evaluation

Tools Evaluation

chatgptopenaigpt-4use-casesevaluationexamplesopenai-apiopenai-cookbook
alph-notebooks/openai-cookbook / tools-evaluation.ipynb

Tool Evaluation with OpenAI Evals

This cookbook shows how to measure and improve a model’s ability to extract structured information from source code with tool evaluation. In this case, the set of symbols (functions, classes, methods, and variables) defined in Python files.

Setup

Install the latest openai Python package ≥ 1.14.0 and set your OPENAI_API_KEY environment variable. If you also want to evaluate an assistant with tools, enable the Assistants v2 beta in your account.

pip install --upgrade openai
export OPENAI_API_KEY=sk‑...

Below we import the SDK, create a client, and define a helper that builds a small dataset from files inside the openai package itself.

[2]

[notice] A new release of pip is available: 24.0 -> 25.1.1
[notice] To update, run: pip install --upgrade pip
Note: you may need to restart the kernel to use updated packages.

Dataset factory & grading rubric

  • get_dataset builds a small in-memory dataset by reading several SDK files.
  • structured_output_grader defines a detailed evaluation rubric.
  • sampled.output_tools[0].function.arguments.symbols specifies the extracted symbols from the code file based on the tool invocation.
  • client.evals.create(...) registers the eval with the platform.
[ ]

Evals Creation

Here we create an eval that will be used to evaluate the quality of extracted information from code files.

[ ]

Kick off model runs

Here we launch two runs against the same eval: one that calls the Completions endpoint, and one that calls the Responses endpoint.

[ ]

Utility Poller

We create a utility poller that will be used to poll for the results of the eval runs.

[ ]
[ ]

Inspecting results

For both completions and responses, we print the symbols dictionary that the model returned. You can diff this against the reference answer or compute precision / recall.

[ ]

Visualize Evals Dashboard

You can navigate to the Evals Dashboard in order to visualize the data.

evals_tool_dashboard

You can also take a look at the explanation of the failed results in the Evals Dashboard after the run is complete as shown in the image below.

evals_tool_failed

This notebook demonstrated how to use OpenAI Evals to assess and improve a model’s ability to extract structured information from Python code using tool calls.

OpenAI Evals provides a robust, reproducible framework for evaluating LLMs on structured extraction tasks. By combining clear tool schemas, rigorous grading rubrics, and well-structured datasets, you can measure and improve overall performance.

For more details, see the OpenAI Evals documentation.