Notebooks
A
Arize AI
Evaluate Code Functionality Classifications

Evaluate Code Functionality Classifications

agentsllmsLlamaIndexarize-phoenixopenaitutorialsevalsllmopsai-monitoringaiengineeringprompt-engineeringdatasetsllm-evalai-observabilityllm-evaluationsmolagentsanthropiclangchain

Open In Colab

phoenix logo
Docs | GitHub | Community

Code Functionality Evals

This tests whether code is written correctly, without bugs, accomplishes the functionality you want, does not have syntax errors.

The purpose of this notebook is:

  • to evaluate the performance of code fuctionality Eval
  • to provide an experimental framework for users to iterate and improve on the default classification template.

Install Dependencies and Import Libraries

[ ]

ℹ️ To enable async request submission in notebook environments like Jupyter or Google Colab, optionally use nest_asyncio. nest_asyncio globally patches asyncio to enable event loops to be re-entrant. This is not required for non-notebook environments.

Without nest_asyncio, eval submission can be much slower, depending on your organization's rate limits. Speed increases of about 5x are typical.

[ ]
[ ]

Download Benchmark Dataset

[ ]

Display Code Functionality Classification Template

View the default template used to code functionality. You can tweak this template and evaluate its performance relative to the default.

[ ]

The template variables are:

  • coding_instruction: What is the code supposed to do as an instruction
  • code: The code to evaluate

Configure the LLM

Configure your OpenAI API key.

[ ]

Benchmark Dataset Sample

Sample size determines run time Recommend iterating small: 100 samples Then increasing to large test set

LLM Evals: Code Functionality Classifications GPT-4

Run Code Functionality against a subset of the data. Instantiate the LLM and set parameters.

[ ]

Run Code Func Classifications

Run code functionality classifications against a subset of the data.

[ ]

Evaluate Classifications

Evaluate the predictions against human-labeled ground-truth code functionality labels.

[ ]

Classifications with explanations

When evaluating a dataset for code functionality, it can be useful to know why the LLM classified a document as relevant or irrelevant. The following code block runs llm_classify with explanations turned on so that we can inspect why the LLM made the classification it did. There is speed tradeoff since more tokens is being generated but it can be highly informative when troubleshooting.

[ ]
[ ]
[ ]

LLM Evals: code functionality Classifications GPT-3.5 Turbo

Run Code functionality against a subset of the data using GPT-3.5. GPT-3.5 can significantly speed up the classification process. However there are tradeoffs as we will see below.

[ ]
[ ]
[ ]

Preview: Running with GPT-4 Turbo

[ ]
[ ]