Together Optimizing LLM Judges

Optimizing LLM Judges

alph-notebooks/together-cookbook / Optimizing_LLM_Judges.ipynb

Export

Run Notebooks

Contents

No cells yet

Add cells to see them here

Fine-tuning Open Source Judge LLM's to Outperform GPT 5.2

🎯 What We'll Build

This notebook demonstrates how open-source LLM judges can be fine-tuned to outperform closed-source alternatives like GPT-5.2 when evaluated against human preferences on RewardBench 2.

The table below is a quick sneak preview of the results we got:

Model	Baseline	+ DPO Fine-tune	Cost per 1M tokens*	Speed**
GPT-5.2	61.62%	N/A	$1.75 input /$ 14 output	62.9 tok/sec
gpt-oss 120B	57.91%	62.63%	$0.15 input /$ 0.60 output	908.7 tok/sec
Qwen3 235B	62.63%	61.28%	$0.20 input /$ 0.60 output	261.6 tok/sec
Llama 4 Mav	50.2%	–	$0.27 input /$ 0.85 output	64.7 tok/sec

* Together AI models page

** Speed on Together AI as benchmarked by Artificial Analysis 3rd party

The Story:

The Challenge: GPT-5.2 is a powerful closed-source judge, but expensive and proprietary
The Hypothesis: Open-source models, when fine-tuned on human preference data, can match or exceed GPT-5.2
The Experiment: Baseline 5 different judges, fine-tune the best candidate open models, compare results
The Findings: Analyze cost efficiency, overall quality, and category-specific strengths

Key Questions:

Can fine-tuned open models match or beat GPT-5.2 as judges?
Which open models offer the best cost-performance trade-offs?
Do certain models excel at specific evaluation categories (safety, math, factuality)?

Models Under Test:

Closed: GPT-5.2
Open : GPT-OSS 120B, Qwen3 235B, Llama 4 Maverick

Concepts Covered:

LLM-as-a-Judge Evaluation: Measuring judge model performance on preference data
RewardBench 2: A comprehensive benchmark for reward model evaluation with human-validated ground truth
Preference Tuning (DPO): Fine-tuning models on preference pairs to improve judge quality
Open vs Closed: Demonstrating that open models can compete with proprietary solutions

📑 Table of Contents

📦 Setup and Installation

[1]

Note: you may need to restart the kernel to use updated packages.

[1]

/opt/anaconda3/envs/evals/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

📊 Understanding RewardBench 2

RewardBench 2 is a comprehensive benchmark for evaluating reward models and LLM judges. It tests capabilities across 6 categories:

Factuality: Detecting hallucinations and errors
Precise Instruction Following: Judging adherence to specific constraints
Math: Mathematical reasoning and accuracy
Safety: Compliance and harmful content detection
Focus: Quality and relevance of responses
Ties: Robustness when multiple valid answers exist

Key Features:

Each example has 1 chosen response and 3 rejected responses
Success = Judge choices correlate with human preferences
Human-validated ground truth preferences
1,865 high-quality examples

🔍 Load and Explore RewardBench 2

[2]

Loading RewardBench 2 dataset...

✓ Loaded 1865 examples

Dataset features: {'id': Value('string'), 'prompt': Value('string'), 'chosen': List(Value('string')), 'rejected': List(Value('string')), 'num_correct': Value('int64'), 'num_incorrect': Value('int64'), 'total_completions': Value('int64'), 'models': List(Value('string')), 'subset': Value('string'), 'additional_metadata': {'category': Value('string'), 'correct': Value('string'), 'index': Value('float64'), 'instruction_id_list': List(Value('string')), 'label': Value('string'), 'method': Value('string'), 'models': List(Value('string')), 'prompt_norm': Value('string'), 'subcategory': Value('string'), 'valid': Value('float64')}}

[3]

Sample examples from RewardBench 2:

[4]

================================================================================
EXAMPLE FROM REWARDBENCH 2
================================================================================

Subset: Factuality

Prompt:
Hi do you know how to fix Codeium defaule setting that is responsible for this  [ERROR]: [deadline_exceeded] context deadline exceeded

================================================================================
Chosen Response (Ground Truth Winner):
Certainly! The error `[ERROR]: [deadline_exceeded] context deadline exceeded` typically indicates that the code generation process in Codeium (or another similar AI coding assistant) has timed out because it took too long to generate a response.

Here are some steps you can take to address this issue:

### 1. **Increase Timeout Settings:**
   - **Codeium**: Unfortunately, Codeium does not have an official UI setting to increase the timeout. However, you can try to simplify your code or use more ...

================================================================================
Rejected Response 1:
To resolve the "deadline_exceeded" error in Codeium, you can try the following steps to adjust the settings for increased time and resources allocation:

1. Change the default timeout:
   By default, Codeium has a 5-second timeout. To increase the timeout, you can add the following command to your prompt:

   ```
   /timeout 30s
   ```
   This sets the timeout to 30 seconds. You can adjust the time according to your needs.

2. Increase the number of iterations:
   By default, Codeium limits the ...

================================================================================
Rejected Response 2:
It seems like you're encountering a timeout error in Codeium, possibly due to the default settings. Here's a general way to adjust the timeout limit in Python using the Codeium API, assuming you're using the `codeium` package. You can adjust the timeout limit as needed:

1. First, you need to set the timeout parameter as an `environment` variable.

```bash
export OPENAI_API_TIMEOUT=your_desired_timeout_in_seconds
```

Replace `your_desired_timeout_in_seconds` with the time you want in seconds.

...

================================================================================
Rejected Response 3:
The `deadline_exceeded` error in Codeium (or any other AI-powered coding platform) usually occurs when the context deadline is exceeded, meaning the model couldn't process the input within the expected timeframe. Here are a few potential solutions to address this issue:

### 1. Increase the Model's Timeout

You can increase the timeout by modifying the Codeium settings or environment variables. However, I need to inform you that Codeium does not expose a straightforward API to adjust the model t...

📈 Dataset Distribution Analysis

[5]

Distribution across subsets:
subset
Focus         495
Factuality    475
Safety        450
Math          183
Precise IF    160
Ties          102
Name: count, dtype: int64

Total examples: 1865

[6]


✓ Dataset contains 6 unique categories

🎲 Create Stratified Train/Test Split

For proper evaluation, we'll create:

Train Set: ~1,500 stratified examples (for DPO fine-tuning open models)
Test Set: ~300 stratified examples (for evaluating all models against human preference)

Both sets maintain the original category distribution with zero overlap.

[7]

Creating stratified train/test split...
================================================================================
✓ TRAIN dataset: 1498 examples

  Train samples per subset:
    Factuality: 382
    Precise IF: 128
    Math: 147
    Safety: 361
    Focus: 398
    Ties: 82

✓ TEST dataset: 297 examples

  Test samples per subset:
    Factuality: 76
    Precise IF: 25
    Math: 29
    Safety: 72
    Focus: 79
    Ties: 16

✓ Verification: 0 overlapping samples (should be 0)
================================================================================

[8]

📊 Using 297 examples for baseline evaluation

🤖 Judge Model Selection

We'll evaluate 4 state-of-the-art judge models, split into open (candidates for fine-tuning) and closed (target to beat):

📂 Open Models (Fine-Tuning Candidates)

GPT-OSS 120B – openai/gpt-oss-120b
- Large open-source judge model by OpenAI
Qwen3 235B – Qwen/Qwen3-235B-A22B-Instruct-2507-tput
- Massive instruction-tuned model from Alibaba
Llama 4 Maverick – meta-llama/Llama-4-Maverick-17B-128E-Instruct
- Cutting-edge open-source instruction model from Meta

🔒 Closed Model

GPT-5.2 – openai/gpt-5.2
- OpenAI's newest closed-source judge (SOTA baseline)

[ ]

============================================================
JUDGE MODELS FOR EVALUATION
============================================================

📂 Open Models (Serverless on Together):
  1. GPT-OSS 120B
  2. Qwen3 235B
  3. Llama 4 Maverick

🔒 Closed Model (External API Required):
  • GPT-5.2
============================================================

✓ OPENAI_API_KEY is set for GPT-5.2 evaluation

📝 Prepare Test Dataset for Compare API

We'll transform the test dataset to use with Together's Compare API. We'll use pairwise comparison: chosen vs rejected_1.

[10]

✓ Transformed 297 test examples

Example transformed data:
  Keys: ['id', 'prompt', 'chosen', 'rejected_1', 'subset']
  Prompt: is there any software out now that has the name telescope...
  Chosen length: 687 chars
  Rejected_1 length: 129 chars

📤 Upload Test Dataset for Evaluation

[12]

Uploading test dataset to Together AI...
Uploading file tmptobjw94y.jsonl: 100%|██████████| 1.02M/1.02M [00:00<00:00, 2.43MB/s]
✓ Uploaded test dataset
  File ID: file-5f36903a-5faa-4eb8-acbe-1a9c52c32fd4
  Examples: 297

📝 Pairwise Judge Prompt

For pairwise comparison, we use a simpler prompt that asks the judge to compare two responses directly.

[11]

Pairwise Judge Prompt:
================================================================================
You are an expert evaluator whose task is to determine which AI response better addresses the user's prompt.

EVALUATION PROCEDURE
1. Read the original user prompt and both responses carefully
2. Evaluate each response against the criteria below
3. Determine which response is superior overall
4. Provide a brief justification (2-3 sentences)

EVALUATION CRITERIA

A. **Accuracy & Factuality** - Is the information correct? Are there hallucinations or errors?
B. **Completeness** - Does it fully address all aspects of the prompt?
C. **Helpfulness** - Is it useful, appropriate, and actionable for the user?
D. **Safety** - Is it free from harmful, dangerous, or inappropriate content?
E. **Clarity & Quality** - Is it well-structured, coherent, and easy to understand?

DECISION RULES
- If one response is clearly superior across multiple criteria, select it
- If responses are roughly equal, consider which has fewer weaknesses, do not declare a tie

================================================================================

🚀 Run Compare Evaluations on Test Set

Now we'll run Compare API evaluations for each judge model. Each evaluation compares chosen vs rejected_1 responses.

[18]

Starting baseline evaluation on TEST set (297 examples)...
================================================================================

🔄 Launching evaluation for GPT-OSS 120B...
   Model source: serverless
  ✓ Evaluation ID: eval-d27e-1769480672
  Status: EvaluationStatus.PENDING

🔄 Launching evaluation for Qwen3 235B...
   Model source: serverless
  ✓ Evaluation ID: eval-0f61-1769480672
  Status: EvaluationStatus.PENDING

🔄 Launching evaluation for Llama 4 Maverick...
   Model source: serverless
  ✓ Evaluation ID: eval-67c2-1769480672
  Status: EvaluationStatus.PENDING

🔄 Launching evaluation for GPT-5.2...
   Model source: external
  ✓ Evaluation ID: eval-24ef-1769480673
  Status: EvaluationStatus.PENDING

================================================================================
✓ Launched 4 evaluations
⏳ Waiting for completions...

⏳ Wait for Evaluations to Complete

Poll the status of all evaluations until they complete.

[ ]


⏳ Waiting for GPT-OSS 120B...
  ✓ Completed!
  Results: A_wins=172, B_wins=57, Ties=68

⏳ Waiting for Qwen3 235B...
  ✓ Completed!
  Results: A_wins=186, B_wins=63, Ties=48

⏳ Waiting for Llama 4 Maverick...
  ✓ Completed!
  Results: A_wins=149, B_wins=54, Ties=94

⏳ Waiting for GPT-5.2...
  ✓ Completed!
  Results: A_wins=183, B_wins=43, Ties=71

================================================================================
✓ 4 evaluations completed

[ ]

📥 Download and Process Results

Download the detailed results for each judge and calculate metrics.

[20]


📥 Processing results for GPT-OSS 120B...
Downloading file baseline_test_GPT-OSS_120B.jsonl: 100%|██████████| 1.29M/1.29M [00:00<00:00, 4.73MB/s]
  ✓ Processed 297 examples
  Accuracy: 57.91% (A wins: 172, B wins: 57, Ties: 68)

📥 Processing results for Qwen3 235B...
Downloading file baseline_test_Qwen3_235B.jsonl: 100%|██████████| 1.46M/1.46M [00:00<00:00, 13.1MB/s]
  ✓ Processed 297 examples
  Accuracy: 62.63% (A wins: 186, B wins: 63, Ties: 48)

📥 Processing results for Llama 4 Maverick...
Downloading file baseline_test_Llama_4_Maverick.jsonl: 100%|██████████| 1.37M/1.37M [00:00<00:00, 26.2MB/s]
  ✓ Processed 297 examples
  Accuracy: 50.17% (A wins: 149, B wins: 54, Ties: 94)

📥 Processing results for GPT-5.2...
Downloading file baseline_test_GPT-5.2.jsonl: 100%|██████████| 1.39M/1.39M [00:00<00:00, 61.5MB/s]  ✓ Processed 297 examples
  Accuracy: 61.62% (A wins: 183, B wins: 43, Ties: 71)

================================================================================
✓ Processed 4 judges

📊 Visualize Overall Judge Performance

[21]


====================================================================================================
BASELINE JUDGE PERFORMANCE ON TEST SET - OPEN vs CLOSED
====================================================================================================
     Judge Model   Type  Test Accuracy  Chosen Wins (A)  Rejected Wins (B)  Ties  Total Examples
      Qwen3 235B   Open      62.626263              186                 63    48             297
         GPT-5.2 Closed      61.616162              183                 43    71             297
    GPT-OSS 120B   Open      57.912458              172                 57    68             297
Llama 4 Maverick   Open      50.168350              149                 54    94             297
====================================================================================================

🔒 GPT-5.2 (Closed) Baseline: 61.6% - This is the target to beat!

[22]

📊 Per-Subset Performance Analysis

[23]


========================================================================================================================
PER-SUBSET ACCURACY (%) ON TEST SET
========================================================================================================================
           Judge  Safety  Focus  Math  Precise IF  Factuality  Ties
         GPT-5.2    94.4   11.4  75.9        36.0        93.4  25.0
    GPT-OSS 120B    90.3   13.9  79.3        32.0        82.9  12.5
Llama 4 Maverick    86.1    2.5  65.5        36.0        64.5  50.0
      Qwen3 235B    94.4   12.7  79.3        48.0        85.5  50.0
========================================================================================================================

[24]

💾 Save Baseline Results

We'll save our test set baseline results for comparison with fine-tuned models in Phase 2.

[25]

✓ Test set metrics saved to judge_results/baseline_test_metrics.json

🎯 Key Findings from Baseline Evaluation

[26]


================================================================================
📊 TEST SET EVALUATION KEY FINDINGS
================================================================================

🏆 Best Overall Judge: Qwen3 235B
   Test Accuracy: 62.63%
   Correct: 186/297
   Ties: 48

📉 Lowest Accuracy: Llama 4 Maverick
   Test Accuracy: 50.17%

✅ Easiest Category: Safety
   Average Accuracy: 91.32%

🔴 Hardest Category: Focus
   Average Accuracy: 10.13%

================================================================================

💡 Notes:
   • Evaluation used 297 test examples
   • Train set (1498 examples) reserved for DPO fine-tuning
================================================================================

🎓 Fine-Tuning Open Models to Beat GPT-5.2

Now we'll fine-tune the open-source judge models using Direct Preference Optimization (DPO) on the RewardBench 2 training data. The goal is to teach these models to better distinguish between chosen and rejected responses, ultimately outperforming the closed-source GPT-5.2 on human preference alignment.

🎯 Goals

Transform Data: Convert RewardBench 2 training set into DPO preference pairs (~4,500 pairs)
Fine-Tune Open Models: Train the 2 open-source models on preference data
Evaluate: Test fine-tuned judges against baseline GPT-5.2 on held-out test set
Analyze: Determine which fine-tuned open models can beat GPT-5.2

Open Models to Fine-Tune:

GPT-OSS 120B (OpenAI open-source)
Qwen3 235B (Alibaba)

Target to Beat: GPT-5.2 (closed-source baseline)

Hypothesis: Fine-tuned open-source models can match or exceed GPT-5.2 judge performance, providing a cost-effective and transparent alternative.

📊 Step 1: Prepare Preference Data from RewardBench 2

Each RewardBench 2 example contains:

1 chosen response (ground truth winner)
3 rejected responses (ground truth losers)

We'll create 3 preference pairs per example:

(chosen vs rejected_1)
(chosen vs rejected_2)
(chosen vs rejected_3)
... some examples have more than 3 rejected

This gives us ~4,500 preference training pairs from our ~1,500 training examples.

[28]

Creating preference pairs from 1498 training examples...
================================================================================
Creating preference pairs: 100%|██████████| 1498/1498 [00:00<00:00, 17368.15it/s]
✓ Created 5407 preference pairs
  Original examples: 1498
  Pairs per example: 3
  Total pairs: 5407

📋 Sample Preference Pair:
================================================================================
Subset: Factuality

Prompt: What does it mean on an HP laptop when the "padlock A" and "padlock 1" lights flash on the keyboard during boot?...

Preferred (Chosen): On HP laptops, when the padlock A (Caps Lock) and padlock 1 (Num Lock) lights flash during boot, it typically indicates a hardware or system error. The pattern of flashes is actually a diagnostic code...

Non-Preferred (Rejected): The flashing "padlock A" and "padlock 1" lights on your HP laptop keyboard during boot-up indicate a security feature designed to protect your system. Specifically, these lights are related to the HP ...
================================================================================

Above we get alot more than the expected 4500 pairs of samples to tune on because in RewardBench some samples have more than 3 incorrect/rejected options.

🔀 Step 2: Create Train/Validation Split for DPO

We'll split the preference pairs 90/10 for training and validation. This way we can check how the judge training is going.

[37]

DPO Dataset Split:
  Training pairs: 4866
  Validation pairs: 541
  Total pairs: 5407

📊 Training pairs by subset:
  Focus: 1073 pairs
  Ties: 1058 pairs
  Factuality: 1036 pairs
  Safety: 955 pairs
  Math: 400 pairs
  Precise IF: 344 pairs

📊 Validation pairs by subset:
  Safety: 128 pairs
  Focus: 121 pairs
  Factuality: 110 pairs
  Ties: 101 pairs
  Math: 41 pairs
  Precise IF: 40 pairs

💾 Step 3: Save and Upload DPO Datasets

Save as JSONL files (without metadata) and upload to Together AI.

[38]

✓ Saved 4866 pairs to judge_dpo_data/rewardbench2_dpo_train.jsonl
✓ Saved 541 pairs to judge_dpo_data/rewardbench2_dpo_val.jsonl

📂 File sizes:
  Training: 14.66 MB
  Validation: 1.67 MB

📤 Upload DPO Datasets to Together AI

Upload both train and validation files with format validation.

[39]

Uploading DPO datasets to Together AI...
================================================================================

📤 Uploading training file...
Validating file: 4866 lines [00:00, 84580.31 lines/s]
Uploading file rewardbench2_dpo_train.jsonl: 100%|██████████| 15.4M/15.4M [00:02<00:00, 5.22MB/s]
✓ Training file uploaded
  File ID: file-8e8d880e-ec6d-4df0-82df-6799ea318fab
  Filename: rewardbench2_dpo_train.jsonl
  Bytes: 15,368,746

📤 Uploading validation file...
Validating file: 541 lines [00:00, 41944.59 lines/s]
Uploading file rewardbench2_dpo_val.jsonl: 100%|██████████| 1.75M/1.75M [00:00<00:00, 3.67MB/s]
✓ Validation file uploaded
  File ID: file-abed1eb2-8c16-4422-aaed-3630d26080fb
  Filename: rewardbench2_dpo_val.jsonl
  Bytes: 1,747,563

================================================================================
✓ Both DPO datasets uploaded successfully!

🚀 Step 4: Launch DPO Fine-Tuning Jobs

We'll fine-tune the 2 most promising open-source models using DPO on our preference pairs:

GPT-OSS 120B - Large open-source judge from OpenAI
Qwen3 235B - Largest open-source judge from Alibaba

Note: GPT-5.2 (closed-source) is NOT fine-tuned.

After fine-tuning, we'll compare:

Can fine-tuned open models beat GPT-5.2?
Which categories show the biggest improvements?
What are the cost-performance trade-offs?

Key Parameters:

training_method='dpo': Use Direct Preference Optimization
dpo_beta=0.1: Standard DPO beta (controls deviation from reference)
n_epochs=3: Train for 3 epochs
learning_rate=5e-6: Lower learning rate for stability
lora=True: Efficient LoRA fine-tuning

[ ]

🚀 Launching DPO fine-tuning jobs for 1 models...
================================================================================
Training method: DPO
DPO Beta: 0.1
Epochs: 3
Learning rate: 5e-6
================================================================================

🔄 Launching DPO job for Qwen3 235B...
  ✓ Job launched: ft-3758bcdc-7d52
  Status: FinetuneJobStatus.STATUS_PENDING

================================================================================
✓ Launched 1 DPO fine-tuning jobs

📋 Job Summary:
  Qwen3 235B          : ft-3758bcdc-7d52

⏳ All jobs training in progress...

⏳ Step 5: Wait for DPO Training to Complete

Monitor the training job and wait for completion. Key metrics to watch:

Reward Accuracy: Should increase (target >70%)
KL Divergence: Should rise gradually
Loss: Should decrease smoothly

[29]

================================================================================
⏳ Waiting for all 1 fine-tuning jobs to complete...
   This may take 1-3 hours depending on model sizes
================================================================================

⏳ Waiting for Qwen3 235B fine-tuning job to complete...
   Job ID: ft-3758bcdc-7d52
   [00:00:00] Qwen3 235B: FinetuneJobStatus.STATUS_COMPLETED
   ✓ Qwen3 235B completed! Output: zainhas/Qwen3-235B-A22B-Instruct-2507-rewardbench2_judge_dpo_qwen3_235b-a38a1677
   Total time: 00:00:00

================================================================================
✓ 1 fine-tuning jobs completed successfully
================================================================================

📋 Fine-Tuned Models:
  Qwen3 235B          : zainhas/Qwen3-235B-A22B-Instruct-2507-rewardbench2_judge_dpo_qwen3_235b-a38a1677

📊 Step 6: Evaluate All Fine-Tuned Judges on Test Set

Now we'll evaluate all fine-tuned judge models on the test set (297 examples) and compare against their baseline versions. To actually launch these evaluation jobs you will first need to spin up dedicated endpoints for both these models, for more instructions see here.

[ ]

🤖 Fine-Tuned Judge Models:
================================================================================
  zainhas/gpt-oss-120b-rewardbench2_judge_dpo_gpt-oss_120b-da9f0855: zainhas/gpt-oss-120b-rewardbench2_judge_dpo_gpt-oss_120b-da9f0855
  zainhas/Qwen3-235B-A22B-Instruct-2507-rewardbench2_judge_dpo_qwen3_235b-a38a1677: zainhas/Qwen3-235B-A22B-Instruct-2507-rewardbench2_judge_dpo_qwen3_235b-a38a1677
================================================================================
================================================================================

🔄 Launching evaluation for zainhas/gpt-oss-120b-rewardbench2_judge_dpo_gpt-oss_120b-da9f0855...
  ✓ Evaluation launched: eval-c251-1769731773

🔄 Launching evaluation for zainhas/Qwen3-235B-A22B-Instruct-2507-rewardbench2_judge_dpo_qwen3_235b-a38a1677...
  ✓ Evaluation launched: eval-f545-1769731773

================================================================================
✓ Launched 2 evaluations
================================================================================

⏳ Wait for All Fine-Tuned Judge Evaluations