Export
Prompt Comparison with LLM Judge Evaluation
Introduction
This notebook demonstrates how to compare different prompts or even model settings(max_tokens, temperature etc.) for the same model to ensure we are using the optimal setup.
Using the Together AI Evaluations API we'll:
- Load the SummEval dataset containing news articles to summarize
- Configure two different prompts for the same model
- Use a judge model to evaluate which prompt produces better summaries
- Analyze the prompt optimization results
The full list of supported models can be found here.
Concepts Covered:
- Prompt Engineering: Comparing simple vs. detailed prompts for the same task
- LLM-as-a-Judge: Using a language model to evaluate prompt effectiveness
- Compare Evaluation: A/B testing prompts to determine optimal configurations
- Summarization Evaluation: Assessing summary quality across multiple criteria
📋 Prompt Comparison Overview
Prompt engineering is crucial for getting optimal performance from language models. This evaluation compares:
- Prompt A (Simple): Basic instruction with minimal guidance
- Prompt B (Structured): Detailed instruction with specific guidelines
We'll test which approach produces better summaries on news articles.
[ ]
[1]
📊 Understanding the SummEval Dataset
The SummEval dataset contains news articles paired with both human and machine-generated summaries, along with quality ratings across multiple dimensions like relevance, coherence, fluency, and consistency.
Dataset Structure:
text: The original news article to be summarizedmachine_summaries: Various automated summarieshuman_summaries: Human-written reference summaries- Quality ratings across multiple evaluation criteria
For our evaluation, we'll focus on the original articles and generate new summaries using different prompts with our target model.
[2]
DatasetDict({
, test: Dataset({
, features: ['machine_summaries', 'human_summaries', 'relevance', 'coherence', 'fluency', 'consistency', 'text', 'id'],
, num_rows: 100
, })
,}) [3]
We are only interested in the 'text' collumn from this dataset
[4]
(CNN)Donald Sterling's racist remarks cost him an NBA team last year. But now it's his former female companion who has lost big. A Los Angeles judge has ordered V. Stiviano to pay back more than $2.6 million in gifts after Sterling's wife sued her. In the lawsuit, Rochelle "Shelly" Sterling accused Stiviano of targeting extremely wealthy older men. She claimed Donald Sterling used the couple's money to buy Stiviano a Ferrari, two Bentleys and a Range Rover, and that he helped her get a $1.8 million duplex. Who is V. Stiviano? Stiviano countered that there was nothing wrong with Donald Sterling giving her gifts and that she never took advantage of the former Los Angeles Clippers owner, who made much of his fortune in real estate. Shelly Sterling was thrilled with the court decision Tuesday, her lawyer told CNN affiliate KABC. "This is a victory for the Sterling family in recovering the $2,630,000 that Donald lavished on a conniving mistress," attorney Pierce O'Donnell said in a statement. "It also sets a precedent that the injured spouse can recover damages from the recipient of these ill-begotten gifts." Stiviano's gifts from Donald Sterling didn't just include uber-expensive items like luxury cars. According to the Los Angeles Times, the list also includes a $391 Easter bunny costume, a $299 two-speed blender and a $12 lace thong. Donald Sterling's downfall came after an audio recording surfaced of the octogenarian arguing with Stiviano. In the tape, Sterling chastises Stiviano for posting pictures on social media of her posing with African-Americans, including basketball legend Magic Johnson. "In your lousy f**ing Instagrams, you don't have to have yourself with -- walking with black people," Sterling said in the audio first posted by TMZ. He also tells Stiviano not to bring Johnson to Clippers games and not to post photos with the Hall of Famer so Sterling's friends can see. "Admire him, bring him here, feed him, f**k him, but don't put (Magic) on an Instagram for the world to have to see so they have to call me," Sterling said. NBA Commissioner Adam Silver banned Sterling from the league, fined him $2.5 million and pushed through a charge to terminate all of his ownership rights in the franchise. Fact check: Donald Sterling's claims vs. reality CNN's Dottie Evans contributed to this report.
🔄 Preparing Data for Evaluation
Before running our comparison, we need to convert the dataset to JSONL format and upload it to the Together AI platform.
The evaluation service requires:
- JSONL format with consistent fields across all examples
- Upload with
purpose="eval"to indicate evaluation usage
[5]
Uploading file tmp8i6p445j.jsonl: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 213k/213k [00:00<00:00, 342kB/s]
Uploaded file: id='file-c8f9574b-165a-4441-b3f0-f7dc11f727c4' object='file' created_at=1762948335 type=None purpose=<FilePurpose.Eval: 'eval'> filename='tmp8i6p445j.jsonl' bytes=213087 line_count=0 processed=True FileType='jsonl' project_id='proj_CPqRA6krcdajTs4Ey7kEa' organization_id='org_CPqRA6kbzxtyMnZvHWk9g'
⚙️ Prompt Configurations
We'll compare the same model but using two different prompts:
- Prompt A: An overly simplistic prompt for comparison
- Prompt B: A well structured second prompt for comparison - in this toy evaluation we expect this prompt to win
- Judge Model: Evaluates which summary is better based on our criteria
We use the prompts below to setup the models being evaluated and the Judge LLM.
[6]
[7]
🏃♂️ Running the Prompt Comparison
The evaluation will:
- Generate summaries using both prompts on the same model
- Have a judge model compare the quality of outputs
- Determine which prompt performs better overall
Key Parameters:
- Same base model for both configurations
- Different system prompts (simple vs. structured)
- Judge evaluates based on accuracy, completeness, and clarity
[8]
Evaluation ID: eval-2755-1762948381 Status: EvaluationStatus.PENDING
📊 Understanding the Results
Once the evaluation is done running we can see results.
The comparison provides key metrics:
- A_wins: Times the simple prompt was preferred
- B_wins: Times the structured prompt was preferred
- Ties: Cases where both prompts performed equally
- Fail counts: Generation errors (should be 0)
A clear winner indicates one prompting approach is more effective.
[9]
{
"A_wins": 14,
"B_wins": 46,
"Ties": 40,
"generation_fail_count": 0,
"judge_fail_count": 0,
"result_file_id": "file-0b469296-0a52-4507-bac2-a830e013d230"
}
🔑 Key Findings
Prompt Optimization Results:
- Structured Prompt (B) significantly outperformed Simple Prompt (A): 46 wins vs 14 wins
- High tie rate of 40% suggests many cases where prompt differences were minimal
- Clear winner: Structured prompts with detailed guidelines produce better summaries
Insights:
- Detailed instructions (8 specific guidelines) dramatically improve summary quality
- The 3:1 win ratio (46 vs 14) shows structured prompts are much more effective
- 40% ties indicate both approaches often produce acceptable results, but structured prompts excel in edge cases