NVIDIA Prompt Optimization

Prompt Optimization

gpu-accelerationEvaluatorretrieval-augmented-generationllm-inferencetensorrtnvidia-generative-ai-exampleslarge-language-modelsmicroservicetriton-inference-serverPrompt OptimizationLLMragnemo

alph-notebooks/nvidia-generative-ai-examples / Prompt Optimization.ipynb

Export

Run Notebooks

Contents

No cells yet

Add cells to see them here

NeMo Evaluator Microservice: Prompt Optimization with MIPROv2

In this notebook, we'll demonstrate how to use NVIDIA NeMo Evaluator Microservice for prompt optimization using MIPROv2 (Multiprompt Instruction PRoposal Optimizer Version 2). This approach uses Bayesian Optimization to improve LLM-as-a-Judge prompts and evaluate their effectiveness.

The Judge model we'll be improving today is the NVIDIA Nemotron Nano 9B V2 model, which is a Large Language Model (LLM) trained from scratch by NVIDIA, and designed as a unified model for both reasoning and non-reasoning tasks.

The model uses a hybrid architecture consisting primarily of Mamba-2 and MLP layers combined with just four Attention layers, making it an effecient and fast model - well suited to this task.

We'll walk through the required steps of:

Setting up the environment and data
Creating and Submitting the optimization job
Analyzing results and comparing baseline vs optimized prompts

Let's get started!

NOTE: You will need access to a deployed instance of NeMo Evaluator and NVIDIA NeMo Data Store Microservice. You can find details here on how to do that!

Setup and Installation

In order to use this notebook, you'll want to set-up the virtual environment with uv.

Get uv - you can start here
Run uv sync to create the virtual environment.
Select the newly created virtual environment to use as the kernel in this Jupyter Notebook.

Configure Endpoints

Set up your Evaluator and Data Store endpoints:

[ ]

Health Check

Verify connectivity to the Evaluator service:

Examine Dataset Format

Let's examine the HelpSteer2 dataset format to understand the structure for prompt optimization:

[251]

Dataset Structure:

Example 1:
Keys: ['prompt', 'response', 'reference_helpfulness']
Prompt: c#...
Response: C# is a high-level, object-oriented programming language developed by Microsoft as part of its .NET ...
Reference Helpfulness: 3

Upload Dataset to NeMo Data Store

Upload the dataset to the NeMo Data Store for use in prompt optimization:

[ ]

hs2.jsonl:   0%|          | 0.00/199k [00:00<?, ?B/s]

Dataset uploaded: https://datastore.aire.nvidia.com/v1/hf/datasets/llm-judge/hs2-short/blob/main/hs2.jsonl

Configure Prompt Optimization with MIPROv2 through an Inline Job!

Now we'll set up the prompt optimization configuration using MIPROv2. This includes:

Initial instruction: The baseline prompt to optimize
Signature: Defines the input/output structure matching our dataset
Metrics: How to evaluate prompt performance
Optimization parameters: Control the optimization process

Target Configuration

Our target configuration tells NeMo Evaluator Microservice what model is the target for our evaluation.

Let's break down the key components:

API Endpoint: model_id, url, and api_key in this example point at a remote hosted model (in this case, hosted on OpenRouter). You can substitute any OpenAI API compatible endpoint here - including, of course, the NVIDIA Nemotron Nano 9B V2 NIM!

NOTE: You can find your OpenRouter API Key through this process!

[1]

Configuration Explanation

Let's break down the key components:

Signature: "prompt, response, reference_helpfulness -> helpfulness" matches our dataset structure
Initial instruction: A baseline prompt for evaluating helpfulness
MIPROv2 parameters:
- auto: "light" - Light optimization intensity
- max_bootstrapped_demos: 2 - Generate up to 2 examples
- max_labeled_demos: 2 - Use up to 2 examples from training set
Metric: Number-check with epsilon=1 allows scores within 1 point to be considered correct

Submit Prompt Optimization Job

Now we'll create and submit the optimization job in a single config!

[271]

Next, we can submit the job to the v1/evaluation/jobs endpoint!

[261]

{'created_at': '2025-09-17T21:27:03.479720', 'updated_at': '2025-09-17T21:27:03.479721', 'id': 'eval-GTtMA2M9SBYGSouHtgv2Cs', 'namespace': 'default', 'description': None, 'target': {'schema_version': '1.0', 'id': 'eval-target-FPZPXByTCnzhneCCTjwNSf', 'description': None, 'type_prefix': 'eval-target', 'namespace': 'default', 'project': None, 'created_at': '2025-09-17T21:27:03.479166', 'updated_at': '2025-09-17T21:27:03.479166', 'custom_fields': {}, 'ownership': None, 'name': 'eval-target-FPZPXByTCnzhneCCTjwNSf', 'type': 'model', 'cached_outputs': None, 'model': {'schema_version': '1.0', 'id': 'model-XLdAWScdr44np3Z7UFygpg', 'description': None, 'type_prefix': 'model', 'namespace': 'default', 'project': None, 'created_at': '2025-09-17T21:27:03.479192', 'updated_at': '2025-09-17T21:27:03.479193', 'custom_fields': {}, 'ownership': None, 'name': 'model-XLdAWScdr44np3Z7UFygpg', 'version_id': 'main', 'version_tags': [], 'spec': None, 'artifact': None, 'base_model': None, 'api_endpoint': {'url': 'https://openrouter.ai/api/v1/chat/completions', 'model_id': 'nvidia/nemotron-nano-9b-v2', 'api_key': '******', 'format': 'nim'}, 'peft': None, 'prompt': None, 'guardrails': None}, 'retriever': None, 'rag': None, 'rows': None, 'dataset': None}, 'config': {'schema_version': '1.0', 'id': 'eval-config-WPVtgfv1aYgutLkUfViUVN', 'description': None, 'type_prefix': 'eval-config', 'namespace': 'default', 'project': None, 'created_at': '2025-09-17T21:27:03.479028', 'updated_at': '2025-09-17T21:27:03.479032', 'custom_fields': {}, 'ownership': None, 'name': 'eval-config-WPVtgfv1aYgutLkUfViUVN', 'type': 'custom', 'params': None, 'tasks': {'helpfulness-prompt-optimization': {'type': 'prompt-optimization', 'params': {'optimizer': {'type': 'miprov2', 'instruction': 'Your task is to evaluate the helpfulness of a response to a given prompt on a scale of 0-4. Output ONLY a single digit (0, 1, 2, 3, or 4) with no additional text.', 'signature': 'prompt, response -> reference_helpfulness: int', 'auto': None, 'num_trials': 1, 'num_candidates': 1, 'max_bootstrapped_demos': 0, 'max_labeled_demos': 0, 'minibatch_size': 2}}, 'metrics': {'number-check': {'type': 'number-check', 'params': {'check': ['absolute difference', '{{item.reference_helpfulness | trim}}', '{{reference_helpfulness | trim}}', 'epsilon', 1]}}}, 'dataset': {'schema_version': '1.0', 'id': 'dataset-Km4n5tApgqZY46wnPeWGfm', 'description': None, 'type_prefix': None, 'namespace': 'default', 'project': None, 'created_at': '2025-09-17T21:27:03.479101', 'updated_at': '2025-09-17T21:27:03.479101', 'custom_fields': {}, 'ownership': None, 'name': 'dataset-Km4n5tApgqZY46wnPeWGfm', 'version_id': 'main', 'version_tags': [], 'format': None, 'files_url': 'hf://datasets/llm-judge/hs2-short', 'hf_endpoint': None, 'split': None, 'limit': None}}}, 'groups': None}, 'result': None, 'output_files_url': None, 'status_details': {'message': None, 'task_status': {}, 'progress': None, 'samples_processed': None}, 'status': 'created', 'project': None, 'custom_fields': {}, 'ownership': None}
Prompt Optimization Job Submitted!
Job ID: eval-GTtMA2M9SBYGSouHtgv2Cs

Job Details:
{
  "created_at": "2025-09-17T21:27:03.479720",
  "updated_at": "2025-09-17T21:27:03.479721",
  "id": "eval-GTtMA2M9SBYGSouHtgv2Cs",
  "namespace": "default",
  "description": null,
  "target": {
    "schema_version": "1.0",
    "id": "eval-target-FPZPXByTCnzhneCCTjwNSf",
    "description": null,
    "type_prefix": "eval-target",
    "namespace": "default",
    "project": null,
    "created_at": "2025-09-17T21:27:03.479166",
    "updated_at": "2025-09-17T21:27:03.479166",
    "custom_fields": {},
    "ownership": null,
    "name": "eval-target-FPZPXByTCnzhneCCTjwNSf",
    "type": "model",
    "cached_outputs": null,
    "model": {
      "schema_version": "1.0",
      "id": "model-XLdAWScdr44np3Z7UFygpg",
      "description": null,
      "type_prefix": "model",
      "namespace": "default",
      "project": null,
      "created_at": "2025-09-17T21:27:03.479192",
      "updated_at": "2025-09-17T21:27:03.479193",
      "custom_fields": {},
      "ownership": null,
      "name": "model-XLdAWScdr44np3Z7UFygpg",
      "version_id": "main",
      "version_tags": [],
      "spec": null,
      "artifact": null,
      "base_model": null,
      "api_endpoint": {
        "url": "https://openrouter.ai/api/v1/chat/completions",
        "model_id": "nvidia/nemotron-nano-9b-v2",
        "api_key": "******",
        "format": "nim"
      },
      "peft": null,
      "prompt": null,
      "guardrails": null
    },
    "retriever": null,
    "rag": null,
    "rows": null,
    "dataset": null
  },
  "config": {
    "schema_version": "1.0",
    "id": "eval-config-WPVtgfv1aYgutLkUfViUVN",
    "description": null,
    "type_prefix": "eval-config",
    "namespace": "default",
    "project": null,
    "created_at": "2025-09-17T21:27:03.479028",
    "updated_at": "2025-09-17T21:27:03.479032",
    "custom_fields": {},
    "ownership": null,
    "name": "eval-config-WPVtgfv1aYgutLkUfViUVN",
    "type": "custom",
    "params": null,
    "tasks": {
      "helpfulness-prompt-optimization": {
        "type": "prompt-optimization",
        "params": {
          "optimizer": {
            "type": "miprov2",
            "instruction": "Your task is to evaluate the helpfulness of a response to a given prompt on a scale of 0-4. Output ONLY a single digit (0, 1, 2, 3, or 4) with no additional text.",
            "signature": "prompt, response -> reference_helpfulness: int",
            "auto": null,
            "num_trials": 1,
            "num_candidates": 1,
            "max_bootstrapped_demos": 0,
            "max_labeled_demos": 0,
            "minibatch_size": 2
          }
        },
        "metrics": {
          "number-check": {
            "type": "number-check",
            "params": {
              "check": [
                "absolute difference",
                "{{item.reference_helpfulness | trim}}",
                "{{reference_helpfulness | trim}}",
                "epsilon",
                1
              ]
            }
          }
        },
        "dataset": {
          "schema_version": "1.0",
          "id": "dataset-Km4n5tApgqZY46wnPeWGfm",
          "description": null,
          "type_prefix": null,
          "namespace": "default",
          "project": null,
          "created_at": "2025-09-17T21:27:03.479101",
          "updated_at": "2025-09-17T21:27:03.479101",
          "custom_fields": {},
          "ownership": null,
          "name": "dataset-Km4n5tApgqZY46wnPeWGfm",
          "version_id": "main",
          "version_tags": [],
          "format": null,
          "files_url": "hf://datasets/llm-judge/hs2-short",
          "hf_endpoint": null,
          "split": null,
          "limit": null
        }
      }
    },
    "groups": null
  },
  "result": null,
  "output_files_url": null,
  "status_details": {
    "message": null,
    "task_status": {},
    "progress": null,
    "samples_processed": null
  },
  "status": "created",
  "project": null,
  "custom_fields": {},
  "ownership": null
}

Monitor Job Progress

Let's monitor the optimization job status. Prompt optimization can take some time as it involves multiple optimization trials:

[ ]

Current Status: created

NOTE: At this time - progress is not captured during the running job

[ ]

Monitoring jobs ['eval-GTtMA2M9SBYGSouHtgv2Cs', 'eval-YbisDgYUhnCHY9BFsHfm5b']...

Job Status eval-GTtMA2M9SBYGSouHtgv2Cs: running

Job Status eval-YbisDgYUhnCHY9BFsHfm5b: completed

Job completed with status: completed
{'created_at': '2025-09-17T20:42:07.698905', 'updated_at': '2025-09-17T21:26:53.154123', 'id': 'eval-YbisDgYUhnCHY9BFsHfm5b', 'namespace': 'default', 'description': None, 'target': {'schema_version': '1.0', 'id': 'eval-target-9ipuYnMsyznDb7zUiZ7muC', 'description': None, 'type_prefix': 'eval-target', 'namespace': 'default', 'project': None, 'created_at': '2025-09-17T20:42:07.698478', 'updated_at': '2025-09-17T20:42:07.698478', 'custom_fields': {}, 'ownership': None, 'name': 'eval-target-9ipuYnMsyznDb7zUiZ7muC', 'type': 'model', 'cached_outputs': None, 'model': {'schema_version': '1.0', 'id': 'model-KrjaH39DaptdzyusEdA2cu', 'description': None, 'type_prefix': 'model', 'namespace': 'default', 'project': None, 'created_at': '2025-09-17T20:42:07.698498', 'updated_at': '2025-09-17T20:42:07.698499', 'custom_fields': {}, 'ownership': None, 'name': 'model-KrjaH39DaptdzyusEdA2cu', 'version_id': 'main', 'version_tags': [], 'spec': None, 'artifact': None, 'base_model': None, 'api_endpoint': {'url': 'https://openrouter.ai/api/v1/chat/completions', 'model_id': 'nvidia/nemotron-nano-9b-v2', 'api_key': '******', 'format': 'nim'}, 'peft': None, 'prompt': None, 'guardrails': None}, 'retriever': None, 'rag': None, 'rows': None, 'dataset': None}, 'config': {'schema_version': '1.0', 'id': 'eval-config-41bbjsnm5KxtcDU8GzPrGh', 'description': None, 'type_prefix': 'eval-config', 'namespace': 'default', 'project': None, 'created_at': '2025-09-17T20:42:07.698371', 'updated_at': '2025-09-17T20:42:07.698373', 'custom_fields': {}, 'ownership': None, 'name': 'eval-config-41bbjsnm5KxtcDU8GzPrGh', 'type': 'custom', 'params': None, 'tasks': {'helpfulness-prompt-optimization': {'type': 'prompt-optimization', 'params': {'optimizer': {'type': 'miprov2', 'instruction': 'Your task is to evaluate the helpfulness of a response to a given prompt on a scale of 0-4. Output ONLY a single digit (0, 1, 2, 3, or 4) with no additional text.', 'signature': 'prompt, response -> reference_helpfulness: int', 'auto': 'light', 'max_bootstrapped_demos': 2, 'max_labeled_demos': 2, 'seed': 42}}, 'metrics': {'number-check': {'type': 'number-check', 'params': {'check': ['absolute difference', '{{item.reference_helpfulness | trim}}', '{{reference_helpfulness | trim}}', 'epsilon', 1]}}}, 'dataset': {'schema_version': '1.0', 'id': 'dataset-DgnUzaiYcc98z7h1UphU3L', 'description': None, 'type_prefix': None, 'namespace': 'default', 'project': None, 'created_at': '2025-09-17T20:42:07.698427', 'updated_at': '2025-09-17T20:42:07.698427', 'custom_fields': {}, 'ownership': None, 'name': 'dataset-DgnUzaiYcc98z7h1UphU3L', 'version_id': 'main', 'version_tags': [], 'format': None, 'files_url': 'hf://datasets/llm-judge/hs2-short', 'hf_endpoint': None, 'split': None, 'limit': None}}}, 'groups': None}, 'result': 'evaluation_result-WPdqDPDuyLdsqz1oZPrdB6', 'output_files_url': 'hf://datasets/evaluation-results/eval-YbisDgYUhnCHY9BFsHfm5b', 'status_details': {'message': 'Job completed successfully.', 'task_status': {'helpfulness-prompt-optimization': 'completed'}, 'progress': 100.0, 'samples_processed': 794}, 'status': 'completed', 'project': None, 'custom_fields': {}, 'ownership': None}

Analyze Optimization Results

Once the job completes, let's examine the results to see how the prompt was optimized:

[269]

{'created_at': '2025-09-17T20:42:17.192537',
, 'updated_at': '2025-09-17T21:26:53.110771',
, 'id': 'evaluation_result-WPdqDPDuyLdsqz1oZPrdB6',
, 'job': 'eval-YbisDgYUhnCHY9BFsHfm5b',
, 'files_url': 'hf://datasets/evaluation-results/eval-YbisDgYUhnCHY9BFsHfm5b',
, 'tasks': {'helpfulness-prompt-optimization': {'metrics': {'number-check': {'scores': {'baseline': {'value': 0.8118000000000001,
,       'stats': {'count': 85}},
,      'optimized': {'value': 0.8941, 'stats': {'count': 85}}}}},
,   'data': {'baseline_prompt': 'Your task is to evaluate the helpfulness of a response to a given prompt on a scale of 0-4. Output ONLY a single digit (0, 1, 2, 3, or 4) with no additional text.',
,    'optimized_prompt': 'Evaluate the helpfulness of the response to the given prompt by systematically analyzing its relevance, clarity, completeness, and alignment with the prompt\'s requirements. Assign a score from 0-4 based on these criteria, ensuring the output is strictly a single digit (0, 1, 2, 3, or 4) with no additional text. Prioritize technical precision and user-centricity in your assessment.\n{"augmented": true, "prompt": "c#", "response": "C# is a high-level, object-oriented programming language developed by Microsoft as part of its .NET initiative. It was created as a modern alternative to Java and supports a variety of programming paradigms, including imperative, functional, and event-driven. C# is primarily used for Windows application development, but it can also be used for web, mobile, and game development. The language is designed to be safe, secure, and efficient, and it provides developers with a rich set of libraries and tools for building robust and scalable applications. C# is also widely used in the game development industry, particularly in the development of games for the Xbox 360 and Xbox One consoles.", "reasoning": "The response provides a general overview of C#, covering its development, features, and use cases. However, the prompt \\"c#\\" is extremely vague, and the response does not address any specific question or context. While it is informative for a broad introduction, it lacks depth or actionable details that would make it highly helpful for a specific query.", "reference_helpfulness": 3}\n{"prompt": "I would like you to audit content for me based on a URL I specify below. I would like you to audit content as if you were a Google Quality Rater following the rules set out by Google (which you can see here (https://developers.google.com/search/blog/2022/08/helpful-content-update)in respect of August 2022 helpful content update (experience, expertise, authority and trust) - I would also like you to consider YMYL (your money your life where applicable) and Google medic factors also depending on the content type and nature. I would like you to provide a content quality rating based on a scale of 1 to 10 where 10 is best and 0 is worst. You should take into consideration - how well the content is written, how well it aligns with Google\'s August 2022 helpful content update guidelines for human quality raters, how well structured the content is, if it makes it clear what is on offer, is it gramatically correct and well written and does it fit the end users intent when comparing the main H1 tag to the body of the content. You should provide clear, actionable recommendations for any areas where the content has an issue as well as guidance to bolster expertise and trust where applicable. You should not self reference and should avoid making any assumptions, the content for you to audit can be found here: \\n\\n\\n\\n\\nhttps://redblink.com/top-ai-content-detector-tools/", "response": "Sure, I can help you with that. I\'ll need access to the URL you specified, so please provide it to me.", "reference_helpfulness": 0}'}}},
, 'groups': {},
, 'namespace': 'default',
, 'custom_fields': {}}

[270]

================================================================================
PROMPT OPTIMIZATION RESULTS
================================================================================

📊 PERFORMANCE METRICS:
----------------------------------------
Baseline Accuracy:  0.8118 (n=85)
Optimized Accuracy: 0.8941 (n=85)
Improvement:        +0.0823 (+10.14%)

📝 PROMPT COMPARISON:
----------------------------------------

🔸 BASELINE PROMPT:
"Your task is to evaluate the helpfulness of a response to a given prompt on a scale of 0-4. Output ONLY a single digit (0, 1, 2, 3, or 4) with no additional text."

🔹 OPTIMIZED PROMPT:
"Evaluate the helpfulness of the response to the given prompt by systematically analyzing its relevance, clarity, completeness, and alignment with the prompt's requirements. Assign a score from 0-4 based on these criteria, ensuring the output is strictly a single digit (0, 1, 2, 3, or 4) with no additional text. Prioritize technical precision and user-centricity in your assessment.
{"augmented": true, "prompt": "c#", "response": "C# is a high-level, object-oriented programming language developed by Microsoft as part of its .NET initiative. It was created as a modern alternative to Java and supports a variety of programming paradigms, including imperative, functional, and event-driven. C# is primarily used for Windows application development, but it can also be used for web, mobile, and game development. The language is designed to be safe, secure, and efficient, and it provides developers with a rich set of libraries and tools for building robust and scalable applications. C# is also widely used in the game development industry, particularly in the development of games for the Xbox 360 and Xbox One consoles.", "reasoning": "The response provides a general overview of C#, covering its development, features, and use cases. However, the prompt \"c#\" is extremely vague, and the response does not address any specific question or context. While it is informative for a broad introduction, it lacks depth or actionable details that would make it highly helpful for a specific query.", "reference_helpfulness": 3}
{"prompt": "I would like you to audit content for me based on a URL I specify below. I would like you to audit content as if you were a Google Quality Rater following the rules set out by Google (which you can see here (https://developers.google.com/search/blog/2022/08/helpful-content-update)in respect of August 2022 helpful content update (experience, expertise, authority and trust) - I would also like you to consider YMYL (your money your life where applicable) and Google medic factors also depending on the content type and nature. I would like you to provide a content quality rating based on a scale of 1 to 10 where 10 is best and 0 is worst. You should take into consideration - how well the content is written, how well it aligns with Google's August 2022 helpful content update guidelines for human quality raters, how well structured the content is, if it makes it clear what is on offer, is it gramatically correct and well written and does it fit the end users intent when comparing the main H1 tag to the body of the content. You should provide clear, actionable recommendations for any areas where the content has an issue as well as guidance to bolster expertise and trust where applicable. You should not self reference and should avoid making any assumptions, the content for you to audit can be found here: \n\n\n\n\nhttps://redblink.com/top-ai-content-detector-tools/", "response": "Sure, I can help you with that. I'll need access to the URL you specified, so please provide it to me.", "reference_helpfulness": 0}"

📋 JOB METADATA:
----------------------------------------
Job ID:        eval-YbisDgYUhnCHY9BFsHfm5b
Created:       2025-09-17T20:42:17.192537
Updated:       2025-09-17T21:26:53.110771
Files URL:     hf://datasets/evaluation-results/eval-YbisDgYUhnCHY9BFsHfm5b

================================================================================

Understanding the Results

The optimization results provide several key insights:

Metrics

Baseline Accuracy: Performance of the original prompt
Optimized Accuracy: Performance of the MIPROv2-optimized prompt
Improvement: Quantified improvement in evaluation accuracy

Prompts

Baseline Prompt: Your original instruction
Optimized Prompt: The improved prompt generated by MIPROv2, which may include:
- Refined instructions
- Few-shot examples
- Better task framing

Summary

In this notebook, we've demonstrated how to:

Set up prompt optimization with MIPROv2 using NeMo Evaluator
Configure the optimization task with proper signature and metrics
Submit and monitor optimization jobs
Analyze results to understand prompt improvements

Key Takeaways:

MIPROv2 uses Bayesian Optimization to systematically improve prompts
Signature definition must match your dataset structure exactly
Metric configuration determines how optimization success is measured
Optimization intensity (auto: light/medium/heavy) controls compute vs. quality tradeoff
Results provide both quantitative metrics and the actual optimized prompts