End2end Tutorial
gpu-accelerationEvaluatorretrieval-augmented-generationllm-inferencetensorrtnvidia-generative-ai-exampleslarge-language-modelsmicroservicetriton-inference-serverLLMragnemo
Export
Nemo Evaluator Microservice Tutorial
About this tutorial
In this tutorial, we will cover the following evaluation types using NeMo Evaluator:
- Agentic Evaluation
- LLM Evaluation on Academic Benchmarks
- Custom Evaluations
- Similarity Metrics Evaluation
- LLM-as-Judge Evaluation
- Tool Calling Evaluation
- Retriever Pipeline Evaluation
- RAG Pipeline Evaluation
1. Prerequisites
1.1 Install Nemo Microservices
While this tutorial focuses on NeMo Evaluator, we recommend installing the NeMo Microservices platform using the NeMo Microservices Platform Helm Chart to avoid manually managing dependencies. For step by step installation guide, refer to the Demo Cluster Setup Guide. This tutorial was done on a single-node cluster with two A100 GPUs.
Check the pods to make sure all required microservices is running before proceeding.
[1]
NAME READY STATUS RESTARTS AGE model-downloader-meta-llama-3-1-8b-instruct-2-0-28trx 0/1 Completed 0 10m model-downloader-meta-llama-3-2-1b-instruct-2-0-b6scn 0/1 Completed 0 10m modeldeployment-meta-llama-3-1-8b-instruct-6b64d56fdc-slctn 1/1 Running 0 9m26s nemo-argo-workflows-server-655f8d755-svgn2 1/1 Running 0 12m nemo-argo-workflows-workflow-controller-8f8877cd4-8t2tf 1/1 Running 0 12m nemo-customizer-5d8554fcf6-rhwfp 1/1 Running 2 (11m ago) 12m nemo-customizerdb-0 1/1 Running 0 12m nemo-data-store-795ccbb97b-nwcf2 1/1 Running 0 12m nemo-deployment-management-646cc67c-l67lq 1/1 Running 0 12m nemo-entity-store-6bfd7c4bd9-p4s4b 1/1 Running 0 12m nemo-entity-storedb-0 1/1 Running 0 12m nemo-evaluator-68c67f6798-f4sqh 1/1 Running 0 12m nemo-evaluatordb-0 1/1 Running 0 12m nemo-guardrails-5985f8c6d4-bkkq2 1/1 Running 0 12m nemo-milvus-standalone-57cc6c46fc-5rctk 1/1 Running 0 12m nemo-nemo-operator-controller-manager-59c98d97f9-7gfsw 2/2 Running 0 12m nemo-nim-operator-765cbf686b-w9n6c 1/1 Running 0 12m nemo-nim-proxy-6fbccbfb45-8qvmz 1/1 Running 0 12m nemo-opentelemetry-collector-6f46646fd9-5tnrf 1/1 Running 0 12m nemo-postgresql-0 1/1 Running 0 12m
Install huggingface_hub which is required to interact with NeMo Data Store:
pip install -U "huggingface_hub[cli]"
[2]
/home/ubuntu/.local/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
Specify the Namespace and API endpoints:
[4]
1.2 Deploy NIM for LLMs
This tutorial will use the Llama-3.1-8b-instruct model as the LLM to be evaluated. You can either choose to deploy a Llama-3.1-8b-instruct NIM locally or use the NIM hosted remotely. As part of the Nemo Microservices Platform, the NeMo Deployment Management service provides an API to deploy NIM on a Kubernetes cluster and manage them through the NIM Operator microservice. The below cells show how to deploy the Llama-3.1-8b-instruct NIM and run inference using NIM Proxy service.
Note: if you use see a pod named modeldeployment-meta-llama-3-1-8b-instruct-xxx from the list of pods above, then a Llama-3.1-8b-instruct NIM has already been deployed and you can skip this step.
[ ]
Check the deployment status and make sure the status is 'ready' before proceeding.
[ ]
We can test LLM inferences to the NIM endpoint
[5]
{'role': 'assistant',
'content': 'LLM (Large Language Model) token log probabilities are used to '
"represent the model's confidence in its predictions, with lower "
'probabilities indicating less likely or less confident '
'predictions.'}
1.3 Set Up API Key and Access Token
This tutorial uses a remote LLM hosted on build.nvidia.com for evaluations that requires a LLM as judge. For this, we will need to set up the API key to access the models. You can generate an NVIDIA API key at Manage API Keys.
[6]
Enter your NVIDIA API Key ········
Next we need to set up the Hugging Face Access Token which needs to have access to Meta's Llama-3.1-8B-instruct model.
[7]
Enter your Hugging Face Token ········
1.4 Running Evaluation Jobs with NeMo Evaluator
Before running evaluations, it is important to understand the typical NeMo Evaluator Workflow:
-
(Optional) Upload your custom dataset to the NeMo Data Store if you’re not using a built-in dataset.
-
Create an evaluation configuration in NeMo Evaluator.
-
Define an evaluation target (the model to evaluate).
-
Submit an evaluation job to NeMo Evaluator. The following steps occur automatically:
a. NeMo Evaluator retrieves any required custom data from the NeMo Data Store.
b. It runs inference using NIM, supporting LLMs, embeddings, and reranking tasks.
c. Results, including generations, logs, and metrics, are written to the NeMo Data Store.
d. The results are returned to the user.
-
Review evaluation results.
2. Agentic Evaluation
Agentic evaluation uses RAGAS metrics to score agent outputs. RAGAS is a library for evaluating retrieval-augmented generation and agentic workflows using standardized, research-backed metrics.
Each task contains a set of metrics relevant to the Agentic evaluation, such as topic adherence, tool call accuracy, agent goal accuracy, or answer accuracy, depending on the metric selected in the job configuration.
2.1 Upload Custom Data to Nemo Data Store
Before uploding the data, we first need to create namespace in NeMo Data Store and Entity Store.
[8]
Follow this dataset format to prepare the dataset for agentic evaluation, example datasets are provided at ./eval_dataset/agent_data. Next, we will upload these example datasets to Data Store.
[14]
agent_tool_call_data.jsonl: 100%|██████████| 3.20k/3.20k [00:00<00:00, 709kB/s]
CommitInfo(commit_url='', commit_message='Upload agent_tool_call_data.jsonl with huggingface_hub', commit_description='', oid='659c7516a96482859db4f4707a32b4a8c5b78ba9', pr_url=None, repo_url=RepoUrl('', endpoint='https://huggingface.co', repo_type='model', repo_id=''), pr_revision=None, pr_num=None) 2.2 Create Evaluation Targets
Currently, agentic evaluation only works with cached_outputs targets which point to files that are stored in NeMo Data Store and that contain pre-generated answers.
We will create two evaluation targets for agentic evaluations: one for agent goal accuracy and one for tool calling accuracy.
[15]
{'created_at': '2025-07-08T19:18:37.063092',
'updated_at': '2025-07-08T19:18:37.063095',
'name': 'agent-goal-target',
'namespace': 'nemo-eval-tutorial',
'type': 'cached_outputs',
'cached_outputs': {'files_url': 'hf://datasets/nemo-eval-tutorial/agent_eval/agent_goal_data.jsonl'},
'id': 'eval-target-GDJ771rB6W3xaowbpE4Bsh',
'custom_fields': {}}
[16]
{'created_at': '2025-07-08T19:18:38.848137',
'updated_at': '2025-07-08T19:18:38.848149',
'name': 'agent-tool-call-target',
'namespace': 'nemo-eval-tutorial',
'type': 'cached_outputs',
'cached_outputs': {'files_url': 'hf://datasets/nemo-eval-tutorial/agent_eval/agent_tool_call_data.jsonl'},
'id': 'eval-target-HBLMJtD4MhBzV6S3AGor2g',
'custom_fields': {}}
2.3 Create Evaluation Configs
Similarily, we will create two evaluation configs: one for agent goal accuracy and one for tool calling accuracy.
[17]
{'created_at': '2025-07-08T19:19:13.374904',...
[18]
{'created_at': '2025-07-08T19:19:30.170735',...
2.4 Submit Evaluation Job
To launch the evaluation job, we simply send a request with the previously created evaluation targets and configs to the /jobs API endpoint.
[28]
{'created_at': '2025-07-08T19:28:46.640661',...
[24]
{'created_at': '2025-07-08T19:27:35.584648',...
2.5 Monitoring job status and get evaluation results
We can monitor job status and get the evaluation results by sending request with the Job ID to the /status and /results API, respectively.
[33]
{'message': None, 'task_status': {}, 'progress': 100.0}
[34]
{'goal-accuracy': {'metrics': {'agent_goal_accuracy': {'scores': {'agent_goal_accuracy': {'value': 1.0}}}}}}
[25]
{'message': None, 'task_status': {}, 'progress': 100.0}
[27]
{'tool-call-accuracy': {'metrics': {'tool_call_accuracy': {'scores': {'tool_call_accuracy': {'value': 1.0}}}}}}
Download Evaluation Eesults: downloads a directory that contains the configuration files, logs, and evaluation results for a specific evaluation job.
[ ]
3. LLM Evaluation on Academic Benchmarks
Create Evaluation Config
[36]
{'created_at': '2025-07-08T19:39:55.734565',
'updated_at': '2025-07-08T19:39:55.734568',
'name': 'gsm8k-chat-config',
'namespace': 'nemo-eval-tutorial',
'type': 'gsm8k',
'params': {'max_tokens': 256,
'temperature': 1e-05,
'top_p': 1e-05,
'stop': ['<|eot|>'],
'extra': {'num_fewshot': 8,
'batch_size': 16,
'bootstrap_iters': 100000,
'dataset_seed': 42,
'use_greedy': True,
'top_k': 1,
'hf_token': '******',
'tokenizer_backend': 'hf',
'tokenizer': 'meta-llama/llama-3.1-8B-Instruct',
'apply_chat_template': True,
'fewshot_as_multiturn': True}},
'id': 'eval-config-6pTREGQfYHFQzq7oWHATgt',
'custom_fields': {}}
Create Evaluation Target
[37]
{'created_at': '2025-07-08T19:40:13.061228',
'updated_at': '2025-07-08T19:40:13.061229',
'name': 'llama-chat-target',
'namespace': 'nemo-eval-tutorial',
'type': 'model',
'model': {'schema_version': '1.0',
'id': 'model-EXZQbF8ZXbjoogTvF9UFKr',
'type_prefix': 'model',
'namespace': 'default',
'created_at': '2025-07-08T19:40:13.061012',
'updated_at': '2025-07-08T19:40:13.061015',
'custom_fields': {},
'name': 'model-EXZQbF8ZXbjoogTvF9UFKr',
'version_id': 'main',
'version_tags': [],
'api_endpoint': {'url': 'http://nim.test/v1/chat/completions',
'model_id': 'meta/llama-3.1-8b-instruct',
'format': 'openai'}},
'id': 'eval-target-GaJYgnmDA4Ta2TN23sV9Wo',
'custom_fields': {}}
Submit Evaluation Job
[38]
{'created_at': '2025-07-08T19:40:19.984771',
'updated_at': '2025-07-08T19:40:19.984774',
'id': 'eval-WF5xprkaQpjTNZLpsNb1Y1',
'namespace': 'nemo-eval-tutorial',
'description': None,
'target': {'schema_version': '1.0',
'id': 'eval-target-GaJYgnmDA4Ta2TN23sV9Wo',
'description': None,
'type_prefix': 'eval-target',
'namespace': 'nemo-eval-tutorial',
'project': None,
'created_at': '2025-07-08T19:40:13.061228',
'updated_at': '2025-07-08T19:40:13.061229',
'custom_fields': {},
'ownership': None,
'name': 'llama-chat-target',
'type': 'model',
'cached_outputs': None,
'model': {'schema_version': '1.0',
'id': 'model-EXZQbF8ZXbjoogTvF9UFKr',
'description': None,
'type_prefix': 'model',
'namespace': 'default',
'project': None,
'created_at': '2025-07-08T19:40:13.061012',
'updated_at': '2025-07-08T19:40:13.061015',
'custom_fields': {},
'ownership': None,
'name': 'model-EXZQbF8ZXbjoogTvF9UFKr',
'version_id': 'main',
'version_tags': [],
'spec': None,
'artifact': None,
'base_model': None,
'api_endpoint': {'url': 'http://nim.test/v1/chat/completions',
'model_id': 'meta/llama-3.1-8b-instruct',
'api_key': None,
'format': 'openai'},
'peft': None,
'prompt': None,
'guardrails': None},
'retriever': None,
'rag': None,
'rows': None,
'dataset': None},
'config': {'schema_version': '1.0',
'id': 'eval-config-6pTREGQfYHFQzq7oWHATgt',
'description': None,
'type_prefix': 'eval-config',
'namespace': 'nemo-eval-tutorial',
'project': None,
'created_at': '2025-07-08T19:39:55.734565',
'updated_at': '2025-07-08T19:39:55.734568',
'custom_fields': {},
'ownership': None,
'name': 'gsm8k-chat-config',
'type': 'gsm8k',
'params': {'parallelism': None,
'request_timeout': None,
'max_retries': None,
'limit_samples': None,
'max_tokens': 256,
'temperature': 1e-05,
'top_p': 1e-05,
'stop': ['<|eot|>'],
'extra': {'num_fewshot': 8,
'batch_size': 16,
'bootstrap_iters': 100000,
'dataset_seed': 42,
'use_greedy': True,
'top_k': 1,
'hf_token': '******',
'tokenizer_backend': 'hf',
'tokenizer': 'meta-llama/llama-3.1-8B-Instruct',
'apply_chat_template': True,
'fewshot_as_multiturn': True}},
'tasks': None,
'groups': None},
'result': None,
'output_files_url': None,
'status_details': {'message': None, 'task_status': {}, 'progress': None},
'status': 'created',
'project': None,
'custom_fields': {},
'ownership': None}
We can check the status of the job using the status API. Note that the status in the API is only regularly updated for custom evaluation. So as long as the status is saying running, it's actually running.
[41]
{'message': 'Job completed successfully', 'task_status': {}, 'progress': 100.0}
Once the job is completed, we can check the evaluation results using the results endpoint.
[42]
{'metrics': {'exact_match': {'scores': {'gsm8k-metric_ranking-1': {'value': 0.7664897649734648},
'gsm8k-metric_ranking-3': {'value': 0.821076573161486}}}}}
{'metrics': {'exact_match_stderr': {'scores': {'gsm8k-metric_ranking-2': {'value': 0.011653286808791036},
'gsm8k-metric_ranking-4': {'value': 0.010557661392901296}}}}}
4. Custom Evaluations
4.1 Similarity Metrics Evaluation
Upload Custom Data to Nemo Data Store
[43]
inputs.jsonl: 100%|██████████| 149k/149k [00:00<00:00, 28.9MB/s]
CommitInfo(commit_url='', commit_message='Upload similarity_metrics/inputs.jsonl with huggingface_hub', commit_description='', oid='6220905767521b90b36e77360dc7213c3dbbb99e', pr_url=None, repo_url=RepoUrl('', endpoint='https://huggingface.co', repo_type='model', repo_id=''), pr_revision=None, pr_num=None) Create Evaluation Config
[44]
{'created_at': '2025-07-08T20:31:57.416755',
'updated_at': '2025-07-08T20:31:57.416756',
'name': 'similarity-configuration',
'namespace': 'nemo-eval-tutorial',
'type': 'similarity_metrics',
'params': {'max_tokens': 200, 'temperature': 0.7, 'extra': {'top_k': 20}},
'tasks': {'my-similarity-metrics-task': {'type': 'default',
'metrics': {'accuracy': {'type': 'accuracy'},
'bleu': {'type': 'bleu'},
'rouge': {'type': 'rouge'},
'em': {'type': 'em'},
'f1': {'type': 'f1'}},
'dataset': {'schema_version': '1.0',
'id': 'dataset-DALY8efJ78UgkXHcrfACGZ',
'namespace': 'default',
'created_at': '2025-07-08T20:31:57.416530',
'updated_at': '2025-07-08T20:31:57.416532',
'custom_fields': {},
'name': 'dataset-DALY8efJ78UgkXHcrfACGZ',
'version_id': 'main',
'version_tags': [],
'files_url': 'hf://datasets/nemo-eval-tutorial/similarity_eval/similarity_metrics/inputs.jsonl'}}},
'id': 'eval-config-XQpRHrKVucSQzLDXFdZqjq',
'custom_fields': {}}
Launch Evaluation Job
[45]
{'created_at': '2025-07-08T20:32:03.445393',...
Monitoring job status and results
[50]
{'message': 'Job completed successfully',
'task_status': {'my-similarity-metrics-task': 'completed'},
'progress': 100.0}
[51]
{'metrics': {'accuracy': {'scores': {'accuracy': {'value': 0.0}}},
'bleu': {'scores': {'bleu_score': {'value': 0.015511131876432806}}},
'em': {'scores': {'em': {'value': 0.0}}},
'f1': {'scores': {'f1': {'value': 0.10128911130270025}}},
'rouge': {'scores': {'rouge_1_score': {'value': 0.1166731565559731},
'rouge_2_score': {'value': 0.03311328362331498},
'rouge_3_score': {'value': 0.01193015710048473},
'rouge_L_score': {'value': 0.09285834070191781}}}}}
4.2 LLM-as-Judge Evaluation
Upload Custom Dataset to Data Store
[52]
math_dataset.csv: 100%|██████████| 449/449 [00:00<00:00, 108kB/s]
CommitInfo(commit_url='', commit_message='Upload llm_as_judge/math_dataset.csv with huggingface_hub', commit_description='', oid='32e22ec06ea372c6cfddf52cb733e86fd1b135ed', pr_url=None, repo_url=RepoUrl('', endpoint='https://huggingface.co', repo_type='model', repo_id=''), pr_revision=None, pr_num=None) - Item — Represents the current item from the dataset.
- Sample — Contains data related to the output from the model. The
sample.output_textrepresents the completion text for completion models and the content of the first message for chat models.
The properties on the item object are derived from the dataset's column names (for CSVs) or keys (for JSONs).
The following rules apply to these properties:
- All non-alphanumeric characters are replaced with underscores.
- Column names are converted to lowercase.
- In case of conflicts, suffixes (
_1,_2, etc.), are appended to the property names.
Templates for Chat Models
Prompt templates are used to structure tasks for evaluating the performance of models, specifically following the NIM/OpenAI format for chat-completion tasks. Templates use the Jinja2 templating syntax. Variables are represented using double-curly brackets, for example, {{item.review}}.
Create Eval Config
We will use llama-3.3-70b-instruct hosted on build.nvidia.com as the judge model.
[53]
{'created_at': '2025-07-08T20:37:58.674883',...
Launch Eval Job
[54]
{'created_at': '2025-07-08T20:38:04.820220',
'updated_at': '2025-07-08T20:38:04.820223',
'id': 'eval-MoXUyarWP3jBR8h6LqyhWU',
'namespace': 'nemo-eval-tutorial',
'description': None,
'target': {'schema_version': '1.0',
'id': 'eval-target-GaJYgnmDA4Ta2TN23sV9Wo',
'description': None,
'type_prefix': 'eval-target',
'namespace': 'nemo-eval-tutorial',
'project': None,
'created_at': '2025-07-08T19:40:13.061228',
'updated_at': '2025-07-08T19:40:13.061229',
'custom_fields': {},
'ownership': None,
'name': 'llama-chat-target',
'type': 'model',
'cached_outputs': None,
'model': {'schema_version': '1.0',
'id': 'model-EXZQbF8ZXbjoogTvF9UFKr',
'description': None,
'type_prefix': 'model',
'namespace': 'default',
'project': None,
'created_at': '2025-07-08T19:40:13.061012',
'updated_at': '2025-07-08T19:40:13.061015',
'custom_fields': {},
'ownership': None,
'name': 'model-EXZQbF8ZXbjoogTvF9UFKr',
'version_id': 'main',
'version_tags': [],
'spec': None,
'artifact': None,
'base_model': None,
'api_endpoint': {'url': 'http://nim.test/v1/chat/completions',
'model_id': 'meta/llama-3.1-8b-instruct',
'api_key': None,
'format': 'openai'},
'peft': None,
'prompt': None,
'guardrails': None},
'retriever': None,
'rag': None,
'rows': None,
'dataset': None},
'config': {'schema_version': '1.0',
'id': 'eval-config-J9aERJok5jacrTdhYcyM1H',
'description': None,
'type_prefix': 'eval-config',
'namespace': 'nemo-eval-tutorial',
'project': None,
'created_at': '2025-07-08T20:37:58.674883',
'updated_at': '2025-07-08T20:37:58.674884',
'custom_fields': {},
'ownership': None,
'name': 'custom_llm_as_judge_config',
'type': 'custom',
'params': None,
'tasks': {'qa': {'type': 'completion',
'params': {'template': {'messages': [{'role': 'system',
'content': 'You '
'are '
'a '
'helpful, '
'respectful '
'and '
'honest '
'assistant. \n'
'Answers '
'the '
'following '
'question '
'as '
'briefly '
'as '
'you '
'can.\n'
'.'},
{'role': 'user',
'content': 'Answer '
'very '
'briefly '
'(no '
'explanation) '
'this '
'question: '
'{{item.question}}'}]}},
'metrics': {'accuracy': {'type': 'string-check',
'params': {'check': ['{{sample.output_text}}',
'contains',
'{{item.answer}}']}},
'bleu': {'type': 'bleu',
'params': {'references': ['{{item.reference_answer}}']}},
'accuracy-llm-judge': {'type': 'llm-judge',
'params': {'model': {'api_endpoint': {'url': 'https://integrate.api.nvidia.com/v1/chat/completions',
'model_id': 'meta/llama-3.3-70b-instruct',
'api_key': '******'}},
'template': {'messages': [{'role': 'system',
'content': 'Your '
'task '
'is '
'to '
'evaluate '
'the '
'semantic '
'similarity '
'between '
'two '
'responses.'},
{'role': 'user',
'content': 'Respond '
'in '
'the '
'following '
'format '
'SIMILARITY: '
'4. '
'The '
'similarity '
'should '
'be '
'a '
'score '
'between '
'0 '
'and '
'10.\n'
'\n'
'RESPONSE '
'1: '
'{{item.reference_answer}}\n'
'\n'
'RESPONSE '
'2: '
'{{sample.output_text}}.\n'
'\n'}]},
'scores': {'similarity': {'type': 'int',
'parser': {'type': 'regex',
'pattern': 'SIMILARITY: '
'(\\d)'}}}}}},
'dataset': {'schema_version': '1.0',
'id': 'dataset-4cBEc7vpZh8aBPpuGpNoFD',
'description': None,
'type_prefix': None,
'namespace': 'default',
'project': None,
'created_at': '2025-07-08T20:37:58.674690',
'updated_at': '2025-07-08T20:37:58.674693',
'custom_fields': {},
'ownership': None,
'name': 'dataset-4cBEc7vpZh8aBPpuGpNoFD',
'version_id': 'main',
'version_tags': [],
'format': None,
'files_url': 'hf://datasets/nemo-eval-tutorial/llm_as_judge_data/llm_as_judge/math_dataset.csv',
'hf_endpoint': None,
'split': None,
'limit': None}}},
'groups': None},
'result': None,
'output_files_url': None,
'status_details': {'message': None, 'task_status': {}, 'progress': None},
'status': 'created',
'project': None,
'custom_fields': {},
'ownership': None}
Monitoring job status and results
[61]
{'message': 'Job completed successfully.',
'task_status': {'qa': 'completed'},
'progress': 100.0}
[62]
{'metrics': {'accuracy': {'scores': {'string-check': {'value': 1.0,
'stats': {'count': 10,
'sum': 10.0,
'mean': 1.0}}}},
'bleu': {'scores': {'sentence': {'value': 9.135501080023044,
'stats': {'count': 10,
'sum': 91.35501080023045,
'mean': 9.135501080023044}},
'corpus': {'value': 0.0}}},
'accuracy-llm-judge': {'scores': {'similarity': {'value': 1.7,
'stats': {'count': 10,
'sum': 17.0,
'mean': 1.7}}}}}}
4.3 Tool Calling Evaluation
Required dataset format for a custom tool calling evaluation:...
Upload Custom Tool Calling Dataset to Data Store
[95]
CommitInfo(commit_url='', commit_message='Upload aiva_tool_call.jsonl with huggingface_hub', commit_description='', oid='8ba99b4e0b4b2224b059c6cacda3b86d724c406e', pr_url=None, repo_url=RepoUrl('', endpoint='https://huggingface.co', repo_type='model', repo_id=''), pr_revision=None, pr_num=None) Create Eval Config
[96]
{'created_at': '2025-07-09T02:18:55.955723',
'updated_at': '2025-07-09T02:18:55.955724',
'name': 'tool-call-eval-config',
'namespace': 'nemo-eval-tutorial',
'type': 'custom',
'tasks': {'custom-tool-calling': {'type': 'chat-completion',
'params': {'template': {'messages': '{{ '
'item.messages '
'| '
'tojson}}',
'tools': '{{ '
'item.tools '
'| tojson '
'}}',
'tool_choice': 'auto'}},
'metrics': {'tool-calling-accuracy': {'type': 'tool-calling',
'params': {'tool_calls_ground_truth': '{{ '
'item.tool_calls '
'| '
'tojson '
'}}'}}},
'dataset': {'schema_version': '1.0',
'id': 'dataset-FHLbB4tDWWXwkeQShawtGg',
'namespace': 'default',
'created_at': '2025-07-09T02:18:55.955557',
'updated_at': '2025-07-09T02:18:55.955560',
'custom_fields': {},
'name': 'dataset-FHLbB4tDWWXwkeQShawtGg',
'version_id': 'main',
'version_tags': [],
'files_url': 'hf://datasets/nemo-eval-tutorial/tool_call_data/aiva_tool_call.jsonl'}}},
'id': 'eval-config-N6PkTGVcZNP8ho4LwHYxqg',
'custom_fields': {}}
Launch Eval Job
[97]
{'created_at': '2025-07-09T02:19:10.833300',...
Monitoring job status and results
[99]
{'message': 'Job completed successfully.',
'task_status': {'custom-tool-calling': 'completed'},
'progress': 100.0}
[100]
{'tool-calling-accuracy': {'scores': {'function_name_accuracy': {'value': 0.9,
'stats': {'count': 10,
'sum': 9.0,
'mean': 0.9}},
'function_name_and_args_accuracy': {'value': 0.0,
'stats': {'count': 10,
'sum': 0.0,
'mean': 0.0}}}}}
5. Retriever Pipeline Evaluation
5.1 Deploy Retriever Models
To evaluate retriever pipelines, retriever models must be deployed locally. For this tutorial, we will create a retriever pipeline with both embedding and reranking models. Specifically, we will deploy two retriever NIMs for llama-3.2-nv-embedqa-1b-v2 and llama-3.2-nv-rerankqa-1b-v2, respectively, using Docker.
First, let's identify a free GPU to deploy the retriever models.
[9]
Wed Jul 9 06:27:12 2025
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.06 Driver Version: 535.183.06 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA A100 80GB PCIe On | 00000002:00:01.0 Off | 0 |
| N/A 34C P0 43W / 300W | 0MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA A100 80GB PCIe On | 00000002:00:02.0 Off | 0 |
| N/A 36C P0 62W / 300W | 70969MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 1 N/A N/A 79075 C /opt/nim/llm/.venv/bin/python3 70958MiB |
+---------------------------------------------------------------------------------------+
Then, we can deploy the llama-3.2-nv-embedqa-1b-v2 embedding NIM (replace <your-api-key> with your NGC API Key below):
[ ]
Next, we will deploy the llama-3.2-nv-rerankqa-1b-v2 reranking NIM:
[ ]
Get Host IP address and specify the API endpoints for embedding and reranking NIMs:
[12]
1.0.0.0 via 172.27.16.1 dev ens3 src 172.27.20.120 uid 1000
[20]
5.2 Set up Milvus Vector Database
To run retriever or RAG evaluations, you must first enable the Milvus document store by setting milvus.enabled: true in values.yaml, and then upgrade your Helm deployment to apply the change.
evaluator:
enabled: true
milvus:
enabled: true
You should see a milvus pod running:
[21]
NAME READY STATUS RESTARTS AGE model-downloader-meta-llama-3-1-8b-instruct-2-0-28trx 0/1 Completed 0 29m model-downloader-meta-llama-3-2-1b-instruct-2-0-b6scn 0/1 Completed 0 29m modeldeployment-meta-llama-3-1-8b-instruct-6b64d56fdc-slctn 1/1 Running 0 28m nemo-argo-workflows-server-655f8d755-svgn2 1/1 Running 0 30m nemo-argo-workflows-workflow-controller-8f8877cd4-8t2tf 1/1 Running 0 30m nemo-customizer-5d8554fcf6-rhwfp 1/1 Running 2 (29m ago) 30m nemo-customizerdb-0 1/1 Running 0 30m nemo-data-store-795ccbb97b-nwcf2 1/1 Running 0 30m nemo-deployment-management-646cc67c-l67lq 1/1 Running 0 30m nemo-entity-store-6bfd7c4bd9-p4s4b 1/1 Running 0 30m nemo-entity-storedb-0 1/1 Running 0 30m nemo-evaluator-68c67f6798-f4sqh 1/1 Running 0 30m nemo-evaluatordb-0 1/1 Running 0 30m nemo-guardrails-5985f8c6d4-bkkq2 1/1 Running 0 30m nemo-milvus-standalone-57cc6c46fc-5rctk 1/1 Running 0 30m nemo-nemo-operator-controller-manager-59c98d97f9-7gfsw 2/2 Running 0 30m nemo-nim-operator-765cbf686b-w9n6c 1/1 Running 0 30m nemo-nim-proxy-6fbccbfb45-8qvmz 1/1 Running 0 30m nemo-opentelemetry-collector-6f46646fd9-5tnrf 1/1 Running 0 30m nemo-postgresql-0 1/1 Running 0 30m
5.3 Evaluate Embedding Pipeline on FIQA Dataset
[26]
[27]
{'created_at': '2025-07-09T06:45:27.786401',...
[33]
{'message': 'Job completed successfully',
'task_status': {'my-beir-task': 'completed'},
'progress': 100.0}
[34]
{'evaluation': {'scores': {'recall_10': {'value': 0.5984280594234299},
'ndcg_cut_10': {'value': 0.5280203494315917},
'ndcg_cut_5': {'value': 0.5054691655963462},
'recall_5': {'value': 0.5225860130952724}}}}
5.4 Evaluate Embedding + Reranking Pipeline on FIQA Dataset
First, let's create a evaluation target for Embedding + Reranking
[35]
Then we can launch the eval job for the embed + rerank pipeline on FIQA data:
[36]
{'created_at': '2025-07-09T07:06:34.280669',...
[38]
{'message': 'Job completed successfully',
'task_status': {'my-beir-task': 'completed'},
'progress': 100.0}
[39]
{'evaluation': {'scores': {'recall_10': {'value': 0.5678767905619758},
'ndcg_cut_10': {'value': 0.5134364317202811},
'recall_5': {'value': 0.5151632575243686},
'ndcg_cut_5': {'value': 0.5011430756367373}}}}
5.5 Evaluate Embedding + Reranking Pipeline on Custom Dataset
Upload the custom dataset:
[40]
[41]
queries.jsonl: 100%|██████████| 16.7k/16.7k [00:00<00:00, 3.90MB/s] corpus.jsonl: 100%|██████████| 11.0k/11.0k [00:00<00:00, 3.04MB/s] test.tsv: 100%|██████████| 7.28k/7.28k [00:00<00:00, 1.79MB/s]
CommitInfo(commit_url='', commit_message='Upload rag_data/qrels/test.tsv with huggingface_hub', commit_description='', oid='c901d2bb70290aa2d898d3f8102d732f774baaec', pr_url=None, repo_url=RepoUrl('', endpoint='https://huggingface.co', repo_type='model', repo_id=''), pr_revision=None, pr_num=None) Create config for the custom data for retriever eval:
[42]
Launch the eval job on the embed + rerank pipeline:
[43]
{'created_at': '2025-07-09T16:16:07.734774',...
[46]
{'message': 'Job completed successfully',
'task_status': {'my-beir-task': 'completed'},
'progress': 100.0}
[47]
{'evaluation': {'scores': {'ndcg_cut_5': {'value': 1.0},
'recall_10': {'value': 1.0},
'recall_5': {'value': 1.0},
'ndcg_cut_10': {'value': 1.0}}}}
6. RAG Pipeline Evaluation
6.1 Evaluate RAG Pipeline on NFCorpus Dataset
[48]
[49]
{'created_at': '2025-07-09T16:30:31.105191',...
[50]
{'created_at': '2025-07-09T16:30:43.855021',
'updated_at': '2025-07-09T16:30:43.855024',
'id': 'eval-VrFzxRNEP1hhjrnUtP8Ug1',
'namespace': 'nemo-eval-tutorial',
'description': None,
'target': {'schema_version': '1.0',
'id': 'eval-target-VTApJbSpoJBj8xQhgzxB4',
'description': None,
'type_prefix': 'eval-target',
'namespace': 'nemo-eval-tutorial',
'project': None,
'created_at': '2025-07-09T16:29:23.468547',
'updated_at': '2025-07-09T16:29:23.468547',
'custom_fields': {},
'ownership': None,
'name': 'rag-target',
'type': 'rag',
'cached_outputs': None,
'model': None,
'retriever': None,
'rag': {'pipeline': {'retriever': {'pipeline': {'query_embedding_model': {'schema_version': '1.0',
'id': 'model-FbFHZ6sEf7pBhmSZj2eTa5',
'description': None,
'type_prefix': 'model',
'namespace': 'default',
'project': None,
'created_at': '2025-07-09T16:29:23.468212',
'updated_at': '2025-07-09T16:29:23.468213',
'custom_fields': {},
'ownership': None,
'name': 'model-FbFHZ6sEf7pBhmSZj2eTa5',
'version_id': 'main',
'version_tags': [],
'spec': None,
'artifact': None,
'base_model': None,
'api_endpoint': {'url': 'http://172.27.20.120:8000/v1/embeddings',
'model_id': 'nvidia/llama-3.2-nv-embedqa-1b-v2',
'api_key': None,
'format': 'nim'},
'peft': None,
'prompt': None,
'guardrails': None},
'index_embedding_model': {'schema_version': '1.0',
'id': 'model-BMxFNMbexdzBy1PgDyMB2f',
'description': None,
'type_prefix': 'model',
'namespace': 'default',
'project': None,
'created_at': '2025-07-09T16:29:23.468261',
'updated_at': '2025-07-09T16:29:23.468261',
'custom_fields': {},
'ownership': None,
'name': 'model-BMxFNMbexdzBy1PgDyMB2f',
'version_id': 'main',
'version_tags': [],
'spec': None,
'artifact': None,
'base_model': None,
'api_endpoint': {'url': 'http://172.27.20.120:8000/v1/embeddings',
'model_id': 'nvidia/llama-3.2-nv-embedqa-1b-v2',
'api_key': None,
'format': 'nim'},
'peft': None,
'prompt': None,
'guardrails': None},
'reranker_model': {'schema_version': '1.0',
'id': 'model-JVYRG5Nxwu24cxXevSD1P',
'description': None,
'type_prefix': 'model',
'namespace': 'default',
'project': None,
'created_at': '2025-07-09T16:29:23.468282',
'updated_at': '2025-07-09T16:29:23.468282',
'custom_fields': {},
'ownership': None,
'name': 'model-JVYRG5Nxwu24cxXevSD1P',
'version_id': 'main',
'version_tags': [],
'spec': None,
'artifact': None,
'base_model': None,
'api_endpoint': {'url': 'http://172.27.20.120:8001/v1/ranking',
'model_id': 'nvidia/llama-3.2-nv-rerankqa-1b-v2',
'api_key': None,
'format': 'nim'},
'peft': None,
'prompt': None,
'guardrails': None},
'top_k': 3},
'cached_outputs': None},
'model': {'schema_version': '1.0',
'id': 'model-fWQ8215bWrwq9sdmSGnno',
'description': None,
'type_prefix': 'model',
'namespace': 'default',
'project': None,
'created_at': '2025-07-09T16:29:23.468307',
'updated_at': '2025-07-09T16:29:23.468307',
'custom_fields': {},
'ownership': None,
'name': 'model-fWQ8215bWrwq9sdmSGnno',
'version_id': 'main',
'version_tags': [],
'spec': None,
'artifact': None,
'base_model': None,
'api_endpoint': {'url': 'http://nim.test/v1/chat/completions',
'model_id': 'meta/llama-3.1-8b-instruct',
'api_key': None,
'format': 'nim'},
'peft': None,
'prompt': None,
'guardrails': None},
'context_ordering': 'desc'},
'cached_outputs': None},
'rows': None,
'dataset': None},
'config': {'schema_version': '1.0',
'id': 'eval-config-Lsvsipxj1cEntDWsDbzJxL',
'description': None,
'type_prefix': 'eval-config',
'namespace': 'nemo-eval-tutorial',
'project': None,
'created_at': '2025-07-09T16:30:31.105191',
'updated_at': '2025-07-09T16:30:31.105192',
'custom_fields': {},
'ownership': None,
'name': 'rag-nfcorpus-config',
'type': 'rag',
'params': None,
'tasks': {'my-beir-task': {'type': 'beir',
'params': {'judge_llm': {'api_endpoint': {'url': 'https://integrate.api.nvidia.com/v1/chat/completions',
'model_id': 'meta/llama-3.3-70b-instruct',
'api_key': '******'}},
'judge_embeddings': {'api_endpoint': {'url': 'https://integrate.api.nvidia.com/v1/embeddings',
'model_id': 'nvidia/nv-embedqa-e5-v5',
'api_key': '******'}},
'judge_timeout': 300,
'judge_max_retries': 5,
'judge_max_workers': 16},
'metrics': {'recall_5': {'type': 'recall_5',
'params': None},
'ndcg_cut_5': {'type': 'ndcg_cut_5',
'params': None},
'recall_10': {'type': 'recall_10',
'params': None},
'ndcg_cut_10': {'type': 'ndcg_cut_10',
'params': None},
'faithfulness': {'type': 'faithfulness',
'params': None},
'answer_relevancy': {'type': 'answer_relevancy',
'params': None}},
'dataset': {'schema_version': '1.0',
'id': 'dataset-3Tgpmmzu7u4AUUrfppFhsN',
'description': None,
'type_prefix': None,
'namespace': 'default',
'project': None,
'created_at': '2025-07-09T16:30:31.105030',
'updated_at': '2025-07-09T16:30:31.105032',
'custom_fields': {},
'ownership': None,
'name': 'dataset-3Tgpmmzu7u4AUUrfppFhsN',
'version_id': 'main',
'version_tags': [],
'format': None,
'files_url': 'file://nfcorpus/',
'hf_endpoint': None,
'split': None,
'limit': None}}},
'groups': None},
'result': None,
'output_files_url': None,
'status_details': {'message': None, 'task_status': {}, 'progress': None},
'status': 'created',
'project': None,
'custom_fields': {},
'ownership': None}
[54]
{'message': 'Job completed successfully',
'task_status': {'my-beir-task': 'completed'},
'progress': 100.0}
[55]
{'evaluation': {'scores': {'ndcg_cut_10': {'value': 0.2679049189207276},
'recall_5': {'value': 0.11898776243879489},
'recall_10': {'value': 0.11898776243879489},
'ndcg_cut_5': {'value': 0.35399013412606123},
'faithfulness': {'value': 0.798750415118147},
'answer_relevancy': {'value': 0.38691755209414236}}}}
6.2 Evaluate RAG Pipeline on Custom Dataset
Create evaluation config for RAG on custom dataset. We will use remote judge LLM and judge embedding model.
[56]
{'created_at': '2025-07-09T17:35:11.033333',
'updated_at': '2025-07-09T17:35:11.033334',
'name': 'custom-rag-config',
'namespace': 'nemo-eval-tutorial',
'type': 'rag',
'tasks': {'rag-beir-task': {'type': 'beir',
'params': {'judge_llm': {'api_endpoint': {'url': 'https://integrate.api.nvidia.com/v1/chat/completions',
'model_id': 'meta/llama-3.3-70b-instruct',
'api_key': '******'}},
'judge_embeddings': {'api_endpoint': {'url': 'https://integrate.api.nvidia.com/v1/embeddings',
'model_id': 'nvidia/nv-embedqa-e5-v5',
'api_key': '******'}},
'judge_timeout': 300,
'judge_max_retries': 5,
'judge_max_workers': 16},
'metrics': {'recall_5': {'type': 'recall_5'},
'ndcg_cut_5': {'type': 'ndcg_cut_5'},
'recall_10': {'type': 'recall_10'},
'ndcg_cut_10': {'type': 'ndcg_cut_10'},
'faithfulness': {'type': 'faithfulness'},
'answer_relevancy': {'type': 'answer_relevancy'}},
'dataset': {'schema_version': '1.0',
'id': 'dataset-4LzRVXqd494p5rSG5BeULA',
'namespace': 'default',
'created_at': '2025-07-09T17:35:11.033156',
'updated_at': '2025-07-09T17:35:11.033158',
'custom_fields': {},
'name': 'dataset-4LzRVXqd494p5rSG5BeULA',
'version_id': 'main',
'version_tags': [],
'files_url': 'hf://datasets/nemo-eval-tutorial/rag_custom_data/rag_data'}}},
'id': 'eval-config-7NBFBYfrYxGgcdqo5M7agU',
'custom_fields': {}}
[57]
{'created_at': '2025-07-09T17:35:43.792368',...
[60]
{'message': 'Job completed successfully',
'task_status': {'rag-beir-task': 'completed'},
'progress': 100.0}
[61]
{'evaluation': {'scores': {'ndcg_cut_10': {'value': 1.0},
'recall_10': {'value': 1.0},
'ndcg_cut_5': {'value': 1.0},
'recall_5': {'value': 1.0},
'faithfulness': {'value': 0.804586038961039},
'answer_relevancy': {'value': 0.5430273571543547}}}}
[ ]