2 Finetuning And Inference
Part II: LoRA Fine-tuning Using NeMo Customizer
This notebook covers the following:
Before you proceed, make sure that you completed the first notebook on data preparation to obtain the assets required to follow along.
Configure NeMo Microservices Endpoints
This section includes importing required libraries, configuring endpoints, and performing health checks to ensure that the NeMo Data Store, NIM, and other services are running correctly.
Data Store endpoint: http://data-store.test Entity Store, Customizer, Evaluator endpoint: http://nemo.test NIM endpoint: http://nim.test Namespace: xlam-tutorial-ns Base Model for Customization: meta/llama-3.2-1b-instruct@v1.0.0+80GB
Configure Path to Prepared data
The following code sets the paths to the prepared dataset files.
Resource Organization Using Namespace
You can use a namespace to isolate and organize the artifacts in this tutorial.
Create Namespace
Both Data Store and Entity Store use namespaces. The following code creates namespaces for the tutorial.
Created namespace in Entity Store: xlam-tutorial-ns Data Store namespace creation response: <Response [201]>
Verify Namespaces
The following Data Store API and Entity Store API list the namespace created in the previous cell.
Data Store - Status Code: 201
Response JSON: {'namespace': 'xlam-tutorial-ns', 'created_at': '2025-06-20T03:56:39Z', 'updated_at': '2025-06-20T03:56:39Z'}
Entity Store - Namespace: xlam-tutorial-ns
Created at: 2025-06-20 03:56:39.457820
Description: None
Project: None
Tips: To list all available namespaces use
requests.get(f"{NDS_URL}/v1/datastore/namespaces/") # For Data Store
nemo_client.namespaces.list() # For Entity Store
To delete a namespace use:
requests.delete(f"{NDS_URL}/v1/datastore/namespaces/{namespace}") # For Data Store
nemo_client.namespaces.delete(namespace) # For Entity Store
The NeMo Data Store supports data management using the Hugging Face HfApi Client.
Note that this step does not interact with Hugging Face at all, it just uses the client library to interact with NeMo Data Store. This is in comparison to the previous notebook, where we used the load_dataset API to download the xLAM dataset from Hugging Face's repository.
More information can be found in documentation
1.1 Create Repository
RepoUrl('datasets/xlam-tutorial-ns/xlam-ft-dataset', endpoint='http://data-store.test/v1/hf', repo_type='dataset', repo_id='xlam-tutorial-ns/xlam-ft-dataset') Next, creating a dataset programmatically requires two steps: uploading and registration. More information can be found in documentation.
1.2 Upload Dataset Files to NeMo Data Store
training.jsonl: 0%| | 0.00/6.06M [00:00<?, ?B/s]
validation.jsonl: 0%| | 0.00/1.30M [00:00<?, ?B/s]
xlam-test-single.jsonl: 0%| | 0.00/1.19M [00:00<?, ?B/s]
CommitInfo(commit_url='', commit_message='Upload testing/xlam-test-single.jsonl with huggingface_hub', commit_description='', oid='e55b3211ba39e6ce80dd8a03a79600183e900f1c', pr_url=None, repo_url=RepoUrl('', endpoint='https://huggingface.co', repo_type='model', repo_id=''), pr_revision=None, pr_num=None) Other tips:
- Take a look at the
path_in_repoargument above. If there are more than one files in the subfolders:- All the .jsonl files in
training/will be merged and used for training by customizer. - All the .jsonl files in
validation/will be merged and used for validation by customizer.
- All the .jsonl files in
- NeMo Data Store generally supports data management using the HfApi API. For example, to delete a repo, you may use -
hf_api.delete_repo(
repo_id=repo_id,
repo_type="dataset"
)
1.3 Register the Dataset with NeMo Entity Store
To use a dataset for operations such as evaluations and customizations, register a dataset using the nemo_client.datasets.create() method.
Register the dataset to refer to it by its namespace and name afterward.
Created dataset: xlam-tutorial-ns/xlam-ft-dataset
Dataset(files_url='hf://datasets/xlam-tutorial-ns/xlam-ft-dataset', id='dataset-3G75hURMVmSLfNahcqQZd5', created_at=datetime.datetime(2025, 6, 20, 3, 58, 16, 182000), custom_fields={}, description='Tool calling xLAM dataset in OpenAI ChatCompletions format', format=None, hf_endpoint=None, limit=None, name='xlam-ft-dataset', namespace='xlam-tutorial-ns', project='tool_calling', split=None, updated_at=datetime.datetime(2025, 6, 20, 3, 58, 16, 182001)) Files URL: hf://datasets/xlam-tutorial-ns/xlam-ft-dataset
2.1 Start the Training Job
Start the training job by calling nemo_client.customization.jobs.create() method.
The following code sets the training parameters and starts the job.
The training job will take approximately 45 minutes to complete.
Created customization job: cust-FarcM8gwhL1XFDXQ57qGLL
CustomizationJobOutput(config='meta/llama-3.2-1b-instruct@v1.0.0+80GB', config_snapshot=ConfigSnapshot(base_model='meta/llama-3.2-1b-instruct', max_seq_length=4096, precision='bf16-mixed', training_option=CustomizationTrainingOption(finetuning_type='lora', micro_batch_size=1, num_gpus=1, training_type='sft', data_parallel_size=1, num_nodes=1, pipeline_parallel_size=1, tensor_parallel_size=1, use_sequence_parallel=False), chat_prompt_template=None, name=None, namespace=None, prompt_template='{prompt} {completion}', tokenizer=None), dataset='xlam-tutorial-ns/xlam-ft-dataset', hyperparameters=Hyperparameters(finetuning_type='lora', batch_size=16, distillation=None, epochs=2, learning_rate=0.0001, log_every_n_steps=None, lora=Lora(adapter_dim=32, adapter_dropout=0.1, alpha=16, target_modules=None), p_tuning=None, sequence_packing_enabled=False, sft=None, training_type='sft', val_check_interval=None, weight_decay=None), id='cust-FarcM8gwhL1XFDXQ57qGLL', created_at=datetime.datetime(2025, 6, 20, 4, 20, 21, 387984), dataset_parameters=None, description=None, integrations=None, namespace='default', output_model='xlam-tutorial-ns/llama-3.2-1b-xlam-run1@cust-FarcM8gwhL1XFDXQ57qGLL', ownership=None, project=None, status='created', status_details={'created_at': '2025-06-20T04:20:22.061480', 'updated_at': '2025-06-20T04:20:22.061480', 'elapsed_time': 0.0, 'steps_completed': 0, 'epochs_completed': 0, 'percentage_done': 0.0, 'status_logs': [{'updated_at': '2025-06-20T04:20:22.061480', 'message': 'created'}]}, updated_at=datetime.datetime(2025, 6, 20, 4, 20, 21, 387989), warnings=None) Note: In the snippet above, the model name and version are passed directly in the config argument. However, in production environments, administrators typically create customization targets and corresponding configs. This approach allows you to configure once and reuse model configurations for multiple customization jobs. In such cases, you simply reference the created configuration in the config argument. For more details, refer to the documentation.
The following code sets variables for storing the job ID and customized model name.
Tips:
-
If you configured the NeMo Customizer microservice with your own Weights & Biases (WandB) API key, you can find the training graphs and logs in your WandB account, "nvidia-nemo-customizer" project. Your run ID is similar to your customization
JOB_ID. -
To cancel a job that you scheduled incorrectly, run the following code.
nemo_client.customization.jobs.cancel(job_id=JOB_ID)
2.2 Get Job Status
Get the job status by using the nemo_client.customization.jobs.status() method.
The following code sets the job ID and sends the request.
Percentage done: 100.0
Job Status: {
"created_at": "2025-06-20 04:20:22.061480",
"status": "failed",
"updated_at": "2025-06-20 04:46:50.376819",
"best_epoch": 2,
"elapsed_time": 0.0,
"epochs_completed": 2,
"metrics": {
"keys": [
"train_loss",
"val_loss"
],
"metrics": {
"train_loss": [
{
"step": 9,
"timestamp": "2025-06-20T04:32:50.973138",
"value": 1.8576767444610596
},
{
"step": 19,
"timestamp": "2025-06-20T04:33:08.341909",
"value": 0.5704246759414673
},
{
"step": 29,
"timestamp": "2025-06-20T04:33:26.712158",
"value": 0.12280075997114182
},
{
"step": 39,
"timestamp": "2025-06-20T04:33:44.524580",
"value": 0.04450385272502899
},
{
"step": 49,
"timestamp": "2025-06-20T04:34:02.972717",
"value": 0.16312028467655182
},
{
"step": 59,
"timestamp": "2025-06-20T04:34:21.251629",
"value": 0.06606246531009674
},
{
"step": 69,
"timestamp": "2025-06-20T04:34:38.958870",
"value": 0.06840228289365768
},
{
"step": 79,
"timestamp": "2025-06-20T04:34:56.712386",
"value": 0.014794548973441124
},
{
"step": 89,
"timestamp": "2025-06-20T04:35:14.584432",
"value": 0.06460676342248917
},
{
"step": 99,
"timestamp": "2025-06-20T04:35:33.584602",
"value": 0.06629155576229095
},
{
"step": 109,
"timestamp": "2025-06-20T04:35:50.791133",
"value": 0.0795985758304596
},
{
"step": 119,
"timestamp": "2025-06-20T04:36:08.793632",
"value": 0.0649154782295227
},
{
"step": 129,
"timestamp": "2025-06-20T04:36:27.129566",
"value": 0.03526812419295311
},
{
"step": 139,
"timestamp": "2025-06-20T04:36:44.795064",
"value": 0.05382717028260231
},
{
"step": 149,
"timestamp": "2025-06-20T04:37:02.464071",
"value": 0.06577281653881073
},
{
"step": 159,
"timestamp": "2025-06-20T04:37:19.491135",
"value": 0.07506256550550461
},
{
"step": 169,
"timestamp": "2025-06-20T04:37:37.801428",
"value": 0.05599074810743332
},
{
"step": 179,
"timestamp": "2025-06-20T04:37:55.431594",
"value": 0.05787814036011696
},
{
"step": 189,
"timestamp": "2025-06-20T04:38:13.119979",
"value": 0.12088055908679962
},
{
"step": 199,
"timestamp": "2025-06-20T04:38:32.153647",
"value": 0.028641901910305023
},
{
"step": 209,
"timestamp": "2025-06-20T04:38:49.627884",
"value": 0.02677938900887966
},
{
"step": 219,
"timestamp": "2025-06-20T04:39:47.214580",
"value": 0.040931180119514465
},
{
"step": 229,
"timestamp": "2025-06-20T04:40:04.984682",
"value": 0.046923473477363586
},
{
"step": 239,
"timestamp": "2025-06-20T04:40:22.354650",
"value": 0.04601616412401199
},
{
"step": 249,
"timestamp": "2025-06-20T04:40:40.309953",
"value": 0.017167748883366585
},
{
"step": 259,
"timestamp": "2025-06-20T04:40:58.106455",
"value": 0.05223283916711807
},
{
"step": 269,
"timestamp": "2025-06-20T04:41:15.422867",
"value": 0.02956080622971058
},
{
"step": 279,
"timestamp": "2025-06-20T04:41:32.624669",
"value": 0.008285900577902794
},
{
"step": 289,
"timestamp": "2025-06-20T04:41:50.030973",
"value": 0.04720735549926758
},
{
"step": 299,
"timestamp": "2025-06-20T04:42:08.389922",
"value": 0.045284420251846313
},
{
"step": 309,
"timestamp": "2025-06-20T04:42:25.731052",
"value": 0.018354125320911407
},
{
"step": 319,
"timestamp": "2025-06-20T04:42:42.964505",
"value": 0.030467215925455093
},
{
"step": 329,
"timestamp": "2025-06-20T04:43:00.917284",
"value": 0.07479127496480942
},
{
"step": 339,
"timestamp": "2025-06-20T04:43:18.372081",
"value": 0.00976032018661499
},
{
"step": 349,
"timestamp": "2025-06-20T04:43:36.098891",
"value": 0.005889382213354111
},
{
"step": 359,
"timestamp": "2025-06-20T04:43:54.259327",
"value": 0.006259406916797161
},
{
"step": 369,
"timestamp": "2025-06-20T04:44:11.483810",
"value": 0.005117898806929588
},
{
"step": 379,
"timestamp": "2025-06-20T04:44:28.295927",
"value": 0.025533676147460938
},
{
"step": 389,
"timestamp": "2025-06-20T04:44:46.095441",
"value": 0.00949533749371767
},
{
"step": 399,
"timestamp": "2025-06-20T04:45:04.837562",
"value": 0.02385464310646057
},
{
"step": 409,
"timestamp": "2025-06-20T04:45:22.326863",
"value": 0.034205105155706406
},
{
"step": 419,
"timestamp": "2025-06-20T04:45:39.639773",
"value": 0.029642079025506973
},
{
"step": 429,
"timestamp": "2025-06-20T04:45:57.049268",
"value": 0.011566996574401855
}
],
"val_loss": [
{
"step": 218,
"timestamp": "2025-06-20T04:39:44.782580",
"value": 0.048260852694511414
},
{
"step": 437,
"timestamp": "2025-06-20T04:46:50.015687",
"value": 0.044774752110242844
}
]
}
},
"percentage_done": 100.0,
"status_logs": [
{
"updated_at": "2025-06-20 04:20:22.061480",
"detail": null,
"message": "created"
}
],
"steps_completed": 438,
"steps_per_epoch": 219,
"train_loss": 0.005117898806929588,
"val_loss": 0.044774752110242844
}
IMPORTANT: At this point, the customization job should be completed. If waiting for the job to finish failed or the status is not "completed", please check the logs (job.status_details.status_logs).
2.3 Validate Availability of Custom Model
The following NeMo Entity Store API should display the model when the training job is complete.
The list below shows all models filtered by your namespace and sorted by the latest first.
For more information about this API, see the NeMo Entity Store API reference.
With the following code, you can find all customized models, including the one trained in the previous cells.
Look for the name fields in the output, which should match your CUSTOMIZED_MODEL.
Found 1 models in namespace xlam-tutorial-ns: Model: llama-3.2-1b-xlam-run1@cust-FarcM8gwhL1XFDXQ57qGLL Namespace: xlam-tutorial-ns Base Model: meta/llama-3.2-1b-instruct Created: 2025-06-20 04:20:22.162792 Fine-tuning Type: lora
The customized model can also be retrieved directly by using its name.
Model: xlam-tutorial-ns/llama-3.2-1b-xlam-run1@cust-FarcM8gwhL1XFDXQ57qGLL Base Model: meta/llama-3.2-1b-instruct Status: upload_completed
NVIDIA NIM directly picks up the LoRA adapters from NeMo Entity Store. You can also query the NIM endpoint to look for it, as shown in the following code.
Step 3: Sanity Test the Customized Model By Running Sample Inference
Once the model is customized, its adapter is automatically saved in NeMo Entity Store and is ready to be picked up by NVIDIA NIM. You can test the model by sending a prompt to its NIM endpoint.
First, choose one of the examples from the test set.
3.1 Get Test Data Sample
There are 713 examples in the test set
([{'role': 'user',
, 'content': "Calculate the integral of the function 'x^2 + 3x + 2' from 0 to 10 using the trapezoidal rule."}],
, [{'type': 'function',
, 'function': {'name': 'merge_sorted_lists',
, 'description': 'Merges two sorted lists into a single sorted list.',
, 'parameters': {'type': 'object',
, 'properties': {'list1': {'description': 'The first sorted list.',
, 'type': 'array'},
, 'list2': {'description': 'The second sorted list.', 'type': 'array'}}}}},
, {'type': 'function',
, 'function': {'name': 'is_power_of_two',
, 'description': 'Checks if a number is a power of two.',
, 'parameters': {'type': 'object',
, 'properties': {'num': {'description': 'The number to check.',
, 'type': 'integer'}}}}},
, {'type': 'function',
, 'function': {'name': 'trapezoidal_integration',
, 'description': 'Calculates the definite integral of a function using the trapezoidal rule.',
, 'parameters': {'type': 'object',
, 'properties': {'func': {'description': 'The function to integrate, expressed as a string (e.g., "x**2 + 2*x").',
, 'type': 'string',
, 'default': 10000},
, 'a': {'description': 'The lower limit of integration.',
, 'type': 'number',
, 'default': 10000},
, 'b': {'description': 'The upper limit of integration.',
, 'type': 'number'},
, 'n': {'description': 'The number of subdivisions for the trapezoidal approximation. Defaults to 10000.',
, 'type': 'integer'}}}}}]) 3.2 Send an Inference Call to NIM
NIM exposes an OpenAI-compatible completions API endpoint, which you can query using the OpenAI client library as shown in the following code.
[ChatCompletionMessageToolCall(id='chatcmpl-tool-1c74f457beee40398662f22c6aaede86', function=Function(arguments='{"a": 56, "b": 98}', name='greatest_common_divisor'), type='function')] The Python SDK also supports the same inference call, as shown in the following code.
[ChoiceMessageToolCall(id='chatcmpl-tool-c616cace9d8e4693a7aa514ef4c6a31a', function=Function(arguments='{"a": 56, "b": 98}', name='greatest_common_divisor'), type='function')] Given that the fine-tuning job was successful, you can get an inference result comparable to the ground truth:
[{'type': 'function',
, 'function': {'name': 'trapezoidal_integration',
, 'arguments': {'func': 'x**2 + 3*x + 2', 'a': 0, 'b': 10}}}] Note: In production environments, application developers typically provide their own set of tools relevant to the specific task. The model must select from these tools based on the given query. To explore this further, you can sample a data point from the dataset to see which tools are available, then experiment by constructing a query and observing the model’s response.
3.3 Take Note of Your Custom Model Name
Take note of your custom model name, as you will use it to run evaluations in the subsequent notebook.
Name of your custom model is: xlam-tutorial-ns/llama-3.2-1b-xlam-run1@cust-FarcM8gwhL1XFDXQ57qGLL