Getting Started with torchtune and Weigths & Biases

In this notebook you will learn how to use torchtune with Weights & Biases to monitor your training runs.

You need to select a machine a GPU, go to Runtime > Change runtime type > select a GPU (L40, A100 ideally)

Setup the libraries

Download a Model

We will download a model from the Hugging Face Hub.

you will need to provide an access token or call huggingface-cli login

Download a model checkpoint using the provided tune download CLI

Let's create a torchtune config that enables W&B, to do so, we can grab the original Mistral 7B LoRA recipe and change the following lines to use W&B as our metric_logger:

# Logging
metric_logger:
  _component_: torchtune.utils.metric_logging.WandBLogger # <---You only need this to enable W&B
  project: mistral_lora # <--- The W&B project to save our logs to
log_every_n_steps: 1

Let's save a modified version of the recipe using %%writefile:

Let's run the recipe with this modified config with W&B enabled

That's it! Now you can click on the URL and continue monitoring your training on the Weights & Biases UI