This Notebook accompanies the report titled "Tensorboard with Accelerators - A Guide"

Most of the code in this notebook is ported from the "Transfer learning and fine-tuning" tutorial from the Tensorflow website in order to demonstrate easy integration of the wandb client into pre-existing workflows.

Author: @SauravMaheshkar

Packages 📦 and Basic Setup

Install Packages

Project Configuration using wandb.config

💿 The Dataset

In this code cell we:-

• download and extract a zip file containing images
• create a tf.data.Dataset using tf.keras.preprocessing.image_dataset_from_directory

✍️ Model Architecture

🏗 Transfer Learning

Excerts from Datacamp Tutorials and MachineLearningMastery blogs

First things first; Transfer learning(TL) is not a machine learning model or technique; it is rather a "design methodology" within machine learning. The general idea of transfer learning is to use knowledge learned from tasks for which a lot of labelled data is available in settings where only little labelled data is available. Creating labelled data is expensive, so optimally leveraging existing datasets is key.

In a traditional machine learning model, the primary goal is to generalise to unseen data based on patterns learned from the training data. With transfer learning, you attempt to kickstart this generalisation process by starting from patterns that have been learned for a different task. Essentially, instead of starting the learning process from a (often randomly initialised) blank sheet, you start from patterns that have been learned to solve a different task.

Convolutional Neural Networks' features are more generic in early layers and more original-dataset-specific in later layers. Thus, we often use these as a backbone / starting point while creating new models. A common practice is to make these base models non-trainable and just learn the later layers. You might think that this will decrease the performance, but as we'll see from training. Transfer Learning is still a viable option.

⚖️ DenseNet121

From PyTorch Hub

Dense Convolutional Network (DenseNet), connects each layer to every other layer in a feed-forward fashion. Whereas traditional convolutional networks with $L$ layers have $L$ connections - one between each layer and its subsequent layer - our network has $L(L+1)/2$ direct connections. For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs into all subsequent layers. DenseNets have several compelling advantages: they alleviate the vanishing-gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters.

In this project we'll use DenseNet121 for training our Binary Classifier. The Model can easily be instantiated using the tf.keras.applications Module, which provides canned architectures with pre-trained weights. For more details kindly visit this link.

🧱 + 🏗 = 🏠 Training

W&B automatically logs metrics from TensorBoard into dashboards. If you've got a pre-existing training workflow based around Tensorboard, switching to wandb requires just 2 lines of code.

1. wandb.tensorboard.patch(root_logdir="...")
2. run = wandb.init(..., sync_tensorboard = True)

W&B client will then automatically log all the metrics generated during training from the logs directory into it's amazing dashboard.

You don't even have to worry about accelerator's !!!

Just create a distribute strategy for either GPUs or TPUs and wandb automatically logs all the system metrics like GPU Power Usage, GPU Memory Allocated, GPU Temp, etc.

Easy Model Versioning with W&B Artifacts 🏪

It's no secret that training deep learning models takes a lot of time, that's why we use Weights and Biases Artifacts to store our models weights and graphs for easy reproducibility.

W&B artifacts allows you to store different versions of your datasets and models in the cloud as Artifacts.

For example all the model files for this project can be found here