Llama3.2 (11B) Vision
To run this, press "Runtime" and press "Run all" on a free Tesla T4 Google Colab instance!
To install Unsloth on your local device, follow our guide. This notebook is licensed LGPL-3.0.
You will learn how to do data prep, how to train, how to run the model, & how to save it
News
Train MoEs - DeepSeek, GLM, Qwen and gpt-oss 12x faster with 35% less VRAM. Blog
You can now train embedding models 1.8-3.3x faster with 20% less VRAM. Blog
Ultra Long-Context Reinforcement Learning is here with 7x more context windows! Blog
3x faster LLM training with 30% less VRAM and 500K context. 3x faster • 500K Context
New in Reinforcement Learning: FP8 RL • Vision RL • Standby • gpt-oss RL
Visit our docs for all our model uploads and notebooks.
Installation
Unsloth
🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning. 🦥 Unsloth Zoo will now patch everything to make training faster! ==((====))== Unsloth 2024.11.9: Fast Mllama vision patching. Transformers = 4.46.2. \\ /| GPU: Tesla T4. Max memory: 14.748 GB. Platform = Linux. O^O/ \_/ \ Pytorch: 2.5.1+cu121. CUDA = 7.5. CUDA Toolkit = 12.1. \ / Bfloat16 = FALSE. FA [Xformers = 0.0.28.post3. FA2 = False] "-____-" Free Apache license: http://github.com/unslothai/unsloth Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
model.safetensors.index.json: 0%| | 0.00/385k [00:00<?, ?B/s]
Downloading shards: 0%| | 0/2 [00:00<?, ?it/s]
model-00001-of-00002.safetensors: 0%| | 0.00/5.00G [00:00<?, ?B/s]
model-00002-of-00002.safetensors: 0%| | 0.00/2.18G [00:00<?, ?B/s]
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
generation_config.json: 0%| | 0.00/210 [00:00<?, ?B/s]
preprocessor_config.json: 0%| | 0.00/477 [00:00<?, ?B/s]
tokenizer_config.json: 0%| | 0.00/55.9k [00:00<?, ?B/s]
tokenizer.json: 0%| | 0.00/17.2M [00:00<?, ?B/s]
special_tokens_map.json: 0%| | 0.00/454 [00:00<?, ?B/s]
chat_template.json: 0%| | 0.00/5.15k [00:00<?, ?B/s]
We now add LoRA adapters for parameter efficient finetuning - this allows us to only efficiently train 1% of all parameters.
[NEW] We also support finetuning ONLY the vision part of the model, or ONLY the language part. Or you can select both! You can also select to finetune the attention or the MLP layers!
Data Prep
We'll be using a sampled version of the ROCO radiography dataset. You can access the dataset here. The full dataset is here.
The dataset includes X-rays, CT scans and ultrasounds showcasing medical conditions and diseases. Each image has a caption written by experts describing it. The goal is to finetune a VLM to make it a useful analysis tool for medical professionals.
README.md: 0%| | 0.00/610 [00:00<?, ?B/s]
train-00000-of-00001.parquet: 0%| | 0.00/481M [00:00<?, ?B/s]
test-00000-of-00001.parquet: 0%| | 0.00/79.2M [00:00<?, ?B/s]
Generating train split: 0%| | 0/1978 [00:00<?, ? examples/s]
Generating test split: 0%| | 0/327 [00:00<?, ? examples/s]
Let's take a look at the dataset, and check what the 1st example shows:
Dataset({
, features: ['image', 'image_id', 'caption', 'cui'],
, num_rows: 1978
,}) 
'Panoramic radiography shows an osteolytic lesion in the right posterior maxilla with resorption of the floor of the maxillary sinus (arrows).'
To format the dataset, all vision finetuning tasks should be formatted as follows:
[
{ "role": "user",
"content": [{"type": "text", "text": instruction}, {"type": "image", "image": image} ]
},
{ "role": "assistant",
"content": [{"type": "text", "text": answer} ]
},
]
We will craft an custom instruction asking the VLM to be an expert radiographer. Notice also instead of just 1 instruction, you can add multiple turns to make it a dynamic conversation.
Let's convert the dataset into the "correct" format for finetuning:
The first example is now structured like below:
{'messages': [{'role': 'user',
, 'content': [{'type': 'text',
, 'text': 'You are an expert radiographer. Describe accurately what you see in this image.'},
, {'type': 'image',
, 'image': <PIL.PngImagePlugin.PngImageFile image mode=L size=657x442>}]},
, {'role': 'assistant',
, 'content': [{'type': 'text',
, 'text': 'Panoramic radiography shows an osteolytic lesion in the right posterior maxilla with resorption of the floor of the maxillary sinus (arrows).'}]}]} Before we do any finetuning, maybe the vision model already knows how to analyse the images? Let's check if this is the case!
This radiograph appears to be a panoramic view of the upper and lower dentition, specifically an Orthopantomogram (OPG). * The panoramic radiograph demonstrates normal dental structures. * There is an abnormal area on the upper right, represented by an area of radiolucent bone, corresponding to the antrum. **Key Observations** * The bone between the left upper teeth is relatively radiopaque. * There are two large arrows above the image, suggesting the need for a closer examination of this area. One of the arrows is in a left-sided position, and the other is in the right-sided position. However, only
Train the model
Now let's train our model. We do 60 steps to speed things up, but you can set num_train_epochs=1 for a full run, and turn off max_steps=None. We also support DPOTrainer and GRPOTrainer for reinforcement learning!!
We use our new UnslothVisionDataCollator which will help in our vision finetuning setup.
max_steps is given, it will override any value given in num_train_epochs
GPU = Tesla T4. Max memory = 14.748 GB. 8.477 GB of memory reserved.
==((====))== Unsloth - 2x faster free finetuning | Num GPUs = 1 \\ /| Num examples = 1,978 | Num Epochs = 1 O^O/ \_/ \ Batch size per device = 2 | Gradient Accumulation steps = 4 \ / Total batch size = 8 | Total steps = 30 "-____-" Number of trainable parameters = 67,174,400 🦥 Unsloth needs about 1-3 minutes to load everything - please wait!
957.9653 seconds used for training. 15.97 minutes used for training. Peak reserved memory = 10.287 GB. Peak reserved memory for training = 1.81 GB. Peak reserved memory % of max memory = 69.752 %. Peak reserved memory for training % of max memory = 12.273 %.
Inference
Let's run the model! You can change the instruction and input - leave the output blank!
We use min_p = 0.1 and temperature = 1.5. Read this Tweet for more information on why.
Panoramic radiography shows multiple small osteolytic lesions in the right mandibular ramus, condyle and ascending ramus, the left mandibular condyle, left first molar and body, right first and second premolar, left first, second and third molars (arrows).<|eot_id|>
[]
Now if you want to load the LoRA adapters we just saved for inference, set False to True:
Panoramic radiograph of the jaw showing well-defined radiolucency (arrowheads) of the right side and unerupted second permanent maxillary molar (arrow).<|eot_id|>
Saving to float16 for VLLM
We also support saving to float16 directly. Select merged_16bit for float16. Use push_to_hub_merged to upload to your Hugging Face account! You can go to https://huggingface.co/settings/tokens for your personal tokens. See our docs for more deployment options.
And we're done! If you have any questions on Unsloth, we have a Discord channel! If you find any bugs or want to keep updated with the latest LLM stuff, or need help, join projects etc, feel free to join our Discord!
Some other resources:
- Looking to use Unsloth locally? Read our Installation Guide for details on installing Unsloth on Windows, Docker, AMD, Intel GPUs.
- Learn how to do Reinforcement Learning with our RL Guide and notebooks.
- Read our guides and notebooks for Text-to-speech (TTS) and vision model support.
- Explore our LLM Tutorials Directory to find dedicated guides for each model.
- Need help with Inference? Read our Inference & Deployment page for details on using vLLM, llama.cpp, Ollama etc.
