HuggingFace Int8 Demo
hf-bloghacktoberfestnotebooks
Export
HuggingFace meets bitsandbytes for lighter models on GPU for inference
You can run your own 8-bit model on any HuggingFace 🤗 model with just few lines of code. Install the dependencies below first!
[ ]
|████████████████████████████████| 55.8 MB 1.2 MB/s
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done
|████████████████████████████████| 101 kB 7.1 MB/s
|████████████████████████████████| 6.6 MB 59.1 MB/s
|████████████████████████████████| 596 kB 69.3 MB/s
Building wheel for transformers (PEP 517) ... done
|████████████████████████████████| 143 kB 34.4 MB/s
Hardware requirements 🔨
To run properly this feature you need to have GPU that supports 8-bit operation modules. Currently, Turing and Ampere GPUs (RTX20s, RTX30s, A40-A100, T4+) are supported, which means on colab we need to use a T4 GPU for this feature. You can check that using this code snippet and make sure you are using a supported GPU
[ ]
Mon Aug 8 09:10:10 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03 Driver Version: 460.32.03 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 38C P8 9W / 70W | 0MiB / 15109MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
Here we are using a Tesla T4 GPU that should support 8-bit tensor cores! We are good to go 🚀
Utility variables & functions 🧰
[ ]
Use 8bit models and pipeline 🤗
You can use 8bit quantized models together with pipeline as follows:
[ ]
The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling `transformers.utils.move_cache()`.
Moving 0 files to the new cache system
0it [00:00, ?it/s]
Downloading: 0%| | 0.00/710 [00:00<?, ?B/s]
Downloading: 0%| | 0.00/65.3k [00:00<?, ?B/s]
===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please use this form: https://docs.google.com/forms/d/e/1FAIpQLScPB8emS3Thkp66nvqwmjTEgxp8Y9ufuWTzFyr9kJ5AoI47dQ/viewform?usp=sf_link ================================================================================ CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64... CUDA SETUP: CUDA path found: /usr/local/cuda/lib64/libcudart.so CUDA SETUP: Highest compute capability among GPUs detected: 7.5 CUDA_SETUP: Detected CUDA version 111 CUDA_SETUP: Loading binary /usr/local/lib/python3.7/dist-packages/bitsandbytes/libbitsandbytes_cuda111.so...
/usr/local/lib/python3.7/dist-packages/bitsandbytes/cuda_setup/paths.py:111: UserWarning: /usr/lib64-nvidia did not contain libcudart.so as expected! Searching further paths...
f'{candidate_env_vars["LD_LIBRARY_PATH"]} did not contain '
WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('"172.28.0.3","jupyterArgs"'), PosixPath('{"kernelManagerProxyPort"'), PosixPath('"/usr/local/bin/dap_multiplexer","enableLsp"'), PosixPath('true}'), PosixPath('["--ip=172.28.0.2"],"debugAdapterMultiplexerPath"'), PosixPath('6000,"kernelManagerProxyHost"')}
WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/env/python')}
WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//ipykernel.pylab.backend_inline'), PosixPath('module')}
Downloading: 0%| | 0.00/6.01G [00:00<?, ?B/s]
Downloading: 0%| | 0.00/222 [00:00<?, ?B/s]
Downloading: 0%| | 0.00/14.5M [00:00<?, ?B/s]
Downloading: 0%| | 0.00/85.0 [00:00<?, ?B/s]
Let's check the output!
[ ]
[{'generated_text': 'Hello my name is John and I am a student at the University of the West of England. I am currently studying for'}] Use 8bit models and .generate 📖
[ ]
The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling `transformers.utils.move_cache()`.
Moving 0 files to the new cache system
0it [00:00, ?it/s]
Downloading: 0%| | 0.00/710 [00:00<?, ?B/s]
===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please use this form: https://docs.google.com/forms/d/e/1FAIpQLScPB8emS3Thkp66nvqwmjTEgxp8Y9ufuWTzFyr9kJ5AoI47dQ/viewform?usp=sf_link ================================================================================ CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64... CUDA SETUP: CUDA path found: /usr/local/cuda/lib64/libcudart.so CUDA SETUP: Highest compute capability among GPUs detected: 7.5 CUDA_SETUP: Detected CUDA version 111 CUDA_SETUP: Loading binary /usr/local/lib/python3.7/dist-packages/bitsandbytes/libbitsandbytes_cuda111.so...
/usr/local/lib/python3.7/dist-packages/bitsandbytes/cuda_setup/paths.py:111: UserWarning: /usr/lib64-nvidia did not contain libcudart.so as expected! Searching further paths...
f'{candidate_env_vars["LD_LIBRARY_PATH"]} did not contain '
WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('6000,"kernelManagerProxyHost"'), PosixPath('true}'), PosixPath('{"kernelManagerProxyPort"'), PosixPath('"/usr/local/bin/dap_multiplexer","enableLsp"'), PosixPath('["--ip=172.28.0.2"],"debugAdapterMultiplexerPath"'), PosixPath('"172.28.0.3","jupyterArgs"')}
WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/env/python')}
WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//ipykernel.pylab.backend_inline'), PosixPath('module')}
Downloading: 0%| | 0.00/6.01G [00:00<?, ?B/s]
Downloading: 0%| | 0.00/222 [00:00<?, ?B/s]
Downloading: 0%| | 0.00/14.5M [00:00<?, ?B/s]
Downloading: 0%| | 0.00/85.0 [00:00<?, ?B/s]
[ ]
/usr/local/lib/python3.7/dist-packages/transformers/generation_utils.py:1233: UserWarning: Neither `max_length` nor `max_new_tokens` has been set, `max_length` will default to 20 (`self.config.max_length`). Controlling `max_length` via the config is deprecated and `max_length` will be removed from the config in v5 of Transformers -- we recommend using `max_new_tokens` to control the maximum length of the generation. UserWarning,
'Hello my name is John and I am a student at the University of the West of England. I'
Let's compare the qualitative results between our quantized model and the original model
[ ]
/usr/local/lib/python3.7/dist-packages/transformers/generation_utils.py:1207: UserWarning: Neither `max_length` nor `max_new_tokens` have been set, `max_length` will default to 20 (`self.config.max_length`). Controlling `max_length` via the config is deprecated and `max_length` will be removed from the config in v5 of Transformers -- we recommend using `max_new_tokens` to control the maximum length of the generation. UserWarning,
'Hello my name is John and I am a student at the University of the West Indies. I am'
Memory footprint comparison 🪶
[ ]
Memory footprint int8 model: 3645818880 | Memory footprint fp16 model: 6005114880 | Relative difference: 1.6471237539918604
We saved 1.65x memory for a 3-billion parameters models! Note that internally we replace all the linear layers by the ones implemented in bitsandbytes. By scaling up the model the number of linear layers will increase therefore the impact of saving memory on those layers will be huge for very large models. For example quantizing BLOOM-176 (176 Billion parameter model) gives a gain of 1.96x memory footprint which can save a lot of compute power in practice.
Hyper-parameter tuning ðŸ“
Warning: you may want to run these cells separately from previous cells to avoid Out Of Memory (OOM) issues.
You can play with the parameter int8_threshold and see its impact in the results of your model. You can directly specify this parameter when loading your model through .from_pretrained method. By default we set this parameter to be 6.0 as described in the paper.
[ ]
Downloading config.json: 0%| | 0.00/710 [00:00<?, ?B/s]
Downloading pytorch_model.bin: 0%| | 0.00/5.59G [00:00<?, ?B/s]
[ ]
'Hello my name is John and I am a student at the University of the West Indies. I am'
[ ]
'Hello my name is John and I am a newbie to the forum. I have a question about'
As you can see the generations can slightly vary by using different thresholds. This is because manipulating 8-bit parameters leads to easier perturbations by small changes! Lowering the threshold means also less parameters in fp16 so breaking down the threshold to 0 leads to a full model in int8.