Unified Tool Calling
Tool Use with 🤗 transformers
We begin by initializing our model as normal. We'll use a small, ungated model so you can run this on Colab - make sure you've got a GPU, though, because it won't fit in CPU memory alone!
/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:89: UserWarning: The secret `HF_TOKEN` does not exist in your Colab secrets. To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session. You will be able to reuse this secret in all of your notebooks. Please note that authentication is recommended but still optional to access public models or datasets. warnings.warn(
tokenizer_config.json: 0%| | 0.00/55.8k [00:00<?, ?B/s]
tokenizer.json: 0%| | 0.00/9.09M [00:00<?, ?B/s]
special_tokens_map.json: 0%| | 0.00/68.0 [00:00<?, ?B/s]
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
config.json: 0%| | 0.00/700 [00:00<?, ?B/s]
model.safetensors.index.json: 0%| | 0.00/23.9k [00:00<?, ?B/s]
Downloading shards: 0%| | 0/4 [00:00<?, ?it/s]
model-00001-of-00004.safetensors: 0%| | 0.00/4.98G [00:00<?, ?B/s]
model-00002-of-00004.safetensors: 0%| | 0.00/5.00G [00:00<?, ?B/s]
model-00003-of-00004.safetensors: 0%| | 0.00/4.92G [00:00<?, ?B/s]
model-00004-of-00004.safetensors: 0%| | 0.00/1.17G [00:00<?, ?B/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
generation_config.json: 0%| | 0.00/142 [00:00<?, ?B/s]
WARNING:accelerate.big_modeling:Some parameters are on the meta device device because they were offloaded to the cpu.
Next, we'll define a simple tool function for our model - but in a real scenario, you'll probably want to use more than one! The type hints and docstrings are mandatory - those are going to be parsed and used by the model to understand what the tools do.
Now, we set up a simple chat.
We pass the tools to the chat template, and generate text from the model using the formatted prompt.
Setting `pad_token_id` to `eos_token_id`:128003 for open-end generation.
<tool_call>
{"arguments": {"location": "Paris, France"}, "name": "get_current_temperature"}
</tool_call><|im_end|>
The model has chosen to call a tool, and picked an argument that matches both the user's request and the format in the tool docstring! Let's add this tool call to the chat as a message!
Next, we add the tool response containing the function output to the chat as well. Both the tool call (containing the arguments passed to the tool) and tool response (containing the tool's output) must be included in the chat history.
Finally, we apply the chat template and generate text once again.
Setting `pad_token_id` to `eos_token_id`:128003 for open-end generation.
The current temperature in Paris, France is 22.0 degrees Celsius. Enjoy your day!<|im_end|>
Success! The model has used the tool response correctly in its response to the user.