Text Embeddings Tutorial
image-searchvector-databasesemantic-searchmilvusembeddingsunstructured-dataquestion-answeringLLMmilvus-bootcampdeep-learningimage-recognitionimage-classificationaudio-searchPythonbootcampragNLP
Export
Text Embeddings with sentence-transformers
We'll start off by installing some dependencies: sentence-transformers for the models and milvus for the vector database. Milvus is known for its scalability and wide adoption among organiziations, but we have an "embedded" version too!
[1]
Requirement already satisfied: sentence_transformers in /Users/zilliz/.pyenv/lib/python3.9/site-packages (2.2.2) Requirement already satisfied: torchvision in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from sentence_transformers) (0.16.2) Requirement already satisfied: numpy in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from sentence_transformers) (1.26.2) Requirement already satisfied: sentencepiece in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from sentence_transformers) (0.1.99) Requirement already satisfied: scikit-learn in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from sentence_transformers) (1.3.2) Requirement already satisfied: torch>=1.6.0 in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from sentence_transformers) (2.1.2) Requirement already satisfied: huggingface-hub>=0.4.0 in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from sentence_transformers) (0.19.4) Requirement already satisfied: nltk in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from sentence_transformers) (3.8.1) Requirement already satisfied: tqdm in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from sentence_transformers) (4.66.1) Requirement already satisfied: transformers<5.0.0,>=4.6.0 in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from sentence_transformers) (4.36.1) Requirement already satisfied: scipy in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from sentence_transformers) (1.12.0) Requirement already satisfied: fsspec>=2023.5.0 in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from huggingface-hub>=0.4.0->sentence_transformers) (2023.10.0) Requirement already satisfied: requests in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from huggingface-hub>=0.4.0->sentence_transformers) (2.31.0) Requirement already satisfied: packaging>=20.9 in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from huggingface-hub>=0.4.0->sentence_transformers) (23.2) Requirement already satisfied: typing-extensions>=3.7.4.3 in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from huggingface-hub>=0.4.0->sentence_transformers) (4.8.0) Requirement already satisfied: pyyaml>=5.1 in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from huggingface-hub>=0.4.0->sentence_transformers) (6.0.1) Requirement already satisfied: filelock in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from huggingface-hub>=0.4.0->sentence_transformers) (3.13.1) Requirement already satisfied: sympy in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from torch>=1.6.0->sentence_transformers) (1.12) Requirement already satisfied: networkx in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from torch>=1.6.0->sentence_transformers) (3.2.1) Requirement already satisfied: jinja2 in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from torch>=1.6.0->sentence_transformers) (3.1.2) Requirement already satisfied: safetensors>=0.3.1 in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from transformers<5.0.0,>=4.6.0->sentence_transformers) (0.4.1) Requirement already satisfied: regex!=2019.12.17 in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from transformers<5.0.0,>=4.6.0->sentence_transformers) (2023.10.3) Requirement already satisfied: tokenizers<0.19,>=0.14 in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from transformers<5.0.0,>=4.6.0->sentence_transformers) (0.15.0) Requirement already satisfied: MarkupSafe>=2.0 in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from jinja2->torch>=1.6.0->sentence_transformers) (2.1.3) Requirement already satisfied: joblib in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from nltk->sentence_transformers) (1.3.2) Requirement already satisfied: click in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from nltk->sentence_transformers) (8.1.7) Requirement already satisfied: idna<4,>=2.5 in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from requests->huggingface-hub>=0.4.0->sentence_transformers) (3.6) Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from requests->huggingface-hub>=0.4.0->sentence_transformers) (2.1.0) Requirement already satisfied: charset-normalizer<4,>=2 in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from requests->huggingface-hub>=0.4.0->sentence_transformers) (3.3.2) Requirement already satisfied: certifi>=2017.4.17 in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from requests->huggingface-hub>=0.4.0->sentence_transformers) (2023.11.17) Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from scikit-learn->sentence_transformers) (3.2.0) Requirement already satisfied: mpmath>=0.19 in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from sympy->torch>=1.6.0->sentence_transformers) (1.3.0) Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /Users/zilliz/.pyenv/lib/python3.9/site-packages (from torchvision->sentence_transformers) (10.1.0) WARNING: You are using pip version 21.2.4; however, version 23.3.2 is available. You should consider upgrading via the '/Users/zilliz/.pyenv/bin/python3 -m pip install --upgrade pip' command. Requirement already satisfied: milvus in /Users/zilliz/.pyenv/lib/python3.9/site-packages (2.3.5) WARNING: You are using pip version 21.2.4; however, version 23.3.2 is available. You should consider upgrading via the '/Users/zilliz/.pyenv/bin/python3 -m pip install --upgrade pip' command.
We'll go over the basics first: specifying a model and computing its embeddings.
[22]
[23]
(1024,)
[24]
tensor([[0.9391, 0.5762, 0.5002, 0.3822, 0.3003]])
[14]
tensor([[1.0000]])
[15]
tensor([[0.8867]])
Now let's check out how to fine-tune our model.
[9]
[10]
[11]
Epoch: 0%| | 0/1 [00:00<?, ?it/s]
Iteration: 0%| | 0/2 [00:00<?, ?it/s]
How about inserting into a vector database?
[16]
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... To disable this warning, you can either: - Avoid using `tokenizers` before the fork if possible - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
[17]
[18]
[19]
(insert count: 5, delete count: 0, upsert count: 0, timestamp: 447271173950799876, success count: 5, err count: 0)
[20]
[ ]