Notebooks
M
Milvus Hybrid Search With Milvus

Hybrid Search With Milvus

image-searchvector-databasesemantic-searchtutorialsmilvusembeddingsunstructured-dataquestion-answeringLLMmilvus-bootcampdeep-learningimage-recognitionimage-classificationaudio-searchPythonquickstartragNLP
alph-notebooks/milvus-bootcamp / hybrid_search_with_milvus.ipynb

Hybrid Search with Dense and Sparse Vectors in Milvus

If you want to experience the final effect of this tutorial, you can go directly to https://demos.milvus.io/hybrid-search/

In this tutorial, we will demonstrate how to conduct hybrid search with Milvus and BGE-M3 model. BGE-M3 model can convert text into dense and sparse vectors. Milvus supports storing both types of vectors in one collection, allowing for hybrid search that enhances the result relevance.

Milvus supports Dense, Sparse, and Hybrid retrieval methods:

  • Dense Retrieval: Utilizes semantic context to understand the meaning behind queries.
  • Sparse Retrieval: Emphasizes keyword matching to find results based on specific terms, equivalent to full-text search.
  • Hybrid Retrieval: Combines both Dense and Sparse approaches, capturing the full context and specific keywords for comprehensive search results.

By integrating these methods, the Milvus Hybrid Search balances semantic and lexical similarities, improving the overall relevance of search outcomes. This notebook will walk through the process of setting up and using these retrieval strategies, highlighting their effectiveness in various search scenarios.

Dependencies and Environment

[1]

Download Dataset

To demonstrate search, we need a corpus of documents. Let's use the Quora Duplicate Questions dataset and place it in the local directory.

Source of the dataset: First Quora Dataset Release: Question Pairs

[2]

Load and Prepare Data

We will load the dataset and prepare a small corpus for search.

[3]
What is the strongest Kevlar cord?

Use BGE-M3 Model for Embeddings

The BGE-M3 model can embed texts as dense and sparse vectors.

[4]
Fetching 30 files: 100%|██████████| 30/30 [00:00<00:00, 302473.85it/s]
Inference Embeddings: 100%|██████████| 32/32 [01:59<00:00,  3.74s/it]

Setup Milvus Collection and Index

We will set up the Milvus collection and create indices for the vector fields.

  • Setting the uri as a local file, e.g. "./milvus.db", is the most convenient method, as it automatically utilizes Milvus Lite to store all data in this file.
  • If you have large scale of data, say more than a million vectors, you can set up a more performant Milvus server on Docker or Kubernetes. In this setup, please use the server uri, e.g.http://localhost:19530, as your uri.
  • If you want to use Zilliz Cloud, the fully managed cloud service for Milvus, adjust the uri and token, which correspond to the Public Endpoint and API key in Zilliz Cloud.
[5]

Insert Data into Milvus Collection

Insert documents and their embeddings into the collection.

[6]
Number of entities inserted: 502

Enter Your Search Query

[7]
How to start learning programming?

Run the Search

We will first prepare some helpful functions to run the search:

  • dense_search: only search across dense vector field
  • sparse_search: only search across sparse vector field
  • hybrid_search: search across both dense and vector fields with a weighted reranker
[8]

Let's run three different searches with defined functions:

[9]

Display Search Results

To display the results for Dense, Sparse, and Hybrid searches, we need some utilities to format the results.

[10]

Then we can display search results in text with highlights:

[11]

Quick Deploy

To learn about how to start an online demo with this tutorial, please refer to the example application.