Semantic Search with Milvus and VoyageAI

This guide showcases how VoyageAI's Embedding API can be used with Milvus vector database to conduct semantic search on text.

Getting started

Before you start, make sure you have the Voyage API key ready, or you get one from the VoyageAI website.

The data used in this example are book titles. You can download the dataset here and put it in the same directory where you run the following code.

First, install the package for Milvus and Voyage AI:

If you are using Google Colab, to enable dependencies just installed, you may need to restart the runtime (click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).

With this, we're ready to generate embeddings and use vector database to conduct semantic search.

Searching book titles with VoyageAI & Milvus

In the following example, we load book title data from the downloaded CSV file, use Voyage AI embedding model to generate vector representations, and store them in Milvus vector database for semantic search.

As for the argument of MilvusClient:

• Setting the uri as a local file, e.g../milvus.db, is the most convenient method, as it automatically utilizes Milvus Lite to store all data in this file.
• If you have large scale of data, you can set up a more performant Milvus server on docker or kubernetes. In this setup, please use the server uri, e.g.http://localhost:19530, as your uri.
• If you want to use Zilliz Cloud, the fully managed cloud service for Milvus, adjust the uri and token, which correspond to the Public Endpoint and Api key in Zilliz Cloud.

With all data in Milvus vector database, we can now perform semantic search by generating vector embedding for the query and conduct vector search.