Load JSON and Metadata filter JSON field or array

In this notebook, we are going to use Kaggle IMDB data, available as either raw JSON or CSV. We'll load it into Milvus vector database, then search with a metadata filter.

Let's get started!

Read JSON data into a pandas dataframe

The JSON data comes from https://www.kaggle.com/datasets/nelepie/imdb-genre-classification

Read CSV data into a pandas dataframe

The data used in this notebook is Kaggle 48K movies which contains a lot of metadata in addition to the raw review text.

Usually there is a data cleaning step. Such as replace empty strings with "" or unusual and empty fields with median values. Below, I'll just drop rows with null values.

Start up a Zilliz free tier cluster.

Code in this notebook uses fully-managed Milvus on Ziliz Cloud free trial.

1. Choose the default "Starter" option when you provision > Create collection > Give it a name > Create cluster and collection.
2. On the Cluster main page, copy your API Key and store it locally in a .env variable. See note below how to do that.
3. Also on the Cluster main page, copy the Public Endpoint URI.

💡 Note: To keep your tokens private, best practice is to use an env variable. See how to save api key in env variable.

👉🏼 In Jupyter, you need a .env file (in same dir as notebooks) containing lines like this:

• ZILLIZ_API_KEY=f370c...
• OPENAI_API_KEY=sk-H...
• VARIABLE_NAME=value...

Load the Embedding Model checkpoint and use it to create vector embeddings

Embedding model: We will use the open-source sentence transformers available on HuggingFace to encode the documentation text. We will download the model from HuggingFace and run it locally.

💡Tip: A good way to choose a sentence transformer model is to check the MTEB Leaderboard. Sort descending by column "Retrieval Average" and choose the best-performing small model.

Two model parameters of note below:

1. EMBEDDING_DIM refers to the dimensionality or length of the embedding vector. In this case, the embeddings generated for EACH token in the input text will have the SAME length = 1024. This size of embedding is often associated with BERT-based models, where the embeddings are used for downstream tasks such as classification, question answering, or text generation.
   
   
2. MAX_SEQ_LENGTH is the maximum Context Length the encoder model can handle for input sequences. In this case, if sequences longer than 512 tokens are given to the model, everything longer will be (silently!) chopped off. This is the reason why a chunking strategy is needed to segment input texts into chunks with lengths that will fit in the model's input.

Create a Milvus collection

You can think of a collection in Milvus like a "table" in SQL databases. The collection will contain the

• Schema (or no-schema Milvus client).
  💡 You'll need the vector EMBEDDING_DIM parameter from your embedding model. Typical values are:
  • 1024 for sbert embedding models
  • 1536 for ada-002 OpenAI embedding models
• Vector index for efficient vector search
• Vector distance metric for measuring nearest neighbor vectors
• Consistency level In Milvus, transactional consistency is possible; however, according to the CAP theorem, some latency must be sacrificed. 💡 Searching movie reviews is not mission-critical, so eventually consistent is fine here.

Add a Vector Index

The vector index determines the vector search algorithm used to find the closest vectors in your data to the query a user submits.

Most vector indexes use different sets of parameters depending on whether the database is:

• inserting vectors (creation mode) - vs -
• searching vectors (search mode)

Scroll down the docs page to see a table listing different vector indexes available on Milvus. For example:

• FLAT - deterministic exhaustive search
• IVF_FLAT or IVF_SQ8 - Hash index (stochastic approximate search)
• HNSW - Graph index (stochastic approximate search)
• AUTOINDEX - Automatically determined based on OSS vs Zilliz cloud, type of GPU, size of data.

Besides a search algorithm, we also need to specify a distance metric, that is, a definition of what is considered "close" in vector space. In the cell below, the HNSW search index is chosen. Its possible distance metrics are one of:

• L2 - L2-norm
• IP - Dot-product
• COSINE - Angular distance

💡 Most use cases work better with normalized embeddings, in which case L2 is useless (every vector has length=1) and IP and COSINE are the same. Only choose L2 if you plan to keep your embeddings unnormalized.

Chunking

Before embedding, it is necessary to decide your chunk strategy, chunk size, and chunk overlap. In this demo, I will use:

• Strategy = Keep movie reveiws as single chunks unless they are too long.
• Chunk size = Use the embedding model's parameter MAX_SEQ_LENGTH
• Overlap = Rule-of-thumb 10-15%
• Function = Langchain's convenient RecursiveCharacterTextSplitter to split up long reviews recursively.

Insert data into Milvus

For each original text chunk, we'll write the quadruplet (vector, text, source, h1, h2) into the database.

The Milvus Client wrapper can only handle loading data from a list of dictionaries.

Otherwise, in general, Milvus supports loading data from:

• pandas dataframes
• list of dictionaries

Below, we use the embedding model provided by HuggingFace, download its checkpoint, and run it locally as the encoder.

Ask a question about your data

So far in this demo notebook:

1. Your custom data has been mapped into a vector embedding space
2. Those vector embeddings have been saved into a vector database

Next, you can ask a question about your custom data!

💡 In LLM vocabulary:

Query is the generic term for user questions.
A query is a list of multiple individual questions, up to maybe 1000 different questions!

Question usually refers to a single user question.
In our example below, the user question is "What is AUTOINDEX in Milvus Client?"

Semantic Search = very fast search of the entire knowledge base to find the TOP_K documentation chunks with the closest embeddings to the user's query.

💡 The same model should always be used for consistency for all the embeddings data and the query.

Execute a vector search

Search Milvus using PyMilvus API.

💡 By their nature, vector searches are "semantic" searches. For example, if you were to search for "leaky faucet":

Traditional Key-word Search - either or both words "leaky", "faucet" would have to match some text in order to return a web page or link text to the document.

Semantic search - results containing words "drippy" "taps" would be returned as well because these words mean the same thing even though they are different words.