Improve Quality of your LLM Application and RAG Relevance with AIMon

Overview

In this tutorial, we'll help you build a retrieval-augmented generation (RAG) chatbot that answers questions on the meeting bank dataset.

In this tutorial you will learn to:

• Build an LLM application that answers a user's query related to the meeting bank dataset.
• Define, measure and improve the quality of your LLM application.
• Improve the RAG relevance score significanly (as high as 97% although your mileage may vary)

Tech Stack

Vector Database

For this application, we will use Weaviate, which is an open-source, AI-native vector database. We will also use Weaviate's fast embedding computation service.

LLM Framework

LlamaIndex is an open-source data orchestration framework that simplifies building large language model (LLM) applications by facilitating the integration of private data with LLMs, enabling context-augmented generative AI applications through a Retrieval-Augmented Generation (RAG) pipeline. We will use LlamaIndex for this tutorial since it offers a good amount of flexibility and better lower level API abstractions.

LLM Output Quality Evaluation

AIMon offers proprietary Judge models for Hallucination, Context Quality issues, Instruction Adherence of LLMs, Retrieval Quality and other LLM reliability tasks. We will use AIMon to judge the quality of the LLM application.

Pre-requisites

1. Signup for an AIMon account here.

Add this secret to the Colab Secrets (the "key" symbol on the left panel)

• AIMON_API_KEY

2. Signup for a Weaviate account here, create a sandbox cluster, and set the following keys in the Colab secrets.

• WCD_URL (REST Endpoint of your Weaviate cluster)
• WCD_API_KEY (Admin API Key of your Weaviate cluster)

3. Signup for an OpenAI account here and add the following key in Colab secrets:

• OPENAI_API_KEY

Required API keys

Dataset

We will use the MeetingBank dataset which is a benchmark dataset created from the city councils of 6 major U.S. cities to supplement existing datasets. It contains 1,366 meetings with over 3,579 hours of video, as well as transcripts, PDF documents of meeting minutes, agenda, and other metadata.

For this exercise, we have created a smaller dataset. It can be found here.

Queries

Below are the 12 queries that we will run on the transcript above

Metric Definition

This quality score metric will help us understand how good the LLM responses are for the set of queries above. To measure quality of our application, we will run a set of queries and aggregate the quality scores across all these queries.

LLM Application Quality Score is a combination of 3 individual quality metrics from AIMon:

1. Hallucination Score (hall_score): checks if the generated text is grounded in the provided context. A score closer to 1.0 means that there is a strong indication of hallucination and a score closer to 0.0 means a lower indication of hallucination. Hence, we will use (1.0-hall_score) here when computing the final quality score.
2. Instruction Adherence Score (ia_score): checks if all explicit instructions provided have been followed by the LLM. The higher the ia_score the better the adherence to instructions. The lower the score, the poorer the adherence to instructions.
3. Retrieval Relevance Score (rr_score): checks if the retrieved documents are relevant to the query. A score closer to 100.0 means perfect relevance of document to query and a score closer to 0.0 means poor relevance of document to query.

quality_score = 0.35 * (1.0 - hall_score) + 0.35 * ia_score + 0.3 * rr_score

Setup AIMon

As mentioned previously, AIMon will be used to judge the quality of the LLM application. Documentation can be found here.

Instantiate the Weaviate client

Create a Weaviate collection

We will use OpenAI's text-embedding-3-small model to create embeddings for the documents. These embeddings are created using Weaviate's embedding service which offers a fast embedding computation option. Note that this option is not available in the open source version of Weaviate. To use the embedding service you need to have a cluster or a sandbox running in Weaviate cloud.

Ingest data to the Weaviate Collection

This step takes about 30 seconds to ingest ~4000 documents.

1. Use Weaviate Vector Database for document retrieval

There are two main components we need to be aware of: Ingestion and RAG based Q&A. The ingestion pipeline processes the transcripts from the Meeting Bank dataset and stores it in the Weaviate Vector database. The RAG Q&A pipeline processes a user query by first retrieving the relevant documents from the vector store. These documents are then be used as grounding documents for the LLM to generate its response. We leverage AIMon to calculate the quality score and continuously monitor the application for hallucination, , instruction adherence, context relevance. These are the same 3 metrics we used to define the quality score above.

We will use LlamaIndex's query embedding to embed the input queries. This option uses the same Open AI embedding model text-embedding-3-small that was used to ingest the documents above.

In the cells below, we will setup a LlamaIndex VectorStoreIndex, setup the LLM and build a LLamaIndex Query Engine that interfaces with the stored embeddings and the LLM to answer a user's questions.

We will use a Hybrid search strategy here. Hybrid search provides a balanced approach to using both the keyword based reverse index search and vector search. The alpha parameter determines how much weight should be given to keyword search bm25 v/s vector search (alpha = 0 -> bm25, alpha=1 -> vector search).

At this point, the query engine. Next, we setup AIMon to help us measure quality scores. We use the same @detect decorator that was created in the previous cells above. The only additional code in ask_and_validate here is to help AIMon interface with LLamaIndex's retrieved document "nodes".

Lets run through all the queries through the LlamaIndex query engine in the queries_df and compute the overall quality score using AIMon.

NOTE: This will take about 2 mins

2. Add Re-ranking to your retrieval

Now, we will add in AIMon's domain adaptable re-ranker using AIMon's LlamaIndex postprocessor re-rank integration.

As shown in the figure below, reranking helps bubble up the most relevant documents to the top by using a more advanced Query-Document matching function. The unique feature of AIMon's re-ranker is the ability to customize it per domain. Similar to how you would prompt engineer an LLM, you can customize reranking performance per domain using the task_definition field. This state-of-the-art reranker runs at ultra low sub second latency (for a ~2k context) and its performance ranks in the top 5 of the MTEB reranking leaderboard.

Let's run through the queries again and recompute the overall quality score to see if there is an improvement.

✨ AIMon's re-ranking should not add additional latency overhead to the total query reponse time since it actually reduces the amount of context documents that need to be sent to the LLM for generating a response making the operation efficient in terms of network I/O and LLM token processing cost (money and time).

NOTE: This step will take 2 mins

Notice the difference in average document relevance scores when using the reranker v/s when not using the reranker

Terminate the connection to the Weaviate client.

🎉 Quality Score improved!

Notice that the overall quality score across all queries improved after using AIMon's reranker.

In sum, as shown in the figure below, we demonstrated the following:

• Computing a quality score using a weighted combination of 3 different quality metrics: hallucination score, instruction adherence score and retrieval relevance score.
• Improved the quality score using AIMon's low-latency, domain adaptable re-ranker.
• We also showed how retrieval relevance improves significantly by adding in AIMon's re-ranker.

We encourage you to experiment with the different components shown in this notebook to further increase the quality score. One idea is to add your own definitions of quality using the instructions field in the instruction_adherence detector above. Another idea is to add another one of AIMon's checker models as part of the quality metric calculation.

Happy experimenting! Join our discord server for fun conversations on AI!

The above table summarizes our results. Your actual numbers will vary depending on various factors such as variations in quality of LLM responses, performance of the nearest neighbor search in the VectorDB etc.

In conclusion, as shown by the figure below, we evaluated quality score, RAG relevance and instruction following capabilities of your LLM application. We used AIMon's re-ranker to improve the overall quality of the application and the average relevance of documents retrieved from your RAG.