02 Filling RAG Outputs For Evaluation

Notebook 2: Filling RAG outputs For Evaluation

In this notebook, we will use the example RAG pipeline to populate the RAG outputs: contexts (retrieved relevant documents) and answer (generated by RAG pipeline).

The example RAG pipeline provided as part of this repository uses langchain to build a chatbot that references a custom knowledge base.

If you want to learn more about how the example RAG works, please see 03_llama_index_simple.ipynb.

• Steps 1-5: Build the RAG pipeline.
• Step 6: Build the Query Engine, exposing the Retriever and Generator outputs
• Step 7: Fill the RAG outputs

Steps 1-5: Build the RAG pipeline

Define the LLM

Here we are using a local llm on triton and the address and gRPC port that the Triton is available on.

***If you are using AI Playground (no local GPU) replace, the code in the cell below with the following: ***

import os
from langchain_nvidia_ai_endpoints import ChatNVIDIA, NVIDIAEmbeddings
os.environ['NVAPI_KEY'] = "REPLACE_WITH_YOUR_API_KEY"

nv_embedder = NVIDIAEmbeddings(model="nvolveqa_40k")
nv_document_embedder = NVIDIAEmbeddings(model="nvolveqa_40k", model_type="passage")
nv_query_embedder = NVIDIAEmbeddings(model="nvolveqa_40k", model_type="query")

Create a Prompt Template

A prompt template is a common paradigm in LLM development.

They are a pre-defined set of instructions provided to the LLM and guide the output produced by the model. They can contain few shot examples and guidance and are a quick way to engineer the responses from the LLM. Llama 2 accepts the prompt format shown in LLAMA_PROMPT_TEMPLATE, which we manipulate to be constructed with:

• The system prompt
• The context
• The user's question

Much like LangChain's abstraction of prompts, LlamaIndex has similar abstractions for you to create prompts.

Load Documents

Follow the step number 1 defined here to upload the pdf's to Milvus server.

In this rest of this section, we will load and split the pdfs of ORAN pdfs. We will use the RecursiveCharacterTextSplitter.

Generate and Store Embeddings

a) Generate Embeddings

Embeddings for documents are created by vectorizing the document text; this vectorization captures the semantic meaning of the text.

We will use intfloat/e5-large-v2 or nvidia retriever for the embeddings.

b) Store Embeddings

We will use the LlamaIndex module ServiceContext to bundle commonly used resources during the indexing and querying stage.

In this example, we bundle the build resources: the LLM, the embedding model, the node parser, and the prompt helper.

Step 6: Build the Query Engine, exposing the Retriever and Generator outputs

Step 7: Fill the RAG outputs

Let's now query the RAG pipeline and fill the outputs contexts and answer on the evaluation JSON file.

First, we need to load the previously generated dataset. So far, the RAG outputs fields are empty.

Let now query the RAG pipeline and populate the contexts and answer fields.