Agentic RAG: turbocharge your RAG with query reformulation and self-query! 🚀
Authored by: Aymeric Roucher
This tutorial is advanced. You should have notions from this other cookbook first!
Reminder: Retrieval-Augmented-Generation (RAG) is “using an LLM to answer a user query, but basing the answer on information retrieved from a knowledge base”. It has many advantages over using a vanilla or fine-tuned LLM: to name a few, it allows to ground the answer on true facts and reduce confabulations, it allows to provide the LLM with domain-specific knowledge, and it allows fine-grained control of access to information from the knowledge base.
But vanilla RAG has limitations, most importantly these two:
- It performs only one retrieval step: if the results are bad, the generation in turn will be bad.
- Semantic similarity is computed with the user query as a reference, which might be suboptimal: for instance, the user query will often be a question and the document containing the true answer will be in affirmative voice, so its similarity score will be downgraded compared to other source documents in the interrogative form, leading to a risk of missing the relevant information.
But we can alleviate these problems by making a RAG agent: very simply, an agent armed with a retriever tool!
This agent will: ✅ Formulate the query itself and ✅ Critique to re-retrieve if needed.
So it should naively recover some advanced RAG techniques!
- Instead of directly using the user query as the reference in semantic search, the agent formulates itself a reference sentence that can be closer to the targeted documents, as in HyDE
- The agent can the generated snippets and re-retrieve if needed, as in Self-Query
Let's build this system. 🛠️
Run the line below to install required dependencies:
Let's login in order to call the HF Inference API:
We first load a knowledge base on which we want to perform RAG: this dataset is a compilation of the documentation pages for many huggingface packages, stored as markdown.
Now we prepare the knowledge base by processing the dataset and storing it into a vector database to be used by the retriever.
We use LangChain for its excellent vector database utilities.
For the embedding model, we use thenlper/gte-small since it performed well in our RAG_evaluation cookbook.
Splitting documents...
100%|██████████| 2647/2647 [00:52<00:00, 50.28it/s] /var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_26437/2798339493.py:37: LangChainDeprecationWarning: The class `HuggingFaceEmbeddings` was deprecated in LangChain 0.2.2 and will be removed in 1.0. An updated version of the class exists in the :class:`~langchain-huggingface package and should be used instead. To use it run `pip install -U :class:`~langchain-huggingface` and import as `from :class:`~langchain_huggingface import HuggingFaceEmbeddings``. embedding_model = HuggingFaceEmbeddings(model_name="thenlper/gte-small")
Embedding documents... This should take a few minutes (5 minutes on MacBook with M1 Pro)
Now the database is ready: let’s build our agentic RAG system!
👉 We only need a RetrieverTool that our agent can leverage to retrieve information from the knowledge base.
Since we need to add a vectordb as an attribute of the tool, we cannot simply use the simple tool constructor with a @tool decorator: so we will follow the advanced setup highlighted in the advanced agents documentation.
Now it’s straightforward to create an agent that leverages this tool!
The agent will need these arguments upon initialization:
tools: a list of tools that the agent will be able to call.model: the LLM that powers the agent.
Our model must be a callable that takes as input a list of messages and returns text. It also needs to accept a stop_sequences argument that indicates when to stop its generation. For convenience, we directly use the InferenceClientModel class provided in the package to get a LLM engine that calls our Inference API.
And we use meta-llama/Llama-3.1-70B-Instruct, served for free on Hugging Face's Inference API!
Note: The Inference API hosts models based on various criteria, and deployed models may be updated or replaced without prior notice. Learn more about it here.
Since we initialized the agent as a ReactJsonAgent, it has been automatically given a default system prompt that tells the LLM engine to process step-by-step and generate tool calls as JSON blobs (you could replace this prompt template with your own as needed).
Then when its .run() method is launched, the agent takes care of calling the LLM engine, parsing the tool call JSON blobs and executing these tool calls, all in a loop that ends only when the final answer is provided.
Final output: To push a model to the Hub, you can use the push_to_hub() method after training. You can also use the PushToHubCallback to upload checkpoints regularly during a longer training run. Additionally, you can push the model up to the hub using the api.upload_folder() method.
Agentic RAG vs. standard RAG
Does the agent setup make a better RAG system? Well, let's compare it to a standard RAG system using LLM Judge!
We will use meta-llama/Meta-Llama-3-70B-Instruct for evaluation since it's one of the strongest OS models we tested for LLM judge use cases.
Before running the test let's make the agent less verbose.
The evaluation prompt follows some of the best principles shown in our llm_judge cookbook: it follows a small integer Likert scale, has clear criteria, and a description for each score.
Average score for agentic RAG: 86.9% Average score for standard RAG: 73.1%
Let us recap: the Agent setup improves scores by 14% compared to a standard RAG! (from 73.1% to 86.9%)
This is a great improvement, with a very simple setup 🚀
(For a baseline, using Llama-3-70B without the knowledge base got 36%)