Use Ragas to evaluate the customized RAG pipeline based on milvus

Please note that this test requires a large amount of OpenAI api token consumption. Please read it carefully and Pay attention to the number of times you request access.

1. Prepare environment and data

Before starting, you must set OPENAI_API_KEY in your environment variables.

You also need to install milvus and start it. You can refer to the official introduction to start quickly.

Install pip dependencies

Download Financial Opinion Mining and Question Answering (fiqa) Dataset data if it not exists in your local space. We convert it into a ragas form that is easier to process, referring from this script.

Now we have the question list and the ground truth list. And the knowledge documents are prepared in fiqa_path.

2. Build RAG pipeline based on milvus and langchain

Split the doc using langchain RecursiveCharacterTextSplitter.

Prepare embedding model and milvus settings.

Build agent using langchain.

3. Start Ragas Evaluation

Note that a large amount of OpenAI api token is consumed. Every time you ask a question and every evaluation, you will ask the OpenAI service. Please pay attention to your token consumption. If you only want to run a small number of tests, you can modify the code to reduce the test size.

You can choose the indicators you care about to test.