Use Ragas to evaluate the OpenAI Assistant

Please note that this test requires a large amount of OpenAI api token consumption. Please read it carefully and Pay attention to the number of times you request access.

1. Prepare environment and data

Before starting, you must set OPENAI_API_KEY in your environment variables.

Install pip dependencies

Download Financial Opinion Mining and Question Answering (fiqa) Dataset data if it not exists in your local space. We convert it into a ragas form that is easier to process, referring from this script.

Now we have the question list and the ground truth list. And the knowledge documents are prepared in fiqa_path.

2. Building RAG using OpenAI assistant

To get the context content from the annotations returned by Open AI.

Build assistant and upload knowledge files.

3. Start Ragas Evaluation

Note that a large amount of OpenAI api token is consumed. Every time you ask a question and every evaluation, you will ask the OpenAI service. Please pay attention to your token consumption. If you only want to run a small number of tests, you can modify the code to reduce the test size.

You can choose the indicators you care about to test.