Notebooks
M
Mistral AI RAG Evaluation

RAG Evaluation

mistral-cookbookevaluationmistral
alph-notebooks/mistral-cookbook / RAG_evaluation.ipynb

Evaluating RAG: Using Mistral Models for LLM as a Judge (With Structured Outputs)

This cookbook shows an example of using the Mistral AI models for LLM As A Judge using structured outputs.

Imports & API Key Setting

You can get your api key from: https://console.mistral.ai/

[2]
Requirement already satisfied: mistralai==1.5.1 in /opt/anaconda3/lib/python3.12/site-packages (1.5.1)
Requirement already satisfied: httpx==0.28.1 in /opt/anaconda3/lib/python3.12/site-packages (0.28.1)
Requirement already satisfied: pydantic==2.10.6 in /opt/anaconda3/lib/python3.12/site-packages (2.10.6)
Requirement already satisfied: python-dateutil==2.9.0.post0 in /opt/anaconda3/lib/python3.12/site-packages (2.9.0.post0)
Requirement already satisfied: jsonpath-python==1.0.6 in /opt/anaconda3/lib/python3.12/site-packages (1.0.6)
Requirement already satisfied: typing-inspect==0.9.0 in /opt/anaconda3/lib/python3.12/site-packages (0.9.0)
Requirement already satisfied: eval-type-backport>=0.2.0 in /opt/anaconda3/lib/python3.12/site-packages (from mistralai==1.5.1) (0.2.2)
Requirement already satisfied: anyio in /opt/anaconda3/lib/python3.12/site-packages (from httpx==0.28.1) (4.8.0)
Requirement already satisfied: certifi in /opt/anaconda3/lib/python3.12/site-packages (from httpx==0.28.1) (2025.1.31)
Requirement already satisfied: httpcore==1.* in /opt/anaconda3/lib/python3.12/site-packages (from httpx==0.28.1) (1.0.7)
Requirement already satisfied: idna in /opt/anaconda3/lib/python3.12/site-packages (from httpx==0.28.1) (3.10)
Requirement already satisfied: annotated-types>=0.6.0 in /opt/anaconda3/lib/python3.12/site-packages (from pydantic==2.10.6) (0.7.0)
Requirement already satisfied: pydantic-core==2.27.2 in /opt/anaconda3/lib/python3.12/site-packages (from pydantic==2.10.6) (2.27.2)
Requirement already satisfied: typing-extensions>=4.12.2 in /opt/anaconda3/lib/python3.12/site-packages (from pydantic==2.10.6) (4.12.2)
Requirement already satisfied: six>=1.5 in /opt/anaconda3/lib/python3.12/site-packages (from python-dateutil==2.9.0.post0) (1.16.0)
Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/anaconda3/lib/python3.12/site-packages (from typing-inspect==0.9.0) (1.0.0)
Requirement already satisfied: h11<0.15,>=0.13 in /opt/anaconda3/lib/python3.12/site-packages (from httpcore==1.*->httpx==0.28.1) (0.14.0)
Requirement already satisfied: sniffio>=1.1 in /opt/anaconda3/lib/python3.12/site-packages (from anyio->httpx==0.28.1) (1.3.1)

Enter Mistral AI API Key ········

Main Code For LLM As A Judge For RAG (With Structured Outputs)

[4]
🏆 RAG Evaluation:

Criteria: Context Relevance
Reasoning: The retrieved context is relevant to the query as it defines renewable energy and lists various types such as solar, wind, hydro, and geothermal energy. It provides a basic understanding of what renewable energy encompasses, which is useful for addressing the benefits of renewable energy.
Score: 3/3

Criteria: Answer Relevance
Reasoning: The generated answer addresses the user's query by highlighting the environmental benefits of renewable energy, specifically mentioning solar and wind energy. It discusses the reduction of carbon emissions, which is a key benefit of renewable energy. However, it does not mention other types of renewable energy like hydro and geothermal, which were included in the context.
Score: 2/3

Criteria: Groundedness
Reasoning: The generated answer is mostly grounded in the retrieved context. It mentions solar and wind energy, which are part of the context. However, it does not mention hydro and geothermal energy, which were also included in the context. Additionally, the answer introduces the benefit of reducing carbon emissions, which is not explicitly stated in the context but is a well-known benefit of renewable energy.
Score: 2/3
[ ]