Notebooks
W
Weaviate Evaluation Metrics Information Retrieval

Evaluation Metrics Information Retrieval

vector-searchvector-databaseretrieval-augmented-generationllm-frameworksweaviate-featuresfunction-callingevaluationweaviate-recipesPythongenerative-ai
alph-notebooks/weaviate-recipes / evaluation_metrics_information_retrieval.ipynb

Open In Colab

Evaluation metrics for Retrieval and Recommendation Systems

[38]
Requirement already satisfied: pytrec_eval in /opt/homebrew/lib/python3.11/site-packages (0.5)
[ ]

Precision and recall

Precision and recall are rank-unaware metrics.

=\frac{\text{Number of relevant items in K}}{\text{Total number of items in K}}$$ $$\text{Recall@K} = \frac{TP}{TP+FN} =\frac{\text{Number of relevant items in K}}{\text{Total number of relevant items}}$$
[39]
{
 "sweet pastry": {
  "P_4": 0.5,
  "recall_4": 0.6666666666666666
 },
 "suitable for lunch": {
  "P_4": 0.5,
  "recall_4": 0.5
 },
 "goes well with jam": {
  "P_4": 0.25,
  "recall_4": 0.3333333333333333
 }
}

Mean Average Precision (MAP)

MAP@K=1U∑u=1UAP@Ku\text{MAP@K} = \frac{1}{U}\sum_{u=1}^{U}AP@K_u

[40]
{
 "sweet pastry": {
  "map_cut_4": 0.8333333333333333
 }
}

Note that pytrec_eval seems to average MAP across the number of relevant items. See, how the MAP@K value changes, when we add 'scone' to the list of relevant items?

[41]
{
 "sweet pastry": {
  "map_cut_4": 0.5555555555555555
 }
}

Normalized Discounted Cumulative Gain (NDCG)

NDCG@K=DCG@KIDCG@K\text{NDCG@K} = \frac{\text{DCG@K}}{\text{IDCG@K}}

[43]
{
 "goes well with jam": {
  "ndcg_cut_4": 0.4672390440360399
 }
}