🤖

Thank you for your detailed questions. Let's address them one by one:

1. The RetrievalEvaluator currently supports MRR (Mean Reciprocal Rank) and Hit Rate because these are the metrics that have been implemented in the METRIC_REGISTRY. The METRIC_REGISTRY is a dictionary that maps metric names to their corresponding classes, and the resolve_metrics function uses this registry to create instances of the metrics for evaluation. As for the additional metrics like NDCG (Normalized Discounted Cumulative Gain), MAP (Mean Average Precision), Recall, and Precision at different K-s, these are not currently supported because they have not been implemented in the METRIC_REGISTRY.

2. The BEIR evaluator supports a different set of metrics because it is designed to evaluate retrieval models in a different context. The BEIR evaluator is designed to evaluate models on the BEIR benchmark, which includes a variety of different tasks and datasets. The metrics supported by the BEIR evaluator are those that are relevant for these tasks and datasets.

3. As for the future plans to include these additional metrics in the METRIC_REGISTRY, the context does not provide any information about this. Therefore, it is not possible to answer whether there are plans to include these additional metrics for better insights into Retrieval and Ranking.

To include these additional metrics, new classes would need to be created for each metric, similar to the HitRate, MRR, and CohereRerankRelevancyMetric classes. These new classes would need to inherit from BaseRetrievalMetric and implement the compute method to calculate the metric. Once these classes are created, they can be added to the METRIC_REGISTRY dictionary.

I hope this answers your questions. If you have any further queries, feel free to ask.

Sources

• llama-index-legacy/llama_index/legacy/evaluation/retrieval/metrics.py
• llama-index-core/llama_index/core/evaluation/retrieval/evaluator.py

It is a pity that there are no explicit plans for the integration of further metrics like MAP@K or NDCG@K, as I now have to implement these metrics locally myself, although I would like to use the RetrievalEvaluator OOTB...

EDIT: @hkristof03 If it is suitable you could use the InformationRetrievalEvaluator from the sentence-transformer framework.

@aoezdTchibo I solved by getting the "embedding_dict" keys (node ids) and the embeddings (values) from the vector store's ".to_dict()" method, then getting the node id - doc id map from the "text_id_to_ref_doc_id" from the "embedding_dict", then creating my own faiss index and evaluating the retrieval using the retrieval component of Torchmetrics. Sounds complicated but not that much in implementation. Although I had to write several workarounds for this library to be able to test the RAG system Retrieval component (and its components) on several datasets. For experimentation and extensive evaluation the LlamaIndex is not yet set up.

@hkristof03 Thanks for the link to Touchmetrics! I hadn't heard of this before and it looks very promising.

Hey @hkristof03,

Quick note incase you are still interested

My PR was recently merged for adding a MRR@K and HitRate@K option, through usage of a "use_granular_..." attribute

So we now have more native flexibility for our evals

Hopefully you find it valuable 💪