Feature Description

This will allow user's to evaluate their RAG pipeline with the synthetic eval dataset generated using llama_index.core.llama_dataset.generator without an OpenAI API Key

Reason

Currently, RagEvaluatorPack only supports OpenAI Embeddings. While judge_llm allows the user to use open source LLMs, the method still requires an OpenAI API Key to run owing to the following code.

Source code from llama-index-packs/llama-index-packs-rag-evaluator/llama_index/packs/rag_evaluator/base.py, line 94

def _prepare_judges(self):
"""Construct the evaluators."""
judges = {}
judges["correctness"] = CorrectnessEvaluator(
llm=self.judge_llm,
)
judges["relevancy"] = RelevancyEvaluator(
llm=self.judge_llm,
)
judges["faithfulness"] = FaithfulnessEvaluator(
llm=self.judge_llm,
)
judges["semantic_similarity"] = SemanticSimilarityEvaluator(
embed_model=OpenAIEmbedding() #this line requires an OpenAI API key from the user's end
)
return judges

Value of Feature

No response