-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Question]: How to use LabelledRagDataset with RetrieverEvaluator or BatchEvalRunner #10445
Comments
🤖 Based on the information provided in the context, it seems that the As for customizing the llm, embedding model, prompts, and adding hit rate and MRR metrics, the context does not provide enough information to give a complete answer. It's possible that these customizations would require modifications to the Here is the code snippet from the context that shows how to use from llama_index.llama_pack import download_llama_pack
RagEvaluatorPack = download_llama_pack("RagEvaluatorPack", "./pack")
rag_evaluator = RagEvaluatorPack(
query_engine=query_engine, # built with the same source Documents as the rag_dataset
rag_dataset=rag_dataset,
)
benchmark_df = await rag_evaluator.run() The You can find more information in the evaluating with LlamaDatasets guide. If you could provide more details about your use case or the specific customizations you're trying to make, I might be able to give a more precise answer. SourcesThis response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot. |
It looks like:
So currently we have three different evaluation mechanisms that require three different dataset formats. It seems |
Bump 😅 |
Hi, @austinmw, I'm helping the LlamaIndex team manage our backlog and am marking this issue as stale. From what I understand, you were inquiring about using a Is this issue still relevant to the latest version of the LlamaIndex repository? If so, please let the LlamaIndex team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days. Thank you for your understanding and contributions to the LlamaIndex project. Dosu |
not stale. |
Question Validation
Question
Hi, I have a
LabelledRagDataset
created withRagDatasetGenerator
. Now how can I useRetrieverEvaluator
orBatchEvalRunner
with this? Are they compatible?The documentation only mentions using it with a
RagEvaluatorPack
which is not customizable enough. My goal is to measure Hit Rate, MRR, Context Relevance, and Faithfulness, using Bedrock LLMs.The text was updated successfully, but these errors were encountered: