RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora
Summary: arXiv:2604.19047v1 Announce Type: cross
Abstract: Existing QA benchmarks typically assume distinct documents with minimal overlap, yet real-world retrieval-augmented generation (RAG) systems operate on corpora such as financial reports, legal codes, and patents, where information is highly redundant and documents exhibit strong inter-document similarity. This mismatch undermines evaluation validity: retrievers can be unfairly undervalued even when they retrieve documents that provide sufficient evidence, because redundancy across documents is not accounted for in evaluation. On the other hand, retrievers that perform well on standard benchmarks often generalize poorly to real-world corpora with highly similar and redundant documents.
We present RARE (Redundancy-Aware Retrieval Evaluation), a framework for constructing realistic benchmarks by:
- Decomposing documents into atomic facts to enable precise redundancy tracking.
- Enhancing LLM-based data generation with CRRF.
RAG benchmark data usually requires multiple quality criteria, but LLMs often yield trivial outputs. CRRF scores criteria separately and fuses decisions by rank, improving the reliability of generated data. Applying RARE to Finance, Legal, and Patent corpora, we introduce RedQA, where a strong retriever baseline drops from 66.4% PerfRecall@10 on 4-hop General-Wiki to 5.0-27.9% PerfRecall@10 at 4-hop depth, revealing robustness gaps that current benchmarks fail to capture. RARE enables practitioners to build domain-specific RAG evaluations that faithfully reflect real-world deployment conditions.
Introduction
The increasing reliance on retrieval-augmented generation (RAG) systems has highlighted a significant gap in the evaluation frameworks currently used. Traditional QA benchmarks have been developed under the assumption that documents are distinct and have minimal overlap. However, many real-world applications, particularly in fields such as finance and law, deal with documents that are often redundant and exhibit a high degree of similarity.
The Challenge of Redundancy
In environments where information is highly redundant, traditional evaluation methods can lead to misleading results. Retrievers may be penalized for retrieving documents that provide sufficient evidence but share content with other documents. Conversely, retrievers that perform well on standard benchmarks may falter in real-world scenarios, where document similarity is the norm rather than the exception.
Introducing RARE
The RARE framework aims to address these issues by providing a structured approach to evaluation that incorporates redundancy awareness. By breaking down documents into atomic facts, RARE allows for a more nuanced tracking of redundancy, ensuring that retrievers are assessed more fairly based on the actual information retrieved.
CRRF and Its Impact
To enhance the quality of data generated by LLMs and ensure that the evaluation criteria are met, RARE incorporates a novel approach known as CRRF (Criteria Rank and Fuse). This method assesses different quality criteria separately and combines the results to yield a more reliable evaluation of generated data.
Application and Results
When applied to specific domains such as Finance, Legal, and Patent corpora, the RARE framework has demonstrated significant advantages. For instance, a strong retriever baseline showed a dramatic decline in performance, dropping from 66.4% PerfRecall@10 on a standard benchmark to only 5.0-27.9% on the more challenging 4-hop depth, underscoring the limitations of existing benchmarks.
Conclusion
The RARE framework provides a crucial advancement in the evaluation of retrieval-augmented generation systems by enabling the construction of benchmarks that reflect real-world conditions. This approach not only improves the assessment of retrievers but also guides practitioners in developing more effective RAG evaluations tailored to specific domains.