FactReview: Evidence-Grounded Reviews with Literature Positioning and Execution-Based Claim Verification
Peer review in machine learning is facing unprecedented challenges due to a sharp increase in submission volumes combined with the limited availability of qualified reviewers. Traditional reviewing systems, particularly those based on large language models (LLMs), primarily rely on the content of the manuscript itself to generate feedback. This reliance on the paper’s own narrative can lead to inconsistencies, as the quality of the review is heavily influenced by how well the paper is presented. Furthermore, crucial evidence for evaluating claims often resides in related literature or associated code repositories, which these systems may overlook.
To address these limitations, researchers have introduced FactReview, an innovative evidence-grounded reviewing system designed to enhance the quality and reliability of peer reviews in machine learning. FactReview integrates several key components, including:
- Claim Extraction: The system identifies and extracts major claims and reported results from the submitted manuscript.
- Literature Positioning: FactReview retrieves relevant nearby research to contextualize the submission within the broader body of work.
- Execution-Based Claim Verification: When the author provides code, FactReview executes it under controlled conditions to verify central empirical claims.
Upon evaluating a submission, FactReview generates a concise review coupled with an evidence report. Each major claim is assigned one of five labels: Supported, Supported by the paper, Partially supported, In conflict, or Inconclusive. This systematic approach aims to provide a more objective and evidence-based assessment of research quality.
In a compelling case study involving the CompGCN model, FactReview successfully reproduced results closely aligning with those initially reported for link prediction and node classification tasks. However, the analysis also revealed that the broader performance claim made by the original authors was only partially substantiated. Specifically, for the MUTAG graph classification task, FactReview’s reproduction achieved an accuracy of 88.4%, while the best baseline cited in the paper stood at 92.6%. This discrepancy highlights the necessity for thorough verification of performance claims in machine learning research.
More broadly, the findings from the CompGCN case study suggest that AI can play a vital role in peer review, not as an ultimate decision-maker but as a powerful tool for gathering critical evidence. By empowering reviewers to produce more evidence-grounded assessments, FactReview aims to improve the overall integrity and reliability of the peer review process in machine learning.
The code for FactReview is publicly available, providing an opportunity for researchers and practitioners alike to explore and contribute to this innovative approach. Interested parties can access the code at https://github.com/DEFENSE-SEU/Review-Assistant.
As the landscape of machine learning research continues to evolve, tools like FactReview are essential for adapting the peer review process to meet the demands of increased submission volumes while maintaining high standards of quality and rigor.