Defend: Automated Rebuttals for Peer Review with Minimal Author Guidance
Summary: arXiv:2603.27360v1 Announce Type: new
Rebuttal generation is an essential aspect of the peer review process for scientific papers. It allows authors to clarify misunderstandings, correct factual inaccuracies, and guide reviewers toward a more informed evaluation. However, the effectiveness of rebuttal generation has come under scrutiny, particularly regarding the capabilities of Large Language Models (LLMs) in this domain.
Recent observations indicate that LLMs frequently struggle with two critical tasks: performing targeted refutation and maintaining accurate factual grounding when applied directly for rebuttal generation. This limitation underscores the necessity for structured reasoning and the inclusion of author intervention in the rebuttal process.
Introducing DEFEND
To address these challenges, the paper introduces DEFEND, an LLM-based tool designed to execute the underlying reasoning process of automated rebuttal generation while ensuring the author remains involved. Unlike traditional methods where authors must draft rebuttals from scratch, DEFEND allows authors to guide the reasoning process with minimal intervention. This approach not only enhances efficiency but also reduces cognitive load on authors.
Comparative Paradigms
The paper conducts a comparative analysis of DEFEND against three other rebuttal generation paradigms:
- Direct Rebuttal Generation using LLM (DRG): This method relies solely on LLMs to generate rebuttals without any author input.
- Segment-wise Rebuttal Generation using LLM (SWRG): This technique divides the rebuttal into segments but still depends on LLMs for generation.
- Sequential Approach (SA): This approach involves segment-wise rebuttal generation without any author intervention, leading to potential inaccuracies.
Methodology and Evaluation
To facilitate a fine-grained evaluation of these paradigms, the authors extend the ReviewCritique dataset. This enhancement includes:
- Review segmentation
- Deficiency and error type annotations
- Rebuttal-action labels
- Mapping to gold rebuttal segments
Experimental Results
The paper presents experimental results alongside a user study, revealing that direct usage of LLMs for rebuttal generation often results in poor performance concerning factual correctness and targeted refutation. In contrast, the introduction of segment-wise generation and the automated sequential approach with author involvement significantly improve both the factual accuracy and the strength of refutations.
Conclusion
DEFEND represents a promising advancement in the field of automated rebuttal generation for peer review processes. By enabling authors to maintain control over the reasoning process while leveraging the capabilities of LLMs, DEFEND not only enhances the quality of rebuttals but also streamlines the effort required from authors. The findings of this study highlight the importance of structured reasoning and collaboration between human authors and AI tools in achieving more effective peer review outcomes.