ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences
Summary: arXiv:2602.11354v2 Announce Type: replace
Abstract: The literature has witnessed an emerging interest in AI agents for automated assessment of scientific papers. Existing benchmarks focus primarily on the computational aspect of this task, testing agents’ ability to reproduce or replicate research outcomes when having access to the code and data. This setting, while foundational, fails to capture the inconsistent availability of new data for replication as opposed to reproduction, and lacks ground-truth diversity by focusing only on reproducible papers, thereby failing to evaluate an agent’s ability to identify non-replicable research. Furthermore, most benchmarks only evaluate outcomes rather than the replication process.
In response, we introduce ReplicatorBench, an end-to-end benchmark, including human-verified replicable and non-replicable research claims in social and behavioral sciences for evaluating AI agents in research replication across three stages:
- Extraction and retrieval of replication data;
- Design and execution of computational experiments;
- Interpretation of results, allowing a test of AI agents’ capability to mimic the activities of human replicators in real-world scenarios.
To set a baseline of AI agents’ capability, we develop ReplicatorAgent, an agentic framework equipped with necessary tools, like web search and iterative interaction with sandboxed environments, to accomplish tasks in ReplicatorBench. We evaluate ReplicatorAgent across four underlying large language models (LLMs), as well as different design choices of programming language and levels of code access.
Our findings reveal that while current LLM agents are capable of effectively designing and executing computational experiments, they struggle with retrieving resources, such as new data, necessary to replicate a claim. This limitation highlights the need for further advancements in AI capabilities, particularly in the context of real-world applications in social and behavioral sciences.
In addition to presenting ReplicatorBench and the ReplicatorAgent, this study emphasizes the importance of robust benchmarks that address the complexities of research replication. By incorporating a variety of research claims, both replicable and non-replicable, we aim to provide a more comprehensive evaluation of AI agents’ performance.
Researchers and practitioners are encouraged to engage with our findings and utilize the publicly available code and data, which can be accessed at https://github.com/CenterForOpenScience/llm-benchmarking. This resource aims to facilitate further exploration and improvement of AI-driven research assessment tools in the field.