AISysRev — LLM-based Tool for Title-abstract Screening
Summary: arXiv:2510.06708v3 Announce Type: replace-cross
Conducting systematic reviews is an essential yet labor-intensive task, especially during the screening or study selection phase. The sheer volume of research papers available can often be overwhelming for researchers. However, recent advancements in artificial intelligence, particularly in large language models (LLMs), have shown promise in facilitating this process. To address these challenges, we present AISysRev, a cutting-edge LLM-based screening tool designed as a containerized web application.
How AISysRev Works
AISysRev streamlines the screening process by allowing users to upload CSV files containing paper titles and abstracts. Researchers can define their specific inclusion and exclusion criteria, enabling the tool to tailor its screening process effectively. Notably, AISysRev supports multiple LLMs, including:
- Gemini
- Claude
- Mistral
- ChatGPT (via OpenRouter)
Additionally, the tool accommodates locally hosted models and any LLM compatible with the OpenAI SDK.
Screening Capabilities
AISysRev employs both zero-shot and few-shot prompting techniques, enhancing its ability to provide accurate results. Furthermore, for researchers who prefer manual verification, the tool offers interfaces that display LLM results, serving as guidance for human reviewers throughout the screening process. One of the standout features of AISysRev is its parallelized LLM calls, which significantly boosts screening speed. Depending on the chosen model and hosting environment, users can expect a screening rate of between 100 to 300 papers per minute.
Trial Study and Findings
To validate the effectiveness of AISysRev, we conducted a qualitative trial study involving 137 research papers. Our analysis revealed that the papers could be classified into four distinct categories:
- Easy Includes
- Easy Excludes
- Boundary Includes
- Boundary Excludes
Notably, the Boundary cases—where LLMs struggled to make definitive classifications—highlighted the essential role of human intervention in the review process. While LLMs like those utilized in AISysRev provide invaluable support, they cannot fully replace human judgment in systematic reviews. Nonetheless, they significantly alleviate the burden of evaluating vast amounts of scientific literature.
Conclusion
AISysRev stands at the forefront of integrating artificial intelligence into the systematic review process. By harnessing the capabilities of LLMs, the tool not only accelerates the screening phase but also enhances the overall efficiency of systematic reviews. Researchers can now focus more on critical analysis and less on the arduous task of paper selection. For more information, you can check out our demonstration video or access the tool on GitHub.