SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration
Summary: arXiv:2603.03823v4 Announce Type: replace-cross
Abstract: Large language model (LLM)-powered agents have demonstrated strong capabilities in automating software engineering tasks such as static bug fixing. However, in the real world, the development of mature software is typically predicated on complex requirement changes and long-term feature iterations — a process that static, one-shot repair paradigms fail to capture. To bridge this gap, we propose SWE-CI, the first repository-level benchmark built upon the Continuous Integration loop, aiming to shift the evaluation paradigm for code generation from static, short-term functional correctness toward dynamic, long-term maintainability. The key insight is simple: Maintainability can be revealed by tracking how functional correctness changes over time. The benchmark comprises 100 tasks, each deriving from a real-world code repository with a development history spanning an average of 233 days and 71 consecutive commits. SWE-CI requires agents to systematically resolve these tasks through dozens of rounds of analysis and coding iterations. SWE-CI provides valuable insights into how well agents can sustain code quality throughout long-term evolution.
Introduction to SWE-CI
The advent of large language models has transformed the landscape of software engineering by enabling more efficient automation of various tasks. Despite these advancements, the traditional methods of evaluating code generation capabilities frequently rely on static assessments. This approach often overlooks the dynamic nature of software development, where requirements evolve and codebases undergo continuous changes.
The Need for Dynamic Testing
Static bug fixing tools, while useful, do not account for the complexities inherent in real-world software development. As software projects grow, the demand for maintaining code quality and ensuring long-term functionality becomes paramount. SWE-CI addresses this need by creating a benchmark that evaluates the performance of LLM-powered agents over extended periods.
Key Features of SWE-CI
- Repository-Level Benchmark: SWE-CI utilizes a repository-level approach that reflects real-world scenarios and maintains authenticity in testing environments.
- Long-Term Evaluation: Each task is designed to track the evolution of functional correctness over a significant timeframe, providing insights into maintainability.
- Comprehensive Task Design: The benchmark consists of 100 tasks, each derived from actual codebases with an average development history of 233 days.
- Iterative Analysis: Agents are required to engage in multiple rounds of analysis and coding iterations, enhancing their ability to adapt to changing requirements.
Conclusion
SWE-CI represents a significant shift in how we evaluate the capabilities of AI agents in software development. By focusing on long-term maintainability and the evolution of functional correctness, this benchmark not only sets a new standard for evaluation but also provides deeper insights into the performance of LLMs as they navigate the complexities of real-world software engineering tasks. As the field progresses, metrics like those proposed by SWE-CI will be crucial in guiding future developments in AI-assisted software engineering.