SWE-chat: Coding Agent Interactions From Real Users in the Wild
Summary: arXiv:2604.20779v1 Announce Type: new
Abstract
AI coding agents are being adopted at scale, yet we lack empirical evidence on how people actually use them and how much of their output is useful in practice. We present SWE-chat, the first large-scale dataset of real coding agent sessions collected from open-source developers in the wild. The dataset currently contains 6,000 sessions, comprising more than 63,000 user prompts and 355,000 agent tool calls. SWE-chat is a living dataset; our collection pipeline automatically and continually discovers and processes sessions from public repositories. Leveraging SWE-chat, we provide an initial empirical characterization of real-world coding agent usage and failure modes.
Key Findings
Our research into SWE-chat reveals several important findings:
- Bimodal Coding Patterns: We find that coding patterns are bimodal: in 41% of sessions, agents author virtually all committed code (“vibe coding”), while in 23%, humans write all code themselves.
- Efficiency Issues: Despite rapidly improving capabilities, coding agents remain inefficient in natural settings. Just 44% of all agent-produced code survives into user commits.
- Security Vulnerabilities: Agent-written code introduces more security vulnerabilities than code authored by humans.
- User Interventions: Users push back against agent outputs through corrections, failure reports, and interruptions in 44% of all turns.
Dataset Overview
The SWE-chat dataset is a milestone in understanding how coding agents interact with developers. It is designed to provide valuable insights into the practical application of AI in software development. The dataset is continuously updated through an automated pipeline that discovers and processes coding sessions from public repositories. This ensures that the dataset remains relevant and reflects the latest trends in coding practices.
Implications for Future Research
By capturing complete interaction traces with human vs. agent code authorship attribution, SWE-chat offers a robust empirical foundation for advancing research in AI-assisted coding. This dataset allows researchers to move beyond curated benchmarks and towards an evidence-based understanding of how AI agents perform in real developer workflows. The insights gained from SWE-chat can inform the development of more effective coding agents and contribute to enhancing their utility in software development.
Conclusion
SWE-chat represents a significant step forward in the analysis of AI coding agents in real-world environments. As the dataset grows and evolves, it will provide crucial insights that can help shape the future of AI in software development. Understanding the interactions between developers and coding agents will not only improve agent capabilities but also ensure that they are aligned with human needs and practices.