Chat2Workflow: A Benchmark for Generating Executable Visual Workflows with Natural Language
Published on: arXiv:2604.19667v1
Article Type: Cross-Disciplinary Research
Abstract
In the realm of industrial applications, executable visual workflows have become a predominant approach due to their reliability and controllability. However, the current methodology for constructing these workflows relies heavily on manual engineering. Developers are tasked with meticulously designing workflows, composing prompts for each step, and continuously revising logic as project requirements evolve. This manual process is costly, time-consuming, and susceptible to errors.
Introduction to Chat2Workflow
To explore the potential of automating this intricate process, we introduce Chat2Workflow, a benchmark aimed at generating executable visual workflows directly from natural language inputs. This initiative also includes a robust agentic framework designed to reduce recurrent execution errors that often plague workflow development.
Key Features of Chat2Workflow
Chat2Workflow is constructed from a comprehensive collection of real-world business workflows. Each instance is crafted to ensure that the generated workflows can be seamlessly transformed and deployed on practical workflow platforms like Dify and Coze. This presents a significant advancement towards automating the workflow generation process.
Experimental Findings
In our experiments, we discovered that while state-of-the-art language models can effectively capture high-level intents from user inputs, they often falter when tasked with generating correct, stable, and executable workflows, particularly when faced with complex or evolving requirements.
Performance Improvements
Our proposed agentic framework has demonstrated a notable improvement, achieving a resolve rate gain of up to 5.34%. Despite these advancements, there remains a considerable gap in achieving real-world applicability. This positions Chat2Workflow as a foundational tool for driving industrial-grade automation.
Conclusion
Chat2Workflow serves as a vital step towards bridging the gap between natural language processing and practical workflow automation. By providing a benchmark and an innovative framework, we aim to enhance the efficiency and reliability of workflow generation in various industrial contexts.
Access and Collaboration
The code for Chat2Workflow is publicly available, allowing researchers and developers to explore and contribute to this pioneering project. You can access the code at the following link:
Future Directions
As we continue to refine Chat2Workflow, we invite collaboration from the research community to further enhance its capabilities. The ongoing challenges in generating executable workflows from natural language inputs present ample opportunities for innovation and improvement.