IndustryCode: A Benchmark for Industry Code Generation
Summary: arXiv:2604.02729v1 Announce Type: cross
Abstract: Code generation and comprehension by Large Language Models (LLMs) have emerged as core drivers of industrial intelligence and decision optimization, finding widespread application in fields such as finance, automation, and aerospace. Although recent advancements have demonstrated the remarkable potential of LLMs in general code generation, existing benchmarks are mainly confined to single domains and languages. Consequently, they fail to effectively evaluate the generalization capabilities required for real-world industrial applications or to reflect the coding proficiency demanded by complex industrial scenarios.
To bridge this gap, we introduce IndustryCode, the first comprehensive benchmark designed to span multiple industrial domains and programming languages. IndustryCode comprises 579 sub-problems derived from 125 primary industrial challenges, accompanied by rigorous problem descriptions and test cases. It covers a wide range of fields, including finance, automation, aerospace, and remote sensing, and incorporates diverse programming languages such as MATLAB, Python, C++, and Stata.
Key Features of IndustryCode
- Multi-domain Coverage: IndustryCode addresses challenges across various sectors, ensuring that the benchmark is relevant to a wide array of industrial applications.
- Diverse Programming Languages: The benchmark includes problems that involve multiple programming languages, allowing for a more holistic evaluation of LLM capabilities.
- Extensive Problem Set: With 579 sub-problems based on 125 primary challenges, IndustryCode provides a robust dataset for evaluating the performance of code generation models.
- Rigorous Evaluation Criteria: Each problem is accompanied by detailed descriptions and test cases, enabling precise measurement of model performance.
Evaluation Results
In our evaluation, the top-performing model, Claude 4.5 Opus, achieved an overall accuracy of 68.1% on sub-problems and 42.5% on main problems. These results underscore the potential of LLMs in tackling complex industrial coding challenges while also highlighting areas for improvement.
Future Directions
The introduction of IndustryCode marks a significant step towards enhancing the evaluation of LLMs in industrial settings. As the field continues to evolve, future work will focus on:
- Expanding the benchmark to include additional industrial domains and programming languages.
- Improving the complexity of problems to better simulate real-world scenarios.
- Incorporating user feedback to refine the benchmark and its evaluation metrics.
Conclusion
IndustryCode represents a crucial advancement in the landscape of code generation benchmarks. By providing a comprehensive framework for evaluating LLMs across multiple domains and languages, it sets the stage for more effective industrial applications of artificial intelligence. The benchmark dataset and automated evaluation code will be made publicly available upon acceptance, fostering collaboration and innovation in the field.