ActuBench: A Multi-Agent LLM Pipeline for Generation and Evaluation of Actuarial Reasoning Tasks
Summary: arXiv:2604.20273v1 Announce Type: new
Abstract: We present ActuBench, a multi-agent LLM pipeline for the automated generation and evaluation of advanced actuarial assessment items aligned with the International Actuarial Association (IAA) Education Syllabus. The pipeline separates four LLM roles by adapter: one agent drafts items, one constructs distractors, a third independently verifies both stages and drives bounded one-shot repair loops, and a cost-optimized auxiliary agent handles Wikipedia-note summarization and topic labeling. The items, per-model responses and complete leaderboard are published as a browsable web interface at https://actubench.de/en/, allowing readers and practitioners to inspect individual items without a repository checkout.
We evaluate 50 language models from eight providers on two complementary benchmarks — 100 empirically hardest multiple-choice items and 100 open-ended items scored by an LLM judge — and report three headline findings.
- Multi-Agent Verification is Load-Bearing: The independent verifier flags a majority of drafted items on the first pass, most of which the one-shot repair loop resolves.
- Cost-Performance Optimization: Locally-hosted open-weights inference sits on the cost-performance Pareto front. A Gemma~4 model running on consumer hardware and a Cerebras-hosted 120B open-weights model dominate the near-zero-cost region, with the latter within one item of the top of the leaderboard.
- MCQ and LLM-as-Judge Rankings Differ Meaningfully: The MCQ scaffold inflates the performance ceiling, and Judge-mode evaluation is needed to discriminate at the frontier.
ActuBench represents a significant leap forward in the field of actuarial education and assessment. By utilizing a multi-agent system, it effectively streamlines the processes of item generation and evaluation, ensuring high-quality outputs that align with educational standards. The system’s architecture not only enhances efficiency but also introduces a robust verification mechanism, thereby increasing the reliability of the generated assessment items.
The implications of these findings are profound, especially for educators and practitioners in the actuarial field. With the ability to access a browsable web interface, users can easily navigate through individual items, gaining insights into the performance of different models and the effectiveness of the assessment items. This transparency fosters a deeper understanding of the evaluation process and encourages further advancements in actuarial education.
As the actuarial profession continues to evolve, tools like ActuBench will be invaluable in preparing future actuaries. The ongoing evaluation of language models and their ability to produce high-quality assessment items signifies a promising future for automated educational tools, enhancing both learning outcomes and professional standards in the actuarial community.