OmniTabBench: Mapping the Empirical Frontiers of GBDTs, Neural Networks, and Foundation Models for Tabular Data at Scale
Summary: arXiv:2604.06814v1 Announce Type: cross
Abstract: While traditional tree-based ensemble methods have long dominated tabular tasks, deep neural networks and emerging foundation models have challenged this primacy, yet no consensus exists on a universally superior paradigm. Existing benchmarks typically contain fewer than 100 datasets, raising concerns about evaluation sufficiency and potential selection biases. To address these limitations, we introduce OmniTabBench, the largest tabular benchmark to date, comprising 3030 datasets spanning diverse tasks that are comprehensively collected from diverse sources and categorized by industry using large language models.
We conduct an unprecedented large-scale empirical evaluation of state-of-the-art models from all model families on OmniTabBench, confirming the absence of a dominant winner. Furthermore, through a decoupled metafeature analysis, which examines individual properties such as dataset size, feature types, feature and target skewness/kurtosis, we elucidate conditions favoring specific model categories, providing clearer, more actionable guidance than prior compound-metric studies.
Introduction
In the realm of machine learning, the competition between various modeling paradigms has led to significant advancements, particularly in the analysis of tabular data. Traditional methods, primarily gradient-boosted decision trees (GBDTs), have been the go-to techniques for many practitioners. However, with the rise of deep learning and foundation models, the landscape has evolved, prompting researchers to reevaluate which models truly excel in handling tabular datasets.
The Need for a Comprehensive Benchmark
Existing benchmarks have often been limited in scope, typically comprising fewer than 100 datasets. This raises critical questions regarding the reliability of the evaluations and the potential for biases in dataset selection. To overcome these challenges, OmniTabBench was developed as the most extensive benchmark for tabular data to date, featuring a staggering 3030 datasets.
Key Features of OmniTabBench
- Diversity of Datasets: The benchmark covers a wide range of tasks, ensuring a comprehensive evaluation across different industries.
- Large-Scale Empirical Evaluation: All state-of-the-art models from various families were tested on OmniTabBench, allowing for a rigorous comparison.
- Decoupled Metafeature Analysis: This innovative approach enables a deeper understanding of model performance based on dataset characteristics.
Findings and Implications
The initial findings from the evaluations conducted on OmniTabBench indicate that there is no single model that outperforms all others across the board. Instead, the performance varies depending on specific dataset features, suggesting that practitioners should consider these characteristics when selecting models for their tasks.
Conclusion
OmniTabBench represents a significant step forward in the evaluation of machine learning models for tabular data. By providing a large and diverse set of datasets, it allows for a more nuanced understanding of model performance and fosters informed decision-making. As the field of machine learning continues to evolve, resources like OmniTabBench will be invaluable in guiding practitioners to select the most suitable models for their specific needs.