Ordered Tokens Enable Efficient Test-Time Search
Summary: arXiv:2604.15453v1 Announce Type: cross
Abstract: Tokenization is a key component of autoregressive (AR) generative models, converting raw data into more manageable units for modeling. Commonly, tokens describe local information, such as regions of pixels in images or word pieces in text, and AR generation predicts these tokens in a fixed order. A worthwhile question is whether token structures affect the ability to steer the generation through test-time search, where multiple candidate generations are explored and evaluated by a verifier. Using image generation as our testbed, we hypothesize that recent 1D ordered tokenizers with coarse-to-fine structure can be more amenable to search than classical 2D grid structures. This is rooted in the fact that the intermediate states in coarse-to-fine sequences carry semantic meaning that verifiers can reliably evaluate, enabling effective steering during generation.
Through controlled experiments, we find that AR models trained on coarse-to-fine ordered tokens exhibit improved test-time scaling behavior compared to grid-based counterparts. Moreover, we demonstrate that, thanks to the ordered structure, pure test-time search over token sequences (i.e., without training an AR model) can perform training-free text-to-image generation when guided by an image-text verifier. Beyond this, we systematically study how classical search algorithms interact with different token structures, as well as the role of different verifiers and AR priors. Our results highlight the impact of token structure on inference-time scalability and provide practical guidance for test-time scaling in AR models.
Key Findings
- Improved Test-Time Scaling: AR models using coarse-to-fine ordered tokens show better scalability during test-time compared to traditional grid-based models. This indicates a significant advantage in real-world applications where efficiency is paramount.
- Training-Free Generation: The study reveals that it is possible to conduct test-time searches over token sequences without prior training of an AR model, enabling efficient text-to-image generation when guided by a verifier.
- Search Algorithms Interaction: The research explores how various classical search algorithms, such as best-of-N, beam search, and lookahead search, interact with different token structures, revealing important insights into their relative effectiveness.
- Verifiers and AR Priors: The role of different verifiers and autoregressive priors is examined, shedding light on how they influence the overall generation process and the efficacy of token structures.
Conclusion
The findings from this research underscore the importance of token structures in enhancing the performance and scalability of autoregressive generative models. By adopting coarse-to-fine ordered tokenization, practitioners can achieve more efficient test-time searches, which is crucial for applications involving complex data generation, such as image synthesis. As the field of AI continues to evolve, these insights provide a pathway for developing more effective generative models capable of meeting the increasing demands for efficiency and quality in outputs.