Autoregressive Mosaics: Probing 2D Spatial Reasoning in Text-Only Language Models
Abstract
Text-only large language models vary in open-ended 2D layout ability despite reliable geometry-to-code translation, with performance shaped by output medium and internal tracking of evolving geometric state rather than fixed plans.
Large language models (LLMs) trained only on text and code can sometimes generate programs that draw recognizable images. However, it is unclear whether this reflects an internal representation of 2D spatial layout or simply the ability to translate spatial descriptions into code. We introduce Autoregressive Mosaics (AM-Bench), a benchmark that separates these factors: First, a translation task gives a model a fully specified geometry of a picture in words as a prompt and asks for the code that produces it. Second, a layout task requires the model to compose an image from an underspecified prompt. Across eight open-weight text-and-code-only models, all models reliably translate specified geometry into code, but their open-ended layout performance differs substantially, indicating that these differences are not explained by code-generation ability alone. An output-medium ablation further shows that the interface or medium of expression that the model uses matters: replacing procedural code with raw SVG improves layout scores across all models. Finally, probing model activations shows that a coarse layout plan is present before generation, but reflects only the layout implied by the prompt. During generation, models track the evolving geometric state instead of executing an initially fixed plan. Overall, these results show that 2D spatial performance in text-only LLMs depends on both the model and the output medium, and is not explained by code-generation ability alone.
Community
TL;DR
We introduce Autoregressive Mosaics (AM-Bench) to evaluate whether text-only LLMs possess genuine 2D spatial reasoning or merely translate spatial text into syntax.
The Core Problem
When LLMs generate code that draws an image, it remains unclear if this reflects an internal understanding of 2D spatial layouts or simply an adept translation of explicit geometric instructions into code.
Methodology
AM-Bench decouples spatial reasoning from code-generation ability through two targeted tasks. The translation task provides a fully specified geometric description and requires the model to generate corresponding code. The layout task provides an underspecified prompt, forcing the model to autonomously compose the 2D spatial arrangement.
Key Findings
Across eight open-weight models, baseline translation ability is universally high, yet open-ended layout performance varies drastically, proving that spatial reasoning is distinct from pure code generation. The output medium also heavily dictates performance; replacing procedural code with raw SVG universally improves layout scores. Finally, activation probing reveals that models form only a coarse initial plan based on the prompt, dynamically tracking the evolving geometric state during autoregressive generation rather than executing a fixed structural plan.
Get this paper in your agent:
hf papers read 2608.30751 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper