EMNLP 2026 · Oral

Left-Branching Transformers Excel at Right-Branching Languages

Data Shapes Word Order Preferences in Language Models

Varvara Arzt, Allan Hanbury, and Terra Blevins

TU Wien · Northeastern University

Oral presentation: 25 October 2026, 11:00–12:30, Session 2, room G15–16.

In short

Do language models handle all word orders equally well, and if not, do their preferences come from the architecture or from the data? We train GPT-2 models on 192 artificial languages that differ only in word order, and evaluate existing pretrained models on more than 70 typologically diverse natural languages: monolingual Goldfish models, which share the GPT-2 architecture and come in matched data sizes, plus multilingual mGPT, XGLM and BLOOM. The two settings point in opposite directions: the architecture prefers left-branching, but with enough data, better-resourced right-branching SVO languages become easiest.

Results

Stacked bar chart of 526 natural languages by number of right-branching switches, from 0R/5L to 5R/0L. Languages cluster at both ends: 59 and 65 mostly SOV languages at the left-branching end (e.g. Japanese), 99 and 257 mostly SVO and VSO languages at the right-branching end (e.g. English, Spanish). — **Natural languages cluster at both extremes.** Our artificial languages cover all six base word orders (the order of subject, verb and object), each combined with five internal constituent switches that are either left- or right-branching (complement clause, complementiser, adposition, adjective, relative clause). We label natural languages with the same scheme, using WALS, Grambank and APiCS, so both settings share one scale: e.g., English is SVO-RRRLR, Japanese SOV-LLLLL. The x-axis shows this directionality scale, the number of right-branching switches. Attested SOV languages tend to be consistently left-branching like Japanese, SVO languages consistently right-branching like English.

Line chart of mean perplexity for GPT-2 trained on 5 MB of artificial languages. For all six base orders, perplexity rises from fully left-branching (0R/5L) towards right-branching (5R/0L); SOV is lowest throughout, VSO highest. — **Left-branching is easiest, whatever the base order.** Across 192 artificial languages (6 base orders × 2⁵ switch settings) with identical content, perplexity rises with every right-branching switch. The easiest configuration for each base order has no natural counterpart, and prenominal relative clauses help most, against typology. GPT-2 small, 5 MB, mean over 10 runs (lower is easier).

Mean bits per English character for Goldfish models trained on 5 MB, 62 languages. SOV and SVO lines overlap in range around 2.1 to 2.3; no clear preference. — **With more data, right-branching SVO becomes easiest.** At 5 MB, monolingual Goldfish models show no clear word order preference (Cliff’s δ = 0.00). On the same 62 languages at 1 GB, SVO is clearly easier (δ = −0.76, p < .001): SOV is about 10% harder to compress, although it is the most frequent order worldwide. The gap is not explained by script, morphology or data composition; language resourcedness accounts for about 20% of it. Multilingual XGLM and mGPT show the same SVO advantage. FLORES-200, mean bits per English character (lower is easier).

Same 62 languages with Goldfish models trained on 1000 MB. SVO languages sit around 1.33 bits per character, clearly below SOV languages at 1.43 to 1.61. — **With more data, right-branching SVO becomes easiest.** At 5 MB, monolingual Goldfish models show no clear word order preference (Cliff’s δ = 0.00). On the same 62 languages at 1 GB, SVO is clearly easier (δ = −0.76, p < .001): SOV is about 10% harder to compress, although it is the most frequent order worldwide. The gap is not explained by script, morphology or data composition; language resourcedness accounts for about 20% of it. Multilingual XGLM and mGPT show the same SVO advantage. FLORES-200, mean bits per English character (lower is easier).

Takeaways

Systematic, but not human-like. On artificial languages, LMs prefer left-branching regardless of base order, against the word order correlations found in natural languages.
Data overrides the architectural bias. On natural languages at scale, better-resourced SVO languages become easiest, in monolingual and multilingual models.
Artificial and natural results go in opposite directions. SVO is the hardest base order on artificial languages and the easiest on natural ones at scale.

Even monolingual models trained on just 1 GB develop the SVO preference; frontier models, with far larger resource gaps between languages, likely amplify it. This is a precondition for silent typological homogenisation: languages with flexible word order may drift towards their dominant word orders.

Citation

@inproceedings{arzt2026leftbranching,
  title     = {Left-Branching Transformers Excel at Right-Branching Languages:
               Data Shapes Word Order Preferences in Language Models},
  author    = {Arzt, Varvara and Hanbury, Allan and Blevins, Terra},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods
               in Natural Language Processing},
  year      = {2026}
}