Truncated backpropagation through time (BPTT) trains only the last \(B\) of \(T\) loops of a looped transformer to save compute. We fit one scaling law over model size, data and both loop counts, in which loops add effective parameters and truncation discounts them. Its answer to the title: truncate only when the number of forward loops is fixed in advance. When both loop counts can be chosen, use full BPTT through fewer loops. With model size free as well, the law favors no looping, and two fully trained loops come closest, saving about 30% of the parameters for little extra compute (figure below).
Looped transformers reuse a shared block of layers to increase computational depth without increasing the number of trainable parameters. They are trained with backpropagation through time (BPTT). To save training compute, BPTT is often truncated to the last few loops, but truncation yields biased gradients that may weaken the benefit of looping. In this work, we study the value of truncation in looped transformer training in a systematic way: we jointly vary model size, training tokens, and forward and backward loop counts across nearly 600 training runs and fit a unified scaling law that captures the effects of looping and truncation through effective parameters. Despite the training-compute savings of truncation, our fitted law favors full BPTT when forward and backward loop counts are jointly optimized. In matched-compute runs, full BPTT through fewer loops consistently outperforms truncated settings of the same compute. The law further shows that at fixed model size, the compute-optimal loop count and training-token budget both grow with compute. When model size is also free, the law identifies two fully trained loops as the best looped setting. Experimentally, this setting achieves lower validation loss than the compute-optimal non-looped model with about 27% fewer non-embedding parameters and only 8% more training compute.
We measure what looping is worth by an effective parameter count: the size of a non-looped model that reaches the same loss on the same tokens. Each fully trained loop adds effective parameters, with diminishing returns toward a finite ceiling. Truncation discounts this gain, and from the same setting, doubling \(B\) adds more effective parameters than doubling \(T\).
A forward pass through the recurrent block costs one unit of compute and a backward pass two, so a loop setting \((T,B)\) costs \(T+2B\) units per token: \((8,2)\) and \((4,4)\) cost the same. Whether truncation pays depends on what is held fixed.
The saved backward compute buys a larger model and more tokens, which more than offset the discounted loop gain. At \(T=8\) the law picks \(B^*=2\); laws fitted separately at \(T=4\) and \(T=8\) agree.
34% less compute at equal loss for \((8,2)\) vs. \((8,8)\)
At fixed loop compute, one backward loop is worth more than the two forward loops of the same cost (\(q\approx1.02>2p\approx0.47\)), so the law is maximized with every loop trained. Rather than stopping gradients on early loops, run fewer loops and train them all.
5 / 5 model sizes where \((4,4)\) beats the equal-cost \((8,2)\)
| Size | Compute \(C\) (\(10^{19}\) FLOPs) | Loss \((8,2)\) | Loss \((4,4)\) | \(\Delta L\) |
|---|---|---|---|---|
| 60M | 2.41 | 2.5211 | 2.4933 | 0.0278 |
| 120M | 4.59 | 2.3843 | 2.3696 | 0.0147 |
| 240M | 8.28 | 2.2778 | 2.2642 | 0.0136 |
| 480M | 15.29 | 2.1845 | 2.1738 | 0.0107 |
| 1B | 28.38 | 2.1144 | 2.1057 | 0.0087 |
When a model must fit a memory budget, extra compute can buy more loops per token or more training tokens. Under the fitted law with full BPTT, the compute-optimal loop count grows as \(C^{0.24}\) and the token count as \(C^{0.76}\), where \(C\) is the training compute; the runs at every size agree.
Loop counts are integers, so the best setting changes in steps: no looping, then two, then four trained loops. Between steps, added compute goes to tokens.
10× more compute at a fixed size
When model size is free, compute spent repeating a block could instead train more unique parameters, so the law favors no looping at every compute (figure above). Among looped settings, \((2,2)\) comes closest: \(g(2,2)\approx1.91\), so two shared loops are worth almost two untied copies of the block.
In the runs, the compute-optimal \((2,2)\) model stays within 0.008 nats of the compute-optimal model without looping at equal training compute, with 27–32% fewer non-embedding parameters. With 8% more compute, its loss falls below.
Reference budget, \(2.1\times10^{19}\) FLOPs
Pick forward loops \(T\) and backward loops \(B\). The numbers come from the fitted law; try a truncated setting such as \((8,2)\) and compare it with full BPTT at the same compute.
forward unit backward unit (2 per trained loop)
Fitted coefficients \(c=1.112\), \(p=0.237\), \(q=1.022\); \(N_{\mathrm{rec}}/N=0.507\) as in the 240M model. Compute per token is the leading-FLOP count \(6N_{\mathrm{comp}}\) with \(N_{\mathrm{comp}}=N_{\mathrm{once}}+\tfrac{T+2B}{3}N_{\mathrm{rec}}\); at equal loss, training compute is proportional to compute per effective parameter. Runs cover \(T\le16\).
The validation loss follows a coupled size–data law in which the model size is replaced by the effective size. The looping coefficient adds three coefficients \(c,p,q\) to the six of the size–data law; all nine are fitted jointly on 230 configurations (\(R^2=0.9992\)).
A single loop recovers the non-looped model.
More loops of either kind raise the coefficient.
An untrained recurrent block adds no effective size.
Finite parameters store finite information.
Size–data law
| \(E\) | \(A\) | \(H\) | \(\alpha\) | \(\beta\) | \(\kappa\) |
|---|---|---|---|---|---|
| 0.2986 | 6.4870 | 7.6334 | 0.3057 | 0.3928 | 0.3251 |
Looping coefficient
| \(c\) | \(p\) | \(q\) |
|---|---|---|
| 1.1120 | 0.2368 | 1.0218 |
Reference scales \(N_0=2.4\times10^{8}\) non-embedding parameters and \(D_0=2^{30}\approx1.07\times10^{9}\) tokens; loss in nats per token on deduplicated held-out data.
@misc{xiong2026truncate,
title = {{Truncate or Not? A Scaling Study of Looped Transformers
with Truncated Backpropagation}},
author = {Xuyuan Xiong and Siyu Chen and Zihan Shao and Yue Su and
Weiyang Jin and Ping Luo and Zhuoran Yang and Xihui Liu},
year = {2026},
eprint = {2610.26030},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2610.26030}
}This work builds on the looped language model architecture, data pipeline, and MuonH + HyperP training recipe of Schwethelm et al. (2026) (kschwethelm/looped-lm-scaling) and on the recurrent-depth design of Huginn (Geiping et al., 2025). We thank the authors for releasing their code.