Don’t feed back the last blockThe best source we found sits just below the top
TL;DR
Several recent models feed the previous token's last-block state into the next token's input. We swept the source over depth and found that reading just below the top works better.
- Read just below the top. The best sources we tested are block 7 of 8 and block 8 of 10, lowering held-out loss by 4.4 and 4.1 ×10−3 nats relative to the last block.
- Add the state to the token input. In the 8-block model, the additive recipe outperforms FBT's gated replacement when both read the last block.
- Changing the source costs no extra compute. It selects a different block's state while keeping the same parameters and FLOPs.
Transformers with feedback
A transformer starts each position from its token embedding. Attention gives it access to earlier positions through their keys and values, but its first block cannot directly read the deeper hidden states computed for the previous token.
Feedback opens that route. A simple additive form takes a hidden state h from position t−1, projects it, and adds it to the embedding e_t to form the next position's input x_0:
x_0 = e_t + W\operatorname{RMSNorm}(h).Here \operatorname{RMSNorm} is RMS normalisation without learned parameters. In our runs, W is initialised at zero, so enabling the path initially leaves the model unchanged.
During sequential decoding, the previous position has already finished its stack. Keeping its source state for the next token makes the extra work just one matrix product per token.
Training computes positions in parallel, so that state is unavailable when the next position needs it. We run the stack twice per step: the first pass produces the states and the second consumes them, roughly doubling training compute. Prompt prefilling has the same dependency and can also use repeated passes.
Where feedback reads and writes
Feedback designs differ in both the state they read and where they write it into the next position. The closest comparisons for this post feed a single state into the token input.
Feedback at the token input. The Full-bandwidth Transformer builds a gated product of the state and the embedding and uses it as the input [1]. Recycler [2] adds the state through RMS normalisation and a learned linear projection. Huang et al. [3] study the same additive form under repeated-data training, comparing the compute and data needed to reach a given loss. These works write the state into the token input in one of two ways:
\begin{aligned} x_0 &= \operatorname{norm}\big(W_s h \odot \sigma(W_g \operatorname{norm}(e_t))\big) &&\quad\text{FBT}\\[2pt] x_0 &= e_t + W\operatorname{RMSNorm}(h) &&\quad\text{Recycler} \end{aligned}All three works use a last-block source. Here h denotes that top-layer state, h^{(t-1)}_L, in a stack of L blocks.
Feedback within the stack. The Transformer with Temporal Middle-Layer Recurrence (T²MLR) [4] feeds a middle-layer state into an earlier layer of the next position. Its experiments show that recurrence confined to a middle portion of the network can outperform full-layer recurrence. Previous to Next (P2N) [5] adds the previous token's output from a middle group of blocks to the current token's input to that group. Turbo Connection [6] uses multiple connections from higher layers of one token to lower layers of the next. These designs explore intermediate states together with the layers that receive them.
The question here. These studies motivate testing intermediate states even when feedback enters at the token embedding. We keep that destination and the additive projection fixed, then vary only the source block b:
h = h^{(t-1)}_b, \qquad b \in \{1, \dots, L\}.This isolates the source choice in the input-feedback setting, with the same parameters and FLOPs at every depth. We then compare the additive and gated input recipes with their source fixed at the last block.
Setup
We use two Marin mixture-of-experts language models [7] trained on Dolma, with 8 and 10 blocks. For each size, we enable feedback at a shared checkpoint and finish the original training schedule. The depth-sweep runs share the data order and differ only in the source block.
We report held-out loss after three passes. Pass 1 evaluates each trained model with feedback disabled, providing a reference for how much the path contributes at fixed weights. Later passes read the states from the preceding pass. Model configurations and evaluation details are in How the runs were made.
Sweeping the source
We test four sources in each model, from the lower half of the stack to the last block. In Figure 3, loss falls as the source gets deeper, reaches its minimum just below the top, and rises again at the last block.
| Feedback source | Pass 1 | Pass 3 | Gain | vs. last block |
|---|---|---|---|---|
| 768 wide, 8 blocks | ||||
| block 3 · 0.375 | 2.0574 | 2.0513 | −6.1 | +12.5 |
| block 5 · 0.625 | 2.0578 | 2.0398 | −18.0 | +1.0 |
| block 7 · 0.875 | 2.0584 | 2.0344 | −24.0 | −4.4 |
| block 8 (last) · 1.000 | 2.0585 | 2.0388 | −19.7 | / |
| 896 wide, 10 blocks | ||||
| block 4 · 0.400 | 1.9400 | 1.9357 | −4.3 | +6.7 |
| block 6 · 0.600 | 1.9405 | 1.9286 | −11.9 | −0.3 |
| block 8 · 0.800 | 1.9409 | 1.9248 | −16.1 | −4.1 |
| block 10 (last) · 1.000 | 1.9408 | 1.9290 | −11.9 | / |
The best tested source lowers loss by 4.4 ×10−3 nats in the smaller model and 4.1 ×10−3 nats in the larger one, relative to the last block.
To put the improvement in context, compare it with the loss reduction from enabling feedback in each trained last-block model (the Gain column in Table 1). The source change is worth 22% of that reduction in the smaller model and 35% in the larger one.
Moving too far down the stack reverses the benefit: the shallowest sources give substantially higher loss than the last block. The useful region in these sweeps is near the top.
Finding 1: feeding back a block just below the top gives the lowest loss at both model sizes, with no extra parameters or FLOPs.
Adding beats replacing
With the source fixed at the last block, we compare additive feedback with FBT's gated replacement in the 8-block model. Both runs start from the same checkpoint with the same data order. We also test FBT with its input jitter disabled (Table 2).
| How the state enters the token input | Pass 1 | Pass 2 | Pass 3 |
|---|---|---|---|
| replace: \operatorname{norm}(W_s h \odot \sigma(W_g \operatorname{norm}(e_t))) · FBT | 2.0621 | 2.0442 | 2.0447 |
| replace, with the jitter switched off · FBT, jitter off | 2.0622 | 2.0442 | 2.0449 |
| add: e_t + W\operatorname{RMSNorm}(h) · Recycler-style | 2.0585 | 2.0395 | 2.0388 |
| adding − replacing | −3.6 | −4.7 | −5.9 |
The additive recipe lowers the final loss by 5.9 ×10−3 nats. Even with feedback disabled at pass 1, the additive model leads by 3.6 ×10−3 nats, so part of the final gap is already present in the weights learned during training.
More passes do not close the gap. FBT's loss rises after pass 2, while the additive model changes little after pass 3. Turning off FBT's jitter also leaves its loss nearly unchanged.
The recipes differ in both gating and normalisation, so this comparison measures their combined effect. The additive recipe also uses fewer parameters.
Finding 2: with the same last-block source, additive feedback outperforms FBT's gated replacement.
Why the last block might be the wrong one
The last block feeds the output head, which trains it to predict the next token. At position t−1, it is trained to predict token t, which position t already receives as an embedding. Our hypothesis is that feeding back this state repeats too much of the information the next position already has.
A state just below the top may retain useful context while being less specialised for the output head. Shallower states have processed less context, which could explain why loss first falls and then rises as the source moves up the stack.
How the runs were made
Feedback is enabled at 75% of the smaller model's training schedule and 88% of the larger model's schedule (Table 3). We compare source depths within each size, where the training budget is shared.
| Model | Backbone params | Active params / token | Blocks | Hidden width | Query / key-value heads | Expert hidden width | MoE latent width | Batch (sequences) | Pretraining steps | Pretraining tokens | Feedback start step | Feedback training steps | Added feedback params |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Marin d768 | 1,591,848,448 | 60,555,264 | 8 | 768 | 6 / 1 | 384 | 384 | 1,024 | 11,420 | 47.9B | 8,565 | 2,855 | 589,824 |
| Marin d896 | 2,601,062,016 | 102,072,320 | 10 | 896 | 7 / 1 | 448 | 448 | 1,440 | 13,689 | 80.7B | 12,000 | 1,689 | 802,816 |
The depth sweep keeps the backbone's attention and residual connections unchanged. Its feedback projection is one matrix initialised at zero, and the first token of a document receives nothing. The FBT recipe uses 0.02 input jitter, which the additional control disables.
During sequential decoding, each position reads a state that was itself computed with feedback. Parallel evaluation approximates this by repeating passes over the sequence, each reading states from the preceding pass. We report pass 3. For the 8-block additive model with a last-block source, loss changes by only 0.04 ×10−3 nats on the next pass. FBT's loss continues to rise, by 0.7 ×10−3 nats from pass 3 to pass 9.
The held-out set contains 42M tokens from six Dolma sources. We report macro loss in nats, the unweighted mean of the per-source losses.
Limitations
We tested two model sizes with one seed per run and measured held-out loss, without downstream evaluation. Repeated seeds are needed to assess how reliably the differences hold. The coarse depth grid also leaves open whether the best source stays a fixed number of blocks below the top or at a fixed fraction of the stack.
The sweep does not establish why intermediate states work better. Feedback may also change how the source block learns. The recipe comparison covers only the last-block source in the smaller model, so its interaction with source depth remains untested. All runs enable feedback partway through pretraining, leaving training from scratch and a matched continuation without feedback for future comparisons.
What we take from this
Source depth is a useful design choice for feedback models. In both sweeps, a block just below the top gives lower loss at the same parameter count and FLOPs. If your model feeds back the last block, try using a deep intermediate layer as the source.
References
Cite this blog
@misc{xiong2026feedbacksource,
title = {{Don't feed back the last block: the best source we found sits just below the top}},
author = {Xiong, Xuyuan and Chen, Siyu and Chen, Mengzhao},
year = {2026},
month = {nov},
note = {Blog post},
url = {https://xjxyys.github.io/blog/feedback-source/}
}

