Don’t feed back the last blockThe best source we found sits just below the top

TL;DR

Several recent models feed the previous token's last-block state into the next token's input. We swept the source over depth and found that reading just below the top works better.

  • Read just below the top. The best sources we tested are block 7 of 8 and block 8 of 10, lowering held-out loss by 4.4 and 4.1 ×10−3 nats relative to the last block.
  • Add the state to the token input. In the 8-block model, the additive recipe outperforms FBT's gated replacement when both read the last block.
  • Changing the source costs no extra compute. It selects a different block's state while keeping the same parameters and FLOPs.

Transformers with feedback

A transformer starts each position from its token embedding. Attention gives it access to earlier positions through their keys and values, but its first block cannot directly read the deeper hidden states computed for the previous token.

Feedback opens that route. A simple additive form takes a hidden state h from position t−1, projects it, and adds it to the embedding e_t to form the next position's input x_0:

x_0 = e_t + W\operatorname{RMSNorm}(h).

Here \operatorname{RMSNorm} is RMS normalisation without learned parameters. In our runs, W is initialised at zero, so enabling the path initially leaves the model unchanged.

Fig. 1 Cross-position feedback. Each position receives a projected hidden state from the previous position. Orange marks the feedback path, blue the current position's forward path, and purple their addition.

During sequential decoding, the previous position has already finished its stack. Keeping its source state for the next token makes the extra work just one matrix product per token.

Training computes positions in parallel, so that state is unavailable when the next position needs it. We run the stack twice per step: the first pass produces the states and the second consumes them, roughly doubling training compute. Prompt prefilling has the same dependency and can also use repeated passes.

Where feedback reads and writes

Feedback designs differ in both the state they read and where they write it into the next position. The closest comparisons for this post feed a single state into the token input.

Feedback at the token input. The Full-bandwidth Transformer builds a gated product of the state and the embedding and uses it as the input [1]. Recycler [2] adds the state through RMS normalisation and a learned linear projection. Huang et al. [3] study the same additive form under repeated-data training, comparing the compute and data needed to reach a given loss. These works write the state into the token input in one of two ways:

\begin{aligned} x_0 &= \operatorname{norm}\big(W_s h \odot \sigma(W_g \operatorname{norm}(e_t))\big) &&\quad\text{FBT}\\[2pt] x_0 &= e_t + W\operatorname{RMSNorm}(h) &&\quad\text{Recycler} \end{aligned}

All three works use a last-block source. Here h denotes that top-layer state, h^{(t-1)}_L, in a stack of L blocks.

Feedback within the stack. The Transformer with Temporal Middle-Layer Recurrence (T²MLR) [4] feeds a middle-layer state into an earlier layer of the next position. Its experiments show that recurrence confined to a middle portion of the network can outperform full-layer recurrence. Previous to Next (P2N) [5] adds the previous token's output from a middle group of blocks to the current token's input to that group. Turbo Connection [6] uses multiple connections from higher layers of one token to lower layers of the next. These designs explore intermediate states together with the layers that receive them.

The question here. These studies motivate testing intermediate states even when feedback enters at the token embedding. We keep that destination and the additive projection fixed, then vary only the source block b:

h = h^{(t-1)}_b, \qquad b \in \{1, \dots, L\}.

This isolates the source choice in the input-feedback setting, with the same parameters and FLOPs at every depth. We then compare the additive and gated input recipes with their source fixed at the last block.

Fig. 2 Changing the source block. The two panels differ only in the source block. Both project its state and add it to the next token's input. Faded blocks still run, but their outputs are not read by the feedback path.

Setup

We use two Marin mixture-of-experts language models [7] trained on Dolma, with 8 and 10 blocks. For each size, we enable feedback at a shared checkpoint and finish the original training schedule. The depth-sweep runs share the data order and differ only in the source block.

We report held-out loss after three passes. Pass 1 evaluates each trained model with feedback disabled, providing a reference for how much the path contributes at fixed weights. Later passes read the states from the preceding pass. Model configurations and evaluation details are in How the runs were made.

Sweeping the source

We test four sources in each model, from the lower half of the stack to the last block. In Figure 3, loss falls as the source gets deeper, reaches its minimum just below the top, and rises again at the last block.

Fig. 3 Both sizes do best below the top. Loss difference from the last-block source, plotted against relative source depth. Lower is better. Each colour denotes a model size, and each point is labelled with its source block and total block count.
Feedback sourcePass 1Pass 3Gainvs. last block
768 wide, 8 blocks
block 3 · 0.3752.05742.0513−6.1+12.5
block 5 · 0.6252.05782.0398−18.0+1.0
block 7 · 0.8752.05842.0344−24.0−4.4
block 8 (last) · 1.0002.05852.0388−19.7/
896 wide, 10 blocks
block 4 · 0.4001.94001.9357−4.3+6.7
block 6 · 0.6001.94051.9286−11.9−0.3
block 8 · 0.8001.94091.9248−16.1−4.1
block 10 (last) · 1.0001.94081.9290−11.9/
Tab. 1 Held-out macro loss, in nats, at the end of training. Pass 1 is the same model with the feedback switched off; the gain and the last column are in 10−3 nats; negative is better in both delta columns. Best source per size in bold.

The best tested source lowers loss by 4.4 ×10−3 nats in the smaller model and 4.1 ×10−3 nats in the larger one, relative to the last block.

To put the improvement in context, compare it with the loss reduction from enabling feedback in each trained last-block model (the Gain column in Table 1). The source change is worth 22% of that reduction in the smaller model and 35% in the larger one.

Moving too far down the stack reverses the benefit: the shallowest sources give substantially higher loss than the last block. The useful region in these sweeps is near the top.

Finding 1: feeding back a block just below the top gives the lowest loss at both model sizes, with no extra parameters or FLOPs.

Adding beats replacing

With the source fixed at the last block, we compare additive feedback with FBT's gated replacement in the 8-block model. Both runs start from the same checkpoint with the same data order. We also test FBT with its input jitter disabled (Table 2).

How the state enters the token inputPass 1Pass 2Pass 3
replace: \operatorname{norm}(W_s h \odot \sigma(W_g \operatorname{norm}(e_t))) · FBT2.06212.04422.0447
replace, with the jitter switched off · FBT, jitter off2.06222.04422.0449
add: e_t + W\operatorname{RMSNorm}(h) · Recycler-style2.05852.03952.0388
adding − replacing−3.6−4.7−5.9
Tab. 2 Additive feedback and FBT's gated replacement. Held-out macro loss in nats for the 8-block model. All runs share the checkpoint, data order and last-block source. The last row is in 10−3 nats, where negative favours addition.

The additive recipe lowers the final loss by 5.9 ×10−3 nats. Even with feedback disabled at pass 1, the additive model leads by 3.6 ×10−3 nats, so part of the final gap is already present in the weights learned during training.

More passes do not close the gap. FBT's loss rises after pass 2, while the additive model changes little after pass 3. Turning off FBT's jitter also leaves its loss nearly unchanged.

The recipes differ in both gating and normalisation, so this comparison measures their combined effect. The additive recipe also uses fewer parameters.

Finding 2: with the same last-block source, additive feedback outperforms FBT's gated replacement.

Why the last block might be the wrong one

The last block feeds the output head, which trains it to predict the next token. At position t−1, it is trained to predict token t, which position t already receives as an embedding. Our hypothesis is that feeding back this state repeats too much of the information the next position already has.

A state just below the top may retain useful context while being less specialised for the output head. Shallower states have processed less context, which could explain why loss first falls and then rises as the source moves up the stack.

How the runs were made

Feedback is enabled at 75% of the smaller model's training schedule and 88% of the larger model's schedule (Table 3). We compare source depths within each size, where the training budget is shared.

ModelBackbone paramsActive params / tokenBlocksHidden widthQuery / key-value headsExpert hidden widthMoE latent widthBatch (sequences)Pretraining stepsPretraining tokensFeedback start stepFeedback training stepsAdded feedback params
Marin d7681,591,848,44860,555,26487686 / 13843841,02411,42047.9B8,5652,855589,824
Marin d8962,601,062,016102,072,320108967 / 14484481,44013,68980.7B12,0001,689802,816
Tab. 3 Backbone configurations and the depth-sweep schedule. Active parameter counts exclude embeddings. Both models use 384 routed experts per block, selecting 8 per token, plus 2 shared experts. The head dimension is 128, the vocabulary has 128,256 tokens, and sequences contain 4,096 tokens. Attention uses a 2,048-token local window with global attention every 4 blocks; short convolutions use kernel size 4. Feedback adds one square projection matrix to each backbone.

The depth sweep keeps the backbone's attention and residual connections unchanged. Its feedback projection is one matrix initialised at zero, and the first token of a document receives nothing. The FBT recipe uses 0.02 input jitter, which the additional control disables.

During sequential decoding, each position reads a state that was itself computed with feedback. Parallel evaluation approximates this by repeating passes over the sequence, each reading states from the preceding pass. We report pass 3. For the 8-block additive model with a last-block source, loss changes by only 0.04 ×10−3 nats on the next pass. FBT's loss continues to rise, by 0.7 ×10−3 nats from pass 3 to pass 9.

The held-out set contains 42M tokens from six Dolma sources. We report macro loss in nats, the unweighted mean of the per-source losses.

Limitations

We tested two model sizes with one seed per run and measured held-out loss, without downstream evaluation. Repeated seeds are needed to assess how reliably the differences hold. The coarse depth grid also leaves open whether the best source stays a fixed number of blocks below the top or at a fixed fraction of the stack.

The sweep does not establish why intermediate states work better. Feedback may also change how the source block learns. The recipe comparison covers only the last-block source in the smaller model, so its interaction with source depth remains untested. All runs enable feedback partway through pretraining, leaving training from scratch and a matched continuation without feedback for future comparisons.

What we take from this

Source depth is a useful design choice for feedback models. In both sweeps, a block just below the top gives lower loss at the same parameter count and FLOPs. If your model feeds back the last block, try using a deep intermediate layer as the source.

References

[1] X. Wang, Z. Cai, Z. Zhan, H. Dong, Y. Fan, G. de Rosa, T. Pearce, J. Langford. Full-bandwidth transformer. arXiv:2608.08888, 2026.
[2] ByteDance Seed. Recycler: Cost-Free Latent Reasoning via Hidden State Recycling. Technical report, 2026.
[3] B. Huang, C. Dwivedi, X. Ma, A. Moreno, H. Liu. Latent Feedback Is a Compute and Data Amplifier: Scaling Law for Latent Feedback Transformers on Repeated Data. alphaXiv preprint, 2026.
[4] Z. Cai, X. Zhu, Y. Dong, Y. He, S. Arora. T²MLR: Transformer with Temporal Middle-Layer Recurrence. arXiv:2607.15178, 2026.
[5] Y. Hao, B. Zeng, Z. Wang, H. Li, S. Song, F. Song, Y. Liu, R. Tao, B. Dai, Z. Lin. P2N: Reusing Deep Representations for Greater Effective Depth. arXiv:2610.25076, 2026.
[6] M. Tang, S. Lu. Turbo Connection: Reasoning as Information Flow from Higher to Lower Layers. arXiv:2602.17993, 2026.
[7] The Marin Authors. Marin. GitHub repository, 2026.

Cite this blog

@misc{xiong2026feedbacksource,
  title  = {{Don't feed back the last block: the best source we found sits just below the top}},
  author = {Xiong, Xuyuan and Chen, Siyu and Chen, Mengzhao},
  year   = {2026},
  month  = {nov},
  note   = {Blog post},
  url    = {https://xjxyys.github.io/blog/feedback-source/}
}