Skip to content
Live
Hugging Face Daily Papers·· yesterday

Loopback Transformer: Towards a Unified View of Residual Looped Transformers

Loopback Transformer: Towards a Unified View of Residual Looped Transformers

Brief overview

Loopback Transformer achieves information storage, transmission, and reuse by treating residual connections as memory operations along the network depth. In 13 zero-shot tasks, both the 1.4B and 3B parameter versions have lower validation loss (3B version decreases from 2.0005 to 1.9849) and higher average accuracy (from 48.56% to 49.45%) compared to standard Transformer, with parameter counts reduced by 32% and 25%, respectively, and outperforming in 11 tasks.

Article

In this paper, we study how residual connections should retain and reuse information when the same Transformer blocks run repeatedly. Inspired by recurrent memory, we develop a unified view of residual connections as memory operating along network depth. The write–transport–read formulation lets us compare different residual designs through what they store, how they carry information forward, and how they retrieve it for subsequent computation.

Guided by this view, we introduce Loopback Transformer. Loopback stores individual block updates, uses geometric forgetting to progressively discount earlier passes, and learns content-dependent attention over the stored updates. This allows the model to retrieve useful information from earlier computation.

13e96ba0103625b69219978d829be3fd

At matched training tokens and executed depth, Loopback achieves lower validation loss and higher mean accuracy across 13 zero-shot tasks than Standard Transformers at both 1.4B and 3B, retaining only 68% and 75% of their Transformer parameters, respectively. At 3B, validation loss improves from 2.0005 to 1.9849, mean accuracy rises from 48.56% to 49.45%, and Loopback scores higher on 11 of 13 tasks. In our main comparisons at 191M, 374M, and 778M, it also achieves the lowest validation loss and highest thirteen-task mean among Standard and seven looped residual baselines. Parameter counts exclude token embeddings and the output head.

Our results highlight residual design as an important part of parameter-efficient modeling: sharing computation works better when the model also learns how to retain and reuse its outputs. We welcome feedback and discussion!

Source: Hugging Face Daily Papers · huggingface.co