Skip to content
Live
Trending storyPast story

Looped Transformers Training Truncation Research

1 reports1 outlets22 days agoUpdate

The story so far

AI overview

Hugging Face published the paper 'Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation' stating that Looped Transformers, when trained with full backtracking under the same computational load, outperform truncated backtracking. Scaling laws indicate that two loops are optimal when the model size is adjustable. Looped Transformers increase computational depth by reusing shared layers without increasing the number of trainable parameters. They typically use backtracking training (BPTT), but are often truncated to the last few loops to save training computational costs, which may weaken the gradient bias and thus the loop advantage.

AI-written from the reports · Updated 22 days ago

Coverage timeline

Follow the coverage to see every side of the story.

28 Oct
  1. Hugging Face Daily Papers
    Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation

    Looped Transformers train better than truncated backpropagation with the same computational cost through full backtracking, extended regularity shows that when the model size is adjustable, two loops are the best. Looped Transformers increase computational depth by reusing shared layers without increasing the number of trainable parameters, usually using backtracking training (BPTT), but are often truncated to the last few loops to save training computational cost, which may lead to gradient bias and weaken the loop advantage.