Looped Transformers Training Truncation Research
The story so far
Hugging Face published the paper 'Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation' stating that Looped Transformers, when trained with full backtracking under the same computational load, outperform truncated backtracking. Scaling laws indicate that two loops are optimal when the model size is adjustable. Looped Transformers increase computational depth by reusing shared layers without increasing the number of trainable parameters. They typically use backtracking training (BPTT), but are often truncated to the last few loops to save training computational costs, which may weaken the gradient bias and thus the loop advantage.
AI-written from the reports · Updated 22 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- Hugging Face Daily PapersTruncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation
Looped Transformers train better than truncated backpropagation with the same computational cost through full backtracking, extended regularity shows that when the model size is adjustable, two loops are the best. Looped Transformers increase computational depth by reusing shared layers without increasing the number of trainable parameters, usually using backtracking training (BPTT), but are often truncated to the last few loops to save training computational cost, which may lead to gradient bias and weaken the loop advantage.