Extended Research on Looped Transformers with Truncated Backpropagation: Should the training loop be truncated?
Looped Transformers train better than truncated backpropagation with the same computational cost through full backtracking, extended regularity shows that when the model size is adjustable, two loops are the best. Looped Transformers increase computational depth by reusing shared layers without increasing the number of trainable parameters, usually using backtracking training (BPTT), but are often truncated to the last few loops to save training computational cost, which may lead to gradient bias and weaken the loop advantage.
















