Transformer high-norm artifacts and the Softmax bottleneck
High-norm spikes in Vision Transformers come from Softmax, not from bad training data. Here is why the sum-to-1 constraint turns background tokens into attention sinks, and what registers, sigmoidal gating and self-distillation actually do about it.