Most people pull a pretrained model off Hugging Face and ship it without ever looking at the feature maps. That works fine until you need dense predictions. On object detection or segmentation, Transformer high-norm artifacts in the attention matrix quietly eat your accuracy.
After 14 years of untangling legacy code and broken architectures, I have one rule: when a system behaves unpredictably, stop adding layers and go look at the math. The high-norm spikes in Vision Transformers (ViTs) are not random noise. They fall out of the Softmax function itself. The spikes run 2 to 10 times larger than the average token norm, and they behave like “attention sinks” that soak up global information and blur local meaning.
Why Softmax creates transformer high-norm artifacts
It comes down to normalization. In a standard Transformer block, the attention weights for a given query have to sum to 1. So a token with nothing useful to look at, a patch of clear sky for instance, still has to put its attention mass somewhere. The model settles on a handful of background tokens and dumps everything into them. Those tokens become the high-norm sinks.
I wrote earlier about solving production ML failures and how training metrics lie to you. This is another case of it. A model can post 90% accuracy on ImageNet and still fall apart on unsupervised object discovery, because the artifacts land in background regions and corners where they confuse the detection heads.
The “naive” attention implementation
This is the standard PyTorch attention block, and the Transformer high-norm artifacts start on the Softmax line.
def forward(self, x):
B, N, C = x.shape
qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4)
q, k, v = qkv[0], qkv[1], qkv[2]
# The bottleneck: Softmax forces weights to sum to 1
attn = (q @ k.transpose(-2, -1))
attn = attn.softmax(dim=-1) # Creation of artifact
x = (attn @ v).transpose(1, 2).reshape(B, N, C)
return self.proj(x)Refactoring for stability with registers
The team behind DINOv2 spotted this and proposed “registers”: extra tokens whose whole job is to be the trash can for Softmax overflow. Jiang et al. went further in 2025 and showed you can operate on a model that already exists, rerouting values from internal MLP neurons into those registers without a full retrain.
Anyone who has dealt with transient bloat or race conditions in WordPress will recognize the shape of the fix. You give the garbage data its own storage slot instead of letting it corrupt the main loop. Here the slot is a register holding global state, which leaves the patch tokens clean and their local semantics intact for something like zero-shot segmentation.
On the LLM side, my guide on stopping AI hallucinations covers how context management heads off the same sink behavior.
Current mitigation strategies
The 2017 Attention Is All You Need paper set the pattern everyone still builds on, but newer work leans toward gated attention. What is on the table right now:
- Test-time registers cost nothing to retrain. They use specific register neurons to move energy away from the patch tokens.
- Sigmoidal gating swaps Softmax for an unnormalized sigmoid, which drops the sum-to-1 constraint entirely.
- Self-distillation uses a teacher model to average the artifacts out with random offsets and flips during a short fine-tuning stage.
If chasing transformer high-norm artifacts is eating your dev hours, I can take it off your plate. I have been working on WordPress and backend architecture since the 4.x days, and I know what survives production.
The bottom line
High-norm spikes in your feature maps are not a sign the model is doing something clever. They are a bottleneck the math created. Registers or gating will stabilize training and recover up to 20% performance on dense tasks, so refactor the attention blocks before your next release instead of patching around the symptoms.
