The usual explanation of spectral bias says neural networks learn low frequencies first and high frequencies last, so the frequency content of your target decides how slow training will be. Conor Rowan and Finn Murphy-Blanchard ran a set of small experiments, posted on Towards Data Science, that show frequency alone is a poor predictor of how training goes. Where the signal is strong inside the domain matters too, and that detail changes which fix you reach for.
What the frequency story leaves out
They fit two-hidden-layer tanh networks, width 100, with ADAM, to targets like sin(26πx) on the unit interval. The fit does not improve everywhere at once. The network locks onto the edges of the domain first, then adds one oscillation at a time and works inward toward the middle. They call this sequential fitting, and it explains the slowness on high-frequency targets: each oscillation gets built by its own pair of smoothed step functions in the last hidden layer, one pair after another.
The stranger result is what they call the boundary effect. One target, with an envelope that drives its amplitude to zero at both edges, made no visible progress in 7500 epochs. A nearly identical target with the dip moved to the center of the domain trained the way you would expect. The two functions have almost the same discrete Fourier spectra, so frequency content cannot be the whole story. When the signal is quiet at the edges, gradient descent struggles to get started.
Why it matters for ordinary regressions
Nobody fits sines for a living around here, but the failure shape transfers. The small regressions that turn up in store work (demand against price, forecast curves, weekly seasonality fits) usually get judged by one error number, and single numbers hide problems of their own. The boundary effect suggests a quiet region of the input range can dominate training time while the average error looks acceptable. So plot residuals across the input range instead of reading one score, and normalize inputs so the interesting part of the signal is not squeezed into a corner of the domain. The normalization advice is my inference from their plots; it is not something they claim.
The three fixes for spectral bias, compared
SIREN, from Sitzmann and colleagues, replaces tanh with a sinusoidal activation. The basis functions the network learns become oscillatory themselves and the sequential sweep disappears. It works, but it is an architecture change: you need the initialization scheme their paper describes, and you are now outside what stock training code assumes.
Fourier features, from Tancik and colleagues, keep the network stock and change the input instead: each coordinate gets mapped through sin and cos at a fixed set of frequencies. The frequencies the network would otherwise have to earn are handed over up front. The price is one real hyperparameter, the frequency scale, and their project page shows the same pipeline sliding from underfitting to overfitting as you turn that scale up, so the scale needs a sweep rather than a guess.
Second-order optimizers attack the problem at the training step. Their post cites recent work where quasi-Newton methods with Hessian-based preconditioning even out convergence across frequencies. Neither the architecture nor the features change, which I like, but you need a real optimizer library and the per-step cost is higher.
Which one I would pick for which project
For a small regression on store data, I would pick Fourier features, because they are one preprocessing step on the input and everything downstream stays ordinary. In PHP that is a small helper that feeds whatever estimator you already run:
function bbioon_fourier_features( float $x, int $terms = 8, float $scale = 6.0 ): array {
$features = [ $x ];
for ( $k = 1; $k <= $terms; $k++ ) {
$features[] = sin( $scale * $k * $x );
$features[] = cos( $scale * $k * $x );
}
return $features;
}For fitting signals where the detail genuinely lives at high frequencies (audio, wavefields, images), SIREN is the right tool. In Python with a scientific stack, second-order methods are worth their setup cost. For the smooth forecasting curves most stores need, skip all three, because a linear model or a small ensemble usually wins on data that thin, which is the small-data reality of most WooCommerce stores.
When not to bother
If you have a target you are trying to fit and cannot tell whether the model or the data is at fault, that is the sort of work I do. Send over the numbers and the question, and I will take a look.
Before trying any of the three fixes, fit the target with something simple and plot the residuals across the whole range. If a linear model or a few splines captures the curve, spectral bias was never your problem, and periodic activations would only hand you new ways to overfit. The fancy options earn their keep only when a simple model visibly smooths over detail that is in the data.
