We need to talk about model complexity. The standard advice in machine learning has become “just add more features,” and it is quietly wrecking project stability. I have spent 14 years debugging systems where the logic was sound and the model underneath was memorizing noise. The penalty term in Lasso Regression is really a spatial constraint: it fences in where your coefficients are allowed to land.
I thought I had seen every way a prediction engine can fail until I inherited a house-pricing model last year. Zero error on the training set, and it valued a 1,000-square-foot shack in the suburbs at $2.4 million. The bias was low and the variance was enormous. It was memorizing data points rather than learning patterns, because it had too much freedom to move. The geometry of Lasso Regression is what reins that in.
The model that scores perfectly and fails anyway
Say you are predicting house prices from size and age. In an ordinary least squares (OLS) setup you have total freedom: move 7 units in the intercept direction, 2 units for size, 1 unit for age, and you hit the target vector exactly. On paper it looks perfect.
Then you hand it a new house, call it House D, and it chokes. It has learned that age, the less reliable feature, pushes the price up, because it spent every available direction matching the training data. That is overfitting. The fix is a budget, a constraint on how big the coefficients are allowed to get.
Step 1: center the data
Before any talk of diamonds, the intercept needs cleaning up. The intercept is the average target level, and it says nothing about how price changes. Skip the centering and your budget gets spent on baseline movement instead of on the feature effects you care about.
# The Naive Approach vs Centered Approach
import numpy as np
# Original Features (Size, Age)
X = np.array([[1, 1], [2, 3], [3, 2]])
y = np.array([4, 8, 9])
# Centering: Subtract the mean to remove the intercept
X_centered = X - X.mean(axis=0)
y_centered = y - y.mean()
print(f"Centered Y: {y_centered}")After centering, the intercept is effectively zero and the whole coefficient budget goes to the features. If you want to know how these models behave once they are in production, I wrote about Machine Learning at Scale.
The diamond constraint and why lasso shrinks to zero
In OLS your coefficients can be anything. Lasso Regression adds a ceiling: the sum of the absolute values of the coefficients has to stay at or below a budget, say 2. That is the L1 norm: |β₁| + |β₂| ≤ 2.
In two dimensions that inequality traces a diamond shape. Your solution can no longer sit anywhere on the infinite plane; it has to sit on or inside the diamond. The diamond has sharp points on the axes, so the closest point from your unconstrained solution often lands right on an axis, and the other coefficient becomes zero. That is lasso doing feature selection for you.
The math: projecting onto the boundary
With the constraint active, the coefficients stop being independent. The boundary equation ties them together (β₁ + β₂ = 2, for example), which collapses the 2D problem into a projection onto a 1D line. You then find the point on that line closest to the target vector. For the implementation details, read the Scikit-Learn Lasso Documentation.
If this Lasso Regression work is eating your dev hours, hand it to me. I have been wrestling with WordPress and messy data logic since the 4.x days.
What to take away
Lasso is less about adding a penalty than about accepting that more flexibility is not always better. Constrain the solution to a diamond and you get a model that stays stable, holds up on unseen data like House D, and tells you which features were garbage all along. If model management is the bigger headache for you, my guide on Managing Machine Learning Projects covers that side. Ship it, with the constraints in place.
