Nearly every introduction to regression starts with a scatter plot and a “best-fit line.” That is fine for a high school stats class. It stops being useful when you are building something that has to hold up in production, because a line through points says very little about how 50 features interact. The Linear Regression Projection view is the precise one. If you are still squashing 50-dimensional data into a 2D line in your head, you are making the job harder than it has to be.
Moving to column space
Most of us get stuck in feature space, where every row of data is a point. With 1,000 features that picture falls apart. So refactor the mental model and work in column space instead. There, each feature is a single vector in a high-dimensional space, and your target, the values you want to predict, is a vector too.
That turns the fit into a geometric search. You are looking for the vector closest to your target inside the space your features span. Once you think in directions instead of points, it is easier to hold in your head.
The intercept is a base vector
I have watched junior devs leave out the intercept and then wonder why the model predicts zero for an empty plot of land. Adding an intercept means adding a base vector of all ones (1, 1, 1…). That gives you one more direction to move in, and it gives the model a baseline that is not zero.
Your feature vector and your base vector together define a plane, or a hyperplane once you add more features. You want to reach the tip of the target “Price” vector, but you can only move on the plane your features create. The closest you can get is by dropping a perpendicular from the target vector onto that plane.
The normal equation as a projection
Put the partial derivatives aside for a moment. The normal equation earns its place in machine learning because it pins down exactly where the error vector runs perpendicular to the feature plane. If the error is e = y - Xβ, then for e to be the shortest distance it has to be orthogonal to the features X. That is where Xᵀ(y - Xβ) = 0 comes from.
// A simple representation of the Normal Equation in PHP
// beta = (X^T * X)^-1 * X^T * y
function bbioon_calculate_regression_weights($matrix_X, $vector_y) {
$XT = bbioon_transpose($matrix_X);
$XTX = bbioon_multiply($XT, $matrix_X);
$XTX_inv = bbioon_inverse($XTX);
$XTy = bbioon_multiply($XT, $vector_y);
return bbioon_multiply($XTX_inv, $XTy);
}I wrote separately about managing machine learning projects for long-term stability. Holding the projection picture in mind is also what lets you diagnose a “singular matrix” error or a collinearity problem instead of guessing your way through it.
What this buys you as an architect
Once regression is a projection, redundant features stop looking like extra information. They are more vectors spanning the same space. The weights then have no principled way to split credit between them, and (XᵀX)⁻¹ fails because the matrix is no longer invertible.
If this kind of modeling work is eating your dev hours, I can take it off your hands. I have been working with WordPress and custom data pipelines since the 4.x days.
Takeaway: it is all geometry
Linear regression is a geometric projection that you can also reach through calculus. With 2 features or 2,000, the rule is the same: find the perpendicular. For the formal proofs, the MIT OpenCourseWare lectures by Gilbert Strang are where I send people.
