Regression: predicting one number from another
So far we have described one quantity at a time — its center, spread, shape. But the interesting part starts when there are two quantities and a relationship between them: more ad spend — more sales? bigger apartment — higher price? Regression answers two questions at once: is there a relationship (and how strong is it), and how to predict one quantity from the other. Drag the points on the chart — the line and the numbers recompute live.
Drag the points. The slope b₁ is how much y grows per +1 in x; r is the strength of the relationship from −1 to +1.
Each point is one observation with two numbers: horizontally x (say, ad spend), vertically y (sales). The cloud of points trends upward: more advertising — usually more sales. The teal line is the regression: the one straight line that best describes this cloud. Drag any point and watch the line adjust.
Regression is used for prediction everywhere: sales from the ad budget, apartment price from floor area, server load from the number of users, delivery time from distance. The idea is always the same — estimate the unknown y from a known x.
The coefficients read in plain language. "b₁ = 3" for sales vs advertising means: each ruble invested brings three rubles of sales on average. This is exactly why regression is built — it not only predicts, it shows which lever is how strong.
But a strong correlation by itself does not prove that x causes y. To claim causality you need an experiment — that same A/B test from the previous module, where we change x ourselves and see whether y changes.
Relationships surround us: height and weight, education and income, outdoor temperature and electricity use. Correlation helps notice and measure them; regression turns them into forecasts.
The internet is full of "spurious correlations" — quantities that move together by accident for years (cheese consumption and drownings, divorces and margarine consumption). They are funny precisely because the relationship is there and the causation is not.
That is why a competent analyst, upon seeing a correlation, first asks: is there a hidden common cause, and can this be tested with an experiment? A line on a chart is grounds for a hypothesis, not proof.
Definitions
Linear regression assumes a roughly linear relationship, independent residuals with roughly constant spread (homoscedasticity), and weak influence of outliers. Nonlinearity or a single strong outlier can flip the slope — always look at the chart itself, not just at r.
A coefficient is a relationship, not causation. Without controlling for extraneous factors (multiple regression) or an experiment, you cannot read b₁ as "the effect of x on y".
Deep dive: the math and the mechanism (optional)
Why squares — deeper than "a fine for big misses". If you predict y with a single number and no x at all, the number that minimizes the sum of squared deviations is exactly the mean (and the one minimizing the sum of absolute deviations is the median). So OLS is a direct generalization of "the mean" to the case with a predictor: regression predicts the conditional mean of y given x. Hence also the link to variance: R² is the share of y's variance the line explains.
The practical bonus of squares: the sum of squares is a smooth, convex function of b₀, b₁, so it has a unique minimum with an explicit formula (no search needed). And OLS coincides with the maximum likelihood estimate when the noise around the line is normal — so squares are not arbitrary but the natural choice for bell-shaped errors.