What Is Linear Regression? How Least Squares Works
Linear regression fits a straight line through data to predict one quantity from another, and least squares picks the line with the smallest squared errors.
Linear regression is a method for finding the straight line that best describes how one quantity changes with another. The line is fitted to a set of data points, and it can then predict the outcome for a new input. The most common way to choose the line is least squares, which picks the line with the smallest total squared error.
What do the slope and intercept mean?
The line has the form y = b₀ + b₁x. The input x might be the floor area of a house, and the outcome y might be its price. The table lists the terms used in the fit.
| Term | Plain meaning |
|---|---|
| Slope (b₁) | Change in the outcome for each one-unit increase in the input |
| Intercept (b₀) | Predicted outcome when the input is zero |
| Residual | Observed outcome minus the outcome the line predicts |
| R squared | Share of the variation in the outcome that the line explains |
If the slope is 3,000, the fitted line predicts a rise of 3,000 in price for each extra square metre. The intercept may mean little on its own, because a house of zero area is outside the data. Its job is to place the line correctly.
Each data point also has a residual, the difference between the observed value and the value the line predicts. Residuals are the errors the line leaves behind, and least squares works by making them small.
How does least squares choose the line?
Think of each data point as attached to the line by a vertical spring. The residual is how far that spring is stretched. A stretched spring stores energy that grows with the square of its stretch. The line settles where the total stored energy is lowest, and that position is the line that minimises the sum of squared residuals.
Squaring the residuals has two effects. Positive and negative errors cannot cancel each other out, and large errors count far more than small ones. Because of this, the best line has a direct formula. The slope equals the sum of each x value's distance from the mean of x, multiplied by the matching distance for y, divided by the sum of the squared distances of x from its mean. The intercept then follows from the two means. For more complicated models, the same minimum is found step by step with gradient descent.
How good is the fit?
The standard summary is R squared, the share of the variation in the outcome that the line accounts for. For a line fitted with an intercept it runs from 0 to 1. A value of 0.9 means the input explains about 90 percent of the variation in the data. A high value shows that the line tracks the data closely, but it does not show that the relationship is causal.
The spread of the residuals matters too. Their standard deviation gives the typical size of the leftover errors, measured in the same units as the outcome. A small spread means the predictions are tight, and a large spread means they are rough even when the line is well placed.
When does linear regression mislead?
Linear regression assumes that the relationship is roughly straight. A curved relationship fitted with a straight line gives misleading predictions, and a plot of the residuals usually reveals the problem. Outliers also have an outsized effect, because squaring makes one extreme residual pull the line strongly toward it.
Predicting beyond the range of the data, called extrapolation, is risky, because the straight line may not hold there. The method also assumes that the errors are roughly independent and that their spread is similar across the range. Many analysts assume the errors follow a normal distribution, which is often reasonable but is not guaranteed.
Finally, a line fitted to correlated data is not proof that one variable causes the other. Regression describes an association, and deciding what causes what needs more than the fit. The reasons are covered in correlation and causation.
Tags
data analysis least squares machine learning mathematics statistics