Step by step
The question
Question: find the least squares line through (−1, 0), (0, 1), (1, 1) and (2, 4).
We want a line y = b + m x.
These 4 points are not all on one line, so we find the line that fits best.
1 One equation per point
Each point gives one equation: put its x and y into y = b + m x.
4 points, 4 equations, but only 2 unknowns, b and m.
As a matrix: A has a column of 1s for b and a column of the x values for m.
The right side holds the y values.
No line goes through all 4 points exactly.
2 Write the normal equations
The best b and m solve A transpose A, times x, equals A transpose y.
These are called the normal equations.
With them, the leftovers are at a right angle to the columns of A: as small as they can be.
A transpose: the rows of A become its columns.
Careful!
A transpose goes in front of A, and in front of y.
Both sides get it.
3 Work out the two sides
A transpose times A.
The top left counts the points: 4.
A transpose times y.
4 Solve for b and m
The two equations as an augmented matrix: the unknowns are b and m.
We call the rows R 1 and R 2.
Row 1 says b = 9/10.
Row 2 says m = 6/5.
So the least squares line is y = 6/5 times x + 9/10.
5 Check
Let's check.
The leftover of a point is its y minus the line's y.
For the best line, the leftovers add to 0, and x times leftover adds to 0.
Each y minus the line's y at that x: 3/10, 1/10, −11/10 and 7/10.
Added: 0.
x times leftover, added: 0.
Both are 0, so the answer is right!
Let's look back.
First: one equation per point.
Then: write the normal equations.
Then: work out the two sides.
Then: solve for b and m.
Last: check.