Step by step
The question
Question: find the least squares line through (0, 1), (1, 1) and (2, 3).
We want a line y = b + m x.
These 3 points are not all on one line, so we find the line that fits best.
1 One equation per point
Each point gives one equation: put its x and y into y = b + m x.
3 points, 3 equations, but only 2 unknowns, b and m.
As a matrix: A has a column of 1s for b and a column of the x values for m.
The right side holds the y values.
No line goes through all 3 points exactly.
2 Write the normal equations
The best b and m solve A transpose A, times x, equals A transpose y.
These are called the normal equations.
With them, the leftovers are at a right angle to the columns of A: as small as they can be.
A transpose: the rows of A become its columns.
Careful!
A transpose goes in front of A, and in front of y.
Both sides get it.
3 Work out the two sides
A transpose times A.
The top left counts the points: 3.
A transpose times y.
4 Solve for b and m
The two equations as an augmented matrix: the unknowns are b and m.
We call the rows R 1 and R 2.
Row 1 says b = 2/3.
Row 2 says m = 1.
So the least squares line is y = x + 2/3.
5 Check
Let's check.
The leftover of a point is its y minus the line's y.
For the best line, the leftovers add to 0, and x times leftover adds to 0.
Each y minus the line's y at that x: 1/3, −2/3 and 1/3.
Added: 0.
x times leftover, added: 0.
Both are 0, so the answer is right!
Let's look back.
First: one equation per point.
Then: write the normal equations.
Then: work out the two sides.
Then: solve for b and m.
Last: check.