Worked examples
The question
Question: find the least squares line through (โ1, 2), (0, 1) and (1, 6).
(โ1, 2), (0, 1), (1, 6)
We want a line y = b + m x.
These 3 points are not all on one line, so we find the line that fits best.
line: y = b + m x (b: where it starts, m: slope)
1 One equation per point
Each point gives one equation: put its x and y into y = b + m x.
3 points, 3 equations, but only 2 unknowns, b and m.
(โ1, 2): b + (โ1)m = 2
(0, 1): b + 0m = 1
As a matrix: A has a column of 1s for b and a column of the x values for m.
The right side holds the y values.
No line goes through all 3 points exactly.
[1, โ1; 1, 0; 1, 1] ร [b; m] = [2; 1; 6]
2 Write the normal equations
The best b and m solve A transpose A, times x, equals A transpose y.
These are called the normal equations.
With them, the leftovers are at a right angle to the columns of A: as small as they can be.
ATA x = ATy
A transpose: the rows of A become its columns.
AT = [1, 1, 1; โ1, 0, 1]
Careful!
A transpose goes in front of A, and in front of y.
Both sides get it.
Careful: Aแต in front, on both sides
3 Work out the two sides
A transpose times A.
ATA = [1, 1, 1; โ1, 0, 1] ร [1, โ1; 1, 0; 1, 1]
ATA = [1 ร 1 + 1 ร 1 + 1 ร 1, 1 ร (โ1) + 1 ร 0 + 1 ร 1; โ1 ร 1 + 0 ร 1 + 1 ร 1, โ1 ร (โ1) + 0 ร 0 + 1 ร 1]
The top left counts the points: 3.
ATA = [3, 0; 0, 2]
A transpose times y.
ATy = [1, 1, 1; โ1, 0, 1] ร [2; 1; 6]
ATy = [1 ร 2 + 1 ร 1 + 1 ร 6; โ1 ร 2 + 0 ร 1 + 1 ร 6]
ATy = [9; 4]
4 Solve for b and m
The two equations as an augmented matrix: the unknowns are b and m.
= [3, 0, 9; 0, 2, 4]
We call the rows R 1 and R 2.
โ [1, 0, 3; 0, 2, 4]
โ [1, 0, 3; 0, 1, 2]
Row 1 says b = 3.
b = 3
Row 2 says m = 2.
m = 2
So the least squares line is y = 2x + 3.
y = 2x + 3
5 Check
Let's check.
The leftover of a point is its y minus the line's y.
For the best line, the leftovers add to 0, and x times leftover adds to 0.
Each y minus the line's y at that x: 1, โ2 and 1.
leftovers: 2 โ 1 = 1, 1 โ 3 = โ2, 6 โ 5 = 1
Added: 0.
added: 1 + (โ2) + 1 = 0
x times leftover, added: 0.
Both are 0, so the answer is right!
x ร leftover: โ1 ร 1 + 0 ร (โ2) + 1 ร 1 = 0
Let's look back.
First: one equation per point.
Then: write the normal equations.
Then: work out the two sides.
Then: solve for b and m.
Last: check.
โถ Watch it solved
The question
Question: find the least squares line through (0, 1), (1, 3) and (2, 4).
(0, 1), (1, 3), (2, 4)
We want a line y = b + m x.
These 3 points are not all on one line, so we find the line that fits best.
line: y = b + m x (b: where it starts, m: slope)
1 One equation per point
Each point gives one equation: put its x and y into y = b + m x.
3 points, 3 equations, but only 2 unknowns, b and m.
(0, 1): b + 0m = 1
(1, 3): b + 1m = 3
As a matrix: A has a column of 1s for b and a column of the x values for m.
The right side holds the y values.
No line goes through all 3 points exactly.
[1, 0; 1, 1; 1, 2] ร [b; m] = [1; 3; 4]
2 Write the normal equations
The best b and m solve A transpose A, times x, equals A transpose y.
These are called the normal equations.
With them, the leftovers are at a right angle to the columns of A: as small as they can be.
ATA x = ATy
A transpose: the rows of A become its columns.
AT = [1, 1, 1; 0, 1, 2]
Careful!
A transpose goes in front of A, and in front of y.
Both sides get it.
Careful: Aแต in front, on both sides
3 Work out the two sides
A transpose times A.
ATA = [1, 1, 1; 0, 1, 2] ร [1, 0; 1, 1; 1, 2]
ATA = [1 ร 1 + 1 ร 1 + 1 ร 1, 1 ร 0 + 1 ร 1 + 1 ร 2; 0 ร 1 + 1 ร 1 + 2 ร 1, 0 ร 0 + 1 ร 1 + 2 ร 2]
The top left counts the points: 3.
ATA = [3, 3; 3, 5]
A transpose times y.
ATy = [1, 1, 1; 0, 1, 2] ร [1; 3; 4]
ATy = [1 ร 1 + 1 ร 3 + 1 ร 4; 0 ร 1 + 1 ร 3 + 2 ร 4]
ATy = [8; 11]
4 Solve for b and m
The two equations as an augmented matrix: the unknowns are b and m.
= [3, 3, 8; 3, 5, 11]
We call the rows R 1 and R 2.
โ [1, 1, 8 over 3; 3, 5, 11]
โ [1, 1, 8 over 3; 0, 2, 3]
โ [1, 1, 8 over 3; 0, 1, 3 over 2]
โ [1, 0, 7 over 6; 0, 1, 3 over 2]
Row 1 says b = 7/6.
b = 76
Row 2 says m = 3/2.
m = 32
So the least squares line is y = 3/2 times x + 7/6.
y = 32x + 76
5 Check
Let's check.
The leftover of a point is its y minus the line's y.
For the best line, the leftovers add to 0, and x times leftover adds to 0.
Each y minus the line's y at that x: โ1/6, 1/3 and โ1/6.
leftovers: 1 โ 76 = โ16, 3 โ 83 = 13, 4 โ 256 = โ16
Added: 0.
added: โ16 + 13 + (โ16) = 0
x times leftover, added: 0.
Both are 0, so the answer is right!
x ร leftover: 0 ร (โ16) + 1 ร 13 + 2 ร (โ16)
= 0
Let's look back.
First: one equation per point.
Then: write the normal equations.
Then: work out the two sides.
Then: solve for b and m.
Last: check.
โถ Watch it solved
The question
Question: find the least squares line through (โ2, 1), (โ1, 0), (0, 2), (1, 2) and (2, 5).
(โ2, 1), (โ1, 0), (0, 2), (1, 2), (2, 5)
We want a line y = b + m x.
These 5 points are not all on one line, so we find the line that fits best.
line: y = b + m x (b: where it starts, m: slope)
1 One equation per point
Each point gives one equation: put its x and y into y = b + m x.
5 points, 5 equations, but only 2 unknowns, b and m.
(โ2, 1): b + (โ2)m = 1
(โ1, 0): b + (โ1)m = 0
As a matrix: A has a column of 1s for b and a column of the x values for m.
The right side holds the y values.
No line goes through all 5 points exactly.
[1, โ2; 1, โ1; 1, 0; 1, 1; 1, 2] ร [b; m] = [1; 0; 2; 2; 5]
2 Write the normal equations
The best b and m solve A transpose A, times x, equals A transpose y.
These are called the normal equations.
With them, the leftovers are at a right angle to the columns of A: as small as they can be.
ATA x = ATy
A transpose: the rows of A become its columns.
AT = [1, 1, 1, 1, 1; โ2, โ1, 0, 1, 2]
Careful!
A transpose goes in front of A, and in front of y.
Both sides get it.
Careful: Aแต in front, on both sides
3 Work out the two sides
A transpose times A.
ATA = [1, 1, 1, 1, 1; โ2, โ1, 0, 1, 2] ร [1, โ2; 1, โ1; 1, 0; 1, 1; 1, 2]
ATA = [1 ร 1 + 1 ร 1 + 1 ร 1 + 1 ร 1 + 1 ร 1, 1 ร (โ2) + 1 ร (โ1) + 1 ร 0 + 1 ร 1 + 1 ร 2; โ2 ร 1 + (โ1) ร 1 + 0 ร 1 + 1 ร 1 + 2 ร 1, โ2 ร (โ2) + (โ1) ร (โ1) + 0 ร 0 + 1 ร 1 + 2 ร 2]
The top left counts the points: 5.
ATA = [5, 0; 0, 10]
A transpose times y.
ATy = [1, 1, 1, 1, 1; โ2, โ1, 0, 1, 2] ร [1; 0; 2; 2; 5]
ATy = [1 ร 1 + 1 ร 0 + 1 ร 2 + 1 ร 2 + 1 ร 5; โ2 ร 1 + (โ1) ร 0 + 0 ร 2 + 1 ร 2 + 2 ร 5]
ATy = [10; 10]
4 Solve for b and m
The two equations as an augmented matrix: the unknowns are b and m.
= [5, 0, 10; 0, 10, 10]
We call the rows R 1 and R 2.
โ [1, 0, 2; 0, 10, 10]
โ [1, 0, 2; 0, 1, 1]
Row 1 says b = 2.
b = 2
Row 2 says m = 1.
m = 1
So the least squares line is y = x + 2.
y = x + 2
5 Check
Let's check.
The leftover of a point is its y minus the line's y.
For the best line, the leftovers add to 0, and x times leftover adds to 0.
Each y minus the line's y at that x: 1, โ1, 0, โ1 and 1.
leftovers: 1 โ 0 = 1, 0 โ 1 = โ1, 2 โ 2 = 0, 2 โ 3 = โ1, 5 โ 4 = 1
Added: 0.
added: 1 + (โ1) + 0 + (โ1) + 1 = 0
x times leftover, added: 0.
Both are 0, so the answer is right!
x ร leftover: โ2 ร 1 + โ1 ร (โ1) + 0 ร 0 + 1 ร (โ1) + 2 ร 1
= 0
Let's look back.
First: one equation per point.
Then: write the normal equations.
Then: work out the two sides.
Then: solve for b and m.
Last: check.
โถ Watch it solved
