Least squares line through (0, 2), (1, 0), (2, 1) Step by Step

Step by step

The question

Question: find the least squares line through (0, 2), (1, 0) and (2, 1).

(0, 2), (1, 0), (2, 1)

We want a line y = b + m x.

These 3 points are not all on one line, so we find the line that fits best.

line: y = b + m x (b: where it starts, m: slope)

1 One equation per point

Each point gives one equation: put its x and y into y = b + m x.

3 points, 3 equations, but only 2 unknowns, b and m.

(0, 2): b + 0m = 2

(1, 0): b + 1m = 0

As a matrix: A has a column of 1s for b and a column of the x values for m.

The right side holds the y values.

No line goes through all 3 points exactly.

[1, 0; 1, 1; 1, 2] × [b; m] = [2; 0; 1]

2 Write the normal equations

The best b and m solve A transpose A, times x, equals A transpose y.

These are called the normal equations.

With them, the leftovers are at a right angle to the columns of A: as small as they can be.

A^TA x = A^Ty

A transpose: the rows of A become its columns.

A^T = [1, 1, 1; 0, 1, 2]

Careful!

A transpose goes in front of A, and in front of y.

Both sides get it.

Careful: Aᵀ in front, on both sides

3 Work out the two sides

A transpose times A.

A^TA = [1, 1, 1; 0, 1, 2] × [1, 0; 1, 1; 1, 2]

A^TA = [1 × 1 + 1 × 1 + 1 × 1, 1 × 0 + 1 × 1 + 1 × 2; 0 × 1 + 1 × 1 + 2 × 1, 0 × 0 + 1 × 1 + 2 × 2]

The top left counts the points: 3.

A^TA = [3, 3; 3, 5]

A transpose times y.

A^Ty = [1, 1, 1; 0, 1, 2] × [2; 0; 1]

A^Ty = [1 × 2 + 1 × 0 + 1 × 1; 0 × 2 + 1 × 0 + 2 × 1]

A^Ty = [3; 2]

4 Solve for b and m

The two equations as an augmented matrix: the unknowns are b and m.

= [3, 3, 3; 3, 5, 2]

We call the rows R 1 and R 2.

→ [1, 1, 1; 3, 5, 2]

→ [1, 1, 1; 0, 2, −1]

→ [1, 1, 1; 0, 1, −1 over 2]

→ [1, 0, 3 over 2; 0, 1, −1 over 2]

Row 1 says b = 3/2.

b = 32

Row 2 says m = −1/2.

m = −12

So the least squares line is y = −1/2 times x + 3/2.

y = −12x + 32

5 Check

Let's check.

The leftover of a point is its y minus the line's y.

For the best line, the leftovers add to 0, and x times leftover adds to 0.

Each y minus the line's y at that x: 1/2, −1 and 1/2.

leftovers: 2 − 32 = 12, 0 − 1 = −1, 1 − 12 = 12

Added: 0.

added: 12 + (−1) + 12 = 0

x times leftover, added: 0.

Both are 0, so the answer is right!

x × leftover: 0 × 12 + 1 × (−1) + 2 × 12 = 0

Let's look back.

First: one equation per point.

Then: write the normal equations.

Then: work out the two sides.

Then: solve for b and m.

Last: check.

Answer