Orthogonal projection

Section 5.3 Orthogonal projection

A trick we learn early on in physics-- specifically, in dynamics problems in

R^{2}

-- is to pick a convenient axis and then decompose any relevant vectors (force, acceleration, velocity, position, etc.) into a sum of two components: one that points along the chosen axis, and one that points perpendicularly to it. As we will see in this section, this technique can be vastly generalized. Namely, instead of

R^{2}

we can take any inner product space

(V, ⟨, ⟩);

and instead of a chosen axis in

R^{2},

we can choose any finite-dimensional subspace

W \subseteq V;

then any

v \in V

can be decomposed in the form

v = w + w^{⊥},

🔗

where

w \in W

and

w^{⊥}

is a vector orthogonal to

W,

in a sense we will make precise below. Just as in our toy physics example, this manner of decomposing vectors helps simplify computations in problems where the subspace

W

chosen is of central importance.

🔗

Subsection 5.3.1 Orthogonal complement

🔗

We begin by making sense of what it means for a vector to be orthogonal to a subspace.

🔗

Definition 5.3.1. Orthogonal complement.

🔗

. Let

(V, ⟨, ⟩)

be an inner product vector space, and let

W \subseteq V

be a subspace.

🔗

A vector

v

is orthogonal to

W

if it is orthogonal to every element of

W :

i.e., if

⟨ v, w ⟩ = 0

for all

w \in W .

🔗

The orthogonal complement of

W,

denoted

W^{⊥},

is the set of all elements of

V

orthogonal to

W :

i.e.,

W^{⊥} = {v \in V : ⟨ v, w ⟩ = 0 for all w \in W} .

🔗

Remark 5.3.2. Computing $W^{⊥}$ .

🔗

According to Definition 5.3.1, to verify that a vector

v

lies in

W^{⊥},

we must show that

⟨ v, w ⟩ = 0

for all

w \in W .

The “for all” quantifier here can potentially make this an onerous task: there are in principle infinitely many

w

to check! In the special case where

W

has a finite spanning set, so that

W = span {w_{1}, w_{2}, \dots, w_{r}}

for some vectors

w_{i},

deciding whether

v \in W^{⊥}

reduces to checking whether

⟨ v, w_{i} ⟩ = 0

for all

1 \leq i \leq r .

In other words, we have

v \in W^{⊥} ⟺ ⟨ v, w_{i} ⟩ = 0 for all 1 \leq i \leq r .

🔗

The forward implication of this equivalence is clear: if

v

is orthogonal to all elements of

W,

then clearly it is orthogonal to each

w_{i} .

The reverse implication is left as an exercise. (See Exercise 5.3.6.15.)

🔗

We illustrate this computational technique in the next examples.

🔗

Example 5.3.3.

🔗

Consider the inner product space

R^{2}

together with the dot product. Let

W = span {(1, 1)} = {(t, t) : t \in R} :

the line

ℓ \subseteq R^{2}

with equation

y = x .

Compute

W^{⊥}

and identify it as a familiar geometric object in

R^{2} .

Solution.

According to Remark 5.3.2, since

W = span {(1, 1)},

we have

x \in W^{⊥} ⟺ x \cdot (1, 1) = 0 .

Letting

x = (x, y),

we see that

x \cdot (1, 1) = 0

if and only if

x + y = 0,

if and only if

y = - x .

Thus

W^{⊥} = {(x, y) : y = - x}

is the line

ℓ^{'} \subseteq R^{2}

with equation

y = - x .

Observe that the lines

ℓ

and

ℓ^{'}

are indeed perpendicular to one another. (Graph them!)

🔗

Example 5.3.4.

🔗

Consider the inner product space

R^{3}

together with the dot product. Let

W \subseteq R^{3}

be the plane with equation

x - 2 y - z = 0 .

Compute

W^{⊥}

and identify this as a familiar geometric object in

R^{3} .

Solution.

First, solving

x - 2 y - z = 0

for

(x, y, z),

we see that

W = {(2 s + t, s, t) : s, t \in R} = span {(2, 1, 0), (1, 0, 1)} .

Next, according to Remark 5.3.2 we have

x \in W^{⊥} ⟺ x \cdot (2, 1, 0) = 0 and x \cdot (1, 0, 1) = 0 .

It follows that

W^{⊥}

is the set of vectors

x = (x, y, z)

satisfying the linear system

\begin{array}{rcr} 2 x & + & y & = & 0 \\ x & + & z & = & 0 \end{array} .

Solving this system using Gaussian elimination we conclude that

W^{⊥} = {(t, - 2 t, - t) : t \in R} = span {(1, - 2, - 1)},

which we recognize as the line

ℓ \subseteq R^{3}

passing through the origin with direction vector

(1, - 2, - 1) .

This is none other than the normal line to the plane

W

passing through the origin.

🔗

Theorem 5.3.5. Orthogonal complement.

🔗

Let

(V, ⟨, ⟩)

be an inner product vector space, and let

W \subseteq V

be a subspace.

🔗
The orthogonal complement $W^{⊥}$ is a subspace of $V .$
🔗
We have $W \cap W^{⊥} = {0} .$
🔗

If $\dim V = n < \infty$ then

$\dim W + \dim W^{⊥} = n .$

🔗

Proof.

The proof is left as an exercise. (See Exercise 5.3.6.17–5.3.6.18.)

🔗

Example 5.3.6.

🔗

Consider the inner product space

R^{3}

with the dot product. Let

W = span {(1, 1, 1)} \subset R^{3},

the line passing through the origin with direction vector

(1, 1, 1) .

The orthogonal complement

W^{⊥}

is the set of vectors orthogonal to

(1, 1, 1) .

Using the definition of dot product, this is the set of solutions

(x, y, z)

to the equation

x + y + z = 0,

🔗

which we recognize as the plane passing through the origin with normal vector

(1, 1, 1) .

Note that we have

\dim W + \dim W^{⊥} = 1 + 2 = 3,

🔗

as predicted in Theorem 5.3.5.

🔗

The notion of orthogonal complement gives us a more conceptual way of understanding the relationship between the various fundamental spaces of a matrix.

🔗

Theorem 5.3.7. Fundamental spaces and orthogonal complements.

🔗

Let

A

m \times n,

and consider

R^{n}

and

R^{m}

as inner product spaces with respect to the dot product. Then:

🔗
$null (A) = {(row (A))}^{⊥},$ and thus $row (A) = {(null (A))}^{⊥} .$
🔗
$null (A^{T}) = {(col (A))}^{⊥},$ and thus $col (A) = {(null (A^{T}))}^{⊥} .$

🔗

Proof.

Using the dot product method of matrix multiplication, we see that a vector $x \in null (A)$ if and only if $x \cdot r_{i} = 0$ for each row $r_{i}$ of $A,$ if and only if $x \cdot w = 0$ for all $w \in span {r_{1}, r_{2}, \dots, r_{m}} = row A$ (see Remark 5.3.2), if and only if $x \in (row A)^{⊥} .$ This shows $null A = (row A)^{⊥} .$

We can use Corollary 5.3.13 to conclude $row A = (null A)^{⊥} .$ Alternatively, and more directly, the argument above shows that $w \in row A ⟹ w \in (null A)^{⊥},$ proving $row A \subseteq (null A)^{⊥} .$ Next, by the rank-nullity theorem we have $\dim row A = n - \dim null A;$ and by Theorem 5.3.5 we have $\dim (null A)^{⊥} = n - \dim null A .$ It follows that $\dim row A = \dim (null A)^{⊥} .$ Since $row A \subseteq (null A)^{⊥}$ and $\dim row A = \dim (null A)^{⊥},$ we conclude by Corollary 3.7.13 that $row A = (null A)^{⊥} .$
This follows from (1) and the fact that $col (A) = row (A^{T}) .$

🔗

Example 5.3.8.

🔗

Understanding the orthogonal relationship between

null A

and

row A

allows us in many cases to quickly determine/visualize the one from the other. As an example, consider

A = [\begin{matrix} 1 & - 1 & 1 \\ 1 & - 1 & - 1 \end{matrix}] .

Looking at the columns, we see easily that

rank A = 2,

which implies that

nullity A = 3 - 2 = 1 .

Since

(1, - 1, 0)

is an element of

null (A)

and

\dim null A = 1,

we must have

null A = span {(1, - 1, 0)},

a line. By orthogonality, we conclude that

row A = (null A)^{⊥},

🔗

which is the plane with normal vector

(1, - 1, 0)

passing through the origin.

🔗

Subsection 5.3.2 Orthogonal Projection

🔗

Theorem 5.3.9. Orthogonal projection theorem.

🔗

Let

(V, ⟨, ⟩)

be an inner product space, and let

W \subseteq V

be a finite-dimensional subspace.

Orthogonal decomposition.
🔗

For all $v \in V$ there are vectors $w$ and $w^{⊥}$ satisfying

$\begin{matrix} (5.3.1) & v = w + w^{⊥}, w \in W, w^{⊥} \in W^{⊥} . \end{matrix}$

Furthermore, the pair $w, w^{⊥}$ is unique in the following sense: if we have $v = u + u^{⊥}$ for some $u \in W$ and $u^{⊥} \in W^{⊥},$ then $u = w$ and $u^{⊥} = w^{⊥} .$ Accordingly, the vector equation (5.3.1) is called the orthogonal decomposition of $v$ with respect to $W;$ and the vector $w$ is called the orthogonal projection of $v$ onto $W,$ denoted $w = {proj}_{W} (v) .$
Orthogonal projection formula.
🔗

Choose any orthogonal basis $B = {w_{1}, w_{2}, \dots, w_{r}}$ of $W .$ We have

$\begin{matrix} (5.3.2) & {proj}_{W} (v) = \sum_{i = 1}^{r} \frac{⟨ v, w_{i} ⟩}{⟨ w_{i}, w_{i} ⟩} w_{i} . \end{matrix}$
Distance to $W$ .
🔗

The orthogonal projection $w = {proj}_{W} (v)$ is the element of $W$ that is closest to $v$ in the following sense: for all $w^{'} \in W$ we have

$d (v, {proj}_{W} (v)) \leq d (v, w^{'}),$

or equivalently,

$\begin{matrix} (5.3.3) & ‖ v - {proj}_{W} (v) ‖ \leq ‖ v - w^{'} ‖ . \end{matrix}$

Accordingly, we define the distance $d (v, W)$ between $v$ and $W$ to be

$d (v, W) = d (v, {proj}_{W} (v)) = ‖ v - {proj}_{W} (v) ‖ .$

🔗

Proof.

Let

B = {w_{1}, w_{2}, \dots, w_{r}} .

We first show that the vectors

\begin{matrix} (5.3.4) & w = \sum_{i = 1}^{r} \frac{⟨ v, w_{i} ⟩}{⟨ w_{i}, w_{i} ⟩} w_{i} \end{matrix}

and

w^{⊥} = v - w

satisfy the conditions in (5.3.1). It is clear that the

w

defined in (5.3.4) is an element of

W,

since it is a linear combination of the

w_{i} .

Furthermore, we see easily that our choice

w^{⊥} = v - w

satisfies

w + w^{⊥} = w + (v - w) = v .

It remains only to show that

w^{⊥} = v - w \in W^{⊥} .

Since

B

is a basis of

W,

it suffices to show that

⟨ w^{⊥}, w_{j} ⟩ = 0

for all

1 \leq i \leq r .

We compute:

\begin{aligned} ⟨ w^{⊥}, w_{j} ⟩ & = ⟨ v - {proj}_{W} (v), w_{i} ⟩ \\ = ⟨ v - \sum_{i = 1}^{r} \frac{⟨ v, w_{i} ⟩}{⟨ w_{i}, w_{i} ⟩} w_{i}, w_{j} ⟩ \\ = ⟨ v, w_{j} ⟩ - \sum_{i = 1}^{r} \frac{⟨ v, w_{i} ⟩}{⟨ w_{i}, w_{i} ⟩} ⟨ w_{i}, w_{j} ⟩ \\ = ⟨ v, w_{j} ⟩ - \frac{⟨ v, w_{j} ⟩}{⟨ w_{j}, w_{j} ⟩} ⟨ w_{j}, w_{j} ⟩ \\ = 0, \end{aligned}

as desired.

Having shown that a decomposition of

v

of the form (5.3.1) exists, we now show it is unique in the sense specified. Suppose we have

v = w + w^{⊥} = u + u^{⊥},

where

w, u \in W

and

w^{⊥}, u^{⊥} \in W^{⊥} .

Rearranging, we see that

w - u = u^{⊥} - w^{⊥} .

We now claim that

w - u = u^{⊥} - w^{⊥} = 0,

in which case

w = u

and

w^{⊥} = u^{⊥},

as desired. To see why the claim is true, consider the vector

v^{'} = w - u = u^{⊥} - w^{⊥} .

Since

v^{'} = w - u,

and

w, u \in W,

we have

v^{'} \in W .

On the other hand, since

v^{'} = u^{⊥} - w^{⊥},

and

u^{⊥}, w^{⊥} \in W^{⊥},

we have

v^{'} \in W^{⊥} .

Thus

v^{'} \in W \cap W^{⊥} .

Since

W \cap W^{⊥} = {0}

(Theorem 5.3.5), we conclude

v^{'} = w - u = u^{⊥} - w^{⊥} = 0,

as claimed.

At this point we have proved both (1) and (2), and it remains only to show that (5.3.3) holds for all

w^{'} \in W .

To this end we compute:

\begin{aligned} {‖ v - w^{'} ‖}^{2} & = {‖ w^{⊥} + (w - w^{'}) ‖}^{2} \\ = {‖ w^{⊥} ‖}^{2} + {‖ w - w^{'} ‖}^{2} & (Exercise 5.2.4.18) \\ \geq {‖ w^{⊥} ‖}^{2} \\ = {‖ v - w ‖}^{2} . \end{aligned}

This shows

{‖ v - w^{'} ‖}^{2} \geq {‖ v - w ‖}^{2} .

Taking square-roots now proves the desired inequality.

🔗

Remark 5.3.10. Orthogonal projection formula.

🔗

The formula (5.3.2) is very convenient for computing an orthogonal projection

{proj}_{W} (v),

but mark well this important detail: to apply the formula we must first provide an orthogonal basis of

W .

Thus unless one is provided, our first step in an orthogonal projection computation is to produce an orthogonal basis of

W .

In some simple cases (e.g., when

W

is 1- or 2-dimensional) this can be done by inspection. Otherwise, we use the Gram-Schmidt procedure.

🔗

Example 5.3.11.

🔗

Consider the inner product space

R^{3}

with the dot product. Let

W \subseteq R^{3}

be the plane with equation

x + y + z = 0 .

Compute

{proj}_{W} (v)

for each

v

below.

🔗
$v = (3, - 2, 2)$
🔗
$v = (2, 1, - 3)$
🔗
$v = (- 7, - 7, - 7)$

Solution.

According to Remark 5.3.10 our first step is to produce an orthogonal basis of

W .

We do so by inspection. Since

\dim W = 2,

we simply need to find two solutions to

x + y + z = 0

that are orthogonal to one another: e.g.,

w_{1} = (1, - 1, 0)

and

w_{2} = (1, 1, - 2) .

Thus we choose

B = {(1, - 1, 0), (1, 1, - 2)}

as our orthogonal basis, and our computations become a matter of applying (5.3.2), which in this case becomes

{proj}_{W} (v) = \frac{v \cdot w_{1}}{w_{1} \cdot w_{1}} w_{1} + \frac{v \cdot w_{2}}{w_{2} \cdot w_{2}} w_{2} = \frac{v \cdot w_{1}}{2} w_{1} + \frac{v \cdot w_{2}}{6} w_{2} .

Now compute:

\begin{aligned} {proj}_{W} ((3, - 2, 2)) & = \frac{5}{2} (1, - 1, 0) + \frac{- 3}{6} (1, 1, - 2) = (2, - 3, 1) \\ {proj}_{W} ((2, 1, - 3)) & = \frac{1}{2} (1, - 1, 0) + \frac{9}{6} (1, 1, - 2) = (2, 1, - 3) \\ {proj}_{W} ((- 7, - 7, - 7)) & = \frac{0}{2} (1, - 1, 0) + \frac{0}{6} (1, 1, - 2) = (0, 0, 0) . \end{aligned}

The last two computations might give you pause. Why do we have

{proj}_{W} ((2, 1, - 3)) = (2, 1, - 3)

and

{proj}_{W} ((- 7, 7 - 7, - 7)) = (0, 0, 0) ?

The answer is that

(2, 1, - 3)

is already an element of

W,

so it stands to reason that its projection is itself; and

(- 7, - 7, - 7)

is already orthogonal to

W

(it is a scalar multiple of

(1, 1, 1)

), so it stands to reason that its projection is equal to

0 .

See Exercise 5.3.6.20 for a rigorous proof of these claims.

🔗

Video example: orthogonal projection in function space.

🔗

Figure 5.3.12. Video: orthogonal projection in function space

🔗

Corollary 5.3.13. Complement of a complement.

🔗

Let

(V, ⟨, ⟩)

be an inner product space, and let

W \subseteq V

be a finite-dimensional subspace. Then

(W^{⊥})^{⊥} = W .

🔗

Proof.

Clearly

W \subseteq (W^{⊥})^{⊥} .

For the other direction, take

v \in (W^{⊥})^{⊥} .

Using the orthogonal projection theorem, we can write

v = w + w^{⊥}

with

w \in W

and

w^{⊥} \in W^{⊥} .

We will show

w^{⊥} = 0 .

Since

v \in (W^{⊥})^{⊥}

we have

⟨ v, w^{⊥} ⟩ = 0 .

Then we have

\begin{aligned} 0 & = ⟨ v, w^{⊥} ⟩ \\ = ⟨ w + w^{⊥}, w^{⊥} ⟩ \\ = ⟨ w, w^{⊥} ⟩ + ⟨ w^{⊥}, w^{⊥} ⟩ & (since W ⊥ W^{⊥}) \\ = 0 + ⟨ w^{⊥}, w^{⊥} ⟩ \end{aligned}

Thus

⟨ w^{⊥}, w^{⊥} ⟩ = 0 .

It follows that

w^{⊥} = 0,

and hence

v = w + 0 = w \in W .

🔗

Corollary 5.3.14. Orthgonal projection is linear.

🔗

Let

(V, ⟨, ⟩)

be an inner product space, and let

W \subseteq V

be a finite-dimensional subspace.

🔗

The function

$\begin{aligned} {proj}_{W} : V & \to V \\ v & \mapsto {proj}_{W} (v) \end{aligned}$

is a linear transformation.
🔗
We have $im {proj}_{W} = W$ and $null {proj}_{W} = W^{⊥} .$

🔗

Proof.

We must show that ${proj}_{W} (c v + d w) = c {proj}_{W} (v) + d {proj}_{W} (w)$ for all $c, d \in R$ and $v, w \in V .$ We pick an orthogonal basis $B = {v_{1}, v_{2}, \dots, v_{r}}$ of $W$ and compute, using formula (5.3.2):

$\begin{aligned} {proj}_{W} (c v + d w) & = \sum_{i = 1}^{r} \frac{⟨ c v + d w, v_{i} ⟩}{⟨ v_{i}, v_{i} ⟩} v_{i} \\ = \sum_{i = 1}^{r} \frac{c ⟨ v, v_{i} ⟩ + d ⟨ w, v_{i} ⟩}{⟨ v_{i}, v_{i} ⟩} v_{i} \\ = c \sum_{i = 1}^{r} \frac{⟨ v, v_{i} ⟩}{⟨ v_{i}, v_{i} ⟩} v_{i} + d \sum_{i = 1}^{r} \frac{⟨ w, v_{i} ⟩}{⟨ v_{i}, v_{i} ⟩} v_{i} \\ = c {proj}_{W} (v) + d {proj}_{W} (w) . \end{aligned}$
By definition we have ${proj}_{W} (v) \in W$ for all $v \in W,$ and thus $im {proj}_{W} \subseteq W .$ For the other direction, if $w \in W,$ then $w = {proj}_{W} (w)$ (Exercise 5.3.6.20), and thus $w \in im proj .$ This proves $im proj = W .$

The fact that $null proj = W^{⊥}$ follows from the equivalence stated in (b) of Exercise 5.3.6.20.

🔗

Subsection 5.3.3 Orthogonal projection in $R^{2}$ and $R^{3}$

🔗

For this subsection we will always work within Euclidean space: i.e.,

V = R^{n}

with the dot product. In applications we often want to compute the projection of a point onto a line (in

R^{2}

R^{3}

) or plane (in

R^{3}

). According to Corollary 5.3.14 the operation of projecting onto any subspace

W \subseteq R^{n}

is in fact a linear transformation

{proj}_{W} : R^{n} \to R^{n} .

By Corollary 3.6.18 we have

{proj}_{W} = T_{A},

where

A = [\begin{matrix} | & | & | \\ {proj}_{W} (e_{1}) & {proj}_{W} (e_{2}) & \dots & {proj}_{W} (e_{n}) \\ | & | & | \end{matrix}] .

🔗

Lastly, (5.3.2) gives us an easy formula for computing

{proj}_{W} (e_{j})

for all

j,

once we have selected an orthogonal basis for

W .

As a result we can easily derive matrix formulas for projection onto any subspace

W

of any Euclidean space

R^{n} .

We illustrate this with some examples in

R^{2}

and

R^{3}

below.

🔗

Example 5.3.15. Projection onto a line $ℓ \subseteq R^{3}$ .

🔗

Any line in

R^{3}

passing through the origin can be described as

ℓ = span {v},

for some

v = (a, b, c) \neq 0 .

The set

{(a, b, c)}

is trivially an orthogonal basis of

ℓ .

Using (5.3.2), we have

{proj}_{ℓ} (x) = \frac{x \cdot v}{v \cdot v} v = \frac{a x + b y + c z}{a^{2} + b^{2} + c^{2}} (a, b, c) .

🔗

It follows that

{proj}_{ℓ} = T_{A},

where

A = [\begin{matrix} | & | & | \\ {proj}_{ℓ} ((1, 0, 0)) & {proj}_{ℓ} ((0, 1, 0)) & {proj}_{ℓ} ((0, 0, 1)) \\ | & | & | \end{matrix}] = \frac{1}{a^{2} + b^{2} + c} [\begin{matrix} a^{2} & a b & a c \\ a b & b^{2} & b c \\ a c & b c & c^{2} \end{matrix}] .

🔗

Example 5.3.16.

🔗

Consider the line

ℓ = span {(1, 2, 1)} \subseteq R^{3} .

🔗
Find the matrix $A$ such that ${proj}_{ℓ} = T_{A} .$
🔗
Use your matrix formula from (a) to compute ${proj}_{ℓ} ((- 2, 3, 1)),$ ${proj}_{ℓ} ((- 2, - 4, - 2)),$ and ${proj}_{ℓ} ((1, - 1, 1)) .$
🔗
Compute $d ((- 2, 3, 1), ℓ)$ and $d ((- 2, - 4, - 2), ℓ) .$

Solution.

Using the general formula described in Example 5.3.15, we have

$A = \frac{1}{6} [\begin{matrix} 1 & 2 & 1 \\ 2 & 4 & 2 \\ 1 & 2 & 1 \end{matrix}] .$
Now compute

$\begin{aligned} {proj}_{ℓ} ((- 2, 3, 1)) & = \frac{1}{6} [\begin{array}{c} 1 & 2 & 1 \\ 2 & 4 & 2 \\ 1 & 2 & 1 \end{array}] [\begin{array}{r} - 2 \\ 3 \\ 1 \end{array}] = \frac{1}{6} [\begin{array}{r} 5 \\ 10 \\ 5 \end{array}] \\ {proj}_{ℓ} ((- 2, - 4, - 2)) & = \frac{1}{6} [\begin{array}{c} 1 & 2 & 1 \\ 2 & 4 & 2 \\ 1 & 2 & 1 \end{array}] [\begin{array}{r} - 2 \\ - 4 \\ - 2 \end{array}] = \frac{1}{6} [\begin{array}{r} - 2 \\ - 4 \\ - 2 \end{array}] \\ {proj}_{ℓ} ((1, - 1, 1)) & = \frac{1}{6} [\begin{array}{c} 1 & 2 & 1 \\ 2 & 4 & 2 \\ 1 & 2 & 1 \end{array}] [\begin{array}{r} 1 \\ - 1 \\ 1 \end{array}] = \frac{1}{6} [\begin{array}{r} 0 \\ 0 \\ 0 \end{array}] . \end{aligned}$

The last two computations, ${proj}_{ℓ} ((- 2, - 4, - 2)) = (- 2, - 4, - 2)$ and ${proj}_{ℓ} ((1, - 1, 1)) = (0, 0, 0),$ should come as no surprise, since $(- 2, - 4, - 2) \in ℓ$ and $(1, - 1, 1) \in ℓ^{⊥} .$ (See Exercise 5.3.6.20.)
We have

$\begin{aligned} d ((- 2, 3, 1), ℓ) & = ‖ (- 2, 3, 1) - {proj}_{ℓ} ((- 2, 3, 1)) ‖ \\ = ‖ \frac{1}{6} (- 17, 8, 1) ‖ = \frac{\sqrt{354}}{6} \\ d ((- 2, - 4, - 2), ℓ) & = ‖ (- 2, - 4, - 2) - {proj}_{ℓ} ((- 2, - 4, - 2)) ‖ \\ = ‖ (0, 0, 0) ‖ = 0 . \end{aligned}$

Again, the second computation should come as no surprise. Since $(- 2, - 4, - 2)$ is itself an element of $ℓ,$ it stands to reason that its distance to $ℓ$ is equal to zero.

🔗

Example 5.3.17. Projection onto planes in $R^{3}$ .

🔗

Any plane

W \subseteq R^{3}

passing through the origin can be described as

W = {(x, y, z) \in R^{3} : a x + b y + c z = 0} .

Equivalently,

W

is the set of all

x \in R^{3}

satisfying

x \cdot (a, b, c) = 0 :

i.e.,

W = ℓ^{⊥},

where

ℓ = span {(a, b, c) .

Consider the orthogonal decomposition with respect to

ℓ :

x = {proj}_{ℓ} (x) + (x - {proj}_{ℓ} (x)) .

🔗

Since

x - {proj}_{ℓ} (x) \in ℓ^{⊥} = W

and

{proj}_{ℓ} (x) \in ℓ = W^{⊥},

we see that this is also an orthogonal decomposition with respect to

W!

Using the matrix formula for

{proj}_{ℓ}

from Example 5.3.15, we have

\begin{aligned} {proj}_{W} (x) & = x - {proj}_{ℓ} (x) \\ = I x - A x & (A = \frac{1}{a^{2} + b^{2} + c} [\begin{array}{c} a^{2} & a b & a c \\ a b & b^{2} & b c \\ a c & b c & c^{2} \end{array}]) \\ = (I - A) x \\ = \frac{1}{a^{2} + b^{2} + c^{2}} [\begin{array}{c} b^{2} + c^{2} & - a b & - a c \\ - a b & a^{2} + c^{2} & - b c \\ - a c & - b c & a^{2} + b^{2} \end{array}] . \end{aligned}

🔗

We conclude that

{proj}_{W} = T_{B},

where

B = \frac{1}{a^{2} + b^{2} + c^{2}} [\begin{matrix} b^{2} + c^{2} & - a b & - a c \\ - a b & a^{2} + c^{2} & - b c \\ - a c & - b c & a^{2} + b^{2} \end{matrix}] .

🔗

Example 5.3.18.

🔗

Consider the plane

W = {(x, y, z) \in R^{3} : x - 2 y + z = 0 -} .

🔗
Find the matrix $A$ such that ${proj}_{W} = T_{A} .$
🔗
Use your matrix formula from (a) to compute ${proj}_{W} ((2, 1, 1))$ and ${proj}_{W} ((1, 1, 1)) .$
🔗
Compute $d ((2, 1, 1), W)$ and $d ((1, 1, 1), W) .$

Solution.

Using the general formula described in Example 5.3.17, we have

$A = \frac{1}{6} [\begin{array}{rrr} 5 & 2 & - 1 \\ 2 & 2 & 2 \\ - 1 & 2 & 5 \end{array}] .$
Now compute

$\begin{aligned} {proj}_{ℓ} ((2, 1, 1)) & = \frac{1}{6} [\begin{array}{rrr} 5 & 2 & - 1 \\ 2 & 2 & 2 \\ - 1 & 2 & 5 \end{array}] [\begin{array}{r} 2 \\ 1 \\ 1 \end{array}] = \frac{1}{6} [\begin{array}{r} 11 \\ 8 \\ 5 \end{array}] \\ {proj}_{ℓ} ((1, 1, 1)) & = \frac{1}{6} [\begin{array}{rrr} 5 & 2 & - 1 \\ 2 & 2 & 2 \\ - 1 & 2 & 5 \end{array}] [\begin{array}{r} 1 \\ 1 \\ 1 \end{array}] = \frac{1}{6} [\begin{array}{r} 0 \\ 0 \\ 0 \end{array}] . \end{aligned}$
We have

$\begin{aligned} d ((2, 1, 1), W) & = ‖ (2, 1, 1) - {proj}_{W} ((2, 1, 1)) ‖ \\ = ‖ \frac{1}{6} (1, - 2, - 1) ‖ = \frac{6}{6} \\ d ((1, 1, 1), W) & = ‖ (1, 1, 1) - {proj}_{W} ((1, 1, 1)) ‖ \\ = ‖ (0, 0, 0) ‖ = 0 . \end{aligned}$

🔗

Subsection 5.3.4 Trigonometric polynomial approximation

🔗

Consider the inner product space consisting of

C ([0, 2 π])

along with the integral inner product

⟨ f, g ⟩ = \int_{0}^{2 π} f (x) g (x) d x .

In Example 5.2.4 we saw that the set

B = {1, \cos (x), \sin (x), \cos (2 x), \sin (2 x), \dots, \cos (n x), \sin (n x)}

🔗

is orthogonal with respect to this inner product. Thus

B

is an orthogonal basis of

span B = {g \in C ([0, 2 π]) : g (x) = a_{0} + \sum_{k = 1}^{n} a_{k} \cos k x + b_{k} \sin k x for some a_{i}, b_{i} \in R} .

🔗

We call

W = span B

the space of trigonometric polynomials of degree at most

n

🔗

Since

B

is an orthogonal basis of

W,

given an arbitrary function

f (x) \in C [0, 2 π],

its orthogonal projection

\hat{f} = {proj}_{W} (f)

is given by

\hat{f} (x) = a_{0} + a_{1} \cos (x) + b_{1} \sin (x) + a_{2} \cos (2 x) + b_{2} \sin (2 x) + \dots + a_{n} \cos (n x) + b_{n} \sin (n x),

🔗

where

a_{0} = \frac{1}{2 π} \int_{0}^{2 π} f (x) d x, a_{j} = \frac{1}{π} \int_{0}^{2 π} f (x) \cos (j x) d x, b_{k} = \frac{1}{π} \int_{0}^{2 π} f (x) \sin (k x) d x .

🔗

Here we are using (5.3.2), as well as the inner product formulas

⟨ 1, 1 ⟩ = 2 π

and

⟨ \cos n x, \cos n x ⟩ = ⟨ \sin n x, \sin n x ⟩ = π

from Example 5.2.4.

🔗

What is the relationship between

f

and

\hat{f} ?

Theorem 5.3.9 tells us that

\hat{f}

is the “best” trigonometric polynomial approximation of

f (x)

of degree at most

n

in the following sense: given any any other trigonometric polynomial

g \in W,

we have

| | f - \hat{f} | | \leq ‖ f - g ‖ .

🔗

Unpacking the definition of norm in this inner product space, we conclude that

\int_{0}^{2 π} (f - \hat{f})^{2} d x \leq \int_{0}^{2 π} (f - g)^{2} d x

🔗

for all

g \in W .

🔗

Thus, given a continuous function

f

[0, 2 π],

linear algebra shows us how to find its best trigonometric polynomial approximation of the form

g (x) = a_{0} + \sum_{k = 1}^{n} a_{k} \cos k x + b_{k} \sin k x .

🔗

However, linear algebra does not tell us just how good this approximation is. This question, among others, is tackled by another mathematical theory: Fourier analysis. There we learn that the trigonometric polynomial approximations get arbitrarily close to

f

as we let

n

increase. More precisely, letting

{\hat{f}}_{n}

be the orthogonal projection of

f

onto the space of trigonometric polynomials of degree at most

n,

we have

lim_{n \to \infty} ‖ f - {\hat{f}}_{n} ‖ = 0 .

🔗

Subsection 5.3.5 Least-squares solution to linear systems

🔗

In statistics we often wish to approximate a scatter plot of points

P_{i} = (X_{i}, Y_{i}),

1 \leq i \leq m,

with a line

ℓ : y = m x + b

that “best fits” the data. “Finding” this line amounts to finding the appropriate slope

m

and

y

-intercept

b :

i.e., in this setup, the points

P_{i} = (X_{i}, Y_{i})

are given, and

m

and

b

are the unknowns we wish to find. For the line to perfectly fit the data, we would want

Y_{i} = m X_{i} + b for all 1 \leq i \leq m .

🔗

In other words

(m, b)

would be a solution to the matrix equation

A x = y,

where

x = [\begin{matrix} m \\ b \end{matrix}], A = [\begin{matrix} X_{1} & 1 \\ X_{2} & 1 \\ ⋮ & ⋮ \\ X_{m} & 1 \end{matrix}], y = [\begin{matrix} Y_{1} \\ Y_{2} \\ ⋮ \\ Y_{m} \end{matrix}] .

🔗

Of course in most situations the provided points do not lie on a line, and thus there is no solution

x

to the given matrix equation

A x = y .

When this is the case we can use the theory of orthogonal projection to find what is called a least-squares solution, which we now describe in detail.

🔗

The least-squares method applies to any matrix equation

\begin{matrix} (5.3.5) & \underset{m \times n}{A} \underset{n \times 1}{x} = \underset{m \times 1}{y}, \end{matrix}

🔗

where

A

and

y

are given, and

x

is treated as an unknown vector. Recall that

\begin{aligned} A x = y has a solution & ⟺ y \in col A & (Theorem 3.8.6) . \end{aligned}

🔗

When

y \notin col A,

and hence (5.3.5) does not have a solution, the least-squares method proceeds by replacing

y

with the element of

W = col A

closest to it: that is, with its orthogonal projection onto

W .

Let

\hat{y} = {proj}_{W} (y),

where orthogonal projection is taken with respect to the dot product on

R^{m},

and consider the adjusted matrix equation

\begin{matrix} (5.3.6) & A x = \hat{y} . \end{matrix}

🔗

By definition of

{proj}_{W},

we have

\hat{y} \in W = col A,

and thus there is a solution

\hat{x}

to (5.3.6). We call

\hat{x}

a least-squares solution to (5.3.5). Observe that

\hat{x}

does not necessarily satisfy

A \hat{x} = y;

rather, it satisfies

A \hat{x} = \hat{y} .

What makes this a “least-squares” solution is that

A \hat{x} = \hat{y}

is the element of

W = col A

closest to

y .

With respect to the dot product, this means that a least-squares solution

\hat{x}

minimizes the quantity

‖ y - A x ‖ = \sqrt{(y_{1} - y_{1}^{'})^{2} + (y_{2} - y_{2}^{'})^{2} + \dots + (y_{n} - y_{n}^{'})^{2}},

🔗

among all

x \in R^{n} .

🔗

Example 5.3.19. Best fitting line.

🔗

Suppose we wish to find a line

ℓ : y = m x + b

that best fits (in the least-square sense) the following data points:

P_{1} = (- 3, 1), P_{2} = (1, 2), P_{3} = (2, 3) .

Following the discussion above, we seek a solution

x = (m, b)

to the matrix equation

A x = y,

where

x = [\begin{matrix} m \\ b \end{matrix}], A = [\begin{array}{rr} - 3 & 1 \\ 1 & 1 \\ 2 & 1 \end{array}], y = [\begin{matrix} 1 \\ 2 \\ 3 \end{matrix}] .

🔗

Using Gaussian elimination, we see easily that this equation has no solution: equivalently,

y \notin W = col A .

Accordingly, we compute

\hat{y} = {proj}_{W} (y)

and find a solution to

A \hat{x} = \hat{y} .

Conveniently, the set

B = {(- 3, 2, 1), (1, 1, 1)}

is already an orthogonal basis of

W = col A,

allowing us to use (5.3.2):

\hat{y} = \frac{y \cdot (- 3, 1, 2)}{(- 3, 2, 1) \cdot (- 3, 1, 2)} (- 3, 1, 2) + \frac{y \cdot (1, 1, 1)}{(1, 1, 1) \cdot (1, 1, 1)} (1, 1, 1) = \frac{1}{14} (13, 33, 38) .

🔗

Lastly, solving

A \hat{x} = \hat{y}

yields

(m, b) = \hat{x} = (5 / 14, 2),

and we conclude the line

ℓ : y = (5 / 14) x + 2

is the one that best fits the data in the least-squares sense.

🔗

Remark 5.3.20. Visualizing least-squares.

🔗

Figure 5.3.21 helps us give a graphical interpretation of how the line

ℓ : y = (5 / 14) x + 2

best approximates the points

P_{1} = (- 3, 1), P_{2} = (1, 2), P_{3} = (2, 3) .

Figure 5.3.21. Least-squares visualization

Let

y = (1, 2, 3) = (y_{1}, y_{2}, y_{3})

be the given

y

-values of the points, and let

\hat{y} = (y_{1}^{'}, y_{2}^{'}, y_{3}^{'})

be the orthogonal projection of

y

onto

col A .

In the graph the values

ϵ_{i}

denote the vertical difference

ϵ_{i} = y_{i} - y_{i}^{'}

between the data points, and our fitting line. The projection

\hat{y}

makes the error

‖ y - \hat{y} ‖ = \sqrt{ϵ_{1}^{2} + ϵ_{2}^{2} + ϵ_{3}^{2}}

as small as possible. This means if I draw any other line and compute the corresponding differences

ϵ_{i}^{'}

at the

x

-values -3, 1 and 2, then

ϵ_{1}^{2} + ϵ_{2}^{2} + ϵ_{3}^{2} \leq (ϵ_{1}^{'})^{2} + (ϵ_{2}^{'})^{2} + (ϵ_{3}^{'})^{2}

🔗

To compute a least-squares solution to

A x = y

we must first compute the orthogonal projection of

y

onto

W = col A;

and this in turn requires first producing an orthogonal basis of

col A,

which may require using the Gram-Schmidt procedure. The following result bypasses these potentially onerous steps by characterizing a least-squares solution to

A x = y

as a solution to the matrix equation

A^{T} A x = A^{T} y .

🔗

Theorem 5.3.22. Least-squares matrix formula.

🔗

Given an

m \times n

matrix

A

and

m \times 1

column vector

y,

a vector

\hat{x}

is a least-squares solution to

A x = y

if and only if

\begin{matrix} (5.3.7) & A^{T} A x = A^{T} y . \end{matrix}

🔗

In other words, we can find a least-squares solution by solving the matrix equation (5.3.7) directly.

🔗

Proof.

Let

W = col A,

and let

\hat{y} = {proj}_{W} (y) .

The key observation is that a vector

\hat{x}

satisfies

A \hat{x} = \hat{y}

if and only if

y = A \hat{x} + (y - A \hat{x})

is an orthogonal decomposition of

y

with respect to

W = col A;

and this is true if and only if

y - A \hat{x} \in (col A)^{⊥} .

Thus we have

\begin{aligned} A \hat{x} = \hat{y} & ⟺ y - A \hat{x} \in (col A)^{⊥} \\ ⟺ y - A \hat{x} \in null A^{T} & ((col A)^{⊥} = null A^{T}, Theorem 5.3.7) \\ ⟺ A^{T} (y - A \hat{x}) = 0 \\ ⟺ A^{T} y - A^{T} A \hat{x} = 0 \\ ⟺ A^{T} A \hat{x} = A^{T} y . \end{aligned}

🔗

Example 5.3.23.

🔗

Consider again the matrix equation

A x = y

from Example 5.3.19. According to Theorem 5.3.22 the least-squares solution can be found by solving the equation

A^{T} A x = A^{T} y

for

x .

We compute

\begin{aligned} A^{T} A & = [\begin{array}{rr} 14 & 0 \\ 0 & 3 \end{array}] & A^{T} y & = [\begin{array}{r} 5 \\ 6 \end{array}] \end{aligned}

🔗

and solve

[\begin{array}{rr} 14 & 0 \\ 0 & 3 \end{array}] x = [\begin{array}{r} 5 \\ 6 \end{array}] ⟺ x = (5 / 14, 2),

🔗

just as before.

🔗

Compute the orthogonal projection of

\vec{v} = [\begin{matrix} - 8 \\ - 9 \\ 1 \end{matrix}]

onto the line

L

through

[\begin{matrix} - 5 \\ 6 \\ - 4 \end{matrix}]

and the origin.

🔗

{p r o j}_{L} (\vec{v}) =

(3 × 1 array).

🔗

2.

🔗

Let

\vec{y} = [\begin{matrix} 9 \\ - 7 \\ 3 \end{matrix}]

and

\vec{u} = [\begin{matrix} - 3 \\ - 6 \\ - 3 \end{matrix}] .

Write

\vec{y}

as the sum of two orthogonal vectors,

{\vec{x}}_{1}

S p a n {u}

and

{\vec{x}}_{2}

orthogonal to

\vec{u} .

🔗

{\vec{x}}_{1} =

(3 × 1 array),

{\vec{x}}_{2} =

(3 × 1 array).

🔗

3.

🔗

(a) Find the distance from the point

P_{0} = (- 2, 7, 6)

to the plane

2 x + 4 y - 7 z = - 5 .

Use sqrt() to enter square roots.

🔗

Distance:

🔗

(b) Find the equation of the plane that passes through the points

P = (2, 0, - 5),

Q = (- 5, 1, - 2),

and

R = (1, - 2, 2) .

Write your answer in terms of the variables

x,

y,

z .

🔗

Answer:

Answer 1.

1.56502

Answer 2.

13 x + 46 y + 15 z = - 49

Solution.

Solution: (a) The idea is to find a convenient point lying in the given plane. For example, let us pick

P_{1} = (1, 0, 1),

and let

v

be the vector from

P_{1}

P_{0} .

Then the distance from

P_{0}

to the plane is the absolute value of the scalar projection of

v

onto the normal vector

N = [\begin{matrix} 2 & 4 & - 7 \end{matrix}]

So, we get that the distance is

\frac{13}{\sqrt{69}} .

(b) To find an equation of the plane in question, we can work with the point

P

and the normal vector

N

given by the cross product of the vector from

P

Q,

and the vector from

P

R .

It turns out that

N = [\begin{matrix} - 13 & - 46 & - 15 \end{matrix}]

So, an equation of the plane is

- 13 x - 46 y - 15 z = 49 .

🔗

4.

🔗

Find bases of the kernel and image of the orthogonal projection onto the plane

- 6 x - 3 y + z = 0

R^{3} .

🔗

A basis for the kernel is

{

(3 × 1 array)

} .

🔗

A basis for the image is

{

(3 × 1 array), (3 × 1 array)

} .

🔗

5.

🔗

Let

\vec{x} = [\begin{matrix} 1 \\ - 3 \\ - 4 \\ 0 \end{matrix}], \vec{y} = [\begin{matrix} - 3 \\ - 3 \\ - 5 \\ 1 \end{matrix}], \vec{z} = [\begin{matrix} 11 \\ 17 \\ - 16.5 \\ 36.5 \end{matrix}] .

🔗

Use the Gram-Schmidt process to determine an orthonormal basis for the subspace of

R^{4}

spanned by

\vec{x},

\vec{y},

and

\vec{z} .

🔗

{

(4 × 1 array), (4 × 1 array), (4 × 1 array)

} .

🔗

6.

🔗

Find the least-squares solution

{\vec{x}}^{*}

of the system

[\begin{array}{cc} 1 & 0 \\ 0 & 1 \\ 0 & 0 \end{array}] \vec{x} = [\begin{matrix} 4 \\ 5 \\ 9 \end{matrix}] .

🔗

{\vec{x}}^{*} =

(2 × 1 array)

🔗

7.

🔗

By using the method of least squares, find the best line through the points:

🔗

(2, 3),

(- 1, - 3),

(0, 0) .

🔗

Step 1. The general equation of a line is

c_{0} + c_{1} x = y .

Plugging the data points into this formula gives a matrix equation

A c = y .

🔗

(3 × 2 array)

[\begin{matrix} c_{0} \\ c_{1} \end{matrix}]

=

(3 × 1 array)

🔗

Step 2. The matrix equation

A c = y

has no solution, so instead we use the normal equation

A^{T} A \hat{c} = A^{T} y

🔗

A^{T} A =

(2 × 2 array)

🔗

A^{T} y =

(2 × 1 array)

🔗

Step 3. Solving the normal equation gives the answer

🔗

\hat{c} =

(2 × 1 array)

🔗

which corresponds to the formula

🔗

y =

🔗

Analysis. Compute the predicted

y

values:

\hat{y} = A \hat{c} .

🔗

\hat{y} =

(3 × 1 array)

🔗

Compute the error vector:

e = y - \hat{y} .

🔗

e =

(3 × 1 array)

🔗

Compute the total error:

SSE = e_{1}^{2} + e_{2}^{2} + e_{3}^{2} .

🔗

SSE =

Answer 1.

- 0.642857 + 1.92857 x

Answer 2.

0.642857

🔗

8.

🔗

By using the method of least squares, find the best parabola through the points:

🔗

(- 1, 0),

(- 2, - 2),

(0, 3),

(1, 1)

🔗

Step 1. The general equation of a parabola is

c_{0} + c_{1} x + c_{2} x^{2} = y .

Plugging the data points into this formula gives a matrix equation

A c = y .

🔗

(4 × 3 array)

[\begin{matrix} c_{0} \\ c_{1} \\ c_{2} \end{matrix}]

=

(4 × 1 array)

🔗

Step 2. The matrix equation

A c = y

has no solution, so instead we use the normal equation

A^{T} A \hat{c} = A^{T} y

🔗

A^{T} A =

(3 × 3 array)

🔗

A^{T} y =

(3 × 1 array)

🔗

Step 3. Solving the normal equation gives the answer

🔗

\hat{c} =

(3 × 1 array)

🔗

which corresponds to the formula

🔗

y =

Answer.

2.1 + 0.2 x + (- 1) x^{2}

🔗

.

In each exercise below you are given an inner product space

V,

a subspace

W = span B

where

B

is orthogonal, and a vector

v \in V .

Compute

{proj}_{W} (v) .

🔗

9.

🔗

V = R^{4}

with the dot product;

W = span {(1, 1, 1, 1), (1, - 1, 1, - 1), (1, 1, - 1, - 1)};

v = (2, 3, - 1, 1)

🔗

10.

🔗

V = R^{3}

with dot product with weights

k_{1} = 1, k_{2} = 2, k_{3} = 1;

W = {(1, 1, 1), (1, - 1, 1), (1, 0, - 1)};

v = (1, 2, 3)

🔗

11.

🔗

V = C ([0, 2 π])

with the integral inner product;

W = span {\cos x, \cos 2 x, \sin x};

f (x) = 3

for all

x \in [0, 2 π]

🔗

12.

🔗

Let

P \subseteq R^{3}

be the plane passing through the origin with normal vector

n = (1, 2, - 1) .

Find the orthogonal projection of

(1, 1, 1)

onto

P

with respect to the dot product.

🔗

13.

🔗

Recall that the trace

tr A

of a square matrix is the sum of its diagonal entries. Let

V = M_{22}

with inner product

⟨ A, B ⟩ = tr (A^{T} B) .

(You may take for granted that this operation is indeed an inner product on

M_{22} .

) Define

W = {A \in M_{22} : tr A = 0} .

🔗
Compute an orthogonal basis for $W .$ You can do this either by inspection (the space is manageable), or by starting with any basis of $W$ and applying the Gram-Schmidt procedure.
🔗

Compute ${proj}_{W} (A),$ where

$A = [\begin{matrix} 1 & 2 \\ 1 & 1 \end{matrix}] .$

🔗

14.

🔗

Let

V = C ([0, 1])

with the integral inner product, and let

f (x) = x .

Find the function of the form

g (x) = a + b \cos (2 π x) + c \sin (2 π x)

that “best approximates”

f (x)

in terms of this inner product: i.e. find the the

g (x)

of this form that minimizes

d (g, f) .

Hint.

The set

S = {f (x) = 1, g (x) = \cos (2 π x), h (x) = \sin (2 π x)}

is orthogonal with respect to the given inner product.

🔗

15.

🔗

Let

(V, ⟨, ⟩)

be an inner product space, let

S = {w_{1}, w_{2}, \dots, w_{r}} \subseteq V,

and let

W = span S .

Prove:

v \in W^{⊥} if and only if ⟨ v, w_{i} ⟩ = 0 for all 1 \leq i \leq r .

🔗

In other words, to check whether an element is in

W^{⊥},

it suffices to check that it is orthogonal to each element of its spanning set

S .

🔗

16.

🔗

Consider the inner product space

R^{4}

together with the dot product. Let

W = {(x_{1}, x_{2}, x_{3}, x_{4}) \in R^{4} : x_{1} = x_{3} and x_{2} = x_{4}} .

🔗

Provide orthogonal bases for

W

and

W^{⊥} .

🔗

17.

🔗

Prove statements (1) and (2) of Theorem 5.3.5.

🔗

18. Dimension of $W^{⊥}$ .

🔗

Prove statement (3) of Theorem 5.3.5: if

(V, ⟨, ⟩)

is an inner product space of dimension

n,

and

W

is a subspace of

V,

then

\dim W + \dim W^{⊥} = n .

Hint.

By Corollary 5.2.7 there is an orthogonal basis

B = {v_{1}, \dots, v_{r}}

W,

and furthermore, we can extend

B

to an orthogonal basis

B^{'} = {v_{1}, v_{2}, \dots, v_{r}, u_{1}, \dots, u_{n - r}}

of all of

V .

Show the

u_{i}

form a basis for

W^{⊥} .

🔗

19.

🔗

Prove Corollary 5.3.14 following the suggestion in the text.

🔗

20.

🔗

Let

V

an inner product space, and let

W \subseteq V

be a finite-dimensional subspace. Prove the following statements:

🔗
$v \in W$ if and only if ${proj}_{W} (v) = v;$
🔗
$v \in W^{⊥}$ if and only if ${proj}_{W} (v) = 0 .$

🔗

21.

🔗

We consider the problem of fitting a collection of data points

(x, y)

with a quadratic curve of the form

y = f (x) = a x^{2} + b x + c .

Thus we are given some collection of points

(x, y),

and we seek parameters

a, b, c

for which the graph of

f (x) = a x^{2} + b x + c

“best fits” the points in some way.

🔗
Show, using linear algebra, that if we are given any three points $(x, y) = (r_{1}, s_{1}), (r_{2}, s_{2}), (r_{3}, s_{3}),$ where the $x$ -coordinates $r_{i}$ are all distinct, then there is a unique choice of $a, b, c$ such that the corresponding quadratic function agrees precisely with the data. In other words, given just about any three points in the plane, there is a unique quadratic curve connecting them.
🔗

Now suppose we are given the four data points

$P_{1} = (0, 2), P_{2} = (1, 0), P_{3} = (2, 2), P_{4} = (3, 6) .$
1. 🔗
  Use the least-squares method described in the lecture notes to come up with a quadratic function $y = f (x)$ that “best fits” the data.
2. 🔗
  Graph the function $f$ you found, along with the points $P_{i} .$ (You may want to use technology.) Use your graph to explain precisely in what sense $f$ “best fits” the data.

Section 5.3 Orthogonal projection

Subsection 5.3.1 Orthogonal complement

Definition 5.3.1. Orthogonal complement.

Remark 5.3.2. Computing W⊥.

Example 5.3.3.

Example 5.3.4.

Theorem 5.3.5. Orthogonal complement.

Proof.

Example 5.3.6.

Theorem 5.3.7. Fundamental spaces and orthogonal complements.

Proof.

Example 5.3.8.

Subsection 5.3.2 Orthogonal Projection

Theorem 5.3.9. Orthogonal projection theorem.

Proof.

Remark 5.3.10. Orthogonal projection formula.

Example 5.3.11.

Video example: orthogonal projection in function space.

Corollary 5.3.13. Complement of a complement.

Proof.

Corollary 5.3.14. Orthgonal projection is linear.

Proof.

Subsection 5.3.3 Orthogonal projection in R2 and R3

Example 5.3.15. Projection onto a line ℓ⊆R3.

Example 5.3.16.

Example 5.3.17. Projection onto planes in R3.

Example 5.3.18.

Subsection 5.3.4 Trigonometric polynomial approximation

Subsection 5.3.5 Least-squares solution to linear systems

Example 5.3.19. Best fitting line.

Remark 5.3.20. Visualizing least-squares.

Theorem 5.3.22. Least-squares matrix formula.

Proof.

Example 5.3.23.

Exercises 5.3.6 Exercises

WeBWork Exercises

1.

2.

3.

4.

5.

6.

7.

8.

.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18. Dimension of W⊥.

19.

20.

21.

Remark 5.3.2. Computing $W^{⊥}$ .

Subsection 5.3.3 Orthogonal projection in $R^{2}$ and $R^{3}$

Example 5.3.15. Projection onto a line $ℓ \subseteq R^{3}$ .

Example 5.3.17. Projection onto planes in $R^{3}$ .

18. Dimension of $W^{⊥}$ .