Inner product spaces

Section 5.1 Inner product spaces

An inner product is an additional layer of structure we can define on a vector space

V .

It takes a pair of elements

v, w \in V

and returns a scalar

⟨ v, w ⟩ \in R .

As with the vector addition and scalar multiplication, we define inner products axiomatically, taking as our model the dot product on

R^{2}

and

R^{3} .

Our definition (5.1.1) simply promulgates a few important properties enjoyed by the dot product that may be familiar to you from studying calculus.

🔗

The addition of an inner product enriches the structure of a vector space considerably, and gives rise to a number of additional useful analytic tools. We highlight a few below.

Distance and angle: 🔗
A notion of distance and angle between two vectors can be defined relative to a given inner product. These provide a numeric measurement of how “close” (distance) or “closely oriented” (angle) two vectors in our space are.
Orthogonality: 🔗
Two vectors $v, w \in V$ are orthogonal, relative to a given inner product, if $⟨ v, w ⟩ = 0 .$ Orthogonality leads further to a general notion of orthogonal projection onto a subspace $W \subseteq V .$
Orthogonal bases: 🔗
An orthogonal basis of a vector space $V,$ relative to a given inner product, is one whose elements are pairwise orthogonal. As we will see there are many computational advantages of working with an orthogonal basis.

🔗

Subsection 5.1.1 Inner products

🔗

Definition 5.1.1. Inner product.

🔗

Let

V

be a vector space. An inner product on

V

is an operation that takes as input a pair of vectors

v, w \in V

and outputs a scalar

⟨ v, w ⟩ \in R .

Using function notation:

\begin{aligned} ⟨, ⟩ : & V \times V \to R \\ (v_{1}, v_{2}) & \mapsto ⟨ v_{1}, v_{2} ⟩ . \end{aligned}

🔗

Furthermore, this operation must satisfy the following axioms.

Symmetry.
🔗

For all $v, w \in V$ we have

$⟨ v, w ⟩ = ⟨ w, v ⟩ .$
Linearity.
🔗

For all $v, w, u \in V$ and $c, d \in R$ we have :

$⟨ c v + d w, u ⟩ = c ⟨ v, u ⟩ + d ⟨ w, u ⟩ .$

It follows by (i) (symmetry) that

$⟨ u, c v + d w ⟩ = c ⟨ u, v ⟩ + d ⟨ u, w ⟩ .$
Positive definiteness.
🔗

For all $v \in V$ we have

$\begin{aligned} ⟨ v, v ⟩ & \geq 0, and & (positivity) \\ ⟨ v, v ⟩ & = 0 if and only if v = 0 & (definiteness) . \end{aligned}$

🔗

An inner product space is a pair

(V, ⟨, ⟩),

where

V

is a vector space, and

⟨, ⟩

is a choice of inner product on

V .

🔗

Remark 5.1.2. Inner products of linear combinations.

🔗

We will have many opportunities to “expand out” an inner product of two linear combinations of vectors. Using axioms (i) and (ii) in series, this process resembles the procedure for multiplying two polynomials. For example, we have

\begin{aligned} ⟨ c v + d w, e v + f w ⟩ & = c ⟨ v, e v + f w ⟩ + d ⟨ w, e v + f w ⟩ & (5.1.1, (ii)) \\ = c e ⟨ v, v ⟩ + c f ⟨ v, w ⟩ + d e ⟨ w, v ⟩ + d f ⟨ w, w ⟩ & (5.1.1, (ii)) \\ = c e ⟨ v, v ⟩ + (c f + d e) ⟨ v, w ⟩ + d f ⟨ w, w ⟩ & (5.1.1, (i)) . \end{aligned}

🔗

Note how in the last step we are able to group the “cross terms”,

⟨ v, w ⟩ = ⟨ w, v ⟩

using the symmetry axiom.

🔗

More generally, given linear combinations

\begin{aligned} v & = c_{1} v_{1} + c_{2} v_{2} + \dots + c_{n} v_{n} = \sum_{i = 1}^{n} c_{i} v_{i} \\ w & = d_{1} v_{1} + d_{2} v_{2} + \dots + d_{n} v_{n} = \sum_{i = 1}^{n} d_{i} v_{i}, \end{aligned}

🔗

the same reasoning shows that

\begin{aligned} ⟨ v, w ⟩ & = c_{1} d_{1} ⟨ v_{1}, v_{1} ⟩ + c_{2} d_{2} ⟨ v_{2}, v_{2} ⟩ + \dots + c_{n} d_{n} ⟨ v_{n}, v_{n} ⟩ + \underset{cross terms}{\underset{⏟}{(c_{1} d_{2} + c_{2} d_{1}) ⟨ v_{1}, v_{2} ⟩ + \dots}} \\ = \sum_{i = 1}^{n} c_{i} d_{i} ⟨ v_{i}, v_{i} ⟩ + \underset{cross terms}{\underset{⏟}{\sum_{1 \leq i < j \leq n} (c_{i} d_{j} + c_{j} d_{i}) ⟨ v_{i}, v_{j} ⟩}} . \end{aligned}

🔗

In particular, we have

⟨ v, v ⟩ = \sum_{i = 1}^{n} c_{i}^{2} ⟨ v_{i}, v_{i} ⟩ + \sum_{1 \leq i < j \leq n} 2 c_{i} c_{j} ⟨ v_{i}, v_{j} ⟩ .

🔗

We now present a series of important examples of inner products defined on our various inner product spaces. Each is presented as a theorem, as we must prove that the proposed operation satisfies the axios of an inner product. The first example, the weighted dot product is itself a vast generalization of the familiar dot product operations defined on

R^{2}

and

R^{3} .

🔗

Theorem 5.1.3. Weighted dot product.

🔗

Let

V = R^{n} .

Let

k_{1}, k_{2}, \dots, k_{n}

be any list of real numbers. Define an operation on

R^{n}

as follows: given

x = (x_{1}, x_{2}, \dots, x_{n}), y = (y_{1}, y_{2}, \dots, y_{n}) \in R^{n},

let

⟨ x, y ⟩ = k_{1} x_{1} y_{1} + k_{2} x_{2} y_{2} + \dots k_{n} x_{n} y_{n} = \sum_{i = 1}^{n} k_{i} x_{i} y_{i} .

🔗

This operation is an inner product if and only if

k_{i} > 0

for all

i .

🔗

We call this inner product a weighted dot product on

R^{n},

or more specifically, the dot product with weights

k_{1}, k_{2}, \dots, k_{n}

. In the special case where

k_{i} = 1

for all

i

we call this the (standard) dot product and write

x \cdot y

instead of

⟨ x, y ⟩ .

🔗

Proof.

First we show that axioms (i) and (ii) are satsified for any choice of

k_{i} .

Let

K = [\begin{array}{rrrr} k_{1} & 0 & \dots & 0 \\ 0 & k_{2} & 0 & \dots \\ ⋮ \\ 0 & \dots & 0 & k_{n} \end{array}],

the diagonal matrix whose

i

-th diagonal entry is

k_{i} .

Then for all

x = (x_{1}, x_{2}, \dots, x_{n}), y = (y_{1}, y_{2}, \dots, y_{n}) \in R^{n}

we have

⟨ x, y ⟩ = x^{T} K y = [\begin{array}{cccc} x_{1} & x_{2} & \dots & x_{n} \end{array}] K [\begin{matrix} y_{1} \\ y_{2} \\ ⋮ \\ y_{n} \end{matrix}] .

Here we treat

x, y

as column vectors, and we treat the resulting

1 \times 1

matrix

x^{T} K y

as a scalar. Axioms (i)-(ii) now follow from various matrix properties. For linearity, for example, we have

\begin{aligned} ⟨ c x_{1} + d x_{2}, y ⟩ & = (c x_{1} + d x_{2})^{T} K y \\ = (c x_{1}^{T} + d x_{2}^{T}) K y & (2.2.11) \\ = c x_{1}^{T} K y + d x_{2}^{T} K y \\ = c ⟨ x_{1}, y ⟩ + d ⟨ x_{2}, y ⟩ . \end{aligned}

Symmetry requires a little more trickery:

\begin{aligned} ⟨ y, x ⟩ & = y^{T} K x \\ = y^{T} K^{T} x & (K^{T} = K) \\ = (x^{T} K y)^{T} & (Theorem 2.2.11) \\ = x^{T} K y \\ = ⟨ x, y ⟩ . \end{aligned}

Note that

(x^{T} K y)^{T} = x^{T} K y

x^{T} K y

is just a

1 \times 1

matrix.

Lastly, we show that axiom (iii) is satisfied if and only if

k_{i} > 0

for all

i .

To this end consider the formula

⟨ x, x ⟩ = k_{1} x_{1}^{2} + k_{2} x_{2}^{2} + \dots k_{n} x_{n}^{2} .

k_{i} > 0,

then since

x_{i}^{2} \geq 0

for all

i,

we have

⟨ x, x ⟩ \geq 0

for any

x,

and

⟨ x, x ⟩ = 0

if and only if

x_{i} = 0

for all

i

if and only if

x = 0 .

For the other direction suppose

k_{i} \leq 0

for some

i .

Let

x = e_{i},

the

i

-th element of the standard basis of

R^{n} .

Then

⟨ x, x ⟩ = k_{i} \leq 0 :

a counterexample to the definiteness property of axiom (iii).

🔗

Definition 5.1.4. (Weighted) Euclidean space.

🔗

Fix a positive integer

n .

Euclidean

n

-space is the inner product space with underlying vector space

R^{n}

and inner product given by the dot product. More generally, a weighted Eulidean space is an inner product space of the form

(R^{n}, ⟨, ⟩),

where

⟨, ⟩

is a weighted dot product.

🔗

Example 5.1.5. Dot product on $R^{4}$ .

🔗

Let

x = (- 1, 2, 0, 1), y = (1, 2, 1, 1) .

Then

x \cdot y = - 1 + 4 + 0 + 1 = 4,

🔗

and

x \cdot x = 1 + 4 + 0 + 1 = 6 .

🔗

Example 5.1.6. Weighted dot product.

🔗

The dot product with weights

2, 1, 3

R^{3}

is defined as

⟨ x, y ⟩ = 2 x_{1} y_{1} + x_{2} y_{2} + 3 x_{3} y_{3} .

🔗

Let

x = (- 1, - 1, - 1)

and

y = (1, 0, 1) .

We have

⟨ x, y ⟩ = 2 (- 1) + 0 - 3 = - 5,

🔗

and

⟨ x, x ⟩ = 2 (- 1)^{2} + 1 (- 1)^{2} + 3 (- 1)^{2} = 2 + 1 + 3 = 6 .

🔗

Example 5.1.7. Why the weights must be positive.

🔗

Consider the operation on

R^{2}

defined as

⟨ x, y ⟩ = (- 1) x_{1} y_{1} + 2 x_{2} y_{2}

🔗

where

x = (x_{1}, x_{2}), y = (y_{1}, y_{2}) .

This operation satisfies axioms (i) and (ii) of Definition 5.1.1. (See proof of Theorem 5.1.3.) However, it fails both the positivity and definiteness properties of axiom (iii):

\begin{aligned} ⟨ (3, 1), (3, 1) ⟩ & = - 9 + 2 = - 7 < 0 \\ ⟨ (1, 1 / \sqrt{2}), (1, 1 / \sqrt{2}) ⟩ & = - 1 + 2 / 2 = 0 . \end{aligned}

🔗

Remark 5.1.8.

🔗

It is worth highlighting the observation in the proof of Theorem 5.1.3 that a dot product with weights

k_{1}, k_{2}, \dots, k_{n}

can be expressed as a matrix product:

⟨ x, y ⟩ = \sum_{i = 1}^{n} k_{i} x_{i} y_{i} = x^{T} K y,

🔗

where

K

is the diagonal

n \times n

matrix whose

i

-th diagonal entry is

k_{i} .

Here

x, y

are treated as column vectors, and we identify the resulting

1 \times 1

matrix

x^{T} K y

with a scalar.

🔗

In particular for the standard dot product this matrix formula reduces to

x \cdot y = x^{T} I y = x^{T} y .

🔗

Conversely, the dot product gives another way to formulate general matrix multiplication. as the next theorem articulates.

🔗

Theorem 5.1.9. Dot product and matrix multiplication.

🔗

Given $n$ -tuples $x = (x_{1}, x_{2}, \dots, x_{n})$ and $y = (y_{1}, y_{2}, \dots, y_{n}),$ we have

$\begin{aligned} x \cdot y & = [\begin{array}{c} x_{1} & x_{2} & \dots & x_{n} \end{array}] [\begin{array}{c} y_{1} \\ y_{2} \\ ⋮ \\ y_{n} \end{array}] \\ = x^{T} y, \end{aligned}$

where in the last equality $x$ and $y$ are treated as column vectors.
🔗

Let $A$ be an $m \times n$ matrix, and let $B$ be a $n \times r$ matrix. Let $r_{i} \in R^{n}$ be the $i$ -th row of $A,$ and let $c_{j} \in R^{n}$ be the $j$ -th column of $B .$ For all $1 \leq i \leq m, 1 \leq j \leq n,$ we have

$(A B)_{i j} = r_{i} \cdot c_{j},$

where $r_{i}$ and $c_{j}$ are considered as $n$ -tuples. In other words, the $i j$ -th entry of $A B$ is the dot product of the $i$ -th row of $A$ and the $j$ -th column of $B .$

🔗

Proof.

See Remark 5.1.8 and the proof of Theorem 5.1.3.

Let

A = [a_{i j}]_{m \times n}

and

B = [b_{i j}]_{n \times r} .

Then

(A B)_{i j} = \sum_{k = 1}^{n} a_{i k} b_{k j} = r_{i} \cdot c_{j},

since

r_{i} = (a_{i 1}, a_{i 2}, \dots, a_{i n})

and

c_{j} = (b_{j 1}, b_{2 j}, \dots, b_{n j}) .

🔗

Next we introduce an important family of inner products defined on polynomials spaces called evaluation inner products. These are useful when we wish to compare polynomials by how they behave at a specified list of inputs.

🔗

Theorem 5.1.10. Evaluation inner products on $P_{n}$ .

🔗

Let

V = P_{n},

and let

c_{0}, c_{1}, \dots, c_{n}

be any list of

n + 1

distinct real numbers. For any

p (x), q (x) \in P_{n}

define

⟨ p (x), q (x) ⟩ = p (c_{0}) q (c_{0}) + p (c_{1}) q (c_{1}) + \dots + p (c_{n}) q (c_{n}) .

🔗

This defines an inner product on

P_{n}

called an evaluation inner product, or more precisely, evaluation at the inputs

c_{0}, c_{1}, \dots, c_{n}

🔗

Proof.

That axioms (i)-(ii) are satisfied is left as an exercise. For axiom (iii), note that

⟨ p (x), p (x) ⟩ = p (c_{0})^{2} + p (c_{1})^{2} + \dots + p (c_{n})^{2} \geq 0,

and we have equality if and only if

p (c_{0}) = p (c_{1}) = \dots p (c_{n}) = 0 .

Since a nonzero polynomial of degree

n

or less has at most

n

distinct roots, we conclude that

p (x) = 0,

the zero polynomial.

🔗

Example 5.1.11. Evaluation at $- 1, 0, 1$ .

🔗

Let

V = P_{2},

and let

⟨ p (x), q (x) ⟩

be the evaluation at

- 1, 0, 1

inner product. Compute

⟨ x^{2} - 1, x^{2} + 2 x + 1 ⟩

and

⟨ x^{2} - 1, x^{2} - 1 ⟩ .

Solution.

Let

p (x) = x^{2} - 1,

q (x) = x^{2} + 2 x + 1 .

We have

⟨ p (x), q (x) ⟩ = p (- 1) q (- 1) + p (0) q (0) + p (1) q (1) = 0 + (- 1) 1 + 0 = - 1

and

⟨ p (x), p (x) ⟩ = p (- 1)^{2} + p (0)^{2} + p (1)^{2} = 0 + (- 1)^{2} + 0 = 1 .

🔗

Our last example defines an integral inner product on the space

C ([a, b])

of continuous functions on an interval

[a, b] .

This inner product plays an important role in Fourier analysis, which studies the approximation of arbibitrary continuous functions with linear combinations of certain trigonometric funtions.

🔗

Theorem 5.1.12. Integral inner product.

🔗

Fix an interval

[a, b],

and let

V = C ([a, b]),

the space of all continuous functions on

[a, b] .

For any

f, g \in C ([a, b])

define

⟨ f, g ⟩ = \int_{a}^{b} f (x) g (x) d x .

🔗

This defines an inner product on

C ([a, b])

called the integral inner product.

🔗

Proof.

First observe that the integral defining the inner product always exists since the product

f g

is a continuous function on the closed interval

[a, b] .

Axioms (i)-(ii) follow directly from the definition and various properties of the integral. This is left as an exercise. As for (iii), we have

⟨ f, f ⟩ = \int_{a}^{b} f^{2} (x) d x \geq 0,

since

f^{2} (x) \geq 0

for all

x \in [a, b] .

(This is a property of integration.) Furthermore, since

f^{2}

is continuous and

f^{2} (x) \geq 0,

we have

⟨ f, f ⟩ = \int_{a}^{b} f^{2} (x) d x = 0

if and only if

f^{2} (x) = 0

for all

x \in [a, b]

(a property of integrals of continuous functions) if and only if

f (x) = 0

for all

x \in [a, b]

if and only if

f = 0,

the zero function.

🔗

Example 5.1.13. Integral inner product.

🔗

Let

V = C ([0, 1]),

equipped with integral inner product. Let

f (x) = x,

g (x) = e^{x} .

Compute

⟨ f, g ⟩

and

⟨ f, f ⟩ .

Solution.

We have

⟨ f, g ⟩ = \int_{0}^{1} x e^{x} d x = (x e^{x} |_{0}^{1} - \int_{0}^{1} e^{x} d x) = e - (e - 1) = 1

and

⟨ f, f ⟩ = \int_{0}^{1} x^{2} d x = \frac{1}{3} .

🔗

Subsection 5.1.2 Norm and distance

🔗

As mentioned above, once an inner product is established, we can define further notions like norm (or length), distance, and angle in terms of the given inner product. When the inner product in question is the standard dot product on

R^{2}

R^{3},

then these are precisely the familiar notions you may have met in multivariable calculus. Things get really interesting when we treat a more exotic inner product space. For example, consider

V = C ([a, b]) :

the integral inner product on

V

(Theorem 5.1.12) gives rise to useful notions of the length of a function

f \in C ([a, b]),

as well as the distance or angle between two functions

f, g \in C ([a, b]) .

🔗

Definition 5.1.14. Norm (or length) of a vector.

🔗

Let

(V, ⟨, ⟩)

be an inner product space. Given

v \in V

we define its norm (or length), denoted

‖ v ‖,

‖ v ‖ = \sqrt{⟨ v, v ⟩} .

🔗

A unit vector is a vector

v

of length one: i.e., a vector

v

satisfying

‖ v ‖ = 1 .

🔗

Example 5.1.15. Norm with respect to dot product.

🔗

Consider

V = R^{4}

with the standard dot product. Compute

‖ (1, - 1, - 2, 1) ‖ .

Solution.

We have

\begin{aligned} ‖ (1, - 1, - 2, 1) ‖ & = \sqrt{(1, - 1, - 2, 1) \cdot (1, - 1, - 2, 1)} \\ = \sqrt{1 + 1 + 4 + 1} \\ = \sqrt{7} . \end{aligned}

🔗

Example 5.1.16. Norm with respect to weighted dot product.

🔗

Consider

V = R^{3}

equipped with the dot product with weights

1, 2, 3 .

Compute

‖ (3, 1, - 2) ‖ .

Solution.

We have

\begin{aligned} ‖ (3, 1, - 2) ‖ & = \sqrt{⟨ (3, 1, - 2), (3, 1, - 2) ⟩} \\ = \sqrt{1 (3^{2}) + 2 (1^{2}) + 3 ((- 2)^{2})} \\ = \sqrt{23} . \end{aligned}

🔗

Example 5.1.17. Norm with respect to integral inner product.

🔗

Consider

V = C ([0, 1])

equipped with the integral inner product. Compute

‖ f ‖,

where

f (x) = e^{x}

Solution.

We have

\begin{aligned} ‖ f ‖ & = \sqrt{⟨ f, f ⟩} \\ = \sqrt{\int_{0}^{1} (e^{x})^{2} d x} \\ = \sqrt{\int_{0}^{1} e^{2 x} d x} \\ = \sqrt{(e^{2} - 1) / 2} . \end{aligned}

🔗

Remark 5.1.18. Unit vectors.

🔗

Given any

v \neq 0 \in V,

the vector

u = \frac{1}{‖ v ‖} v

is a unit vector. To verify this, let

c = ‖ v ‖

and compute

\begin{aligned} ‖ u ‖ & = \sqrt{⟨ \frac{1}{c} v, \frac{1}{c} v ⟩} \\ = \sqrt{\frac{1}{c^{2}} ⟨ v, v ⟩} & (Definition 5.1.1, (ii)) \\ = \frac{1}{{| c |}_{}} \sqrt{⟨ v, v ⟩} \\ = \frac{1}{c} \sqrt{⟨ v, v ⟩} & (c = ‖ v ‖ \geq 0) \\ = \frac{‖ v ‖}{‖ v ‖} = 1 . \end{aligned}

🔗

Example 5.1.19. Unit vectors.

🔗

For each inner product space

(V, ⟨, ⟩)

and

v \in V

compute the associated unit vector

u = \frac{1}{‖ v ‖} v

🔗
$V = R^{4}$ with dot product, $v = (1, - 1, 2, 1)$
🔗
$V = R^{3}$ with dot product with weights $1, 2, 3,$ $v = (3, 1, - 2)$
🔗
$V = C ([0, 1])$ with integral inner product, $v = e^{x}$

Solution.

The norms of the vectors in each case were computed in Example 5.1.15–5.1.17. We simply scale to compute the corresponding unit vectors.

$u = \frac{1}{\sqrt{7}} (1, - 1, 2, 1) = (\sqrt{7} / 7, - \sqrt{7} / 7, 2 \sqrt{7} / 7, \sqrt{7} / 7)$
$u = \frac{1}{\sqrt{23}} (3, 1, - 2) = (3 \sqrt{23} / 23, \sqrt{23} / 23, - 2 \sqrt{23} / 23)$
$u = \frac{1}{\sqrt{(e^{2} - 1) / 2}} e^{x} = \frac{\sqrt{2 (e^{2} - 1)}}{e^{2} - 1} e^{x}$

🔗

Next, we define the distance between two vectors in an inner product space as the length of their vector difference.

🔗

Definition 5.1.20. Distance between vectors.

🔗

Let

(V, ⟨, ⟩)

be an inner product space. The distance between

v, w \in V

, denoted

d (v, w),

is defined as

d (v, w) = ‖ v - w ‖ = \sqrt{⟨ v - w, v - w ⟩} .

🔗

Example 5.1.21.

🔗

For each inner product space

V,

compute the distance between the given vectors.

🔗
$V = R^{3}$ with the dot product, $x = (x_{1}, x_{2}, x_{3}),$ $y = (y_{1}, y_{2}, y_{3})$
🔗
$V = P_{2}$ with the evaluation at $1, - 1, 0$ inner product, $p_{1} (x) = x^{2} + 1,$ $p_{2} (x) = x + 2$
🔗
$V = C ([- π, π])$ with the integral inner product, $f (x) = \sin x + x,$ $g (x) = x$

Solution.

We have

$\begin{aligned} d (x, y) & = ‖ x - y ‖ \\ = \sqrt{⟨ x - y, x - y ⟩} \\ = \sqrt{(x_{1} - y_{1}, x_{2} - y_{2}, x_{3} - y_{3}) \cdot (x_{1} - y_{1}, x_{2} - y_{2}, x_{3} - y_{3})} \\ = \sqrt{(x_{1} - y_{1})^{2} + (x_{2} - y_{2})^{2} + (x_{3} - y_{3})^{2}} . \end{aligned}$
We have

$\begin{aligned} d (p_{1}, p_{2}) & = ‖ p_{1} - p_{2} ‖ \\ = \sqrt{⟨ p_{1} - p_{2}, p_{1} - p_{2} ⟩} \\ = \sqrt{(p_{1} (1) - p_{2} (1))^{2} + (p_{1} (- 1) - p_{2} (- 1))^{2} + (p_{1} (0) - p_{2} (0))^{2}} \\ = \sqrt{(- 1)^{2} + (1)^{2} + (- 1)^{2}} = \sqrt{3} \end{aligned}$
We have

$\begin{aligned} d (f, g) & = ‖ f - g ‖ \\ = \sqrt{⟨ f - g, f - g ⟩} \\ = \sqrt{⟨ \sin x, \sin x ⟩} \\ = \sqrt{\int_{- p i}^{p i} \sin^{2} x d x} \\ = \sqrt{π} \end{aligned}$

🔗

Theorem 5.1.22. Basic properties of norm and distance.

🔗

Let

(V, ⟨, ⟩)

be an inner product space.

🔗

For all $v \in V$ we have

$‖ v ‖ \geq 0,$

and equality holds if and only if $v = 0 .$
🔗

For all $c \in R$ and $v \in V$ we have

$‖ c v ‖ = {| c |}_{} ‖ v ‖ .$
🔗

For all $v, w \in V$ we have

$d (v, w) = d (w, v) \geq 0,$

and equality holds if and only if $v = w .$

🔗

Proof.

We prove (2) and leave the rest as an exercise (Exercise 5.1.5.16).

Given

c \in R

and

v \in V

we have

\begin{aligned} ‖ c v ‖ & = \sqrt{⟨ c v, c v ⟩} \\ = \sqrt{c^{2} ⟨ v, v ⟩} & (Definition 5.1.1) \\ = {| c |}_{} \sqrt{⟨ v, v ⟩} & (\sqrt{c^{2}} = {| c |}_{}) . \end{aligned}

🔗

Subsection 5.1.3 Cauchy-Schwarz inequality, triangle inequalities, and angles between vectors

🔗

The famous Cauchy-Schwarz inequality has a knack of cropping up all over the world of science: from properties of covariance in statistics, to the Heisenberg uncertainty principle of quantum mechanics. More directly pertinent to our discussion, the Cauchy-Schwarz inequality implies the triangle inequalities (5.1.24) and ensures that our notion of the angle between two nonzero vectors (Definition 5.1.25) is well-defined.

🔗

Theorem 5.1.23. Cauchy-Schwarz inequality.

🔗

Let

(V, ⟨, ⟩)

be an inner product space. For all

v, w \in V

we have

{| ⟨ v, w ⟩ |}_{} \leq ‖ v ‖ ‖ w ‖,

🔗

and equality holds if and only if

v = c w

for some

c \in R .

🔗

Proof.

Fix vectors

v

and

w .

For any

t \in R

we have by positivity

0 \leq ⟨ t v - w, t v - w ⟩ = ⟨ v, v ⟩ t^{2} - 2 ⟨ v, w ⟩ t + ⟨ w, w ⟩ = a t^{2} - 2 b t + c,

where

\begin{matrix} (5.1.1) & a = ⟨ v, v ⟩, b = ⟨ v, w ⟩, c = ⟨ w, w ⟩ = {‖ w ‖}^{2} . \end{matrix}

Since

a t^{2} - 2 b t + c \geq 0

for all

t \in R

the quadratic polynomial

p (t) = a t^{2} - 2 b t + c

has at most one root. Using the quadratic formula we conclude that we must have

4 b^{2} - 4 a c \leq 0,

since otherwise

p (t)

would have two distinct roots. It follows that

4 ⟨ v, w ⟩^{2} - 4 {‖ v ‖}^{2} {‖ w ‖}^{2} \leq 0,

or equivalently

⟨ v, w ⟩^{2} \leq {‖ v ‖}^{2} {‖ w ‖}^{2} .

Taking square-roots yields the desired inequality.

The same reasoning shows that the Cauchy-Schwarz inequality is an actual equality if and only if

p (t) = 0

for some

t

if and only if

0 = ⟨ t v - w, t v - w ⟩

if and only if

v = t w

for some

t

(by positivity).

🔗

The following triangle inequalities are more or less direct consequences of the Cauchy-Schwarz inequality.

🔗

Theorem 5.1.24. Triangle Inequalities.

🔗

Let

(V, ⟨, ⟩)

be an inner product space.

🔗

For all $v, w \in V$ we have

$‖ v + w ‖ \leq ‖ v ‖ + ‖ w ‖ .$
🔗

For all $v, w, u \in V$ we have

$d (v, w) \leq d (v, u) + d (u, w)$

🔗

Proof.

This is an elementary exercise of unpacking the definitions of norm and distance in terms of the inner product, and then applying the Cauchy-Schwarz inequality appropriately. The proof is left as an exercise.

🔗

Let

(V, ⟨, ⟩)

be an inner product space. For any nonzero vectors

v, w,

the Cauchy-Schwarz inequality tells us that

{| ⟨ v, w ⟩ |}_{} \leq ‖ v ‖ ‖ w ‖,

🔗

or equivalently,

- 1 \leq \frac{⟨ v, w ⟩}{‖ v ‖ ‖ w ‖} \leq 1 .

🔗

It follows that there is a unique real number

θ \in [0, π]

satisfying

\cos θ = \frac{⟨ v, w ⟩}{‖ v ‖ ‖ w ‖} .

🔗

We call

θ

the angle between $v$ and $w$ .

🔗

Definition 5.1.25. Angle between vectors.

🔗

Let

(V, ⟨, ⟩)

be an inner product space. Given nonzero vectors

v, w \in V,

the angle between

v

and

w

is defined to be the unique

θ \in [0, π]

satisfying

\cos θ = \frac{⟨ v, w ⟩}{‖ v ‖ ‖ w ‖} .

🔗

Equivalently, we have

θ = \arccos (\frac{⟨ v, w ⟩}{‖ v ‖ ‖ w ‖}) .

🔗

Remark 5.1.26.

🔗

Our definition of the angle between two vectors may remind you of the dot product angle formula for vectors in

R^{3} :

\begin{matrix} (5.1.2) & \cos θ = \frac{x \cdot y}{‖ x ‖ ‖ y ‖} . \end{matrix}

🔗

Interestingly, whereas (5.1.2) is typically treated as a theorem, derived from properties of the dot product and the law of cosines, in a general inner product space the equation

\cos θ = \frac{⟨ v, w ⟩}{‖ v ‖ ‖ w ‖}

🔗

is understood as the definition of the angle between two vectors.

🔗

Example 5.1.27.

🔗

Consider

R^{2}

along with the dot product. Verify that our definition of the angle

θ

between

(1, 1)

and

(1, 0)

is consistent with our planar geometry notion of angle.

Solution.

According to Definition 5.1.25,

θ

is the unique element of

[0, π]

satisfying

\cos θ = \frac{(1, 1) \cdot (1, 0)}{‖ (1, 1) ‖ ‖ (1, 0) ‖} = \frac{1}{\sqrt{2}} = \frac{\sqrt{2}}{2} .

We recognize

θ

as the familiar angle

π / 4,

as expected.

🔗

Example 5.1.28.

🔗

Consider

R^{2}

with the weighted dot product

⟨ (x_{1}, x_{2}), (y_{1}, y_{2}) ⟩ = 2 x_{1} x_{2} + y_{1} y_{2}

🔗

Compute the angle

θ

between

(1, 1)

and

(0, 0)

with respect to this inner product

Solution.

First compute

\begin{aligned} ⟨ (1, 1), (1, 0) ⟩ & = 2 (1) + 1 (0) = 2 \\ ‖ (1, 1) ‖ & = \sqrt{⟨ (1, 1), (1, 1) ⟩} \\ = \sqrt{2 + 1} = \sqrt{3} \\ ‖ (1, 0) ‖ & = \sqrt{⟨ (1, 0), (1, 0) ⟩} \\ = \sqrt{2} \end{aligned}

By definition

θ

is the unique value in

[0, π]

satisfying

\cos θ = \frac{⟨ (1, 1), (1, 0) ⟩}{‖ (1, 1) ‖ ‖ (1, 0) ‖} = \frac{2}{\sqrt{3} \sqrt{2}} = \frac{\sqrt{6}}{3} .

We see that

θ

is not one of our familiar angles from the unit circle (e.g.,

π / 6, π / 4,

etc.) and so express

θ

in terms of the

\arccos

function:

θ = \arccos (\sqrt{6} / 3) \approx {35.3}^{\circ} .

🔗

Example 5.1.29.

🔗

Consider

V = C ([- π, π])

with the integral inner product. Compute the angle

θ

between

f (x) = \sin x

and

g (x) = \sin 2 x

with respect to this inner product.

Solution.

First compute

\begin{aligned} ⟨ f, g ⟩ & = \int_{- π}^{π} \sin x \sin 2 x d x \\ = \frac{1}{2} \int_{- π}^{π} \cos (x - 2 x) - \cos (x + 2 x) d x & (trig. identity) \\ = \frac{1}{2} \int_{- π}^{π} \cos (- x) - \cos 3 x d x \\ = 0 \end{aligned}

It follows that

\cos θ = \frac{⟨ f, g ⟩}{‖ f ‖ ‖ g ‖} = 0,

and hence that

θ = π / 2 .

🔗

Subsection 5.1.4 Choosing your inner product

🔗

Why, given a fixed vector space

V,

would we prefer one inner product definition to another? One way of understanding a particular choice of inner product is to ask what its corresponding notion of distance measures.

🔗

Example 5.1.30. Weighted dot product distance.

🔗

Consider

R^{n}

with a choice of weighted dot product

⟨ (x_{1}, x_{2}, \dots, x_{n}), (y_{1}, y_{2}, \dots, y_{n}) ⟩ = k_{1} x_{1} y_{1} + k_{2} x_{2} y_{2} + \dots + k_{n} x_{n} y_{n},

🔗

where

k_{1}, k_{2}, \dots, k_{n}

are fixed positive constants. With respect to this inner product the distance between two vectors

x = (x_{1}, x_{2}, \dots, x_{n})

and

y = (y_{1}, y_{2}, \dots, y_{n})

d (x, y) = ‖ x - y ‖ = \sqrt{k_{1} (x_{1} - y_{1})^{2} + k_{2} (x_{2} - y_{2})^{2} + \dots + k_{n} (x_{n} - y_{n})^{2}} .

🔗

Thus

d (x, y)

is an aggregate measure of the difference between the corresponding entries of

x

and

y,

as weighted by our choice of the constants

k_{i} .

🔗

Imagine that each element of

x \in R^{n}

is a data point collected by measuring

n

different properties of a sample

s

: i.e.,

x_{i}

is the measured value of property

P_{i}

s

for all

1 \leq i \leq n .

Given samples

s

and

s^{'}

with corresponding measurement vectors

x

and

y,

the weighted distance

d (x, y)

is then a quantitative way of saying how “close” the two samples are to one another. The choice of weights

k_{i}

allows us to adjust the relative influence of a given property

P_{i}

in determining this closeness. For example, the standard dot product (

k_{i} = 1

for all

i

) yields a notion of distance that gives each property equal standing.

🔗

Example 5.1.31. Evaluation inner product distance.

🔗

Consider

P_{n}

with the evaluation inner product at a fixed choice of inputs

x = c_{0}, c_{1}, \dots, c_{n} .

Given two polynomials

p (x), q (x),

the distance between them with respect to this inner product is

‖ p (x) - q (x) ‖ = \sqrt{(p (c_{0}) - q (c_{0}))^{2} + (p (c_{1}) - q (c_{1}))^{2} + \dots + (p (c_{n}) - q (c_{n}))^{2}} .

🔗

We see that with respect to this inner product, the distance between two polynomials is a measure of how much their values at the inputs

x = c_{0}, c_{1}, \dots, c_{n}

differ. This inner product may be useful if you are interested in how a polynomial behaves at this finite list of inputs.

🔗

Example 5.1.32. Integral inner product and distance.

🔗

Take

C [a, b]

with the standard inner product

⟨ f, g ⟩ = \int_{a}^{b} f (x) g (x) d x .

Here the distance between two functions is defined as

‖ f - g ‖ = \sqrt{\int_{a}^{b} (f (x) - g (x))^{2} d x},

🔗

which we can think of as an aggregate measure of the difference of values

f (x) - g (x)

for all

x \in [a, b] .

Thus

d (f, g)

is a global measure of the similarity between

f

and

g

that takes into account their values over the entire interval

[a, b] .

🔗

Find the norm of

\vec{x}

and the unit vector

\vec{u}

in the direction of

\vec{x}

\vec{x} = [\begin{matrix} - 5 \\ 5 \\ 2 \\ 5 \end{matrix}] .

🔗

‖ \vec{x} ‖ =

🔗

\vec{u} =

(4 × 1 array)

Answer.

\sqrt{79}

🔗

2.

🔗

Find the angle

α

between the vectors

[\begin{matrix} - 4 \\ - 4 \\ - 3 \end{matrix}] and [\begin{matrix} 1 \\ 2 \\ 3 \end{matrix}] .

🔗

α =

Answer.

\cos^{- 1} (- 0.876523)

🔗

3.

🔗

f (x)

and

g (x)

are arbitrary polynomials of degree at most 2, then the mapping

⟨ f, g ⟩ = f (- 3) g (- 3) + f (0) g (0) + f (1) g (1)

🔗

defines an inner product in

P_{2} .

Use this inner product to find

⟨ f, g ⟩,

‖ f ‖,

‖ g ‖,

and the angle

α_{f, g}

between

f (x)

and

g (x)

for

f (x) = 3 x^{2} + 4 x - 5 and g (x) = 2 x^{2} - 2 x + 7.

🔗

⟨ f, g ⟩ =

🔗

‖ f ‖ =

🔗

‖ g ‖ =

🔗

α_{f, g} =

Answer 1.

289

Answer 2.

11.3578

Answer 3.

32.5423

Answer 4.

0.673078

🔗

4.

🔗

A

and

B

are arbitrary real

m \times n

matrices, then the mapping

⟨ A, B ⟩ = t r a c e (A^{T} B)

🔗

defines an inner product in

R^{m \times n} .

Use this inner product to find

⟨ A, B ⟩,

the norms

‖ A ‖

and

‖ B ‖,

and the angle

α_{A, B}

between

A

and

B

for

A = [\begin{array}{cc} - 1 & 3 \\ 3 & - 2 \\ - 2 & 3 \end{array}] and B = [\begin{array}{cc} - 1 & - 1 \\ - 3 & 1 \\ - 1 & 3 \end{array}] .

🔗

⟨ A, B ⟩ =

🔗

‖ A ‖ =

🔗

‖ B ‖ =

🔗

α_{A, B} =

Answer 1.

- 2

Answer 2.

6

Answer 3.

4.69042

Answer 4.

1.64192

🔗

5.

🔗

Use the inner product

⟨ f, g ⟩ = \int_{0}^{1} f (x) g (x) d x

🔗

in the vector space

C^{0} [0, 1]

of continuous functions on the domain

[0, 1]

to find

⟨ f, g ⟩,

‖ f ‖,

‖ g ‖,

and the angle

α_{f, g}

between

f (x)

and

g (x)

for

f (x) = - 5 x^{2} - 9 and g (x) = - 6 x - 10.

🔗

⟨ f, g ⟩ =

🔗

‖ f ‖ =

🔗

‖ g ‖ =

🔗

α_{f, g}

Answer 1.

141.167

Answer 2.

10.7703

Answer 3.

13.1149

Answer 4.

0.0346695

🔗

6.

🔗

For each of the following operations on

R^{2},

determine whether it defines an inner product on

R^{2} .

If it fails to be an inner product, identify which of the three inner product axioms (if any) it does satisfy, and provide explicit counterexamples for any axiom that fails.

🔗
$⟨ (x_{1}, x_{2}), (y_{1}, y_{2}) ⟩ = x_{1} y_{2} + x_{2} y_{1} .$
🔗
$⟨ (x_{1}, x_{2}), (y_{1}, y_{2}) ⟩ = 2 x_{1} y_{1} + x_{1} y_{2} + x_{2} y_{1} + 3 x_{2} y_{2} .$
🔗
$⟨ (x_{1}, x_{2}), (y_{1}, y_{2}) ⟩ = x_{1}^{2} y_{1}^{2} + x_{2}^{2} y_{2}^{2} .$

Hint.

The operation in (b) is an inner product. Use that fact that

⟨ x, y ⟩ = x^{T} [\begin{array}{cc} 2 & 1 \\ 1 & 3 \end{array}] y,

where we treat

x, y

as column vectors. This helps to prove axioms (i)-(ii). For axiom (iii), use either a “complete the square” or quadratic formula argument on the expression

⟨ x, x ⟩ = 2 x_{1}^{2} + 2 x_{1} x_{2} + 3 x_{2}^{2} .

🔗

7.

🔗

We work within the inner product space given by

V = P_{2}

together with the evaluation at 0, 1, 2 inner product.

🔗

Let

q (x) = x .

Give a parametric description of the set

W = {p (x) \in P_{2} : ⟨ p (x), q (x) ⟩ = 0} .

🔗

8.

🔗

We work in the inner product space given by

V = C ([- π, π])

together with the integral inner product.

🔗
Let $f (x) = \cos x, g (x) = \sin x .$ Compute $⟨ f, g ⟩$ and $‖ g ‖ .$
🔗
Show that if $f (x)$ is an odd function (i.e., $f (x) = - f (- x)$ for all $x$ ) and $g (x)$ is an even function ( $g (- x) = g (x)$ for all $x$ ), then $⟨ f, g ⟩ = 0 .$ Hint: use the area interpretation of the integral and properties of even/odd functions.

🔗

.

Compute the angle between the given vectors with respect to the given inner product. The exercises are designed to be done by hand: i.e., do not express your answer in terms of

\arccos .

🔗

9.

🔗

V = R^{4}

with the standard dot product;

v = (1, 1, 1, 1), w = (1, - 1, 1, 1)

🔗

10.

🔗

V = R^{2}

with the dot product with weights

1, 2;

v = (1, 0), w = (- 2, \sqrt{2})

🔗

11.

🔗

V = C ([0, 1])

with the integral inner product;

f (x) = 1, g (x) = x

🔗

12.

🔗

V = P_{2}

with evaluation at

- 1, 1

inner product;

p (x) = - \frac{1}{2} x + \frac{1}{2}, q (x) = 2 x

🔗

13.

🔗

Let

(V, ⟨, ⟩)

be an inner product space. Prove that

⟨ v, 0 ⟩ = 0

for all

v \in V .

🔗

14.

🔗

Let

v

and

w

be nonzero vectors of the inner product space

(V, ⟨, ⟩),

and let

θ

be the angle between them. Prove the following equivalence:

‖ v + w ‖ = ‖ v ‖ + ‖ w ‖ if and only if θ = 0 .

🔗

Your proof should be a chain of equivalences with each step justified.

Hint.

The equality is true if and only if it is true after squaring both sides. (Why?) Use the definition

{‖ v + w ‖}^{2} = ⟨ v + w, v + w ⟩

and expand the inner product.

🔗

15.

🔗

Let

(V, ⟨, ⟩)

be an inner product space. Suppose vectors

v, w \in V

satisfy

‖ v ‖ = 2

and

‖ w ‖ = 3 .

Using the Cauchy-Schwarz inequality (5.1.23) find the maximum and minimum possible values of

‖ v - w ‖,

and give explicit examples where those values occur.

🔗

16.

🔗

Prove statements (1) and (3) of Theorem 5.1.22.

🔗

17.

🔗

Prove each inequality below using the Cauchy-Schwarz inequality (5.1.23) applied to a judicious choice of inner product space, and possibly a judicious choice of vector in said inner product space.

🔗

For all $f, g \in C ([a, b])$

${(\int_{a}^{b} f (x) g (x) d x)}^{2} \leq \int_{a}^{b} f^{2} (x) d x \int_{a}^{b} g^{2} (x) d x .$
🔗

For all $(x_{1}, x_{2}, \dots, x_{n}) \in R^{n},$

$(x_{1} + x_{2} + \dots + x_{n}) \leq \sqrt{x_{1}^{2} + x_{2}^{2} + \dots + x_{n}^{2}} \sqrt{n} .$
🔗

For all $a, b, θ \in R$

$(a \cos θ + b \sin θ)^{2} \leq a^{2} + b^{2} .$

🔗

18. Isometries of inner product spaces.

🔗

Let

(V, ⟨, ⟩)

be an inner product space. An isometry of

V

is a function

f : V \to V

that preserves distance: i.e.,

d (f (v), f (w)) = d (v, w) for all v, w \in V .

🔗

In this exercise we will show that any isometry that maps

0

0

is a linear transformation. This is a very useful fact. For example, it implies the linearity of many geometric transformations we have considered: rotation about the origin in

R^{2},

reflection through a line in

R^{2},

etc..

🔗

In what follows assume that

f

is an isometry of

V

satisfying

f (0) = 0 .

🔗
Prove that $‖ f (v) ‖ = ‖ v ‖ :$ i.e., $f$ preserves norms.
🔗
Prove $⟨ f (v), f (w) ⟩ = ⟨ v, w ⟩ :$ i.e., $f$ preserves inner products. Hint: first prove that $⟨ v, w ⟩ = \frac{1}{2} ({‖ v ‖}^{2} + {‖ w ‖}^{2} - {‖ v - w ‖}^{2}) .$
🔗

To prove $f$ is linear it is enough to show $f (v + c w) = f (v) + c f (w)$ for all $v, w \in V,$ $c \in R .$ To do so, use the above parts to show that

$‖ f (v + c w) - (f (v) + c f (w)) ‖ = 0 .$

Section 5.1 Inner product spaces

Subsection 5.1.1 Inner products

Definition 5.1.1. Inner product.

Remark 5.1.2. Inner products of linear combinations.

Theorem 5.1.3. Weighted dot product.

Proof.

Definition 5.1.4. (Weighted) Euclidean space.

Example 5.1.5. Dot product on R4.

Example 5.1.6. Weighted dot product.

Example 5.1.7. Why the weights must be positive.

Remark 5.1.8.

Theorem 5.1.9. Dot product and matrix multiplication.

Proof.

Theorem 5.1.10. Evaluation inner products on Pn.

Proof.

Example 5.1.11. Evaluation at −1,0,1.

Theorem 5.1.12. Integral inner product.

Proof.

Example 5.1.13. Integral inner product.

Subsection 5.1.2 Norm and distance

Definition 5.1.14. Norm (or length) of a vector.

Example 5.1.15. Norm with respect to dot product.

Example 5.1.16. Norm with respect to weighted dot product.

Example 5.1.17. Norm with respect to integral inner product.

Remark 5.1.18. Unit vectors.

Example 5.1.19. Unit vectors.

Definition 5.1.20. Distance between vectors.

Example 5.1.21.

Theorem 5.1.22. Basic properties of norm and distance.

Proof.

Subsection 5.1.3 Cauchy-Schwarz inequality, triangle inequalities, and angles between vectors

Theorem 5.1.23. Cauchy-Schwarz inequality.

Proof.

Theorem 5.1.24. Triangle Inequalities.

Proof.

Definition 5.1.25. Angle between vectors.

Remark 5.1.26.

Example 5.1.27.

Example 5.1.28.

Example 5.1.29.

Subsection 5.1.4 Choosing your inner product

Example 5.1.30. Weighted dot product distance.

Example 5.1.31. Evaluation inner product distance.

Example 5.1.32. Integral inner product and distance.

Exercises 5.1.5 Exercises

WeBWork Exercises

1.

2.

3.

4.

5.

6.

7.

8.

.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18. Isometries of inner product spaces.

Example 5.1.5. Dot product on $R^{4}$ .

Theorem 5.1.10. Evaluation inner products on $P_{n}$ .

Example 5.1.11. Evaluation at $- 1, 0, 1$ .