Inner product structure of \R^n

Section 1.2 Inner product structure of \(\R^n\)

In this section we introduce another operation on \(\R^n\) called the dot product. As we will see, the dot product is an additional layer added to the vector space structure of \(\R^n\) that gives rise to a number of useful analytic tools. More generally, the dot product turns out to be just one example of what is called an inner product on a vector space. Inner products imbue vector spaces with valuable geometric content, a few of which are highlighted below.

Distance and angle.
A notion of distance and angle between two vectors can be defined relative to a given inner product. These provide a numeric measurement of how “close” (distance) or “closely oriented” (angle) two vectors in our space are.
🔗

🔗
Orthogonality.
Two vectors of an inner product space are orthogonal if their inner product is equal to zero. Orthogonality leads to a general notion of orthogonal projection, and is part of the definition of an orthogonal basis.
🔗

🔗

We will get into these geometric notions in detail later. For now we content ourselves simply with defining what an inner product is. We will also show that the dot product is just one example of infinitely many inner products that can be defined on \(\R^n\text{.}\) In particular, we will define a family of natural inner products on \(\R^n\) called weighted dot products, which play an important role in statistics and data science.

🔗

Definition 1.2.1. Dot product.

Given \(n\)-vectors \(\boldx=(x_1,x_2,\dots, x_n)\) and \(\boldy=(y_1,y_2,\dots, y_n)\) we define their dot product \(\boldx\cdot\boldy\) as

\begin{equation*} \boldx\cdot\boldy=\sum_{i=1}^nx_iy_i=x_1y_1+x_2y_2+\cdots +x_ny_n\text{.} \end{equation*}

The operation

\begin{align*} \R^n\times \R^n \amp\rightarrow \R^n \\ (\boldx,\boldy) \amp \mapsto \boldx\cdot \boldy \end{align*}

is called the dot product on \(\R^n\text{.}\)

🔗

The vector space \(\R^n\) together with the dot product is called Euclidean \(n\)-space

🔗

Example 1.2.2. Dot product on \(\R^4\).

Let \(\boldx=(-1,2,0,1), \boldy=(1,2,1,1)\text{.}\) Then

\begin{equation*} \boldx\cdot \boldy=-1+4+0+1=4\text{,} \end{equation*}

and

\begin{equation*} \boldx\cdot\boldx=1+4+0+1=6\text{.} \end{equation*}

🔗

As mentioned above, the dot product is just one example of a more general notion called an inner product, which is an additional operation defined on a vector space.

🔗

Definition 1.2.3. Inner product.

Let \(V\) be a vector space. An inner product on \(V\) is an operation that takes as input a pair of vectors \(\boldv, \boldw\in V\) and outputs a scalar \(\langle \boldv, \boldw \rangle \in \R\text{.}\) Using function notation:

\begin{align*} \langle \ , \rangle \colon \amp V\times V\rightarrow \R\\ (\boldv_1,\boldv_2)\amp \mapsto \langle \boldv_1,\boldv_2\rangle\text{.} \end{align*}

Furthermore, this operation must satisfy the following axioms.

Symmetry.

For all \(\boldv, \boldw\in V\) we have

\begin{equation*} \langle \boldv, \boldw \rangle =\langle \boldw, \boldv \rangle\text{.} \end{equation*}

🔗

🔗
Linearity.

For all \(\boldv, \boldw, \boldu\in V\) and \(c, d\in \R\) we have :

\begin{equation*} \langle c\boldv+d\boldw, \boldu \rangle =c \langle \boldv, \boldu \rangle +d \langle \boldw, \boldu \rangle\text{.} \end{equation*}

It follows by (i) (symmetry) that

\begin{equation*} \langle \boldu, c\boldv+d\boldw \rangle =c \langle \boldu, \boldv \rangle +d \langle \boldu, \boldw \rangle\text{.} \end{equation*}

🔗

🔗
Positive definiteness.

For all \(\boldv\in V\) we have

\begin{align*} \langle \boldv, \boldv \rangle \amp\geq 0,\text{ and} \amp (\text{positivity})\\ \langle \boldv, \boldv \rangle \amp=0 \text{ if and only if } \boldv=\boldzero \amp (\text{definiteness}) \text{.} \end{align*}

🔗

🔗

An inner product space is a pair \((V, \langle , \rangle )\text{,}\) where \(V\) is a vector space, and \(\langle , \rangle \) is a choice of inner product on \(V\text{.}\)

🔗

Although we will almost exclusively work with the dot product in this treatment of linear algebra, it is worth considering a natural family of inner products on \(\R^n\) that the dot product fits nicely into: namely, weighted dot products. These examples of inner products are especially important in data science.

🔗

Definition 1.2.4. Weighted dot product.

Let \(k_1, k_2, \dots , k_n\) be a list of positive real numbers: i.e., \(k_i> 0\) for all \(1\leq i\leq n\text{.}\) The weighted dot product with weights \((k_1,k_2,\dots, k_n)\) is the operation

\begin{align*} \R^n\times\R^n \amp \rightarrow \R\\ (\boldx,\boldy) \amp \mapsto \angvec{\boldx,\boldy} \end{align*}

defined as on \(n\)-vectors \(\boldx=(x_1,x_2,\dots, x_n)\) and \(\boldy=(y_1,y_2,\dots, y_n)\) as follows:

\begin{equation} \angvec{\boldx,\boldy}=\sum_{i=1}^nk_ix_iy_i=k_1x_1y_1+k_2x_2y_2+\cdots k_nx_ny_n\text{.}\tag{1.3} \end{equation}

When \(k_i=1\) for all \(1\leq i\leq n\text{,}\) the weighted dot product is equal to the dot product.

🔗

Example 1.2.5. Weighted dot product.

The dot product with weights \((2, 1, 3)\) on \(\R^3\) is defined as

\begin{equation*} \langle \boldx, \boldy \rangle= 2x_1y_1+x_2y_2+3x_3y_3\text{.} \end{equation*}

Let \(\boldx=(-1,-1,-1)\) and \(\boldy=(1,0,1)\text{.}\) We have

\begin{equation*} \langle \boldx, \boldy \rangle =2(-1)+0-3=-5\text{,} \end{equation*}

and

\begin{equation*} \langle \boldx, \boldx \rangle =2(-1)^2+1(-1)^2+3(-1)^2=2+1+3=6\text{.} \end{equation*}

🔗

We now show that any weighted dot product (including the dot product itself) is an inner product on \(\R^n\text{.}\)

🔗

Theorem 1.2.6. Weighted dot product.

Let \(k_1, k_2, \dots , k_n\) be a list of positive real numbers. The weighted dot product on \(\R^n\) with weights \((k_1,k_2,\dots, k_n)\) is an inner product. In particular, the dot product is an inner product.

🔗

Proof.

We verify each axiom in turn. Throughout we assume \(\boldx=(x_1,x_2,\dots, x_n)\text{,}\) \(\boldy=(y_1,y_2,\dots, y_n)\text{,}\) and \(\boldw=(w_1,w_2,\dots, w_n)\) are arbitrary elements of \(\R^n\text{.}\)

🔗

Axiom i.

We have

\begin{align*} \angvec{\boldx,\boldy} \amp = \sum_{k=1}^n k_ix_iy_i \amp \text{(def.)}\\ \amp = \sum_{k=1}^n k_iy_ix_i \amp \text{(real mult. is commut.)}\\ \amp = \angvec{\boldy,\boldx} \amp \text{(def.)}\text{.} \end{align*}

🔗

Axiom ii.

Given scalars \(c,d\in \R\text{,}\) we have

\begin{equation*} c\boldx+d\boldy=(cx_1+dy_1,cx_2+dy_2,\dots, cx_n+dy_n)\text{,} \end{equation*}

and thus

\begin{align*} \angvec{c\boldx+d\boldy, \boldw} \amp = \sum_{i=1}^n k_i(cx_i+dy_i)w_i \amp \text{(def.)}\\ \amp = \sum_{i=1}^nck_ix_iw_i+dk_iy_iw_i \amp \text{(dist. prop. of reals)} \\ \amp = c\sum_{i=1}^nk_ix_iw_i+d\sum_{i=1}^nk_iy_iw_i \amp \text{(props. of real arith.)} \\ \amp = c\angvec{\boldx,\boldw}+d\angvec{\boldy,\boldw}\text{.} \end{align*}

🔗

Axiom iii.

We have

\begin{equation*} \langle \boldx, \boldx \rangle=k_1x_1^2+k_2x_2^2+\cdots k_nx_n^2\text{.} \end{equation*}

Since \(k_i> 0\) and \(x_i^2\geq 0\) for all \(i\) (squares of real numbers are nonnegative), we have \(\langle \boldx, \boldx \rangle\geq 0\) for any \(\boldx\text{.}\)

🔗

We now show that \(\angvec{\boldx,\boldx}=0\) if and only if \(\boldx=\boldzero=(0,0,\dots, 0)\text{.}\) The reverse implication is clear: if \(x_i=0\) for all \(i\text{,}\) then \(\sum_{i=1}^nk_ix_i^2=\sum_{i=1}^n 0=0\text{.}\) We prove the forward implication by showing that its contrapositive is true: i.e., if \(\boldx\ne \boldzero\text{,}\) then \(\angvec{\boldx,\boldx} \ne 0\text{.}\) If \(\boldx\ne \boldzero\text{,}\) then we have \(x_{i_0}\ne 0\) for some \(1\leq i_0\leq n\text{,}\) in which case \(k_{i_0}x_{i_0}^2 > 0\text{.}\) But then

\begin{align*} \angvec{\boldx,\boldx} \amp = k_{i_0}x_{i_0}^2+\sum_{i\ne i_0}k_ix_i^2 \\ \amp \geq k_{i_0}x_{i_0}^2 \amp (k_ix_i^2\geq 0)\\ \amp > 0 \text{.} \end{align*}

In particular, \(\angvec{\boldx,\boldx}\ne 0\text{,}\) as desired.

🔗

Example 1.2.7. Why the weights must be positive.

Consider the operation on \(\R^2\) defined as

\begin{equation*} \langle \boldx, \boldy \rangle =(-1)x_1y_1+2x_2y_2 \end{equation*}

where \(\boldx=(x_1,x_2), \boldy=(y_1,y_2)\text{.}\) This operation satisfies axioms i-ii of Definition 1.2.3. (See proof of Theorem 1.2.6.) However, it fails both the positivity and definiteness properties of axiom (iii): e.g.,

\begin{align*} \langle (3,1), (3,1) \rangle \amp =-9+2=-7\lt 0\\ \langle (1,1/\sqrt{2}), (1,1/\sqrt{2})\rangle \amp=-1+2/2=0 \text{.} \end{align*}

🔗

Are there other examples of inner products besides weighted dot products that can be defined on \(\R^n\text{?}\) Indeed there are! For example, it is possible to show that the operation

\begin{equation} \angvec{(x_1,x_2),(y_1,y_2)}=2x_1y_1+x_1y_2+x_2y_1+2x_2y_2\tag{1.4} \end{equation}

defines an inner product on \(\R^2\text{.}\) In fact, getting ahead of ourselves once again, inner products on \(\R^n\) are in one-to-one correspondence with symmetric positive-definite \(n\times n\) matrices. For example, the inner product (1.4) corresponds to the matrix

\begin{equation*} A=\begin{bmatrix}2\amp 1\\ 1\amp 2 \end{bmatrix}\text{.} \end{equation*}

For our purposes, however, the family of weighted dot products will serve as a sufficiently rich source of examples of inner products. Furthermore, as we will see later, any inner product on \(\R^n\) can be viewed as being equal to the dot product “after a change of variables”. For example, letting \(\boldx=(x_1,x_2)\) and \(\boldy=(y_1,y_2)\text{,}\) the inner product (1.4) satisfies

\begin{equation*} \angvec{\boldx,\boldy}=\boldx'\cdot \boldy'\text{,} \end{equation*}

where

\begin{align*} \boldx'\amp =\frac{1}{4}((\sqrt{3}+1)x_1+(\sqrt{3}-1)x_2,(\sqrt{3}-1)x_1+(\sqrt{3}+1)x_2)\\ \boldy'\amp =\frac{1}{4}((\sqrt{3}+1)y_1+(\sqrt{3}-1)y_2,(\sqrt{3}-1)y_1+(\sqrt{3}+1)y_2)\text{.} \end{align*}