Add column space orthogonality justification

Author: surajrampure

06_linear_transformations_and_projections/03-projecting-onto-column-space.ipynb

@@ -3225,6 +3225,17 @@
     "\n",
     "we need to find the $\\vec w$ that makes the error vector ${\\color{#d81a60} \\vec e} = {\\color{orange} \\vec y} - {\\color{#3d81f6} X} \\vec w$ orthogonal to $\\text{colsp}({\\color{#3d81f6} X})$. $\\text{colsp}({\\color{#3d81f6} X})$ is the set of all linear combinations of ${\\color{#3d81f6} X}$'s columns. So, if we can find an $\\color{#d81a60} \\vec e$ that is orthogonal to every column of ${\\color{#3d81f6} X}$, then it must be orthogonal to any of their linear combinations, too.\n",
     "\n",
+    "To see why, take any vector in $\\text{colsp}({\\color{#3d81f6} X})$. It has the form $a_1 {\\color{#3d81f6} \\vec x^{(1)}} + a_2 {\\color{#3d81f6} \\vec x^{(2)}} + \\ldots + a_d {\\color{#3d81f6} \\vec x^{(d)}}$. Then\n",
+    "\n",
+    "$$\\begin{align*}\n",
+    "\\left(a_1 {\\color{#3d81f6} \\vec x^{(1)}} + a_2 {\\color{#3d81f6} \\vec x^{(2)}} + \\ldots + a_d {\\color{#3d81f6} \\vec x^{(d)}}\\right) \\cdot {\\color{#d81a60} \\vec e}\n",
+    "&= a_1 ({\\color{#3d81f6} \\vec x^{(1)}} \\cdot {\\color{#d81a60} \\vec e}) + a_2 ({\\color{#3d81f6} \\vec x^{(2)}} \\cdot {\\color{#d81a60} \\vec e}) + \\ldots + a_d ({\\color{#3d81f6} \\vec x^{(d)}} \\cdot {\\color{#d81a60} \\vec e}) \\\\\n",
+    "&= a_1(0) + a_2(0) + \\ldots + a_d(0) \\\\\n",
+    "&= 0\n",
+    "\\end{align*}$$\n",
+    "\n",
+    "So checking orthogonality against the columns of ${\\color{#3d81f6} X}$ is enough to guarantee orthogonality against everything in $\\text{colsp}({\\color{#3d81f6} X})$.\n",
+    "\n",
     "So, we're looking for a ${\\color{#d81a60} \\vec e} = {\\color{orange} \\vec y} - {\\color{#3d81f6} X} \\vec w$ that satisfies\n",
     "\n",
     "$$\\begin{align*} {\\color{#3d81f6} \\vec x^{(1)}} \\cdot ({\\color{orange} \\vec y} - {\\color{#3d81f6} X} \\vec w) &= 0 \\\\ {\\color{#3d81f6} \\vec x^{(2)}} \\cdot ({\\color{orange} \\vec y} - {\\color{#3d81f6} X} \\vec w) &= 0 \\\\ &\\vdots \\\\ {\\color{#3d81f6} \\vec x^{(d)}} \\cdot ({\\color{orange} \\vec y} - {\\color{#3d81f6} X} \\vec w) &= 0  \\end{align*}$$\n",