{ "cells": [ { "cell_type": "markdown", "metadata": { "origin_pos": 0 }, "source": [ "# Eigendecompositions\n", ":label:`sec_eigendecompositions`\n", "\n", "Eigenvalues are often one of the most useful notions \n", "we will encounter when studying linear algebra, \n", "however, as a beginner, it is easy to overlook their importance.\n", "Below, we introduce eigendecomposition and \n", "try to convey some sense of just why it is so important. \n", "\n", "Suppose that we have a matrix $A$ with the following entries:\n", "\n", "$$\n", "\\mathbf{A} = \\begin{bmatrix}\n", "2 & 0 \\\\\n", "0 & -1\n", "\\end{bmatrix}.\n", "$$\n", "\n", "If we apply $A$ to any vector $\\mathbf{v} = [x, y]^\\top$, \n", "we obtain a vector $\\mathbf{A}\\mathbf{v} = [2x, -y]^\\top$.\n", "This has an intuitive interpretation:\n", "stretch the vector to be twice as wide in the $x$-direction,\n", "and then flip it in the $y$-direction.\n", "\n", "However, there are *some* vectors for which something remains unchanged.\n", "Namely $[1, 0]^\\top$ gets sent to $[2, 0]^\\top$\n", "and $[0, 1]^\\top$ gets sent to $[0, -1]^\\top$.\n", "These vectors are still in the same line,\n", "and the only modification is that the matrix stretches them\n", "by a factor of $2$ and $-1$ respectively.\n", "We call such vectors *eigenvectors*\n", "and the factor they are stretched by *eigenvalues*.\n", "\n", "In general, if we can find a number $\\lambda$ \n", "and a vector $\\mathbf{v}$ such that \n", "\n", "$$\n", "\\mathbf{A}\\mathbf{v} = \\lambda \\mathbf{v}.\n", "$$\n", "\n", "We say that $\\mathbf{v}$ is an eigenvector for $A$ and $\\lambda$ is an eigenvalue.\n", "\n", "## Finding Eigenvalues\n", "Let us figure out how to find them. By subtracting off the $\\lambda \\mathbf{v}$ from both sides,\n", "and then factoring out the vector,\n", "we see the above is equivalent to:\n", "\n", "$$(\\mathbf{A} - \\lambda \\mathbf{I})\\mathbf{v} = 0.$$\n", ":eqlabel:`eq_eigvalue_der`\n", "\n", "For :eqref:`eq_eigvalue_der` to happen, we see that $(\\mathbf{A} - \\lambda \\mathbf{I})$ \n", "must compress some direction down to zero, \n", "hence it is not invertible, and thus the determinant is zero.\n", "Thus, we can find the *eigenvalues* \n", "by finding for what $\\lambda$ is $\\det(\\mathbf{A}-\\lambda \\mathbf{I}) = 0$.\n", "Once we find the eigenvalues, we can solve \n", "$\\mathbf{A}\\mathbf{v} = \\lambda \\mathbf{v}$ \n", "to find the associated *eigenvector(s)*.\n", "\n", "### An Example\n", "Let us see this with a more challenging matrix\n", "\n", "$$\n", "\\mathbf{A} = \\begin{bmatrix}\n", "2 & 1\\\\\n", "2 & 3 \n", "\\end{bmatrix}.\n", "$$\n", "\n", "If we consider $\\det(\\mathbf{A}-\\lambda \\mathbf{I}) = 0$, \n", "we see this is equivalent to the polynomial equation\n", "$0 = (2-\\lambda)(3-\\lambda)-2 = (4-\\lambda)(1-\\lambda)$.\n", "Thus, two eigenvalues are $4$ and $1$.\n", "To find the associated vectors, we then need to solve\n", "\n", "$$\n", "\\begin{bmatrix}\n", "2 & 1\\\\\n", "2 & 3 \n", "\\end{bmatrix}\\begin{bmatrix}x \\\\ y\\end{bmatrix} = \\begin{bmatrix}x \\\\ y\\end{bmatrix} \\; \\text{and} \\;\n", "\\begin{bmatrix}\n", "2 & 1\\\\\n", "2 & 3 \n", "\\end{bmatrix}\\begin{bmatrix}x \\\\ y\\end{bmatrix} = \\begin{bmatrix}4x \\\\ 4y\\end{bmatrix} .\n", "$$\n", "\n", "We can solve this with the vectors $[1, -1]^\\top$ and $[1, 2]^\\top$ respectively.\n", "\n", "We can check this in code using the built-in `numpy.linalg.eig` routine.\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "origin_pos": 3, "tab": [ "tensorflow" ] }, "outputs": [ { "data": { "text/plain": [ "(,\n", " )" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%matplotlib inline\n", "import tensorflow as tf\n", "from IPython import display\n", "from d2l import tensorflow as d2l\n", "\n", "tf.linalg.eig(tf.constant([[2, 1], [2, 3]], dtype=tf.float64))" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 4 }, "source": [ "Note that `numpy` normalizes the eigenvectors to be of length one,\n", "whereas we took ours to be of arbitrary length.\n", "Additionally, the choice of sign is arbitrary.\n", "However, the vectors computed are parallel \n", "to the ones we found by hand with the same eigenvalues.\n", "\n", "## Decomposing Matrices\n", "Let us continue the previous example one step further. Let\n", "\n", "$$\n", "\\mathbf{W} = \\begin{bmatrix}\n", "1 & 1 \\\\\n", "-1 & 2\n", "\\end{bmatrix},\n", "$$\n", "\n", "be the matrix where the columns are the eigenvectors of the matrix $\\mathbf{A}$. Let\n", "\n", "$$\n", "\\boldsymbol{\\Sigma} = \\begin{bmatrix}\n", "1 & 0 \\\\\n", "0 & 4\n", "\\end{bmatrix},\n", "$$\n", "\n", "be the matrix with the associated eigenvalues on the diagonal.\n", "Then the definition of eigenvalues and eigenvectors tells us that\n", "\n", "$$\n", "\\mathbf{A}\\mathbf{W} =\\mathbf{W} \\boldsymbol{\\Sigma} .\n", "$$\n", "\n", "The matrix $W$ is invertible, so we may multiply both sides by $W^{-1}$ on the right,\n", "we see that we may write\n", "\n", "$$\\mathbf{A} = \\mathbf{W} \\boldsymbol{\\Sigma} \\mathbf{W}^{-1}.$$\n", ":eqlabel:`eq_eig_decomp`\n", "\n", "In the next section we will see some nice consequences of this,\n", "but for now we need only know that such a decomposition \n", "will exist as long as we can find a full collection \n", "of linearly independent eigenvectors (so that $W$ is invertible).\n", "\n", "## Operations on Eigendecompositions\n", "One nice thing about eigendecompositions :eqref:`eq_eig_decomp` is that \n", "we can write many operations we usually encounter cleanly \n", "in terms of the eigendecomposition. As a first example, consider:\n", "\n", "$$\n", "\\mathbf{A}^n = \\overbrace{\\mathbf{A}\\cdots \\mathbf{A}}^{\\text{$n$ times}} = \\overbrace{(\\mathbf{W}\\boldsymbol{\\Sigma} \\mathbf{W}^{-1})\\cdots(\\mathbf{W}\\boldsymbol{\\Sigma} \\mathbf{W}^{-1})}^{\\text{$n$ times}} = \\mathbf{W}\\overbrace{\\boldsymbol{\\Sigma}\\cdots\\boldsymbol{\\Sigma}}^{\\text{$n$ times}}\\mathbf{W}^{-1} = \\mathbf{W}\\boldsymbol{\\Sigma}^n \\mathbf{W}^{-1}.\n", "$$\n", "\n", "This tells us that for any positive power of a matrix,\n", "the eigendecomposition is obtained by just raising the eigenvalues to the same power.\n", "The same can be shown for negative powers,\n", "so if we want to invert a matrix we need only consider\n", "\n", "$$\n", "\\mathbf{A}^{-1} = \\mathbf{W}\\boldsymbol{\\Sigma}^{-1} \\mathbf{W}^{-1},\n", "$$\n", "\n", "or in other words, just invert each eigenvalue.\n", "This will work as long as each eigenvalue is non-zero,\n", "so we see that invertible is the same as having no zero eigenvalues. \n", "\n", "Indeed, additional work can show that if $\\lambda_1, \\ldots, \\lambda_n$ \n", "are the eigenvalues of a matrix, then the determinant of that matrix is\n", "\n", "$$\n", "\\det(\\mathbf{A}) = \\lambda_1 \\cdots \\lambda_n,\n", "$$\n", "\n", "or the product of all the eigenvalues.\n", "This makes sense intuitively because whatever stretching $\\mathbf{W}$ does, \n", "$W^{-1}$ undoes it, so in the end the only stretching that happens is \n", "by multiplication by the diagonal matrix $\\boldsymbol{\\Sigma}$, \n", "which stretches volumes by the product of the diagonal elements.\n", "\n", "Finally, recall that the rank was the maximum number \n", "of linearly independent columns of your matrix.\n", "By examining the eigendecomposition closely,\n", "we can see that the rank is the same \n", "as the number of non-zero eigenvalues of $\\mathbf{A}$.\n", "\n", "The examples could continue, but hopefully the point is clear:\n", "eigendecomposition can simplify many linear-algebraic computations\n", "and is a fundamental operation underlying many numerical algorithms\n", "and much of the analysis that we do in linear algebra. \n", "\n", "## Eigendecompositions of Symmetric Matrices\n", "It is not always possible to find enough linearly independent eigenvectors \n", "for the above process to work. For instance the matrix\n", "\n", "$$\n", "\\mathbf{A} = \\begin{bmatrix}\n", "1 & 1 \\\\\n", "0 & 1\n", "\\end{bmatrix},\n", "$$\n", "\n", "has only a single eigenvector, namely $(1, 0)^\\top$. \n", "To handle such matrices, we require more advanced techniques \n", "than we can cover (such as the Jordan Normal Form, or Singular Value Decomposition).\n", "We will often need to restrict our attention to those matrices \n", "where we can guarantee the existence of a full set of eigenvectors.\n", "\n", "The most commonly encountered family are the *symmetric matrices*,\n", "which are those matrices where $\\mathbf{A} = \\mathbf{A}^\\top$. \n", "In this case, we may take $W$ to be an *orthogonal matrix*—a matrix whose columns are all length one vectors that are at right angles to one another, where \n", "$\\mathbf{W}^\\top = \\mathbf{W}^{-1}$—and all the eigenvalues will be real. \n", "Thus, in this special case, we can write :eqref:`eq_eig_decomp` as\n", "\n", "$$\n", "\\mathbf{A} = \\mathbf{W}\\boldsymbol{\\Sigma}\\mathbf{W}^\\top .\n", "$$\n", "\n", "## Gershgorin Circle Theorem\n", "Eigenvalues are often difficult to reason with intuitively.\n", "If presented an arbitrary matrix, there is little that can be said\n", "about what the eigenvalues are without computing them.\n", "There is, however, one theorem that can make it easy to approximate well \n", "if the largest values are on the diagonal.\n", "\n", "Let $\\mathbf{A} = (a_{ij})$ be any square matrix ($n\\times n$).\n", "We will define $r_i = \\sum_{j \\neq i} |a_{ij}|$.\n", "Let $\\mathcal{D}_i$ represent the disc in the complex plane \n", "with center $a_{ii}$ radius $r_i$.\n", "Then, every eigenvalue of $\\mathbf{A}$ is contained in one of the $\\mathcal{D}_i$.\n", "\n", "This can be a bit to unpack, so let us look at an example. \n", "Consider the matrix:\n", "\n", "$$\n", "\\mathbf{A} = \\begin{bmatrix}\n", "1.0 & 0.1 & 0.1 & 0.1 \\\\\n", "0.1 & 3.0 & 0.2 & 0.3 \\\\\n", "0.1 & 0.2 & 5.0 & 0.5 \\\\\n", "0.1 & 0.3 & 0.5 & 9.0\n", "\\end{bmatrix}.\n", "$$\n", "\n", "We have $r_1 = 0.3$, $r_2 = 0.6$, $r_3 = 0.8$ and $r_4 = 0.9$.\n", "The matrix is symmetric, so all eigenvalues are real.\n", "This means that all of our eigenvalues will be in one of the ranges of \n", "\n", "$$[a_{11}-r_1, a_{11}+r_1] = [0.7, 1.3], $$\n", "\n", "$$[a_{22}-r_2, a_{22}+r_2] = [2.4, 3.6], $$\n", "\n", "$$[a_{33}-r_3, a_{33}+r_3] = [4.2, 5.8], $$\n", "\n", "$$[a_{44}-r_4, a_{44}+r_4] = [8.1, 9.9]. $$\n", "\n", "\n", "Performing the numerical computation shows \n", "that the eigenvalues are approximately $0.99$, $2.97$, $4.95$, $9.08$,\n", "all comfortably inside the ranges provided.\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "origin_pos": 7, "tab": [ "tensorflow" ] }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "A = tf.constant([[1.0, 0.1, 0.1, 0.1],\n", " [0.1, 3.0, 0.2, 0.3],\n", " [0.1, 0.2, 5.0, 0.5],\n", " [0.1, 0.3, 0.5, 9.0]])\n", "\n", "v, _ = tf.linalg.eigh(A)\n", "v" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 8 }, "source": [ "In this way, eigenvalues can be approximated, \n", "and the approximations will be fairly accurate \n", "in the case that the diagonal is \n", "significantly larger than all the other elements. \n", "\n", "It is a small thing, but with a complex \n", "and subtle topic like eigendecomposition, \n", "it is good to get any intuitive grasp we can.\n", "\n", "## A Useful Application: The Growth of Iterated Maps\n", "\n", "Now that we understand what eigenvectors are in principle,\n", "let us see how they can be used to provide a deep understanding \n", "of a problem central to neural network behavior: proper weight initialization. \n", "\n", "### Eigenvectors as Long Term Behavior\n", "\n", "The full mathematical investigation of the initialization \n", "of deep neural networks is beyond the scope of the text, \n", "but we can see a toy version here to understand\n", "how eigenvalues can help us see how these models work.\n", "As we know, neural networks operate by interspersing layers \n", "of linear transformations with non-linear operations.\n", "For simplicity here, we will assume that there is no non-linearity,\n", "and that the transformation is a single repeated matrix operation $A$,\n", "so that the output of our model is\n", "\n", "$$\n", "\\mathbf{v}_{out} = \\mathbf{A}\\cdot \\mathbf{A}\\cdots \\mathbf{A} \\mathbf{v}_{in} = \\mathbf{A}^N \\mathbf{v}_{in}.\n", "$$\n", "\n", "When these models are initialized, $A$ is taken to be \n", "a random matrix with Gaussian entries, so let us make one of those. \n", "To be concrete, we start with a mean zero, variance one Gaussian distributed $5 \\times 5$ matrix.\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "origin_pos": 11, "tab": [ "tensorflow" ] }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "k = 5\n", "A = tf.random.normal((k, k), dtype=tf.float64)\n", "A" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 12 }, "source": [ "### Behavior on Random Data\n", "For simplicity in our toy model, \n", "we will assume that the data vector we feed in $\\mathbf{v}_{in}$ \n", "is a random five dimensional Gaussian vector.\n", "Let us think about what we want to have happen.\n", "For context, lets think of a generic ML problem,\n", "where we are trying to turn input data, like an image, into a prediction, \n", "like the probability the image is a picture of a cat.\n", "If repeated application of $\\mathbf{A}$ \n", "stretches a random vector out to be very long, \n", "then small changes in input will be amplified \n", "into large changes in output---tiny modifications of the input image\n", "would lead to vastly different predictions.\n", "This does not seem right!\n", "\n", "On the flip side, if $\\mathbf{A}$ shrinks random vectors to be shorter,\n", "then after running through many layers, the vector will essentially shrink to nothing, \n", "and the output will not depend on the input. This is also clearly not right either!\n", "\n", "We need to walk the narrow line between growth and decay \n", "to make sure that our output changes depending on our input, but not much!\n", "\n", "Let us see what happens when we repeatedly multiply our matrix $\\mathbf{A}$ \n", "against a random input vector, and keep track of the norm.\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "origin_pos": 15, "tab": [ "tensorflow" ] }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-03-24T13:09:20.520169\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.1, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Calculate the sequence of norms after repeatedly applying `A`\n", "v_in = tf.random.normal((k, 1), dtype=tf.float64)\n", "\n", "norm_list = [tf.norm(v_in).numpy()]\n", "for i in range(1, 100):\n", " v_in = tf.matmul(A, v_in)\n", " norm_list.append(tf.norm(v_in).numpy())\n", "\n", "d2l.plot(tf.range(0, 100), norm_list, 'Iteration', 'Value')" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 16 }, "source": [ "The norm is growing uncontrollably! \n", "Indeed if we take the list of quotients, we will see a pattern.\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "origin_pos": 19, "tab": [ "tensorflow" ] }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-03-24T13:09:20.696175\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.1, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Compute the scaling factor of the norms\n", "norm_ratio_list = []\n", "for i in range(1, 100):\n", " norm_ratio_list.append(norm_list[i]/norm_list[i - 1])\n", "\n", "d2l.plot(tf.range(1, 100), norm_ratio_list, 'Iteration', 'Ratio')" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 20 }, "source": [ "If we look at the last portion of the above computation, \n", "we see that the random vector is stretched by a factor of `1.974459321485[...]`,\n", "where the portion at the end shifts a little, \n", "but the stretching factor is stable. \n", "\n", "### Relating Back to Eigenvectors\n", "\n", "We have seen that eigenvectors and eigenvalues correspond \n", "to the amount something is stretched, \n", "but that was for specific vectors, and specific stretches.\n", "Let us take a look at what they are for $\\mathbf{A}$.\n", "A bit of a caveat here: it turns out that to see them all,\n", "we will need to go to complex numbers.\n", "You can think of these as stretches and rotations.\n", "By taking the norm of the complex number\n", "(square root of the sums of squares of real and imaginary parts)\n", "we can measure that stretching factor. Let us also sort them.\n" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "origin_pos": 23, "tab": [ "tensorflow" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "norms of eigenvalues: [, , , , ]\n" ] } ], "source": [ "# Compute the eigenvalues\n", "eigs = tf.linalg.eigh(A)[0].numpy().tolist()\n", "norm_eigs = [tf.abs(tf.constant(x, dtype=tf.float64)) for x in eigs]\n", "norm_eigs.sort()\n", "print(f'norms of eigenvalues: {norm_eigs}')" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 24 }, "source": [ "### An Observation\n", "\n", "We see something a bit unexpected happening here: \n", "that number we identified before for the \n", "long term stretching of our matrix $\\mathbf{A}$ \n", "applied to a random vector is *exactly* \n", "(accurate to thirteen decimal places!) \n", "the largest eigenvalue of $\\mathbf{A}$.\n", "This is clearly not a coincidence!\n", "\n", "But, if we now think about what is happening geometrically,\n", "this starts to make sense. Consider a random vector. \n", "This random vector points a little in every direction, \n", "so in particular, it points at least a little bit \n", "in the same direction as the eigenvector of $\\mathbf{A}$\n", "associated with the largest eigenvalue.\n", "This is so important that it is called \n", "the *principle eigenvalue* and *principle eigenvector*.\n", "After applying $\\mathbf{A}$, our random vector \n", "gets stretched in every possible direction,\n", "as is associated with every possible eigenvector,\n", "but it is stretched most of all in the direction \n", "associated with this principle eigenvector.\n", "What this means is that after apply in $A$, \n", "our random vector is longer, and points in a direction \n", "closer to being aligned with the principle eigenvector.\n", "After applying the matrix many times, \n", "the alignment with the principle eigenvector becomes closer and closer until, \n", "for all practical purposes, our random vector has been transformed \n", "into the principle eigenvector!\n", "Indeed this algorithm is the basis \n", "for what is known as the *power iteration*\n", "for finding the largest eigenvalue and eigenvector of a matrix. For details see, for example, :cite:`Van-Loan.Golub.1983`.\n", "\n", "### Fixing the Normalization\n", "\n", "Now, from above discussions, we concluded \n", "that we do not want a random vector to be stretched or squished at all,\n", "we would like random vectors to stay about the same size throughout the entire process.\n", "To do so, we now rescale our matrix by this principle eigenvalue \n", "so that the largest eigenvalue is instead now just one.\n", "Let us see what happens in this case.\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "origin_pos": 27, "tab": [ "tensorflow" ] }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-03-24T13:09:20.914246\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.1, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Rescale the matrix `A`\n", "A /= norm_eigs[-1]\n", "\n", "# Do the same experiment again\n", "v_in = tf.random.normal((k, 1), dtype=tf.float64)\n", "\n", "norm_list = [tf.norm(v_in).numpy()]\n", "for i in range(1, 100):\n", " v_in = tf.matmul(A, v_in)\n", " norm_list.append(tf.norm(v_in).numpy())\n", "\n", "d2l.plot(tf.range(0, 100), norm_list, 'Iteration', 'Value')" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 28 }, "source": [ "We can also plot the ratio between consecutive norms as before and see that indeed it stabilizes.\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "origin_pos": 31, "tab": [ "tensorflow" ] }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-03-24T13:09:21.084170\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.1, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Also plot the ratio\n", "norm_ratio_list = []\n", "for i in range(1, 100):\n", " norm_ratio_list.append(norm_list[i]/norm_list[i-1])\n", "\n", "d2l.plot(tf.range(1, 100), norm_ratio_list, 'Iteration', 'Ratio')" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 32 }, "source": [ "## Conclusions\n", "\n", "We now see exactly what we hoped for!\n", "After normalizing the matrices by the principle eigenvalue,\n", "we see that the random data does not explode as before,\n", "but rather eventually equilibrates to a specific value.\n", "It would be nice to be able to do these things from first principles,\n", "and it turns out that if we look deeply at the mathematics of it,\n", "we can see that the largest eigenvalue \n", "of a large random matrix with independent mean zero,\n", "variance one Gaussian entries is on average about $\\sqrt{n}$,\n", "or in our case $\\sqrt{5} \\approx 2.2$,\n", "due to a fascinating fact known as the *circular law* :cite:`Ginibre.1965`.\n", "The relationship between the eigenvalues (and a related object called singular values) of random matrices has been shown to have deep connections to proper initialization of neural networks as was discussed in :cite:`Pennington.Schoenholz.Ganguli.2017` and subsequent works.\n", "\n", "## Summary\n", "* Eigenvectors are vectors which are stretched by a matrix without changing direction.\n", "* Eigenvalues are the amount that the eigenvectors are stretched by the application of the matrix.\n", "* The eigendecomposition of a matrix can allow for many operations to be reduced to operations on the eigenvalues.\n", "* The Gershgorin Circle Theorem can provide approximate values for the eigenvalues of a matrix.\n", "* The behavior of iterated matrix powers depends primarily on the size of the largest eigenvalue. This understanding has many applications in the theory of neural network initialization.\n", "\n", "## Exercises\n", "1. What are the eigenvalues and eigenvectors of\n", "$$\n", "\\mathbf{A} = \\begin{bmatrix}\n", "2 & 1 \\\\\n", "1 & 2\n", "\\end{bmatrix}?\n", "$$\n", "1. What are the eigenvalues and eigenvectors of the following matrix, and what is strange about this example compared to the previous one?\n", "$$\n", "\\mathbf{A} = \\begin{bmatrix}\n", "2 & 1 \\\\\n", "0 & 2\n", "\\end{bmatrix}.\n", "$$\n", "1. Without computing the eigenvalues, is it possible that the smallest eigenvalue of the following matrix is less that $0.5$? *Note*: this problem can be done in your head.\n", "$$\n", "\\mathbf{A} = \\begin{bmatrix}\n", "3.0 & 0.1 & 0.3 & 1.0 \\\\\n", "0.1 & 1.0 & 0.1 & 0.2 \\\\\n", "0.3 & 0.1 & 5.0 & 0.0 \\\\\n", "1.0 & 0.2 & 0.0 & 1.8\n", "\\end{bmatrix}.\n", "$$\n" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 35, "tab": [ "tensorflow" ] }, "source": [ "[Discussions](https://discuss.d2l.ai/t/1087)\n" ] } ], "metadata": { "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 4 }