{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 0
},
"source": [
"# Maximum Likelihood\n",
":label:`sec_maximum_likelihood`\n",
"\n",
"One of the most commonly encountered way of thinking in machine learning is the maximum likelihood point of view. This is the concept that when working with a probabilistic model with unknown parameters, the parameters which make the data have the highest probability are the most likely ones.\n",
"\n",
"## The Maximum Likelihood Principle\n",
"\n",
"This has a Bayesian interpretation which can be helpful to think about. Suppose that we have a model with parameters $\\boldsymbol{\\theta}$ and a collection of data examples $X$. For concreteness, we can imagine that $\\boldsymbol{\\theta}$ is a single value representing the probability that a coin comes up heads when flipped, and $X$ is a sequence of independent coin flips. We will look at this example in depth later.\n",
"\n",
"If we want to find the most likely value for the parameters of our model, that means we want to find\n",
"\n",
"$$\\mathop{\\mathrm{argmax}} P(\\boldsymbol{\\theta}\\mid X).$$\n",
":eqlabel:`eq_max_like`\n",
"\n",
"By Bayes' rule, this is the same thing as\n",
"\n",
"$$\n",
"\\mathop{\\mathrm{argmax}} \\frac{P(X \\mid \\boldsymbol{\\theta})P(\\boldsymbol{\\theta})}{P(X)}.\n",
"$$\n",
"\n",
"The expression $P(X)$, a parameter agnostic probability of generating the data, does not depend on $\\boldsymbol{\\theta}$ at all, and so can be dropped without changing the best choice of $\\boldsymbol{\\theta}$. Similarly, we may now posit that we have no prior assumption on which set of parameters are better than any others, so we may declare that $P(\\boldsymbol{\\theta})$ does not depend on theta either! This, for instance, makes sense in our coin flipping example where the probability it comes up heads could be any value in $[0,1]$ without any prior belief it is fair or not (often referred to as an *uninformative prior*). Thus we see that our application of Bayes' rule shows that our best choice of $\\boldsymbol{\\theta}$ is the maximum likelihood estimate for $\\boldsymbol{\\theta}$:\n",
"\n",
"$$\n",
"\\hat{\\boldsymbol{\\theta}} = \\mathop{\\mathrm{argmax}} _ {\\boldsymbol{\\theta}} P(X \\mid \\boldsymbol{\\theta}).\n",
"$$\n",
"\n",
"As a matter of common terminology, the probability of the data given the parameters ($P(X \\mid \\boldsymbol{\\theta})$) is referred to as the *likelihood*.\n",
"\n",
"### A Concrete Example\n",
"\n",
"Let us see how this works in a concrete example. Suppose that we have a single parameter $\\theta$ representing the probability that a coin flip is heads. Then the probability of getting a tails is $1-\\theta$, and so if our observed data $X$ is a sequence with $n_H$ heads and $n_T$ tails, we can use the fact that independent probabilities multiply to see that \n",
"\n",
"$$\n",
"P(X \\mid \\theta) = \\theta^{n_H}(1-\\theta)^{n_T}.\n",
"$$\n",
"\n",
"If we flip $13$ coins and get the sequence \"HHHTHTTHHHHHT\", which has $n_H = 9$ and $n_T = 4$, we see that this is\n",
"\n",
"$$\n",
"P(X \\mid \\theta) = \\theta^9(1-\\theta)^4.\n",
"$$\n",
"\n",
"One nice thing about this example will be that we know the answer going in. Indeed, if we said verbally, \"I flipped 13 coins, and 9 came up heads, what is our best guess for the probability that the coin comes us heads?, \" everyone would correctly guess $9/13$. What this maximum likelihood method will give us is a way to get that number from first principals in a way that will generalize to vastly more complex situations.\n",
"\n",
"For our example, the plot of $P(X \\mid \\theta)$ is as follows:\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"origin_pos": 3,
"tab": [
"tensorflow"
]
},
"outputs": [
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"%matplotlib inline\n",
"import tensorflow as tf\n",
"from d2l import tensorflow as d2l\n",
"\n",
"theta = tf.range(0, 1, 0.001)\n",
"p = theta**9 * (1 - theta)**4.\n",
"\n",
"d2l.plot(theta, p, 'theta', 'likelihood')"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 4
},
"source": [
"This has its maximum value somewhere near our expected $9/13 \\approx 0.7\\ldots$. To see if it is exactly there, we can turn to calculus. Notice that at the maximum, the gradient of the function is flat. Thus, we could find the maximum likelihood estimate :eqref:`eq_max_like` by finding the values of $\\theta$ where the derivative is zero, and finding the one that gives the highest probability. We compute:\n",
"\n",
"$$\n",
"\\begin{aligned}\n",
"0 & = \\frac{d}{d\\theta} P(X \\mid \\theta) \\\\\n",
"& = \\frac{d}{d\\theta} \\theta^9(1-\\theta)^4 \\\\\n",
"& = 9\\theta^8(1-\\theta)^4 - 4\\theta^9(1-\\theta)^3 \\\\\n",
"& = \\theta^8(1-\\theta)^3(9-13\\theta).\n",
"\\end{aligned}\n",
"$$\n",
"\n",
"This has three solutions: $0$, $1$ and $9/13$. The first two are clearly minima, not maxima as they assign probability $0$ to our sequence. The final value does *not* assign zero probability to our sequence, and thus must be the maximum likelihood estimate $\\hat \\theta = 9/13$.\n",
"\n",
"## Numerical Optimization and the Negative Log-Likelihood\n",
"\n",
"The previous example is nice, but what if we have billions of parameters and data examples?\n",
"\n",
"First, notice that if we make the assumption that all the data examples are independent, we can no longer practically consider the likelihood itself as it is a product of many probabilities. Indeed, each probability is in $[0,1]$, say typically of value about $1/2$, and the product of $(1/2)^{1000000000}$ is far below machine precision. We cannot work with that directly. \n",
"\n",
"However, recall that the logarithm turns products to sums, in which case \n",
"\n",
"$$\n",
"\\log((1/2)^{1000000000}) = 1000000000\\cdot\\log(1/2) \\approx -301029995.6\\ldots\n",
"$$\n",
"\n",
"This number fits perfectly within even a single precision $32$-bit float. Thus, we should consider the *log-likelihood*, which is\n",
"\n",
"$$\n",
"\\log(P(X \\mid \\boldsymbol{\\theta})).\n",
"$$\n",
"\n",
"Since the function $x \\mapsto \\log(x)$ is increasing, maximizing the likelihood is the same thing as maximizing the log-likelihood. Indeed in :numref:`sec_naive_bayes` we will see this reasoning applied when working with the specific example of the naive Bayes classifier.\n",
"\n",
"We often work with loss functions, where we wish to minimize the loss. We may turn maximum likelihood into the minimization of a loss by taking $-\\log(P(X \\mid \\boldsymbol{\\theta}))$, which is the *negative log-likelihood*.\n",
"\n",
"To illustrate this, consider the coin flipping problem from before, and pretend that we do not know the closed form solution. We may compute that\n",
"\n",
"$$\n",
"-\\log(P(X \\mid \\boldsymbol{\\theta})) = -\\log(\\theta^{n_H}(1-\\theta)^{n_T}) = -(n_H\\log(\\theta) + n_T\\log(1-\\theta)).\n",
"$$\n",
"\n",
"This can be written into code, and freely optimized even for billions of coin flips.\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"origin_pos": 7,
"tab": [
"tensorflow"
]
},
"outputs": [
{
"data": {
"text/plain": [
"(,\n",
" 0.9970550284664874)"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Set up our data\n",
"n_H = 8675309\n",
"n_T = 25624\n",
"\n",
"# Initialize our paramteres\n",
"theta = tf.Variable(tf.constant(0.5))\n",
"\n",
"# Perform gradient descent\n",
"lr = 0.00000000001\n",
"for iter in range(10):\n",
" with tf.GradientTape() as t:\n",
" loss = -(n_H * tf.math.log(theta) + n_T * tf.math.log(1 - theta))\n",
" theta.assign_sub(lr * t.gradient(loss, theta))\n",
"\n",
"# Check output\n",
"theta, n_H / (n_H + n_T)"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 8
},
"source": [
"Numerical convenience is not the only reason why people like to use negative log-likelihoods. There are several other reasons why it is preferable.\n",
"\n",
"\n",
"\n",
"The second reason we consider the log-likelihood is the simplified application of calculus rules. As discussed above, due to independence assumptions, most probabilities we encounter in machine learning are products of individual probabilities.\n",
"\n",
"$$\n",
"P(X\\mid\\boldsymbol{\\theta}) = p(x_1\\mid\\boldsymbol{\\theta})\\cdot p(x_2\\mid\\boldsymbol{\\theta})\\cdots p(x_n\\mid\\boldsymbol{\\theta}).\n",
"$$\n",
"\n",
"This means that if we directly apply the product rule to compute a derivative we get\n",
"\n",
"$$\n",
"\\begin{aligned}\n",
"\\frac{\\partial}{\\partial \\boldsymbol{\\theta}} P(X\\mid\\boldsymbol{\\theta}) & = \\left(\\frac{\\partial}{\\partial \\boldsymbol{\\theta}}P(x_1\\mid\\boldsymbol{\\theta})\\right)\\cdot P(x_2\\mid\\boldsymbol{\\theta})\\cdots P(x_n\\mid\\boldsymbol{\\theta}) \\\\\n",
"& \\quad + P(x_1\\mid\\boldsymbol{\\theta})\\cdot \\left(\\frac{\\partial}{\\partial \\boldsymbol{\\theta}}P(x_2\\mid\\boldsymbol{\\theta})\\right)\\cdots P(x_n\\mid\\boldsymbol{\\theta}) \\\\\n",
"& \\quad \\quad \\quad \\quad \\quad \\quad \\quad \\quad \\quad \\quad \\vdots \\\\\n",
"& \\quad + P(x_1\\mid\\boldsymbol{\\theta})\\cdot P(x_2\\mid\\boldsymbol{\\theta}) \\cdots \\left(\\frac{\\partial}{\\partial \\boldsymbol{\\theta}}P(x_n\\mid\\boldsymbol{\\theta})\\right).\n",
"\\end{aligned}\n",
"$$\n",
"\n",
"This requires $n(n-1)$ multiplications, along with $(n-1)$ additions, so it is proportional to quadratic time in the inputs! Sufficient cleverness in grouping terms will reduce this to linear time, but it requires some thought. For the negative log-likelihood we have instead\n",
"\n",
"$$\n",
"-\\log\\left(P(X\\mid\\boldsymbol{\\theta})\\right) = -\\log(P(x_1\\mid\\boldsymbol{\\theta})) - \\log(P(x_2\\mid\\boldsymbol{\\theta})) \\cdots - \\log(P(x_n\\mid\\boldsymbol{\\theta})),\n",
"$$\n",
"\n",
"which then gives\n",
"\n",
"$$\n",
"- \\frac{\\partial}{\\partial \\boldsymbol{\\theta}} \\log\\left(P(X\\mid\\boldsymbol{\\theta})\\right) = \\frac{1}{P(x_1\\mid\\boldsymbol{\\theta})}\\left(\\frac{\\partial}{\\partial \\boldsymbol{\\theta}}P(x_1\\mid\\boldsymbol{\\theta})\\right) + \\cdots + \\frac{1}{P(x_n\\mid\\boldsymbol{\\theta})}\\left(\\frac{\\partial}{\\partial \\boldsymbol{\\theta}}P(x_n\\mid\\boldsymbol{\\theta})\\right).\n",
"$$\n",
"\n",
"This requires only $n$ divides and $n-1$ sums, and thus is linear time in the inputs.\n",
"\n",
"The third and final reason to consider the negative log-likelihood is the relationship to information theory, which we will discuss in detail in :numref:`sec_information_theory`. This is a rigorous mathematical theory which gives a way to measure the degree of information or randomness in a random variable. The key object of study in that field is the entropy which is \n",
"\n",
"$$\n",
"H(p) = -\\sum_{i} p_i \\log_2(p_i),\n",
"$$\n",
"\n",
"which measures the randomness of a source. Notice that this is nothing more than the average $-\\log$ probability, and thus if we take our negative log-likelihood and divide by the number of data examples, we get a relative of entropy known as cross-entropy. This theoretical interpretation alone would be sufficiently compelling to motivate reporting the average negative log-likelihood over the dataset as a way of measuring model performance.\n",
"\n",
"## Maximum Likelihood for Continuous Variables\n",
"\n",
"Everything that we have done so far assumes we are working with discrete random variables, but what if we want to work with continuous ones?\n",
"\n",
"The short summary is that nothing at all changes, except we replace all the instances of the probability with the probability density. Recalling that we write densities with lower case $p$, this means that for example we now say\n",
"\n",
"$$\n",
"-\\log\\left(p(X\\mid\\boldsymbol{\\theta})\\right) = -\\log(p(x_1\\mid\\boldsymbol{\\theta})) - \\log(p(x_2\\mid\\boldsymbol{\\theta})) \\cdots - \\log(p(x_n\\mid\\boldsymbol{\\theta})) = -\\sum_i \\log(p(x_i \\mid \\theta)).\n",
"$$\n",
"\n",
"The question becomes, \"Why is this OK?\" After all, the reason we introduced densities was because probabilities of getting specific outcomes themselves was zero, and thus is not the probability of generating our data for any set of parameters zero?\n",
"\n",
"Indeed, this is the case, and understanding why we can shift to densities is an exercise in tracing what happens to the epsilons.\n",
"\n",
"Let us first re-define our goal. Suppose that for continuous random variables we no longer want to compute the probability of getting exactly the right value, but instead matching to within some range $\\epsilon$. For simplicity, we assume our data is repeated observations $x_1, \\ldots, x_N$ of identically distributed random variables $X_1, \\ldots, X_N$. As we have seen previously, this can be written as\n",
"\n",
"$$\n",
"\\begin{aligned}\n",
"&P(X_1 \\in [x_1, x_1+\\epsilon], X_2 \\in [x_2, x_2+\\epsilon], \\ldots, X_N \\in [x_N, x_N+\\epsilon]\\mid\\boldsymbol{\\theta}) \\\\\n",
"\\approx &\\epsilon^Np(x_1\\mid\\boldsymbol{\\theta})\\cdot p(x_2\\mid\\boldsymbol{\\theta}) \\cdots p(x_n\\mid\\boldsymbol{\\theta}).\n",
"\\end{aligned}\n",
"$$\n",
"\n",
"Thus, if we take negative logarithms of this we obtain\n",
"\n",
"$$\n",
"\\begin{aligned}\n",
"&-\\log(P(X_1 \\in [x_1, x_1+\\epsilon], X_2 \\in [x_2, x_2+\\epsilon], \\ldots, X_N \\in [x_N, x_N+\\epsilon]\\mid\\boldsymbol{\\theta})) \\\\\n",
"\\approx & -N\\log(\\epsilon) - \\sum_{i} \\log(p(x_i\\mid\\boldsymbol{\\theta})).\n",
"\\end{aligned}\n",
"$$\n",
"\n",
"If we examine this expression, the only place that the $\\epsilon$ occurs is in the additive constant $-N\\log(\\epsilon)$. This does not depend on the parameters $\\boldsymbol{\\theta}$ at all, so the optimal choice of $\\boldsymbol{\\theta}$ does not depend on our choice of $\\epsilon$! If we demand four digits or four-hundred, the best choice of $\\boldsymbol{\\theta}$ remains the same, thus we may freely drop the epsilon to see that what we want to optimize is\n",
"\n",
"$$\n",
"- \\sum_{i} \\log(p(x_i\\mid\\boldsymbol{\\theta})).\n",
"$$\n",
"\n",
"Thus, we see that the maximum likelihood point of view can operate with continuous random variables as easily as with discrete ones by replacing the probabilities with probability densities.\n",
"\n",
"## Summary\n",
"* The maximum likelihood principle tells us that the best fit model for a given dataset is the one that generates the data with the highest probability.\n",
"* Often people work with the negative log-likelihood instead for a variety of reasons: numerical stability, conversion of products to sums (and the resulting simplification of gradient computations), and theoretical ties to information theory.\n",
"* While simplest to motivate in the discrete setting, it may be freely generalized to the continuous setting as well by maximizing the probability density assigned to the datapoints.\n",
"\n",
"## Exercises\n",
"1. Suppose that you know that a random variable has density $\\frac{1}{\\alpha}e^{-\\alpha x}$ for some value $\\alpha$. You obtain a single observation from the random variable which is the number $3$. What is the maximum likelihood estimate for $\\alpha$?\n",
"2. Suppose that you have a dataset of samples $\\{x_i\\}_{i=1}^N$ drawn from a Gaussian with unknown mean, but variance $1$. What is the maximum likelihood estimate for the mean?\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 11,
"tab": [
"tensorflow"
]
},
"source": [
"[Discussions](https://discuss.d2l.ai/t/1097)\n"
]
}
],
"metadata": {
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 4
}