{ "cells": [ { "cell_type": "markdown", "metadata": { "origin_pos": 0 }, "source": [ "# Implementation of Softmax Regression from Scratch\n", ":label:`sec_softmax_scratch`\n", "\n", "(**Just as we implemented linear regression from scratch, we believe that**)\n", "softmax regression\n", "is similarly fundamental and\n", "(**you ought to know the gory details of**) (~~softmax regression~~) and how to implement it yourself.\n", "We will work with the Fashion-MNIST dataset, just introduced in :numref:`sec_fashion_mnist`,\n", "setting up a data iterator with batch size 256.\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "origin_pos": 3, "tab": [ "tensorflow" ] }, "outputs": [], "source": [ "import tensorflow as tf\n", "from IPython import display\n", "from d2l import tensorflow as d2l" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "origin_pos": 4, "tab": [ "tensorflow" ] }, "outputs": [], "source": [ "batch_size = 256\n", "train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size)" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 5 }, "source": [ "## Initializing Model Parameters\n", "\n", "As in our linear regression example,\n", "each example here will be represented by a fixed-length vector.\n", "Each example in the raw dataset is a $28 \\times 28$ image.\n", "In this section, [**we will flatten each image,\n", "treating them as vectors of length 784.**]\n", "In the future, we will talk about more sophisticated strategies\n", "for exploiting the spatial structure in images,\n", "but for now we treat each pixel location as just another feature.\n", "\n", "Recall that in softmax regression,\n", "we have as many outputs as there are classes.\n", "(**Because our dataset has 10 classes,\n", "our network will have an output dimension of 10.**)\n", "Consequently, our weights will constitute a $784 \\times 10$ matrix\n", "and the biases will constitute a $1 \\times 10$ row vector.\n", "As with linear regression, we will initialize our weights `W`\n", "with Gaussian noise and our biases to take the initial value 0.\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "origin_pos": 8, "tab": [ "tensorflow" ] }, "outputs": [], "source": [ "num_inputs = 784\n", "num_outputs = 10\n", "\n", "W = tf.Variable(tf.random.normal(shape=(num_inputs, num_outputs),\n", " mean=0, stddev=0.01))\n", "b = tf.Variable(tf.zeros(num_outputs))" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 9 }, "source": [ "## Defining the Softmax Operation\n", "\n", "Before implementing the softmax regression model,\n", "let us briefly review how the sum operator works\n", "along specific dimensions in a tensor,\n", "as discussed in :numref:`subseq_lin-alg-reduction` and :numref:`subseq_lin-alg-non-reduction`.\n", "[**Given a matrix `X` we can sum over all elements (by default) or only\n", "over elements in the same axis,**]\n", "i.e., the same column (axis 0) or the same row (axis 1).\n", "Note that if `X` is a tensor with shape (2, 3)\n", "and we sum over the columns,\n", "the result will be a vector with shape (3,).\n", "When invoking the sum operator,\n", "we can specify to keep the number of axes in the original tensor,\n", "rather than collapsing out the dimension that we summed over.\n", "This will result in a two-dimensional tensor with shape (1, 3).\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "origin_pos": 11, "tab": [ "tensorflow" ] }, "outputs": [ { "data": { "text/plain": [ "(,\n", " )" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])\n", "tf.reduce_sum(X, 0, keepdims=True), tf.reduce_sum(X, 1, keepdims=True)" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 12 }, "source": [ "We are now ready to (**implement the softmax operation**).\n", "Recall that softmax consists of three steps:\n", "(i) we exponentiate each term (using `exp`);\n", "(ii) we sum over each row (we have one row per example in the batch)\n", "to get the normalization constant for each example;\n", "(iii) we divide each row by its normalization constant,\n", "ensuring that the result sums to 1.\n", "Before looking at the code, let us recall\n", "how this looks expressed as an equation:\n", "\n", "(**\n", "$$\\mathrm{softmax}(\\mathbf{X})_{ij} = \\frac{\\exp(\\mathbf{X}_{ij})}{\\sum_k \\exp(\\mathbf{X}_{ik})}.$$\n", "**)\n", "\n", "The denominator, or normalization constant,\n", "is also sometimes called the *partition function*\n", "(and its logarithm is called the log-partition function).\n", "The origins of that name are in [statistical physics](https://en.wikipedia.org/wiki/Partition_function_(statistical_mechanics))\n", "where a related equation models the distribution\n", "over an ensemble of particles.\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "origin_pos": 13, "tab": [ "tensorflow" ] }, "outputs": [], "source": [ "def softmax(X):\n", " X_exp = tf.exp(X)\n", " partition = tf.reduce_sum(X_exp, 1, keepdims=True)\n", " return X_exp / partition # The broadcasting mechanism is applied here" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 15 }, "source": [ "As you can see, for any random input,\n", "[**we turn each element into a non-negative number.\n", "Moreover, each row sums up to 1,**]\n", "as is required for a probability.\n" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "origin_pos": 17, "tab": [ "tensorflow" ] }, "outputs": [ { "data": { "text/plain": [ "(,\n", " )" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X = tf.random.normal((2, 5), 0, 1)\n", "X_prob = softmax(X)\n", "X_prob, tf.reduce_sum(X_prob, 1)" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 18 }, "source": [ "Note that while this looks correct mathematically,\n", "we were a bit sloppy in our implementation\n", "because we failed to take precautions against numerical overflow or underflow\n", "due to large or very small elements of the matrix.\n", "\n", "## Defining the Model\n", "\n", "Now that we have defined the softmax operation,\n", "we can [**implement the softmax regression model.**]\n", "The below code defines how the input is mapped to the output through the network.\n", "Note that we flatten each original image in the batch\n", "into a vector using the `reshape` function\n", "before passing the data through our model.\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "origin_pos": 19, "tab": [ "tensorflow" ] }, "outputs": [], "source": [ "def net(X):\n", " return softmax(tf.matmul(tf.reshape(X, (-1, W.shape[0])), W) + b)" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 20 }, "source": [ "## Defining the Loss Function\n", "\n", "Next, we need to implement the cross-entropy loss function,\n", "as introduced in :numref:`sec_softmax`.\n", "This may be the most common loss function\n", "in all of deep learning because, at the moment,\n", "classification problems far outnumber regression problems.\n", "\n", "Recall that cross-entropy takes the negative log-likelihood\n", "of the predicted probability assigned to the true label.\n", "Rather than iterating over the predictions with a Python for-loop\n", "(which tends to be inefficient),\n", "we can pick all elements by a single operator.\n", "Below, we [**create sample data `y_hat`\n", "with 2 examples of predicted probabilities over 3 classes and their corresponding labels `y`.**]\n", "With `y` we know that in the first example the first class is the correct prediction and\n", "in the second example the third class is the ground-truth.\n", "[**Using `y` as the indices of the probabilities in `y_hat`,**]\n", "we pick the probability of the first class in the first example\n", "and the probability of the third class in the second example.\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "origin_pos": 22, "tab": [ "tensorflow" ] }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y_hat = tf.constant([[0.1, 0.3, 0.6], [0.3, 0.2, 0.5]])\n", "y = tf.constant([0, 2])\n", "tf.boolean_mask(y_hat, tf.one_hot(y, depth=y_hat.shape[-1]))" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 23 }, "source": [ "Now we can (**implement the cross-entropy loss function**) efficiently with just one line of code.\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "origin_pos": 25, "tab": [ "tensorflow" ] }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def cross_entropy(y_hat, y):\n", " return -tf.math.log(tf.boolean_mask(\n", " y_hat, tf.one_hot(y, depth=y_hat.shape[-1])))\n", "\n", "cross_entropy(y_hat, y)" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 26 }, "source": [ "## Classification Accuracy\n", "\n", "Given the predicted probability distribution `y_hat`,\n", "we typically choose the class with the highest predicted probability\n", "whenever we must output a hard prediction.\n", "Indeed, many applications require that we make a choice.\n", "Gmail must categorize an email into \"Primary\", \"Social\", \"Updates\", or \"Forums\".\n", "It might estimate probabilities internally,\n", "but at the end of the day it has to choose one among the classes.\n", "\n", "When predictions are consistent with the label class `y`, they are correct.\n", "The classification accuracy is the fraction of all predictions that are correct.\n", "Although it can be difficult to optimize accuracy directly (it is not differentiable),\n", "it is often the performance measure that we care most about,\n", "and we will nearly always report it when training classifiers.\n", "\n", "To compute accuracy we do the following.\n", "First, if `y_hat` is a matrix,\n", "we assume that the second dimension stores prediction scores for each class.\n", "We use `argmax` to obtain the predicted class by the index for the largest entry in each row.\n", "Then we [**compare the predicted class with the ground-truth `y` elementwise.**]\n", "Since the equality operator `==` is sensitive to data types,\n", "we convert `y_hat`'s data type to match that of `y`.\n", "The result is a tensor containing entries of 0 (false) and 1 (true).\n", "Taking the sum yields the number of correct predictions.\n" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "origin_pos": 27, "tab": [ "tensorflow" ] }, "outputs": [], "source": [ "def accuracy(y_hat, y): #@save\n", " \"\"\"Compute the number of correct predictions.\"\"\"\n", " if len(y_hat.shape) > 1 and y_hat.shape[1] > 1:\n", " y_hat = tf.argmax(y_hat, axis=1)\n", " cmp = tf.cast(y_hat, y.dtype) == y\n", " return float(tf.reduce_sum(tf.cast(cmp, y.dtype)))" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 28 }, "source": [ "We will continue to use the variables `y_hat` and `y`\n", "defined before\n", "as the predicted probability distributions and labels, respectively.\n", "We can see that the first example's prediction class is 2\n", "(the largest element of the row is 0.6 with the index 2),\n", "which is inconsistent with the actual label, 0.\n", "The second example's prediction class is 2\n", "(the largest element of the row is 0.5 with the index of 2),\n", "which is consistent with the actual label, 2.\n", "Therefore, the classification accuracy rate for these two examples is 0.5.\n" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "origin_pos": 29, "tab": [ "tensorflow" ] }, "outputs": [ { "data": { "text/plain": [ "0.5" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "accuracy(y_hat, y) / len(y)" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 30 }, "source": [ "[**Similarly, we can evaluate the accuracy for any model `net` on a dataset**]\n", "that is accessed via the data iterator `data_iter`.\n" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "origin_pos": 31, "tab": [ "tensorflow" ] }, "outputs": [], "source": [ "def evaluate_accuracy(net, data_iter): #@save\n", " \"\"\"Compute the accuracy for a model on a dataset.\"\"\"\n", " metric = Accumulator(2) # No. of correct predictions, no. of predictions\n", " for X, y in data_iter:\n", " metric.add(accuracy(net(X), y), d2l.size(y))\n", " return metric[0] / metric[1]" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 33 }, "source": [ "Here `Accumulator` is a utility class to accumulate sums over multiple variables.\n", "In the above `evaluate_accuracy` function,\n", "we create 2 variables in the `Accumulator` instance for storing both\n", "the number of correct predictions and the number of predictions, respectively.\n", "Both will be accumulated over time as we iterate over the dataset.\n" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "origin_pos": 34, "tab": [ "tensorflow" ] }, "outputs": [], "source": [ "class Accumulator: #@save\n", " \"\"\"For accumulating sums over `n` variables.\"\"\"\n", " def __init__(self, n):\n", " self.data = [0.0] * n\n", "\n", " def add(self, *args):\n", " self.data = [a + float(b) for a, b in zip(self.data, args)]\n", "\n", " def reset(self):\n", " self.data = [0.0] * len(self.data)\n", "\n", " def __getitem__(self, idx):\n", " return self.data[idx]" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 35 }, "source": [ "[**Because we initialized the `net` model with random weights,\n", "the accuracy of this model should be close to random guessing,**]\n", "i.e., 0.1 for 10 classes.\n" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "origin_pos": 36, "tab": [ "tensorflow" ] }, "outputs": [ { "data": { "text/plain": [ "0.1058" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "evaluate_accuracy(net, test_iter)" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 37 }, "source": [ "## Training\n", "\n", "[**The training loop**]\n", "for softmax regression should look strikingly familiar\n", "if you read through our implementation\n", "of linear regression in :numref:`sec_linear_scratch`.\n", "Here we refactor the implementation to make it reusable.\n", "First, we define a function to train for one epoch.\n", "Note that `updater` is a general function to update the model parameters,\n", "which accepts the batch size as an argument.\n", "It can be either a wrapper of the `d2l.sgd` function\n", "or a framework's built-in optimization function.\n" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "origin_pos": 40, "tab": [ "tensorflow" ] }, "outputs": [], "source": [ "def train_epoch_ch3(net, train_iter, loss, updater): #@save\n", " \"\"\"The training loop defined in Chapter 3.\"\"\"\n", " # Sum of training loss, sum of training accuracy, no. of examples\n", " metric = Accumulator(3)\n", " for X, y in train_iter:\n", " # Compute gradients and update parameters\n", " with tf.GradientTape() as tape:\n", " y_hat = net(X)\n", " # Keras implementations for loss takes (labels, predictions)\n", " # instead of (predictions, labels) that users might implement\n", " # in this book, e.g. `cross_entropy` that we implemented above\n", " if isinstance(loss, tf.keras.losses.Loss):\n", " l = loss(y, y_hat)\n", " else:\n", " l = loss(y_hat, y)\n", " if isinstance(updater, tf.keras.optimizers.Optimizer):\n", " params = net.trainable_variables\n", " grads = tape.gradient(l, params)\n", " updater.apply_gradients(zip(grads, params))\n", " else:\n", " updater(X.shape[0], tape.gradient(l, updater.params))\n", " # Keras loss by default returns the average loss in a batch\n", " l_sum = l * float(tf.size(y)) if isinstance(\n", " loss, tf.keras.losses.Loss) else tf.reduce_sum(l)\n", " metric.add(l_sum, accuracy(y_hat, y), tf.size(y))\n", " # Return training loss and training accuracy\n", " return metric[0] / metric[2], metric[1] / metric[2]" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 41 }, "source": [ "Before showing the implementation of the training function,\n", "we define [**a utility class that plot data in animation.**]\n", "Again, it aims to simplify code in the rest of the book.\n" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "origin_pos": 42, "tab": [ "tensorflow" ] }, "outputs": [], "source": [ "class Animator: #@save\n", " \"\"\"For plotting data in animation.\"\"\"\n", " def __init__(self, xlabel=None, ylabel=None, legend=None, xlim=None,\n", " ylim=None, xscale='linear', yscale='linear',\n", " fmts=('-', 'm--', 'g-.', 'r:'), nrows=1, ncols=1,\n", " figsize=(3.5, 2.5)):\n", " # Incrementally plot multiple lines\n", " if legend is None:\n", " legend = []\n", " d2l.use_svg_display()\n", " self.fig, self.axes = d2l.plt.subplots(nrows, ncols, figsize=figsize)\n", " if nrows * ncols == 1:\n", " self.axes = [self.axes, ]\n", " # Use a lambda function to capture arguments\n", " self.config_axes = lambda: d2l.set_axes(\n", " self.axes[0], xlabel, ylabel, xlim, ylim, xscale, yscale, legend)\n", " self.X, self.Y, self.fmts = None, None, fmts\n", "\n", " def add(self, x, y):\n", " # Add multiple data points into the figure\n", " if not hasattr(y, \"__len__\"):\n", " y = [y]\n", " n = len(y)\n", " if not hasattr(x, \"__len__\"):\n", " x = [x] * n\n", " if not self.X:\n", " self.X = [[] for _ in range(n)]\n", " if not self.Y:\n", " self.Y = [[] for _ in range(n)]\n", " for i, (a, b) in enumerate(zip(x, y)):\n", " if a is not None and b is not None:\n", " self.X[i].append(a)\n", " self.Y[i].append(b)\n", " self.axes[0].cla()\n", " for x, y, fmt in zip(self.X, self.Y, self.fmts):\n", " self.axes[0].plot(x, y, fmt)\n", " self.config_axes()\n", " display.display(self.fig)\n", " display.clear_output(wait=True)" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 43 }, "source": [ "[~~The training function~~]\n", "The following training function then\n", "trains a model `net` on a training dataset accessed via `train_iter`\n", "for multiple epochs, which is specified by `num_epochs`.\n", "At the end of each epoch,\n", "the model is evaluated on a testing dataset accessed via `test_iter`.\n", "We will leverage the `Animator` class to visualize\n", "the training progress.\n" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "origin_pos": 44, "tab": [ "tensorflow" ] }, "outputs": [], "source": [ "def train_ch3(net, train_iter, test_iter, loss, num_epochs, updater): #@save\n", " \"\"\"Train a model (defined in Chapter 3).\"\"\"\n", " animator = Animator(xlabel='epoch', xlim=[1, num_epochs], ylim=[0.3, 0.9],\n", " legend=['train loss', 'train acc', 'test acc'])\n", " for epoch in range(num_epochs):\n", " train_metrics = train_epoch_ch3(net, train_iter, loss, updater)\n", " test_acc = evaluate_accuracy(net, test_iter)\n", " animator.add(epoch + 1, train_metrics + (test_acc,))\n", " train_loss, train_acc = train_metrics\n", " assert train_loss < 0.5, train_loss\n", " assert train_acc <= 1 and train_acc > 0.7, train_acc\n", " assert test_acc <= 1 and test_acc > 0.7, test_acc" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 45 }, "source": [ "As an implementation from scratch,\n", "we [**use the minibatch stochastic gradient descent**] defined in :numref:`sec_linear_scratch`\n", "to optimize the loss function of the model with a learning rate 0.1.\n" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "origin_pos": 47, "tab": [ "tensorflow" ] }, "outputs": [], "source": [ "class Updater(): #@save\n", " \"\"\"For updating parameters using minibatch stochastic gradient descent.\"\"\"\n", " def __init__(self, params, lr):\n", " self.params = params\n", " self.lr = lr\n", "\n", " def __call__(self, batch_size, grads):\n", " d2l.sgd(self.params, grads, self.lr, batch_size)\n", "\n", "updater = Updater([W, b], lr=0.1)" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 48 }, "source": [ "Now we [**train the model with 10 epochs.**]\n", "Note that both the number of epochs (`num_epochs`),\n", "and learning rate (`lr`) are adjustable hyperparameters.\n", "By changing their values, we may be able\n", "to increase the classification accuracy of the model.\n" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "origin_pos": 49, "tab": [ "tensorflow" ] }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-03-24T13:09:36.643253\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.1, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "num_epochs = 10\n", "train_ch3(net, train_iter, test_iter, cross_entropy, num_epochs, updater)" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 50 }, "source": [ "## Prediction\n", "\n", "Now that training is complete,\n", "our model is ready to [**classify some images.**]\n", "Given a series of images,\n", "we will compare their actual labels\n", "(first line of text output)\n", "and the predictions from the model\n", "(second line of text output).\n" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "origin_pos": 51, "tab": [ "tensorflow" ] }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-03-24T13:09:37.050684\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.1, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "def predict_ch3(net, test_iter, n=6): #@save\n", " \"\"\"Predict labels (defined in Chapter 3).\"\"\"\n", " for X, y in test_iter:\n", " break\n", " trues = d2l.get_fashion_mnist_labels(y)\n", " preds = d2l.get_fashion_mnist_labels(tf.argmax(net(X), axis=1))\n", " titles = [true +'\\n' + pred for true, pred in zip(trues, preds)]\n", " d2l.show_images(\n", " tf.reshape(X[0:n], (n, 28, 28)), 1, n, titles=titles[0:n])\n", "\n", "predict_ch3(net, test_iter)" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 52 }, "source": [ "## Summary\n", "\n", "* With softmax regression, we can train models for multiclass classification.\n", "* The training loop of softmax regression is very similar to that in linear regression: retrieve and read data, define models and loss functions, then train models using optimization algorithms. As you will soon find out, most common deep learning models have similar training procedures.\n", "\n", "## Exercises\n", "\n", "1. In this section, we directly implemented the softmax function based on the mathematical definition of the softmax operation. What problems might this cause? Hint: try to calculate the size of $\\exp(50)$.\n", "1. The function `cross_entropy` in this section was implemented according to the definition of the cross-entropy loss function. What could be the problem with this implementation? Hint: consider the domain of the logarithm.\n", "1. What solutions you can think of to fix the two problems above?\n", "1. Is it always a good idea to return the most likely label? For example, would you do this for medical diagnosis?\n", "1. Assume that we want to use softmax regression to predict the next word based on some features. What are some problems that might arise from a large vocabulary?\n" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 55, "tab": [ "tensorflow" ] }, "source": [ "[Discussions](https://discuss.d2l.ai/t/225)\n" ] } ], "metadata": { "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 4 }