{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 0
},
"source": [
"# Implementation of Multilayer Perceptrons from Scratch\n",
":label:`sec_mlp_scratch`\n",
"\n",
"Now that we have characterized\n",
"multilayer perceptrons (MLPs) mathematically,\n",
"let us try to implement one ourselves. To compare against our previous results\n",
"achieved with softmax regression\n",
"(:numref:`sec_softmax_scratch`),\n",
"we will continue to work with\n",
"the Fashion-MNIST image classification dataset\n",
"(:numref:`sec_fashion_mnist`).\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"origin_pos": 3,
"tab": [
"tensorflow"
]
},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"from d2l import tensorflow as d2l"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"origin_pos": 4,
"tab": [
"tensorflow"
]
},
"outputs": [],
"source": [
"batch_size = 256\n",
"train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size)"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 5
},
"source": [
"## Initializing Model Parameters\n",
"\n",
"Recall that Fashion-MNIST contains 10 classes,\n",
"and that each image consists of a $28 \\times 28 = 784$\n",
"grid of grayscale pixel values.\n",
"Again, we will disregard the spatial structure\n",
"among the pixels for now,\n",
"so we can think of this as simply a classification dataset\n",
"with 784 input features and 10 classes.\n",
"To begin, we will [**implement an MLP\n",
"with one hidden layer and 256 hidden units.**]\n",
"Note that we can regard both of these quantities\n",
"as hyperparameters.\n",
"Typically, we choose layer widths in powers of 2,\n",
"which tend to be computationally efficient because\n",
"of how memory is allocated and addressed in hardware.\n",
"\n",
"Again, we will represent our parameters with several tensors.\n",
"Note that *for every layer*, we must keep track of\n",
"one weight matrix and one bias vector.\n",
"As always, we allocate memory\n",
"for the gradients of the loss with respect to these parameters.\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"origin_pos": 8,
"tab": [
"tensorflow"
]
},
"outputs": [],
"source": [
"num_inputs, num_outputs, num_hiddens = 784, 10, 256\n",
"\n",
"W1 = tf.Variable(tf.random.normal(\n",
" shape=(num_inputs, num_hiddens), mean=0, stddev=0.01))\n",
"b1 = tf.Variable(tf.zeros(num_hiddens))\n",
"W2 = tf.Variable(tf.random.normal(\n",
" shape=(num_hiddens, num_outputs), mean=0, stddev=0.01))\n",
"b2 = tf.Variable(tf.random.normal([num_outputs], stddev=.01))\n",
"\n",
"params = [W1, b1, W2, b2]"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 9
},
"source": [
"## Activation Function\n",
"\n",
"To make sure we know how everything works,\n",
"we will [**implement the ReLU activation**] ourselves\n",
"using the maximum function rather than\n",
"invoking the built-in `relu` function directly.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"origin_pos": 12,
"tab": [
"tensorflow"
]
},
"outputs": [],
"source": [
"def relu(X):\n",
" return tf.math.maximum(X, 0)"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 13
},
"source": [
"## Model\n",
"\n",
"Because we are disregarding spatial structure,\n",
"we `reshape` each two-dimensional image into\n",
"a flat vector of length `num_inputs`.\n",
"Finally, we (**implement our model**)\n",
"with just a few lines of code.\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"origin_pos": 16,
"tab": [
"tensorflow"
]
},
"outputs": [],
"source": [
"def net(X):\n",
" X = tf.reshape(X, (-1, num_inputs))\n",
" H = relu(tf.matmul(X, W1) + b1)\n",
" return tf.matmul(H, W2) + b2"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 17
},
"source": [
"## Loss Function\n",
"\n",
"To ensure numerical stability,\n",
"and because we already implemented\n",
"the softmax function from scratch\n",
"(:numref:`sec_softmax_scratch`),\n",
"we leverage the integrated function from high-level APIs\n",
"for calculating the softmax and cross-entropy loss.\n",
"Recall our earlier discussion of these intricacies\n",
"in :numref:`subsec_softmax-implementation-revisited`.\n",
"We encourage the interested reader\n",
"to examine the source code for the loss function\n",
"to deepen their knowledge of implementation details.\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"origin_pos": 20,
"tab": [
"tensorflow"
]
},
"outputs": [],
"source": [
"def loss(y_hat, y):\n",
" return tf.losses.sparse_categorical_crossentropy(\n",
" y, y_hat, from_logits=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 21
},
"source": [
"## Training\n",
"\n",
"Fortunately, [**the training loop for MLPs\n",
"is exactly the same as for softmax regression.**]\n",
"Leveraging the `d2l` package again,\n",
"we call the `train_ch3` function\n",
"(see :numref:`sec_softmax_scratch`),\n",
"setting the number of epochs to 10\n",
"and the learning rate to 0.1.\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"origin_pos": 24,
"tab": [
"tensorflow"
]
},
"outputs": [
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"num_epochs, lr = 10, 0.1\n",
"updater = d2l.Updater([W1, W2, b1, b2], lr)\n",
"d2l.train_ch3(net, train_iter, test_iter, loss, num_epochs, updater)"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 25
},
"source": [
"To evaluate the learned model,\n",
"we [**apply it on some test data**].\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"origin_pos": 26,
"tab": [
"tensorflow"
]
},
"outputs": [
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"d2l.predict_ch3(net, test_iter)"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 27
},
"source": [
"## Summary\n",
"\n",
"* We saw that implementing a simple MLP is easy, even when done manually.\n",
"* However, with a large number of layers, implementing MLPs from scratch can still get messy (e.g., naming and keeping track of our model's parameters).\n",
"\n",
"\n",
"## Exercises\n",
"\n",
"1. Change the value of the hyperparameter `num_hiddens` and see how this hyperparameter influences your results. Determine the best value of this hyperparameter, keeping all others constant.\n",
"1. Try adding an additional hidden layer to see how it affects the results.\n",
"1. How does changing the learning rate alter your results? Fixing the model architecture and other hyperparameters (including number of epochs), what learning rate gives you the best results?\n",
"1. What is the best result you can get by optimizing over all the hyperparameters (learning rate, number of epochs, number of hidden layers, number of hidden units per layer) jointly?\n",
"1. Describe why it is much more challenging to deal with multiple hyperparameters.\n",
"1. What is the smartest strategy you can think of for structuring a search over multiple hyperparameters?\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 30,
"tab": [
"tensorflow"
]
},
"source": [
"[Discussions](https://discuss.d2l.ai/t/227)\n"
]
}
],
"metadata": {
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 4
}