{ "cells": [ { "cell_type": "markdown", "metadata": { "origin_pos": 0 }, "source": [ "# Concise Implementation of Linear Regression\n", ":label:`sec_linear_concise`\n", "\n", "Broad and intense interest in deep learning for the past several years\n", "has inspired companies, academics, and hobbyists\n", "to develop a variety of mature open source frameworks\n", "for automating the repetitive work of implementing\n", "gradient-based learning algorithms.\n", "In :numref:`sec_linear_scratch`, we relied only on\n", "(i) tensors for data storage and linear algebra;\n", "and (ii) auto differentiation for calculating gradients.\n", "In practice, because data iterators, loss functions, optimizers,\n", "and neural network layers\n", "are so common, modern libraries implement these components for us as well.\n", "\n", "In this section, (**we will show you how to implement\n", "the linear regression model**) from :numref:`sec_linear_scratch`\n", "(**concisely by using high-level APIs**) of deep learning frameworks.\n", "\n", "\n", "## Generating the Dataset\n", "\n", "To start, we will generate the same dataset as in :numref:`sec_linear_scratch`.\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "origin_pos": 2, "tab": [ "pytorch" ] }, "outputs": [], "source": [ "import numpy as np\n", "import torch\n", "from torch.utils import data\n", "from d2l import torch as d2l" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "origin_pos": 4, "tab": [ "pytorch" ] }, "outputs": [], "source": [ "true_w = torch.tensor([2, -3.4])\n", "true_b = 4.2\n", "features, labels = d2l.synthetic_data(true_w, true_b, 1000)" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 5 }, "source": [ "## Reading the Dataset\n", "\n", "Rather than rolling our own iterator,\n", "we can [**call upon the existing API in a framework to read data.**]\n", "We pass in `features` and `labels` as arguments and specify `batch_size`\n", "when instantiating a data iterator object.\n", "Besides, the boolean value `is_train`\n", "indicates whether or not\n", "we want the data iterator object to shuffle the data\n", "on each epoch (pass through the dataset).\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "origin_pos": 7, "tab": [ "pytorch" ] }, "outputs": [], "source": [ "def load_array(data_arrays, batch_size, is_train=True): #@save\n", " \"\"\"Construct a PyTorch data iterator.\"\"\"\n", " dataset = data.TensorDataset(*data_arrays)\n", " return data.DataLoader(dataset, batch_size, shuffle=is_train)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "origin_pos": 9, "tab": [ "pytorch" ] }, "outputs": [], "source": [ "batch_size = 10\n", "data_iter = load_array((features, labels), batch_size)" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 10 }, "source": [ "Now we can use `data_iter` in much the same way as we called\n", "the `data_iter` function in :numref:`sec_linear_scratch`.\n", "To verify that it is working, we can read and print\n", "the first minibatch of examples.\n", "Comparing with :numref:`sec_linear_scratch`,\n", "here we use `iter` to construct a Python iterator and use `next` to obtain the first item from the iterator.\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "origin_pos": 11, "tab": [ "pytorch" ] }, "outputs": [ { "data": { "text/plain": [ "[tensor([[-0.0385, -1.5909],\n", " [-0.1432, 0.9797],\n", " [-1.2551, 0.1154],\n", " [ 2.0345, 0.3657],\n", " [ 0.0179, 0.4323],\n", " [-1.9013, -0.6234],\n", " [-1.7319, 0.3589],\n", " [-0.2995, 0.0825],\n", " [-1.7680, -1.3965],\n", " [-0.8613, -0.5467]]),\n", " tensor([[ 9.5418],\n", " [ 0.5854],\n", " [ 1.3041],\n", " [ 7.0231],\n", " [ 2.7599],\n", " [ 2.5210],\n", " [-0.4856],\n", " [ 3.3295],\n", " [ 5.4176],\n", " [ 4.3249]])]" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "next(iter(data_iter))" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 12 }, "source": [ "## Defining the Model\n", "\n", "When we implemented linear regression from scratch\n", "in :numref:`sec_linear_scratch`,\n", "we defined our model parameters explicitly\n", "and coded up the calculations to produce output\n", "using basic linear algebra operations.\n", "You *should* know how to do this.\n", "But once your models get more complex,\n", "and once you have to do this nearly every day,\n", "you will be glad for the assistance.\n", "The situation is similar to coding up your own blog from scratch.\n", "Doing it once or twice is rewarding and instructive,\n", "but you would be a lousy web developer\n", "if every time you needed a blog you spent a month\n", "reinventing the wheel.\n", "\n", "For standard operations, we can [**use a framework's predefined layers,**]\n", "which allow us to focus especially\n", "on the layers used to construct the model\n", "rather than having to focus on the implementation.\n", "We will first define a model variable `net`,\n", "which will refer to an instance of the `Sequential` class.\n", "The `Sequential` class defines a container\n", "for several layers that will be chained together.\n", "Given input data, a `Sequential` instance passes it through\n", "the first layer, in turn passing the output\n", "as the second layer's input and so forth.\n", "In the following example, our model consists of only one layer,\n", "so we do not really need `Sequential`.\n", "But since nearly all of our future models\n", "will involve multiple layers,\n", "we will use it anyway just to familiarize you\n", "with the most standard workflow.\n", "\n", "Recall the architecture of a single-layer network as shown in :numref:`fig_single_neuron`.\n", "The layer is said to be *fully-connected*\n", "because each of its inputs is connected to each of its outputs\n", "by means of a matrix-vector multiplication.\n" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 14, "tab": [ "pytorch" ] }, "source": [ "In PyTorch, the fully-connected layer is defined in the `Linear` class. Note that we passed two arguments into `nn.Linear`. The first one specifies the input feature dimension, which is 2, and the second one is the output feature dimension, which is a single scalar and therefore 1.\n" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "origin_pos": 17, "tab": [ "pytorch" ] }, "outputs": [], "source": [ "# `nn` is an abbreviation for neural networks\n", "from torch import nn\n", "\n", "net = nn.Sequential(nn.Linear(2, 1))" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 19 }, "source": [ "## Initializing Model Parameters\n", "\n", "Before using `net`, we need to (**initialize the model parameters,**)\n", "such as the weights and bias in the linear regression model.\n", "Deep learning frameworks often have a predefined way to initialize the parameters.\n", "Here we specify that each weight parameter\n", "should be randomly sampled from a normal distribution\n", "with mean 0 and standard deviation 0.01.\n", "The bias parameter will be initialized to zero.\n" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 21, "tab": [ "pytorch" ] }, "source": [ "As we have specified the input and output dimensions when constructing `nn.Linear`,\n", "now we can access the parameters directly to specify their initial values.\n", "We first locate the layer by `net[0]`, which is the first layer in the network,\n", "and then use the `weight.data` and `bias.data` methods to access the parameters.\n", "Next we use the replace methods `normal_` and `fill_` to overwrite parameter values.\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "origin_pos": 24, "tab": [ "pytorch" ] }, "outputs": [ { "data": { "text/plain": [ "tensor([0.])" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "net[0].weight.data.normal_(0, 0.01)\n", "net[0].bias.data.fill_(0)" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 27, "tab": [ "pytorch" ] }, "source": [ "\n" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 29 }, "source": [ "## Defining the Loss Function\n" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 31, "tab": [ "pytorch" ] }, "source": [ "[**The `MSELoss` class computes the mean squared error (without the $1/2$ factor in :eqref:`eq_mse`).**]\n", "By default it returns the average loss over examples.\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "origin_pos": 34, "tab": [ "pytorch" ] }, "outputs": [], "source": [ "loss = nn.MSELoss()" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 36 }, "source": [ "## Defining the Optimization Algorithm\n" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 38, "tab": [ "pytorch" ] }, "source": [ "Minibatch stochastic gradient descent is a standard tool\n", "for optimizing neural networks\n", "and thus PyTorch supports it alongside a number of\n", "variations on this algorithm in the `optim` module.\n", "When we (**instantiate an `SGD` instance,**)\n", "we will specify the parameters to optimize over\n", "(obtainable from our net via `net.parameters()`), with a dictionary of hyperparameters\n", "required by our optimization algorithm.\n", "Minibatch stochastic gradient descent just requires that\n", "we set the value `lr`, which is set to 0.03 here.\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "origin_pos": 41, "tab": [ "pytorch" ] }, "outputs": [], "source": [ "trainer = torch.optim.SGD(net.parameters(), lr=0.03)" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 43 }, "source": [ "## Training\n", "\n", "You might have noticed that expressing our model through\n", "high-level APIs of a deep learning framework\n", "requires comparatively few lines of code.\n", "We did not have to individually allocate parameters,\n", "define our loss function, or implement minibatch stochastic gradient descent.\n", "Once we start working with much more complex models,\n", "advantages of high-level APIs will grow considerably.\n", "However, once we have all the basic pieces in place,\n", "[**the training loop itself is strikingly similar\n", "to what we did when implementing everything from scratch.**]\n", "\n", "To refresh your memory: for some number of epochs,\n", "we will make a complete pass over the dataset (`train_data`),\n", "iteratively grabbing one minibatch of inputs\n", "and the corresponding ground-truth labels.\n", "For each minibatch, we go through the following ritual:\n", "\n", "* Generate predictions by calling `net(X)` and calculate the loss `l` (the forward propagation).\n", "* Calculate gradients by running the backpropagation.\n", "* Update the model parameters by invoking our optimizer.\n", "\n", "For good measure, we compute the loss after each epoch and print it to monitor progress.\n" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "origin_pos": 45, "tab": [ "pytorch" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "epoch 1, loss 0.000207\n", "epoch 2, loss 0.000105\n", "epoch 3, loss 0.000105\n" ] } ], "source": [ "num_epochs = 3\n", "for epoch in range(num_epochs):\n", " for X, y in data_iter:\n", " l = loss(net(X) ,y)\n", " trainer.zero_grad()\n", " l.backward()\n", " trainer.step()\n", " l = loss(net(features), labels)\n", " print(f'epoch {epoch + 1}, loss {l:f}')" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 47 }, "source": [ "Below, we [**compare the model parameters learned by training on finite data\n", "and the actual parameters**] that generated our dataset.\n", "To access parameters,\n", "we first access the layer that we need from `net`\n", "and then access that layer's weights and bias.\n", "As in our from-scratch implementation,\n", "note that our estimated parameters are\n", "close to their ground-truth counterparts.\n" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "origin_pos": 49, "tab": [ "pytorch" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "error in estimating w: tensor([-0.0005, -0.0003])\n", "error in estimating b: tensor([-0.0002])\n" ] } ], "source": [ "w = net[0].weight.data\n", "print('error in estimating w:', true_w - w.reshape(true_w.shape))\n", "b = net[0].bias.data\n", "print('error in estimating b:', true_b - b)" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 51 }, "source": [ "## Summary\n" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 53, "tab": [ "pytorch" ] }, "source": [ "* Using PyTorch's high-level APIs, we can implement models much more concisely.\n", "* In PyTorch, the `data` module provides tools for data processing, the `nn` module defines a large number of neural network layers and common loss functions.\n", "* We can initialize the parameters by replacing their values with methods ending with `_`.\n" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 55 }, "source": [ "## Exercises\n" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 57, "tab": [ "pytorch" ] }, "source": [ "1. If we replace `nn.MSELoss(reduction='sum')` with `nn.MSELoss()`, how can we change the learning rate for the code to behave identically. Why?\n", "1. Review the PyTorch documentation to see what loss functions and initialization methods are provided. Replace the loss by Huber's loss.\n", "1. How do you access the gradient of `net[0].weight`?\n", "\n", "[Discussions](https://discuss.d2l.ai/t/45)\n" ] } ], "metadata": { "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 4 }