{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 0
},
"source": [
"# Linear Regression Implementation from Scratch\n",
":label:`sec_linear_scratch`\n",
"\n",
"Now that you understand the key ideas behind linear regression,\n",
"we can begin to work through a hands-on implementation in code.\n",
"In this section, (**we will implement the entire method from scratch,\n",
"including the data pipeline, the model,\n",
"the loss function, and the minibatch stochastic gradient descent optimizer.**)\n",
"While modern deep learning frameworks can automate nearly all of this work,\n",
"implementing things from scratch is the only way\n",
"to make sure that you really know what you are doing.\n",
"Moreover, when it comes time to customize models,\n",
"defining our own layers or loss functions,\n",
"understanding how things work under the hood will prove handy.\n",
"In this section, we will rely only on tensors and auto differentiation.\n",
"Afterwards, we will introduce a more concise implementation,\n",
"taking advantage of bells and whistles of deep learning frameworks.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"origin_pos": 3,
"tab": [
"tensorflow"
]
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"import random\n",
"import tensorflow as tf\n",
"from d2l import tensorflow as d2l"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 4
},
"source": [
"## Generating the Dataset\n",
"\n",
"To keep things simple, we will [**construct an artificial dataset\n",
"according to a linear model with additive noise.**]\n",
"Our task will be to recover this model's parameters\n",
"using the finite set of examples contained in our dataset.\n",
"We will keep the data low-dimensional so we can visualize it easily.\n",
"In the following code snippet, we generate a dataset\n",
"containing 1000 examples, each consisting of 2 features\n",
"sampled from a standard normal distribution.\n",
"Thus our synthetic dataset will be a matrix\n",
"$\\mathbf{X}\\in \\mathbb{R}^{1000 \\times 2}$.\n",
"\n",
"(**The true parameters generating our dataset will be\n",
"$\\mathbf{w} = [2, -3.4]^\\top$ and $b = 4.2$,\n",
"and**) our synthetic labels will be assigned according\n",
"to the following linear model with the noise term $\\epsilon$:\n",
"\n",
"(**$$\\mathbf{y}= \\mathbf{X} \\mathbf{w} + b + \\mathbf\\epsilon.$$**)\n",
"\n",
"You could think of $\\epsilon$ as capturing potential\n",
"measurement errors on the features and labels.\n",
"We will assume that the standard assumptions hold and thus\n",
"that $\\epsilon$ obeys a normal distribution with mean of 0.\n",
"To make our problem easy, we will set its standard deviation to 0.01.\n",
"The following code generates our synthetic dataset.\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"origin_pos": 6,
"tab": [
"tensorflow"
]
},
"outputs": [],
"source": [
"def synthetic_data(w, b, num_examples): #@save\n",
" \"\"\"Generate y = Xw + b + noise.\"\"\"\n",
" X = tf.zeros((num_examples, w.shape[0]))\n",
" X += tf.random.normal(shape=X.shape)\n",
" y = tf.matmul(X, tf.reshape(w, (-1, 1))) + b\n",
" y += tf.random.normal(shape=y.shape, stddev=0.01)\n",
" y = tf.reshape(y, (-1, 1))\n",
" return X, y"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"origin_pos": 7,
"tab": [
"tensorflow"
]
},
"outputs": [],
"source": [
"true_w = tf.constant([2, -3.4])\n",
"true_b = 4.2\n",
"features, labels = synthetic_data(true_w, true_b, 1000)"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 8
},
"source": [
"Note that [**each row in `features` consists of a 2-dimensional data example\n",
"and that each row in `labels` consists of a 1-dimensional label value (a scalar).**]\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"origin_pos": 9,
"tab": [
"tensorflow"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"features: tf.Tensor([ 0.8988941 -0.41589838], shape=(2,), dtype=float32) \n",
"label: tf.Tensor([7.4150643], shape=(1,), dtype=float32)\n"
]
}
],
"source": [
"print('features:', features[0],'\\nlabel:', labels[0])"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 10
},
"source": [
"By generating a scatter plot using the second feature `features[:, 1]` and `labels`,\n",
"we can clearly observe the linear correlation between the two.\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"origin_pos": 11,
"tab": [
"tensorflow"
]
},
"outputs": [
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"d2l.set_figsize()\n",
"# The semicolon is for displaying the plot only\n",
"d2l.plt.scatter(features[:, (1)].numpy(), labels.numpy(), 1);"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 12
},
"source": [
"## Reading the Dataset\n",
"\n",
"Recall that training models consists of\n",
"making multiple passes over the dataset,\n",
"grabbing one minibatch of examples at a time,\n",
"and using them to update our model.\n",
"Since this process is so fundamental\n",
"to training machine learning algorithms,\n",
"it is worth defining a utility function\n",
"to shuffle the dataset and access it in minibatches.\n",
"\n",
"In the following code, we [**define the `data_iter` function**] (~~that~~)\n",
"to demonstrate one possible implementation of this functionality.\n",
"The function (**takes a batch size, a matrix of features,\n",
"and a vector of labels, yielding minibatches of the size `batch_size`.**)\n",
"Each minibatch consists of a tuple of features and labels.\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"origin_pos": 14,
"tab": [
"tensorflow"
]
},
"outputs": [],
"source": [
"def data_iter(batch_size, features, labels):\n",
" num_examples = len(features)\n",
" indices = list(range(num_examples))\n",
" # The examples are read at random, in no particular order\n",
" random.shuffle(indices)\n",
" for i in range(0, num_examples, batch_size):\n",
" j = tf.constant(indices[i: min(i + batch_size, num_examples)])\n",
" yield tf.gather(features, j), tf.gather(labels, j)"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 15
},
"source": [
"In general, note that we want to use reasonably sized minibatches\n",
"to take advantage of the GPU hardware,\n",
"which excels at parallelizing operations.\n",
"Because each example can be fed through our models in parallel\n",
"and the gradient of the loss function for each example can also be taken in parallel,\n",
"GPUs allow us to process hundreds of examples in scarcely more time\n",
"than it might take to process just a single example.\n",
"\n",
"To build some intuition, let us read and print\n",
"the first small batch of data examples.\n",
"The shape of the features in each minibatch tells us\n",
"both the minibatch size and the number of input features.\n",
"Likewise, our minibatch of labels will have a shape given by `batch_size`.\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"origin_pos": 16,
"tab": [
"tensorflow"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tf.Tensor(\n",
"[[-0.41198364 -0.995215 ]\n",
" [-0.23361789 -1.533465 ]\n",
" [ 1.7380656 -0.03695551]\n",
" [-0.08670451 -1.4108223 ]\n",
" [ 0.36844295 -0.8031616 ]\n",
" [-1.1682457 -0.82453495]\n",
" [-2.1918385 0.565052 ]\n",
" [ 1.249257 0.48241305]\n",
" [ 1.4234517 0.5527354 ]\n",
" [-0.4905975 1.9714203 ]], shape=(10, 2), dtype=float32) \n",
" tf.Tensor(\n",
"[[ 6.762945 ]\n",
" [ 8.9574995]\n",
" [ 7.7989793]\n",
" [ 8.819844 ]\n",
" [ 7.670209 ]\n",
" [ 4.658944 ]\n",
" [-2.1091151]\n",
" [ 5.0411334]\n",
" [ 5.1664157]\n",
" [-3.4750023]], shape=(10, 1), dtype=float32)\n"
]
}
],
"source": [
"batch_size = 10\n",
"\n",
"for X, y in data_iter(batch_size, features, labels):\n",
" print(X, '\\n', y)\n",
" break"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 17
},
"source": [
"As we run the iteration, we obtain distinct minibatches\n",
"successively until the entire dataset has been exhausted (try this).\n",
"While the iteration implemented above is good for didactic purposes,\n",
"it is inefficient in ways that might get us in trouble on real problems.\n",
"For example, it requires that we load all the data in memory\n",
"and that we perform lots of random memory access.\n",
"The built-in iterators implemented in a deep learning framework\n",
"are considerably more efficient and they can deal\n",
"with both data stored in files and data fed via data streams.\n",
"\n",
"\n",
"## Initializing Model Parameters\n",
"\n",
"[**Before we can begin optimizing our model's parameters**] by minibatch stochastic gradient descent,\n",
"(**we need to have some parameters in the first place.**)\n",
"In the following code, we initialize weights by sampling\n",
"random numbers from a normal distribution with mean 0\n",
"and a standard deviation of 0.01, and setting the bias to 0.\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"origin_pos": 20,
"tab": [
"tensorflow"
]
},
"outputs": [],
"source": [
"w = tf.Variable(tf.random.normal(shape=(2, 1), mean=0, stddev=0.01),\n",
" trainable=True)\n",
"b = tf.Variable(tf.zeros(1), trainable=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 21
},
"source": [
"After initializing our parameters,\n",
"our next task is to update them until\n",
"they fit our data sufficiently well.\n",
"Each update requires taking the gradient\n",
"of our loss function with respect to the parameters.\n",
"Given this gradient, we can update each parameter\n",
"in the direction that may reduce the loss.\n",
"\n",
"Since nobody wants to compute gradients explicitly\n",
"(this is tedious and error prone),\n",
"we use automatic differentiation,\n",
"as introduced in :numref:`sec_autograd`, to compute the gradient.\n",
"\n",
"\n",
"## Defining the Model\n",
"\n",
"Next, we must [**define our model,\n",
"relating its inputs and parameters to its outputs.**]\n",
"Recall that to calculate the output of the linear model,\n",
"we simply take the matrix-vector dot product\n",
"of the input features $\\mathbf{X}$ and the model weights $\\mathbf{w}$,\n",
"and add the offset $b$ to each example.\n",
"Note that below $\\mathbf{Xw}$ is a vector and $b$ is a scalar.\n",
"Recall the broadcasting mechanism as described in :numref:`subsec_broadcasting`.\n",
"When we add a vector and a scalar,\n",
"the scalar is added to each component of the vector.\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"origin_pos": 22,
"tab": [
"tensorflow"
]
},
"outputs": [],
"source": [
"def linreg(X, w, b): #@save\n",
" \"\"\"The linear regression model.\"\"\"\n",
" return tf.matmul(X, w) + b"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 23
},
"source": [
"## Defining the Loss Function\n",
"\n",
"Since [**updating our model requires taking\n",
"the gradient of our loss function,**]\n",
"we ought to (**define the loss function first.**)\n",
"Here we will use the squared loss function\n",
"as described in :numref:`sec_linear_regression`.\n",
"In the implementation, we need to transform the true value `y`\n",
"into the predicted value's shape `y_hat`.\n",
"The result returned by the following function\n",
"will also have the same shape as `y_hat`.\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"origin_pos": 24,
"tab": [
"tensorflow"
]
},
"outputs": [],
"source": [
"def squared_loss(y_hat, y): #@save\n",
" \"\"\"Squared loss.\"\"\"\n",
" return (y_hat - tf.reshape(y, y_hat.shape)) ** 2 / 2"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 25
},
"source": [
"## Defining the Optimization Algorithm\n",
"\n",
"As we discussed in :numref:`sec_linear_regression`,\n",
"linear regression has a closed-form solution.\n",
"However, this is not a book about linear regression:\n",
"it is a book about deep learning.\n",
"Since none of the other models that this book introduces\n",
"can be solved analytically, we will take this opportunity to introduce your first working example of\n",
"minibatch stochastic gradient descent.\n",
"[~~Despite linear regression has a closed-form solution, other models in this book don't. Here we introduce minibatch stochastic gradient descent.~~]\n",
"\n",
"At each step, using one minibatch randomly drawn from our dataset,\n",
"we will estimate the gradient of the loss with respect to our parameters.\n",
"Next, we will update our parameters\n",
"in the direction that may reduce the loss.\n",
"The following code applies the minibatch stochastic gradient descent update,\n",
"given a set of parameters, a learning rate, and a batch size.\n",
"The size of the update step is determined by the learning rate `lr`.\n",
"Because our loss is calculated as a sum over the minibatch of examples,\n",
"we normalize our step size by the batch size (`batch_size`),\n",
"so that the magnitude of a typical step size\n",
"does not depend heavily on our choice of the batch size.\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"origin_pos": 28,
"tab": [
"tensorflow"
]
},
"outputs": [],
"source": [
"def sgd(params, grads, lr, batch_size): #@save\n",
" \"\"\"Minibatch stochastic gradient descent.\"\"\"\n",
" for param, grad in zip(params, grads):\n",
" param.assign_sub(lr*grad/batch_size)"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 29
},
"source": [
"## Training\n",
"\n",
"Now that we have all of the parts in place,\n",
"we are ready to [**implement the main training loop.**]\n",
"It is crucial that you understand this code\n",
"because you will see nearly identical training loops\n",
"over and over again throughout your career in deep learning.\n",
"\n",
"In each iteration, we will grab a minibatch of training examples,\n",
"and pass them through our model to obtain a set of predictions.\n",
"After calculating the loss, we initiate the backwards pass through the network,\n",
"storing the gradients with respect to each parameter.\n",
"Finally, we will call the optimization algorithm `sgd`\n",
"to update the model parameters.\n",
"\n",
"In summary, we will execute the following loop:\n",
"\n",
"* Initialize parameters $(\\mathbf{w}, b)$\n",
"* Repeat until done\n",
" * Compute gradient $\\mathbf{g} \\leftarrow \\partial_{(\\mathbf{w},b)} \\frac{1}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}} l(\\mathbf{x}^{(i)}, y^{(i)}, \\mathbf{w}, b)$\n",
" * Update parameters $(\\mathbf{w}, b) \\leftarrow (\\mathbf{w}, b) - \\eta \\mathbf{g}$\n",
"\n",
"In each *epoch*,\n",
"we will iterate through the entire dataset\n",
"(using the `data_iter` function) once\n",
"passing through every example in the training dataset\n",
"(assuming that the number of examples is divisible by the batch size).\n",
"The number of epochs `num_epochs` and the learning rate `lr` are both hyperparameters,\n",
"which we set here to 3 and 0.03, respectively.\n",
"Unfortunately, setting hyperparameters is tricky\n",
"and requires some adjustment by trial and error.\n",
"We elide these details for now but revise them\n",
"later in\n",
":numref:`chap_optimization`.\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"origin_pos": 30,
"tab": [
"tensorflow"
]
},
"outputs": [],
"source": [
"lr = 0.03\n",
"num_epochs = 3\n",
"net = linreg\n",
"loss = squared_loss"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"origin_pos": 33,
"tab": [
"tensorflow"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"epoch 1, loss 0.031967\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"epoch 2, loss 0.000111\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"epoch 3, loss 0.000049\n"
]
}
],
"source": [
"for epoch in range(num_epochs):\n",
" for X, y in data_iter(batch_size, features, labels):\n",
" with tf.GradientTape() as g:\n",
" l = loss(net(X, w, b), y) # Minibatch loss in `X` and `y`\n",
" # Compute gradient on l with respect to [`w`, `b`]\n",
" dw, db = g.gradient(l, [w, b])\n",
" # Update parameters using their gradient\n",
" sgd([w, b], [dw, db], lr, batch_size)\n",
" train_l = loss(net(features, w, b), labels)\n",
" print(f'epoch {epoch + 1}, loss {float(tf.reduce_mean(train_l)):f}')"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 34
},
"source": [
"In this case, because we synthesized the dataset ourselves,\n",
"we know precisely what the true parameters are.\n",
"Thus, we can [**evaluate our success in training\n",
"by comparing the true parameters\n",
"with those that we learned**] through our training loop.\n",
"Indeed they turn out to be very close to each other.\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"origin_pos": 35,
"tab": [
"tensorflow"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"error in estimating w: [-6.4849854e-05 -2.1266937e-04]\n",
"error in estimating b: [0.00053024]\n"
]
}
],
"source": [
"print(f'error in estimating w: {true_w - tf.reshape(w, true_w.shape)}')\n",
"print(f'error in estimating b: {true_b - b}')"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 36
},
"source": [
"Note that we should not take it for granted\n",
"that we are able to recover the parameters perfectly.\n",
"However, in machine learning, we are typically less concerned\n",
"with recovering true underlying parameters,\n",
"and more concerned with parameters that lead to highly accurate prediction.\n",
"Fortunately, even on difficult optimization problems,\n",
"stochastic gradient descent can often find remarkably good solutions,\n",
"owing partly to the fact that, for deep networks,\n",
"there exist many configurations of the parameters\n",
"that lead to highly accurate prediction.\n",
"\n",
"\n",
"## Summary\n",
"\n",
"* We saw how a deep network can be implemented and optimized from scratch, using just tensors and auto differentiation, without any need for defining layers or fancy optimizers.\n",
"* This section only scratches the surface of what is possible. In the following sections, we will describe additional models based on the concepts that we have just introduced and learn how to implement them more concisely.\n",
"\n",
"\n",
"## Exercises\n",
"\n",
"1. What would happen if we were to initialize the weights to zero. Would the algorithm still work?\n",
"1. Assume that you are\n",
" [Georg Simon Ohm](https://en.wikipedia.org/wiki/Georg_Ohm) trying to come up\n",
" with a model between voltage and current. Can you use auto differentiation to learn the parameters of your model?\n",
"1. Can you use [Planck's Law](https://en.wikipedia.org/wiki/Planck%27s_law) to determine the temperature of an object using spectral energy density?\n",
"1. What are the problems you might encounter if you wanted to compute the second derivatives? How would you fix them?\n",
"1. Why is the `reshape` function needed in the `squared_loss` function?\n",
"1. Experiment using different learning rates to find out how fast the loss function value drops.\n",
"1. If the number of examples cannot be divided by the batch size, what happens to the `data_iter` function's behavior?\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 39,
"tab": [
"tensorflow"
]
},
"source": [
"[Discussions](https://discuss.d2l.ai/t/201)\n"
]
}
],
"metadata": {
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 4
}