{ "cells": [ { "cell_type": "markdown", "metadata": { "origin_pos": 0 }, "source": [ "# Pooling\n", ":label:`sec_pooling`\n", "\n", "\n", "Often, as we process images, we want to gradually\n", "reduce the spatial resolution of our hidden representations,\n", "aggregating information so that\n", "the higher up we go in the network,\n", "the larger the receptive field (in the input)\n", "to which each hidden node is sensitive.\n", "\n", "Often our ultimate task asks some global question about the image,\n", "e.g., *does it contain a cat?*\n", "So typically the units of our final layer should be sensitive\n", "to the entire input.\n", "By gradually aggregating information, yielding coarser and coarser maps,\n", "we accomplish this goal of ultimately learning a global representation,\n", "while keeping all of the advantages of convolutional layers at the intermediate layers of processing.\n", "\n", "Moreover, when detecting lower-level features, such as edges\n", "(as discussed in :numref:`sec_conv_layer`),\n", "we often want our representations to be somewhat invariant to translation.\n", "For instance, if we take the image `X`\n", "with a sharp delineation between black and white\n", "and shift the whole image by one pixel to the right,\n", "i.e., `Z[i, j] = X[i, j + 1]`,\n", "then the output for the new image `Z` might be vastly different.\n", "The edge will have shifted by one pixel.\n", "In reality, objects hardly ever occur exactly at the same place.\n", "In fact, even with a tripod and a stationary object,\n", "vibration of the camera due to the movement of the shutter\n", "might shift everything by a pixel or so\n", "(high-end cameras are loaded with special features to address this problem).\n", "\n", "This section introduces *pooling layers*,\n", "which serve the dual purposes of\n", "mitigating the sensitivity of convolutional layers to location\n", "and of spatially downsampling representations.\n", "\n", "## Maximum Pooling and Average Pooling\n", "\n", "Like convolutional layers, *pooling* operators\n", "consist of a fixed-shape window that is slid over\n", "all regions in the input according to its stride,\n", "computing a single output for each location traversed\n", "by the fixed-shape window (sometimes known as the *pooling window*).\n", "However, unlike the cross-correlation computation\n", "of the inputs and kernels in the convolutional layer,\n", "the pooling layer contains no parameters (there is no *kernel*).\n", "Instead, pooling operators are deterministic,\n", "typically calculating either the maximum or the average value\n", "of the elements in the pooling window.\n", "These operations are called *maximum pooling* (*max pooling* for short)\n", "and *average pooling*, respectively.\n", "\n", "\n", "In both cases, as with the cross-correlation operator,\n", "we can think of the pooling window\n", "as starting from the upper-left of the input tensor\n", "and sliding across the input tensor from left to right and top to bottom.\n", "At each location that the pooling window hits,\n", "it computes the maximum or average\n", "value of the input subtensor in the window,\n", "depending on whether max or average pooling is employed.\n", "\n", "\n", "![Maximum pooling with a pooling window shape of $2\\times 2$. The shaded portions are the first output element as well as the input tensor elements used for the output computation: $\\max(0, 1, 3, 4)=4$.](../img/pooling.svg)\n", ":label:`fig_pooling`\n", "\n", "The output tensor in :numref:`fig_pooling` has a height of 2 and a width of 2.\n", "The four elements are derived from the maximum value in each pooling window:\n", "\n", "$$\n", "\\max(0, 1, 3, 4)=4,\\\\\n", "\\max(1, 2, 4, 5)=5,\\\\\n", "\\max(3, 4, 6, 7)=7,\\\\\n", "\\max(4, 5, 7, 8)=8.\\\\\n", "$$\n", "\n", "A pooling layer with a pooling window shape of $p \\times q$\n", "is called a $p \\times q$ pooling layer.\n", "The pooling operation is called $p \\times q$ pooling.\n", "\n", "Let us return to the object edge detection example\n", "mentioned at the beginning of this section.\n", "Now we will use the output of the convolutional layer\n", "as the input for $2\\times 2$ maximum pooling.\n", "Set the convolutional layer input as `X` and the pooling layer output as `Y`. Whether or not the values of `X[i, j]` and `X[i, j + 1]` are different,\n", "or `X[i, j + 1]` and `X[i, j + 2]` are different,\n", "the pooling layer always outputs `Y[i, j] = 1`.\n", "That is to say, using the $2\\times 2$ maximum pooling layer,\n", "we can still detect if the pattern recognized by the convolutional layer\n", "moves no more than one element in height or width.\n", "\n", "In the code below, we (**implement the forward propagation\n", "of the pooling layer**) in the `pool2d` function.\n", "This function is similar to the `corr2d` function\n", "in :numref:`sec_conv_layer`.\n", "However, here we have no kernel, computing the output\n", "as either the maximum or the average of each region in the input.\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "origin_pos": 4, "tab": [ "tensorflow" ] }, "outputs": [], "source": [ "import tensorflow as tf\n", "\n", "\n", "def pool2d(X, pool_size, mode='max'):\n", " p_h, p_w = pool_size\n", " Y = tf.Variable(tf.zeros((X.shape[0] - p_h + 1, X.shape[1] - p_w +1)))\n", " for i in range(Y.shape[0]):\n", " for j in range(Y.shape[1]):\n", " if mode == 'max':\n", " Y[i, j].assign(tf.reduce_max(X[i: i + p_h, j: j + p_w]))\n", " elif mode =='avg':\n", " Y[i, j].assign(tf.reduce_mean(X[i: i + p_h, j: j + p_w]))\n", " return Y" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 5 }, "source": [ "We can construct the input tensor `X` in :numref:`fig_pooling` to [**validate the output of the two-dimensional maximum pooling layer**].\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "origin_pos": 6, "tab": [ "tensorflow" ] }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X = tf.constant([[0.0, 1.0, 2.0], [3.0, 4.0, 5.0], [6.0, 7.0, 8.0]])\n", "pool2d(X, (2, 2))" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 7 }, "source": [ "Also, we experiment with (**the average pooling layer**).\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "origin_pos": 8, "tab": [ "tensorflow" ] }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pool2d(X, (2, 2), 'avg')" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 9 }, "source": [ "## [**Padding and Stride**]\n", "\n", "As with convolutional layers, pooling layers\n", "can also change the output shape.\n", "And as before, we can alter the operation to achieve a desired output shape\n", "by padding the input and adjusting the stride.\n", "We can demonstrate the use of padding and strides\n", "in pooling layers via the built-in two-dimensional maximum pooling layer from the deep learning framework.\n", "We first construct an input tensor `X` whose shape has four dimensions,\n", "where the number of examples (batch size) and number of channels are both 1.\n" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 10, "tab": [ "tensorflow" ] }, "source": [ "It is important to note that tensorflow\n", "prefers and is optimized for *channels-last* input.\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "origin_pos": 12, "tab": [ "tensorflow" ] }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X = tf.reshape(tf.range(16, dtype=tf.float32), (1, 4, 4, 1))\n", "X" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 13 }, "source": [ "By default, (**the stride and the pooling window in the instance from the framework's built-in class\n", "have the same shape.**)\n", "Below, we use a pooling window of shape `(3, 3)`,\n", "so we get a stride shape of `(3, 3)` by default.\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "origin_pos": 16, "tab": [ "tensorflow" ] }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pool2d = tf.keras.layers.MaxPool2D(pool_size=[3, 3])\n", "pool2d(X)" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 17 }, "source": [ "[**The stride and padding can be manually specified.**]\n" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "origin_pos": 20, "tab": [ "tensorflow" ] }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "paddings = tf.constant([[0, 0], [1,0], [1,0], [0,0]])\n", "X_padded = tf.pad(X, paddings, \"CONSTANT\")\n", "pool2d = tf.keras.layers.MaxPool2D(pool_size=[3, 3], padding='valid',\n", " strides=2)\n", "pool2d(X_padded)" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 23, "tab": [ "tensorflow" ] }, "source": [ "Of course, we can specify an arbitrary rectangular pooling window\n", "and specify the padding and stride for height and width, respectively.\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "origin_pos": 26, "tab": [ "tensorflow" ] }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "paddings = tf.constant([[0, 0], [0, 0], [1, 1], [0, 0]])\n", "X_padded = tf.pad(X, paddings, \"CONSTANT\")\n", "\n", "pool2d = tf.keras.layers.MaxPool2D(pool_size=[2, 3], padding='valid',\n", " strides=(2, 3))\n", "pool2d(X_padded)" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 27 }, "source": [ "## Multiple Channels\n", "\n", "When processing multi-channel input data,\n", "[**the pooling layer pools each input channel separately**],\n", "rather than summing the inputs up over channels\n", "as in a convolutional layer.\n", "This means that the number of output channels for the pooling layer\n", "is the same as the number of input channels.\n", "Below, we will concatenate tensors `X` and `X + 1`\n", "on the channel dimension to construct an input with 2 channels.\n" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 28, "tab": [ "tensorflow" ] }, "source": [ "Note that this will require a\n", "concatenation along the last dimension for TensorFlow due to the channels-last syntax.\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "origin_pos": 30, "tab": [ "tensorflow" ] }, "outputs": [], "source": [ "X = tf.concat([X, X + 1], 3) # Concatenate along `dim=3` due to channels-last syntax" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 31 }, "source": [ "As we can see, the number of output channels is still 2 after pooling.\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "origin_pos": 34, "tab": [ "tensorflow" ] }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "paddings = tf.constant([[0, 0], [1,0], [1,0], [0,0]])\n", "X_padded = tf.pad(X, paddings, \"CONSTANT\")\n", "pool2d = tf.keras.layers.MaxPool2D(pool_size=[3, 3], padding='valid',\n", " strides=2)\n", "pool2d(X_padded)" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 35, "tab": [ "tensorflow" ] }, "source": [ "Note that the output for the tensorflow pooling appears at first glance to be different, however\n", "numerically the same results are presented as MXNet and PyTorch.\n", "The difference lies in the dimensionality, and reading the\n", "output vertically yields the same output as the other implementations.\n" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 36 }, "source": [ "## Summary\n", "\n", "* Taking the input elements in the pooling window, the maximum pooling operation assigns the maximum value as the output and the average pooling operation assigns the average value as the output.\n", "* One of the major benefits of a pooling layer is to alleviate the excessive sensitivity of the convolutional layer to location.\n", "* We can specify the padding and stride for the pooling layer.\n", "* Maximum pooling, combined with a stride larger than 1 can be used to reduce the spatial dimensions (e.g., width and height).\n", "* The pooling layer's number of output channels is the same as the number of input channels.\n", "\n", "\n", "## Exercises\n", "\n", "1. Can you implement average pooling as a special case of a convolution layer? If so, do it.\n", "1. Can you implement maximum pooling as a special case of a convolution layer? If so, do it.\n", "1. What is the computational cost of the pooling layer? Assume that the input to the pooling layer is of size $c\\times h\\times w$, the pooling window has a shape of $p_h\\times p_w$ with a padding of $(p_h, p_w)$ and a stride of $(s_h, s_w)$.\n", "1. Why do you expect maximum pooling and average pooling to work differently?\n", "1. Do we need a separate minimum pooling layer? Can you replace it with another operation?\n", "1. Is there another operation between average and maximum pooling that you could consider (hint: recall the softmax)? Why might it not be so popular?\n" ] }, { "cell_type": "markdown", "metadata": { "origin_pos": 39, "tab": [ "tensorflow" ] }, "source": [ "[Discussions](https://discuss.d2l.ai/t/274)\n" ] } ], "metadata": { "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 4 }