{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 0
   },
   "source": [
    "# Image Classification (CIFAR-10) on Kaggle\n",
    ":label:`sec_kaggle_cifar10`\n",
    "\n",
    "So far, we have been using high-level APIs of deep learning frameworks to directly obtain image datasets in tensor format.\n",
    "However, custom image datasets\n",
    "often come in the form of image files.\n",
    "In this section, we will start from\n",
    "raw image files,\n",
    "and organize, read, then transform them\n",
    "into tensor format step by step.\n",
    "\n",
    "We experimented with the CIFAR-10 dataset in :numref:`sec_image_augmentation`,\n",
    "which is an important dataset in computer vision.\n",
    "In this section,\n",
    "we will apply the knowledge we learned\n",
    "in previous sections\n",
    "to practice the Kaggle competition of\n",
    "CIFAR-10 image classification.\n",
    "(**The web address of the competition is https://www.kaggle.com/c/cifar-10**)\n",
    "\n",
    ":numref:`fig_kaggle_cifar10` shows the information on the competition's webpage.\n",
    "In order to submit the results,\n",
    "you need to register a Kaggle account.\n",
    "\n",
    "![CIFAR-10 image classification competition webpage information. The competition dataset can be obtained by clicking the \"Data\" tab.](../img/kaggle-cifar10.png)\n",
    ":width:`600px`\n",
    ":label:`fig_kaggle_cifar10`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "origin_pos": 1,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [],
   "source": [
    "import collections\n",
    "import math\n",
    "import os\n",
    "import shutil\n",
    "import pandas as pd\n",
    "from mxnet import gluon, init, npx\n",
    "from mxnet.gluon import nn\n",
    "from d2l import mxnet as d2l\n",
    "\n",
    "npx.set_np()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 3
   },
   "source": [
    "## Obtaining and Organizing the Dataset\n",
    "\n",
    "The competition dataset is divided into\n",
    "a training set and a test set,\n",
    "which contain 50000 and 300000 images, respectively.\n",
    "In the test set,\n",
    "10000 images will be used for evaluation,\n",
    "while the remaining 290000 images will not\n",
    "be evaluated:\n",
    "they are included just\n",
    "to make it hard\n",
    "to cheat with\n",
    "*manually* labeled results of the test set.\n",
    "The images in this dataset\n",
    "are all png color (RGB channels) image files,\n",
    "whose height and width are both 32 pixels.\n",
    "The images cover a total of 10 categories, namely airplanes, cars, birds, cats, deer, dogs, frogs, horses, boats, and trucks.\n",
    "The upper-left corner of :numref:`fig_kaggle_cifar10` shows some images of airplanes, cars, and birds in the dataset.\n",
    "\n",
    "\n",
    "### Downloading the Dataset\n",
    "\n",
    "After logging in to Kaggle, we can click the \"Data\" tab on the CIFAR-10 image classification competition webpage shown in :numref:`fig_kaggle_cifar10` and download the dataset by clicking the \"Download All\" button.\n",
    "After unzipping the downloaded file in `../data`, and unzipping `train.7z` and `test.7z` inside it, you will find the entire dataset in the following paths:\n",
    "\n",
    "* `../data/cifar-10/train/[1-50000].png`\n",
    "* `../data/cifar-10/test/[1-300000].png`\n",
    "* `../data/cifar-10/trainLabels.csv`\n",
    "* `../data/cifar-10/sampleSubmission.csv`\n",
    "\n",
    "where the `train` and `test` directories contain the training and testing images, respectively, `trainLabels.csv` provides labels for the training images, and `sample_submission.csv` is a sample submission file.\n",
    "\n",
    "To make it easier to get started, [**we provide a small-scale sample of the dataset that\n",
    "contains the first 1000 training images and 5 random testing images.**]\n",
    "To use the full dataset of the Kaggle competition, you need to set the following `demo` variable to `False`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "origin_pos": 4,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading ../data/kaggle_cifar10_tiny.zip from http://d2l-data.s3-accelerate.amazonaws.com/kaggle_cifar10_tiny.zip...\n"
     ]
    }
   ],
   "source": [
    "#@save\n",
    "d2l.DATA_HUB['cifar10_tiny'] = (d2l.DATA_URL + 'kaggle_cifar10_tiny.zip',\n",
    "                                '2068874e4b9a9f0fb07ebe0ad2b29754449ccacd')\n",
    "\n",
    "# If you use the full dataset downloaded for the Kaggle competition, set\n",
    "# `demo` to False\n",
    "demo = True\n",
    "\n",
    "if demo:\n",
    "    data_dir = d2l.download_extract('cifar10_tiny')\n",
    "else:\n",
    "    data_dir = '../data/cifar-10/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 5
   },
   "source": [
    "### [**Organizing the Dataset**]\n",
    "\n",
    "We need to organize datasets to facilitate model training and testing.\n",
    "Let us first read the labels from the csv file.\n",
    "The following function returns a dictionary that maps\n",
    "the non-extension part of the filename to its label.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "origin_pos": 6,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# training examples: 1000\n",
      "# classes: 10\n"
     ]
    }
   ],
   "source": [
    "#@save\n",
    "def read_csv_labels(fname):\n",
    "    \"\"\"Read `fname` to return a filename to label dictionary.\"\"\"\n",
    "    with open(fname, 'r') as f:\n",
    "        # Skip the file header line (column name)\n",
    "        lines = f.readlines()[1:]\n",
    "    tokens = [l.rstrip().split(',') for l in lines]\n",
    "    return dict(((name, label) for name, label in tokens))\n",
    "\n",
    "labels = read_csv_labels(os.path.join(data_dir, 'trainLabels.csv'))\n",
    "print('# training examples:', len(labels))\n",
    "print('# classes:', len(set(labels.values())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 7
   },
   "source": [
    "Next, we define the `reorg_train_valid` function to [**split the validation set out of the original training set.**]\n",
    "The argument `valid_ratio` in this function is the ratio of the number of examples in the validation set to the number of examples in the original training set.\n",
    "More concretely,\n",
    "let $n$ be the number of images of the class with the least examples, and $r$ be the ratio.\n",
    "The validation set will split out\n",
    "$\\max(\\lfloor nr\\rfloor,1)$ images for each class.\n",
    "Let us use `valid_ratio=0.1` as an example. Since the original training set has 50000 images,\n",
    "there will be 45000 images used for training in the path `train_valid_test/train`,\n",
    "while the other 5000 images will be split out\n",
    "as validation set in the path `train_valid_test/valid`. After organizing the dataset, images of the same class will be placed under the same folder.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "origin_pos": 8,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "def copyfile(filename, target_dir):\n",
    "    \"\"\"Copy a file into a target directory.\"\"\"\n",
    "    os.makedirs(target_dir, exist_ok=True)\n",
    "    shutil.copy(filename, target_dir)\n",
    "\n",
    "#@save\n",
    "def reorg_train_valid(data_dir, labels, valid_ratio):\n",
    "    \"\"\"Split the validation set out of the original training set.\"\"\"\n",
    "    # The number of examples of the class that has the fewest examples in the\n",
    "    # training dataset\n",
    "    n = collections.Counter(labels.values()).most_common()[-1][1]\n",
    "    # The number of examples per class for the validation set\n",
    "    n_valid_per_label = max(1, math.floor(n * valid_ratio))\n",
    "    label_count = {}\n",
    "    for train_file in os.listdir(os.path.join(data_dir, 'train')):\n",
    "        label = labels[train_file.split('.')[0]]\n",
    "        fname = os.path.join(data_dir, 'train', train_file)\n",
    "        copyfile(fname, os.path.join(data_dir, 'train_valid_test',\n",
    "                                     'train_valid', label))\n",
    "        if label not in label_count or label_count[label] < n_valid_per_label:\n",
    "            copyfile(fname, os.path.join(data_dir, 'train_valid_test',\n",
    "                                         'valid', label))\n",
    "            label_count[label] = label_count.get(label, 0) + 1\n",
    "        else:\n",
    "            copyfile(fname, os.path.join(data_dir, 'train_valid_test',\n",
    "                                         'train', label))\n",
    "    return n_valid_per_label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 9
   },
   "source": [
    "The `reorg_test` function below [**organizes the testing set for data loading during prediction.**]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "origin_pos": 10,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "def reorg_test(data_dir):\n",
    "    \"\"\"Organize the testing set for data loading during prediction.\"\"\"\n",
    "    for test_file in os.listdir(os.path.join(data_dir, 'test')):\n",
    "        copyfile(os.path.join(data_dir, 'test', test_file),\n",
    "                 os.path.join(data_dir, 'train_valid_test', 'test',\n",
    "                              'unknown'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 11
   },
   "source": [
    "Finally, we use a function to [**invoke**]\n",
    "the `read_csv_labels`, `reorg_train_valid`, and `reorg_test` (**functions defined above.**)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "origin_pos": 12,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [],
   "source": [
    "def reorg_cifar10_data(data_dir, valid_ratio):\n",
    "    labels = read_csv_labels(os.path.join(data_dir, 'trainLabels.csv'))\n",
    "    reorg_train_valid(data_dir, labels, valid_ratio)\n",
    "    reorg_test(data_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 13
   },
   "source": [
    "Here we only set the batch size to 32 for the small-scale sample of the dataset.\n",
    "When training and testing\n",
    "the complete dataset of the Kaggle competition,\n",
    "`batch_size` should be set to a larger integer, such as 128.\n",
    "We split out 10% of the training examples as the validation set for tuning hyperparameters.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "origin_pos": 14,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [],
   "source": [
    "batch_size = 32 if demo else 128\n",
    "valid_ratio = 0.1\n",
    "reorg_cifar10_data(data_dir, valid_ratio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 15
   },
   "source": [
    "## [**Image Augmentation**]\n",
    "\n",
    "We use image augmentation to address overfitting.\n",
    "For example, images can be flipped horizontally at random during training.\n",
    "We can also perform standardization for the three RGB channels of color images. Below lists some of these operations that you can tweak.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "origin_pos": 16,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [],
   "source": [
    "transform_train = gluon.data.vision.transforms.Compose([\n",
    "    # Scale the image up to a square of 40 pixels in both height and width\n",
    "    gluon.data.vision.transforms.Resize(40),\n",
    "    # Randomly crop a square image of 40 pixels in both height and width to\n",
    "    # produce a small square of 0.64 to 1 times the area of the original\n",
    "    # image, and then scale it to a square of 32 pixels in both height and\n",
    "    # width\n",
    "    gluon.data.vision.transforms.RandomResizedCrop(32, scale=(0.64, 1.0),\n",
    "                                                   ratio=(1.0, 1.0)),\n",
    "    gluon.data.vision.transforms.RandomFlipLeftRight(),\n",
    "    gluon.data.vision.transforms.ToTensor(),\n",
    "    # Standardize each channel of the image\n",
    "    gluon.data.vision.transforms.Normalize([0.4914, 0.4822, 0.4465],\n",
    "                                           [0.2023, 0.1994, 0.2010])])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 18
   },
   "source": [
    "During testing,\n",
    "we only perform standardization on images\n",
    "so as to\n",
    "remove randomness in the evaluation results.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "origin_pos": 19,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [],
   "source": [
    "transform_test = gluon.data.vision.transforms.Compose([\n",
    "    gluon.data.vision.transforms.ToTensor(),\n",
    "    gluon.data.vision.transforms.Normalize([0.4914, 0.4822, 0.4465],\n",
    "                                           [0.2023, 0.1994, 0.2010])])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 21
   },
   "source": [
    "## Reading the Dataset\n",
    "\n",
    "Next, we [**read the organized dataset consisting of raw image files**]. Each example includes an image and a label.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "origin_pos": 22,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [],
   "source": [
    "train_ds, valid_ds, train_valid_ds, test_ds = [\n",
    "    gluon.data.vision.ImageFolderDataset(\n",
    "        os.path.join(data_dir, 'train_valid_test', folder))\n",
    "    for folder in ['train', 'valid', 'train_valid', 'test']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 24
   },
   "source": [
    "During training,\n",
    "we need to [**specify all the image augmentation operations defined above**].\n",
    "When the validation set\n",
    "is used for model evaluation during hyperparameter tuning,\n",
    "no randomness from image augmentation should be introduced.\n",
    "Before final prediction,\n",
    "we train the model on the combined training set and validation set to make full use of all the labeled data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "origin_pos": 25,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [],
   "source": [
    "train_iter, train_valid_iter = [gluon.data.DataLoader(\n",
    "    dataset.transform_first(transform_train), batch_size, shuffle=True,\n",
    "    last_batch='discard') for dataset in (train_ds, train_valid_ds)]\n",
    "\n",
    "valid_iter = gluon.data.DataLoader(\n",
    "    valid_ds.transform_first(transform_test), batch_size, shuffle=False,\n",
    "    last_batch='discard')\n",
    "\n",
    "test_iter = gluon.data.DataLoader(\n",
    "    test_ds.transform_first(transform_test), batch_size, shuffle=False,\n",
    "    last_batch='keep')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 27
   },
   "source": [
    "## Defining the [**Model**]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 28,
    "tab": [
     "mxnet"
    ]
   },
   "source": [
    "Here, we build the residual blocks based on the `HybridBlock` class, which is\n",
    "slightly different from the implementation described in\n",
    ":numref:`sec_resnet`.\n",
    "This is for improving computational efficiency.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "origin_pos": 29,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [],
   "source": [
    "class Residual(nn.HybridBlock):\n",
    "    def __init__(self, num_channels, use_1x1conv=False, strides=1, **kwargs):\n",
    "        super(Residual, self).__init__(**kwargs)\n",
    "        self.conv1 = nn.Conv2D(num_channels, kernel_size=3, padding=1,\n",
    "                               strides=strides)\n",
    "        self.conv2 = nn.Conv2D(num_channels, kernel_size=3, padding=1)\n",
    "        if use_1x1conv:\n",
    "            self.conv3 = nn.Conv2D(num_channels, kernel_size=1,\n",
    "                                   strides=strides)\n",
    "        else:\n",
    "            self.conv3 = None\n",
    "        self.bn1 = nn.BatchNorm()\n",
    "        self.bn2 = nn.BatchNorm()\n",
    "\n",
    "    def hybrid_forward(self, F, X):\n",
    "        Y = F.npx.relu(self.bn1(self.conv1(X)))\n",
    "        Y = self.bn2(self.conv2(Y))\n",
    "        if self.conv3:\n",
    "            X = self.conv3(X)\n",
    "        return F.npx.relu(Y + X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 30,
    "tab": [
     "mxnet"
    ]
   },
   "source": [
    "Next, we define the ResNet-18 model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "origin_pos": 31,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [],
   "source": [
    "def resnet18(num_classes):\n",
    "    net = nn.HybridSequential()\n",
    "    net.add(nn.Conv2D(64, kernel_size=3, strides=1, padding=1),\n",
    "            nn.BatchNorm(), nn.Activation('relu'))\n",
    "\n",
    "    def resnet_block(num_channels, num_residuals, first_block=False):\n",
    "        blk = nn.HybridSequential()\n",
    "        for i in range(num_residuals):\n",
    "            if i == 0 and not first_block:\n",
    "                blk.add(Residual(num_channels, use_1x1conv=True, strides=2))\n",
    "            else:\n",
    "                blk.add(Residual(num_channels))\n",
    "        return blk\n",
    "\n",
    "    net.add(resnet_block(64, 2, first_block=True),\n",
    "            resnet_block(128, 2),\n",
    "            resnet_block(256, 2),\n",
    "            resnet_block(512, 2))\n",
    "    net.add(nn.GlobalAvgPool2D(), nn.Dense(num_classes))\n",
    "    return net"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 32,
    "tab": [
     "mxnet"
    ]
   },
   "source": [
    "We use Xavier initialization described in :numref:`subsec_xavier` before training begins.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "origin_pos": 34,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [],
   "source": [
    "def get_net(devices):\n",
    "    num_classes = 10\n",
    "    net = resnet18(num_classes)\n",
    "    net.initialize(ctx=devices, init=init.Xavier())\n",
    "    return net\n",
    "\n",
    "loss = gluon.loss.SoftmaxCrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 36
   },
   "source": [
    "## Defining the [**Training Function**]\n",
    "\n",
    "We will select models and tune hyperparameters according to the model's performance on the validation set.\n",
    "In the following, we define the model training function `train`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "origin_pos": 37,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [],
   "source": [
    "def train(net, train_iter, valid_iter, num_epochs, lr, wd, devices, lr_period,\n",
    "          lr_decay):\n",
    "    trainer = gluon.Trainer(net.collect_params(), 'sgd',\n",
    "                            {'learning_rate': lr, 'momentum': 0.9, 'wd': wd})\n",
    "    num_batches, timer = len(train_iter), d2l.Timer()\n",
    "    legend = ['train loss', 'train acc']\n",
    "    if valid_iter is not None:\n",
    "        legend.append('valid acc')\n",
    "    animator = d2l.Animator(xlabel='epoch', xlim=[1, num_epochs],\n",
    "                            legend=legend)\n",
    "    for epoch in range(num_epochs):\n",
    "        metric = d2l.Accumulator(3)\n",
    "        if epoch > 0 and epoch % lr_period == 0:\n",
    "            trainer.set_learning_rate(trainer.learning_rate * lr_decay)\n",
    "        for i, (features, labels) in enumerate(train_iter):\n",
    "            timer.start()\n",
    "            l, acc = d2l.train_batch_ch13(\n",
    "                net, features, labels.astype('float32'), loss, trainer,\n",
    "                devices, d2l.split_batch)\n",
    "            metric.add(l, acc, labels.shape[0])\n",
    "            timer.stop()\n",
    "            if (i + 1) % (num_batches // 5) == 0 or i == num_batches - 1:\n",
    "                animator.add(epoch + (i + 1) / num_batches,\n",
    "                             (metric[0] / metric[2], metric[1] / metric[2],\n",
    "                              None))\n",
    "        if valid_iter is not None:\n",
    "            valid_acc = d2l.evaluate_accuracy_gpus(net, valid_iter,\n",
    "                                                   d2l.split_batch)\n",
    "            animator.add(epoch + 1, (None, None, valid_acc))\n",
    "    measures = (f'train loss {metric[0] / metric[2]:.3f}, '\n",
    "                f'train acc {metric[1] / metric[2]:.3f}')\n",
    "    if valid_iter is not None:\n",
    "        measures += f', valid acc {valid_acc:.3f}'\n",
    "    print(measures + f'\\n{metric[2] * num_epochs / timer.sum():.1f}'\n",
    "          f' examples/sec on {str(devices)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 39
   },
   "source": [
    "## [**Training and Validating the Model**]\n",
    "\n",
    "Now, we can train and validate the model.\n",
    "All the following hyperparameters can be tuned.\n",
    "For example, we can increase the number of epochs.\n",
    "When `lr_period` and `lr_decay` are set to 4 and 0.9, respectively, the learning rate of the optimization algorithm will be multiplied by 0.9 after every 4 epochs. Just for ease of demonstration,\n",
    "we only train 20 epochs here.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "origin_pos": 40,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train loss 0.838, train acc 0.701, valid acc 0.391\n",
      "1103.7 examples/sec on [gpu(0), gpu(1)]\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"229.425pt\" height=\"180.65625pt\" viewBox=\"0 0 229.425 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2022-03-24T11:10:34.967618</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 180.65625 \n",
       "L 229.425 180.65625 \n",
       "L 229.425 0 \n",
       "L 0 0 \n",
       "L 0 180.65625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 20.5625 143.1 \n",
       "L 215.8625 143.1 \n",
       "L 215.8625 7.2 \n",
       "L 20.5625 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 61.678289 143.1 \n",
       "L 61.678289 7.2 \n",
       "\" clip-path=\"url(#p9321c92340)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m8940567eaa\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8940567eaa\" x=\"61.678289\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(58.497039 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 113.073026 143.1 \n",
       "L 113.073026 7.2 \n",
       "\" clip-path=\"url(#p9321c92340)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8940567eaa\" x=\"113.073026\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(106.710526 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 164.467763 143.1 \n",
       "L 164.467763 7.2 \n",
       "\" clip-path=\"url(#p9321c92340)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8940567eaa\" x=\"164.467763\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 15 -->\n",
       "      <g transform=\"translate(158.105263 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 215.8625 143.1 \n",
       "L 215.8625 7.2 \n",
       "\" clip-path=\"url(#p9321c92340)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8940567eaa\" x=\"215.8625\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(209.5 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_5\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(102.984375 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 20.5625 100.306229 \n",
       "L 215.8625 100.306229 \n",
       "\" clip-path=\"url(#p9321c92340)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <defs>\n",
       "       <path id=\"mead1053204\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mead1053204\" x=\"20.5625\" y=\"100.306229\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- 1 -->\n",
       "      <g transform=\"translate(7.2 104.105448)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 20.5625 56.989115 \n",
       "L 215.8625 56.989115 \n",
       "\" clip-path=\"url(#p9321c92340)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mead1053204\" x=\"20.5625\" y=\"56.989115\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(7.2 60.788334)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 20.5625 13.672001 \n",
       "L 215.8625 13.672001 \n",
       "\" clip-path=\"url(#p9321c92340)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mead1053204\" x=\"20.5625\" y=\"13.672001\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 3 -->\n",
       "      <g transform=\"translate(7.2 17.47122)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_15\">\n",
       "    <path d=\"M 12.119079 23.694445 \n",
       "L 13.954605 16.532701 \n",
       "L 15.790132 13.377273 \n",
       "L 17.625658 13.428299 \n",
       "L 19.461184 16.055478 \n",
       "L 20.5625 17.502276 \n",
       "L 22.398026 29.0702 \n",
       "L 24.233553 38.132595 \n",
       "L 26.069079 41.249413 \n",
       "L 27.904605 44.172939 \n",
       "L 29.740132 44.508694 \n",
       "L 30.841447 44.582267 \n",
       "L 32.676974 57.530706 \n",
       "L 34.5125 55.371329 \n",
       "L 36.348026 55.291653 \n",
       "L 38.183553 54.920367 \n",
       "L 40.019079 55.27832 \n",
       "L 41.120395 54.96546 \n",
       "L 42.955921 63.701601 \n",
       "L 44.791447 61.133871 \n",
       "L 46.626974 61.952782 \n",
       "L 48.4625 62.138139 \n",
       "L 50.298026 62.679647 \n",
       "L 51.399342 62.560327 \n",
       "L 53.234868 60.338425 \n",
       "L 55.070395 64.658508 \n",
       "L 56.905921 65.39075 \n",
       "L 58.741447 65.938711 \n",
       "L 60.576974 67.147043 \n",
       "L 61.678289 67.398911 \n",
       "L 63.513816 76.76787 \n",
       "L 65.349342 74.619237 \n",
       "L 67.184868 73.338301 \n",
       "L 69.020395 73.789738 \n",
       "L 70.855921 73.629051 \n",
       "L 71.957237 73.736786 \n",
       "L 73.792763 81.640884 \n",
       "L 75.628289 82.221607 \n",
       "L 77.463816 80.084077 \n",
       "L 79.299342 79.472948 \n",
       "L 81.134868 78.593079 \n",
       "L 82.236184 77.577052 \n",
       "L 84.071711 74.045089 \n",
       "L 85.907237 75.115642 \n",
       "L 87.742763 77.225971 \n",
       "L 89.578289 77.017344 \n",
       "L 91.413816 76.429793 \n",
       "L 92.515132 75.763104 \n",
       "L 94.350658 74.962445 \n",
       "L 96.186184 80.486968 \n",
       "L 98.021711 78.647312 \n",
       "L 99.857237 79.846121 \n",
       "L 101.692763 81.064189 \n",
       "L 102.794079 80.188441 \n",
       "L 104.629605 92.579886 \n",
       "L 106.465132 91.92705 \n",
       "L 108.300658 88.548859 \n",
       "L 110.136184 89.153506 \n",
       "L 111.971711 88.647944 \n",
       "L 113.073026 87.707859 \n",
       "L 114.908553 94.048343 \n",
       "L 116.744079 93.080317 \n",
       "L 118.579605 90.664647 \n",
       "L 120.415132 88.024814 \n",
       "L 122.250658 87.913943 \n",
       "L 123.351974 87.8698 \n",
       "L 125.1875 99.191655 \n",
       "L 127.023026 94.996382 \n",
       "L 128.858553 91.419697 \n",
       "L 130.694079 91.825758 \n",
       "L 132.529605 91.200939 \n",
       "L 133.630921 90.384297 \n",
       "L 135.466447 88.70032 \n",
       "L 137.301974 88.592839 \n",
       "L 139.1375 90.125437 \n",
       "L 140.973026 90.688047 \n",
       "L 142.808553 89.649822 \n",
       "L 143.909868 89.579646 \n",
       "L 145.745395 91.952945 \n",
       "L 147.580921 90.687742 \n",
       "L 149.416447 92.381426 \n",
       "L 151.251974 90.962102 \n",
       "L 153.0875 91.448075 \n",
       "L 154.188816 90.6544 \n",
       "L 156.024342 94.543482 \n",
       "L 157.859868 95.270992 \n",
       "L 159.695395 94.816324 \n",
       "L 161.530921 96.123175 \n",
       "L 163.366447 94.18288 \n",
       "L 164.467763 94.21628 \n",
       "L 166.303289 106.588155 \n",
       "L 168.138816 105.940489 \n",
       "L 169.974342 105.78342 \n",
       "L 171.809868 102.816433 \n",
       "L 173.645395 102.136459 \n",
       "L 174.746711 101.958773 \n",
       "L 176.582237 103.884718 \n",
       "L 178.417763 100.681573 \n",
       "L 180.253289 102.613696 \n",
       "L 182.088816 101.772554 \n",
       "L 183.924342 102.239533 \n",
       "L 185.025658 103.671106 \n",
       "L 186.861184 108.106499 \n",
       "L 188.696711 110.972861 \n",
       "L 190.532237 108.860185 \n",
       "L 192.367763 108.438 \n",
       "L 194.203289 107.319206 \n",
       "L 195.304605 106.565282 \n",
       "L 197.140132 110.695216 \n",
       "L 198.975658 112.828055 \n",
       "L 200.811184 112.387516 \n",
       "L 202.646711 111.620259 \n",
       "L 204.482237 111.650442 \n",
       "L 205.583553 111.32474 \n",
       "L 207.419079 111.253936 \n",
       "L 209.254605 109.322749 \n",
       "L 211.090132 109.906544 \n",
       "L 212.925658 107.8918 \n",
       "L 214.761184 107.006911 \n",
       "L 215.8625 107.31739 \n",
       "\" clip-path=\"url(#p9321c92340)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_16\">\n",
       "    <path d=\"M 12.119079 136.584312 \n",
       "L 13.954605 135.63675 \n",
       "L 15.790132 136.674556 \n",
       "L 17.625658 136.922727 \n",
       "L 19.461184 136.584312 \n",
       "L 20.5625 136.516629 \n",
       "L 22.398026 134.689189 \n",
       "L 24.233553 132.794065 \n",
       "L 26.069079 132.794065 \n",
       "L 27.904605 132.591016 \n",
       "L 29.740132 132.848211 \n",
       "L 30.841447 133.03579 \n",
       "L 32.676974 129.274549 \n",
       "L 34.5125 130.222111 \n",
       "L 36.348026 130.898941 \n",
       "L 38.183553 130.560526 \n",
       "L 40.019079 130.844795 \n",
       "L 41.120395 131.0053 \n",
       "L 42.955921 130.086745 \n",
       "L 44.791447 130.357477 \n",
       "L 46.626974 130.447721 \n",
       "L 48.4625 130.019062 \n",
       "L 50.298026 129.545281 \n",
       "L 51.399342 129.45826 \n",
       "L 53.234868 130.086745 \n",
       "L 55.070395 127.514792 \n",
       "L 56.905921 128.011133 \n",
       "L 58.741447 127.92089 \n",
       "L 60.576974 127.216986 \n",
       "L 61.678289 127.331081 \n",
       "L 63.513816 122.50625 \n",
       "L 65.349342 124.672106 \n",
       "L 67.184868 125.123326 \n",
       "L 69.020395 125.078204 \n",
       "L 70.855921 125.051131 \n",
       "L 71.957237 125.107211 \n",
       "L 73.792763 124.130642 \n",
       "L 75.628289 124.53674 \n",
       "L 77.463816 124.130642 \n",
       "L 79.299342 124.266008 \n",
       "L 81.134868 124.942838 \n",
       "L 82.236184 124.913831 \n",
       "L 84.071711 126.296498 \n",
       "L 85.907237 124.672106 \n",
       "L 87.742763 123.679422 \n",
       "L 89.578289 123.724544 \n",
       "L 91.413816 124.076496 \n",
       "L 92.515132 124.430381 \n",
       "L 94.350658 126.025766 \n",
       "L 96.186184 123.724544 \n",
       "L 98.021711 124.130642 \n",
       "L 99.857237 123.589178 \n",
       "L 101.692763 123.210153 \n",
       "L 102.794079 123.560171 \n",
       "L 104.629605 117.633075 \n",
       "L 106.465132 119.392833 \n",
       "L 108.300658 120.791614 \n",
       "L 110.136184 120.272712 \n",
       "L 111.971711 120.881858 \n",
       "L 113.073026 121.336301 \n",
       "L 114.908553 119.528199 \n",
       "L 116.744079 119.122101 \n",
       "L 118.579605 119.437955 \n",
       "L 120.415132 119.866614 \n",
       "L 122.250658 120.286248 \n",
       "L 123.351974 120.321057 \n",
       "L 125.1875 114.925755 \n",
       "L 127.023026 117.362343 \n",
       "L 128.858553 118.625759 \n",
       "L 130.694079 118.919052 \n",
       "L 132.529605 119.040881 \n",
       "L 133.630921 119.209122 \n",
       "L 135.466447 119.798931 \n",
       "L 137.301974 119.934297 \n",
       "L 139.1375 119.437955 \n",
       "L 140.973026 119.189784 \n",
       "L 142.808553 119.474052 \n",
       "L 143.909868 119.547537 \n",
       "L 145.745395 120.069663 \n",
       "L 147.580921 120.069663 \n",
       "L 149.416447 119.347711 \n",
       "L 151.251974 120.00198 \n",
       "L 153.0875 119.744784 \n",
       "L 154.188816 119.885952 \n",
       "L 156.024342 118.174539 \n",
       "L 157.859868 117.497709 \n",
       "L 159.695395 118.355027 \n",
       "L 161.530921 117.700758 \n",
       "L 163.366447 117.957953 \n",
       "L 164.467763 118.145532 \n",
       "L 166.303289 113.572095 \n",
       "L 168.138816 113.572095 \n",
       "L 169.974342 112.940388 \n",
       "L 171.809868 113.978193 \n",
       "L 173.645395 114.167706 \n",
       "L 174.746711 114.519657 \n",
       "L 176.582237 113.842827 \n",
       "L 178.417763 115.196487 \n",
       "L 180.253289 114.203803 \n",
       "L 182.088816 115.128804 \n",
       "L 183.924342 114.70917 \n",
       "L 185.025658 114.181242 \n",
       "L 186.861184 113.301364 \n",
       "L 188.696711 112.489168 \n",
       "L 190.532237 112.7599 \n",
       "L 192.367763 112.7599 \n",
       "L 194.203289 112.976485 \n",
       "L 195.304605 113.165998 \n",
       "L 197.140132 113.030632 \n",
       "L 198.975658 111.676972 \n",
       "L 200.811184 111.586728 \n",
       "L 202.646711 111.812338 \n",
       "L 204.482237 111.676972 \n",
       "L 205.583553 111.667303 \n",
       "L 207.419079 111.947704 \n",
       "L 209.254605 112.353802 \n",
       "L 211.090132 112.30868 \n",
       "L 212.925658 113.030632 \n",
       "L 214.761184 113.35551 \n",
       "L 215.8625 113.262688 \n",
       "\" clip-path=\"url(#p9321c92340)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_17\">\n",
       "    <path d=\"M 20.5625 132.117235 \n",
       "L 30.841447 132.117235 \n",
       "L 41.120395 130.763575 \n",
       "L 51.399342 130.086745 \n",
       "L 61.678289 127.379426 \n",
       "L 71.957237 128.733085 \n",
       "L 82.236184 123.995276 \n",
       "L 92.515132 132.794065 \n",
       "L 102.794079 129.409915 \n",
       "L 113.073026 127.379426 \n",
       "L 123.351974 125.348936 \n",
       "L 133.630921 126.025766 \n",
       "L 143.909868 128.056255 \n",
       "L 154.188816 128.056255 \n",
       "L 164.467763 123.318446 \n",
       "L 174.746711 125.348936 \n",
       "L 185.025658 123.995276 \n",
       "L 195.304605 125.348936 \n",
       "L 205.583553 124.672106 \n",
       "L 215.8625 126.702596 \n",
       "\" clip-path=\"url(#p9321c92340)\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 20.5625 143.1 \n",
       "L 20.5625 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 215.8625 143.1 \n",
       "L 215.8625 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 20.5625 143.1 \n",
       "L 215.8625 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 20.5625 7.2 \n",
       "L 215.8625 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 131.09375 59.234375 \n",
       "L 208.8625 59.234375 \n",
       "Q 210.8625 59.234375 210.8625 57.234375 \n",
       "L 210.8625 14.2 \n",
       "Q 210.8625 12.2 208.8625 12.2 \n",
       "L 131.09375 12.2 \n",
       "Q 129.09375 12.2 129.09375 14.2 \n",
       "L 129.09375 57.234375 \n",
       "Q 129.09375 59.234375 131.09375 59.234375 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_18\">\n",
       "     <path d=\"M 133.09375 20.298438 \n",
       "L 143.09375 20.298438 \n",
       "L 153.09375 20.298438 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_9\">\n",
       "     <!-- train loss -->\n",
       "     <g transform=\"translate(161.09375 23.798438)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"232.763672\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"264.550781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"292.333984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"353.515625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"405.615234\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_19\">\n",
       "     <path d=\"M 133.09375 34.976562 \n",
       "L 143.09375 34.976562 \n",
       "L 153.09375 34.976562 \n",
       "\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- train acc -->\n",
       "     <g transform=\"translate(161.09375 38.476562)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"232.763672\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"264.550781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"325.830078\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"380.810547\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_20\">\n",
       "     <path d=\"M 133.09375 49.654688 \n",
       "L 143.09375 49.654688 \n",
       "L 153.09375 49.654688 \n",
       "\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- valid acc -->\n",
       "     <g transform=\"translate(161.09375 53.154688)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-76\" d=\"M 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 563 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2284 0 \n",
       "L 1503 0 \n",
       "L 191 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \n",
       "L 2906 4863 \n",
       "L 3481 4863 \n",
       "L 3481 0 \n",
       "L 2906 0 \n",
       "L 2906 525 \n",
       "Q 2725 213 2448 61 \n",
       "Q 2172 -91 1784 -91 \n",
       "Q 1150 -91 751 415 \n",
       "Q 353 922 353 1747 \n",
       "Q 353 2572 751 3078 \n",
       "Q 1150 3584 1784 3584 \n",
       "Q 2172 3584 2448 3432 \n",
       "Q 2725 3281 2906 2969 \n",
       "z\n",
       "M 947 1747 \n",
       "Q 947 1113 1208 752 \n",
       "Q 1469 391 1925 391 \n",
       "Q 2381 391 2643 752 \n",
       "Q 2906 1113 2906 1747 \n",
       "Q 2906 2381 2643 2742 \n",
       "Q 2381 3103 1925 3103 \n",
       "Q 1469 3103 1208 2742 \n",
       "Q 947 2381 947 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-76\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"59.179688\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"120.458984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"148.242188\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-64\" x=\"176.025391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"239.501953\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"271.289062\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"332.568359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"387.548828\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p9321c92340\">\n",
       "   <rect x=\"20.5625\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "devices, num_epochs, lr, wd = d2l.try_all_gpus(), 20, 0.02, 5e-4\n",
    "lr_period, lr_decay, net = 4, 0.9, get_net(devices)\n",
    "net.hybridize()\n",
    "train(net, train_iter, valid_iter, num_epochs, lr, wd, devices, lr_period,\n",
    "      lr_decay)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 42
   },
   "source": [
    "## [**Classifying the Testing Set**] and Submitting Results on Kaggle\n",
    "\n",
    "After obtaining a promising model with hyperparameters,\n",
    "we use all the labeled data (including the validation set) to retrain the model and classify the testing set.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "origin_pos": 43,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train loss 0.741, train acc 0.757\n",
      "1104.8 examples/sec on [gpu(0), gpu(1)]\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"229.425pt\" height=\"180.65625pt\" viewBox=\"0 0 229.425 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2022-03-24T11:11:28.225478</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 180.65625 \n",
       "L 229.425 180.65625 \n",
       "L 229.425 0 \n",
       "L 0 0 \n",
       "L 0 180.65625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 20.5625 143.1 \n",
       "L 215.8625 143.1 \n",
       "L 215.8625 7.2 \n",
       "L 20.5625 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 61.678289 143.1 \n",
       "L 61.678289 7.2 \n",
       "\" clip-path=\"url(#p76247765d3)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"mb322dc1a51\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb322dc1a51\" x=\"61.678289\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(58.497039 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 113.073026 143.1 \n",
       "L 113.073026 7.2 \n",
       "\" clip-path=\"url(#p76247765d3)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb322dc1a51\" x=\"113.073026\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(106.710526 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 164.467763 143.1 \n",
       "L 164.467763 7.2 \n",
       "\" clip-path=\"url(#p76247765d3)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb322dc1a51\" x=\"164.467763\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 15 -->\n",
       "      <g transform=\"translate(158.105263 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 215.8625 143.1 \n",
       "L 215.8625 7.2 \n",
       "\" clip-path=\"url(#p76247765d3)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb322dc1a51\" x=\"215.8625\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(209.5 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_5\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(102.984375 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 20.5625 141.553501 \n",
       "L 215.8625 141.553501 \n",
       "\" clip-path=\"url(#p76247765d3)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <defs>\n",
       "       <path id=\"m6b947898fa\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6b947898fa\" x=\"20.5625\" y=\"141.553501\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(7.2 145.35272)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 20.5625 101.139474 \n",
       "L 215.8625 101.139474 \n",
       "\" clip-path=\"url(#p76247765d3)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6b947898fa\" x=\"20.5625\" y=\"101.139474\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 1 -->\n",
       "      <g transform=\"translate(7.2 104.938693)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 20.5625 60.725446 \n",
       "L 215.8625 60.725446 \n",
       "\" clip-path=\"url(#p76247765d3)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6b947898fa\" x=\"20.5625\" y=\"60.725446\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(7.2 64.524665)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 20.5625 20.311419 \n",
       "L 215.8625 20.311419 \n",
       "\" clip-path=\"url(#p76247765d3)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6b947898fa\" x=\"20.5625\" y=\"20.311419\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 3 -->\n",
       "      <g transform=\"translate(7.2 24.110638)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_17\">\n",
       "    <path d=\"M 12.273026 13.377273 \n",
       "L 14.2625 21.621005 \n",
       "L 16.251974 21.499377 \n",
       "L 18.241447 25.194181 \n",
       "L 20.230921 28.332951 \n",
       "L 20.5625 28.748998 \n",
       "L 22.551974 58.619885 \n",
       "L 24.541447 59.048452 \n",
       "L 26.530921 56.999181 \n",
       "L 28.520395 55.606344 \n",
       "L 30.509868 56.037467 \n",
       "L 30.841447 55.954671 \n",
       "L 32.830921 63.975838 \n",
       "L 34.820395 65.410701 \n",
       "L 36.809868 65.392146 \n",
       "L 38.799342 63.748754 \n",
       "L 40.788816 64.835704 \n",
       "L 41.120395 64.760852 \n",
       "L 43.109868 72.78332 \n",
       "L 45.099342 72.347159 \n",
       "L 47.088816 70.295641 \n",
       "L 49.078289 70.366053 \n",
       "L 51.067763 68.662522 \n",
       "L 51.399342 68.002952 \n",
       "L 53.388816 66.216356 \n",
       "L 55.378289 69.648242 \n",
       "L 57.367763 69.072891 \n",
       "L 59.357237 68.133747 \n",
       "L 61.346711 68.331755 \n",
       "L 61.678289 69.131871 \n",
       "L 63.667763 74.867407 \n",
       "L 65.657237 75.912154 \n",
       "L 67.646711 76.801981 \n",
       "L 69.636184 75.928706 \n",
       "L 71.625658 76.040209 \n",
       "L 71.957237 76.062984 \n",
       "L 73.946711 82.662393 \n",
       "L 75.936184 81.428232 \n",
       "L 77.925658 79.940574 \n",
       "L 79.915132 79.879125 \n",
       "L 81.904605 78.860128 \n",
       "L 82.236184 79.090132 \n",
       "L 84.225658 84.502389 \n",
       "L 86.215132 82.01764 \n",
       "L 88.204605 80.853471 \n",
       "L 90.194079 80.879529 \n",
       "L 92.183553 80.945927 \n",
       "L 92.515132 80.574229 \n",
       "L 94.504605 87.908389 \n",
       "L 96.494079 86.626832 \n",
       "L 98.483553 85.600206 \n",
       "L 100.473026 83.507384 \n",
       "L 102.4625 82.963218 \n",
       "L 102.794079 82.928663 \n",
       "L 104.783553 88.102383 \n",
       "L 106.773026 85.382849 \n",
       "L 108.7625 86.475828 \n",
       "L 110.751974 88.015779 \n",
       "L 112.741447 87.751713 \n",
       "L 113.073026 88.065929 \n",
       "L 115.0625 90.68344 \n",
       "L 117.051974 89.817679 \n",
       "L 119.041447 90.668936 \n",
       "L 121.030921 90.35676 \n",
       "L 123.020395 89.530164 \n",
       "L 123.351974 89.741374 \n",
       "L 125.341447 92.714445 \n",
       "L 127.330921 92.642755 \n",
       "L 129.320395 91.142925 \n",
       "L 131.309868 90.725242 \n",
       "L 133.299342 88.700054 \n",
       "L 133.630921 88.315051 \n",
       "L 135.620395 91.789238 \n",
       "L 137.609868 92.108592 \n",
       "L 139.599342 90.272963 \n",
       "L 141.588816 91.478626 \n",
       "L 143.578289 90.815013 \n",
       "L 143.909868 90.873016 \n",
       "L 145.899342 102.539059 \n",
       "L 147.888816 101.554112 \n",
       "L 149.878289 99.969097 \n",
       "L 151.867763 100.571273 \n",
       "L 153.857237 99.202826 \n",
       "L 154.188816 99.396849 \n",
       "L 156.178289 101.923463 \n",
       "L 158.167763 100.557963 \n",
       "L 160.157237 100.617301 \n",
       "L 162.146711 100.88501 \n",
       "L 164.136184 99.578051 \n",
       "L 164.467763 99.953232 \n",
       "L 166.457237 101.121634 \n",
       "L 168.446711 101.952787 \n",
       "L 170.436184 102.408486 \n",
       "L 172.425658 103.127152 \n",
       "L 174.415132 102.444681 \n",
       "L 174.746711 102.23017 \n",
       "L 176.736184 99.702191 \n",
       "L 178.725658 102.203068 \n",
       "L 180.715132 100.756017 \n",
       "L 182.704605 101.189399 \n",
       "L 184.694079 101.043308 \n",
       "L 185.025658 101.364723 \n",
       "L 187.015132 105.788321 \n",
       "L 189.004605 105.178952 \n",
       "L 190.994079 106.248658 \n",
       "L 192.983553 106.66882 \n",
       "L 194.973026 106.417061 \n",
       "L 195.304605 106.255587 \n",
       "L 197.294079 107.834548 \n",
       "L 199.283553 108.740576 \n",
       "L 201.273026 107.872585 \n",
       "L 203.2625 107.960136 \n",
       "L 205.251974 107.742243 \n",
       "L 205.583553 107.331598 \n",
       "L 207.573026 109.884796 \n",
       "L 209.5625 110.425372 \n",
       "L 211.551974 109.725668 \n",
       "L 213.541447 111.08455 \n",
       "L 215.530921 111.468284 \n",
       "L 215.8625 111.612658 \n",
       "\" clip-path=\"url(#p76247765d3)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_18\">\n",
       "    <path d=\"M 12.273026 136.922727 \n",
       "L 14.2625 135.02832 \n",
       "L 16.251974 134.467014 \n",
       "L 18.241447 134.554718 \n",
       "L 20.230921 134.986222 \n",
       "L 20.5625 134.83141 \n",
       "L 22.551974 128.713628 \n",
       "L 24.541447 129.239852 \n",
       "L 26.530921 130.257219 \n",
       "L 28.520395 130.71328 \n",
       "L 30.509868 131.029015 \n",
       "L 30.841447 131.042595 \n",
       "L 32.830921 128.924118 \n",
       "L 34.820395 128.924118 \n",
       "L 36.809868 128.152322 \n",
       "L 38.799342 128.76625 \n",
       "L 40.788816 128.46104 \n",
       "L 41.120395 128.353758 \n",
       "L 43.109868 127.02971 \n",
       "L 45.099342 126.292996 \n",
       "L 47.088816 127.380526 \n",
       "L 49.078289 127.345445 \n",
       "L 51.067763 127.534885 \n",
       "L 51.399342 127.620439 \n",
       "L 53.388816 129.345097 \n",
       "L 55.378289 127.661179 \n",
       "L 57.367763 127.520853 \n",
       "L 59.357237 127.819047 \n",
       "L 61.346711 127.492788 \n",
       "L 61.678289 127.09082 \n",
       "L 63.667763 127.2402 \n",
       "L 65.657237 126.292996 \n",
       "L 67.646711 125.766772 \n",
       "L 69.636184 125.451037 \n",
       "L 71.625658 125.556282 \n",
       "L 71.957237 125.542702 \n",
       "L 73.946711 123.030405 \n",
       "L 75.936184 122.609426 \n",
       "L 77.925658 123.100569 \n",
       "L 79.915132 123.030405 \n",
       "L 81.904605 123.240895 \n",
       "L 82.236184 123.139045 \n",
       "L 84.225658 121.977957 \n",
       "L 86.215132 123.13565 \n",
       "L 88.204605 123.240895 \n",
       "L 90.194079 123.451385 \n",
       "L 92.183553 123.240895 \n",
       "L 92.515132 123.383485 \n",
       "L 94.504605 122.398936 \n",
       "L 96.494079 121.556977 \n",
       "L 98.483553 121.697304 \n",
       "L 100.473026 121.872712 \n",
       "L 102.4625 122.062153 \n",
       "L 102.794079 122.161286 \n",
       "L 104.783553 122.609426 \n",
       "L 106.773026 122.504181 \n",
       "L 108.7625 121.346488 \n",
       "L 110.751974 120.662396 \n",
       "L 112.741447 120.88341 \n",
       "L 113.073026 120.776128 \n",
       "L 115.0625 119.66257 \n",
       "L 117.051974 120.399284 \n",
       "L 119.041447 120.153712 \n",
       "L 121.030921 120.136172 \n",
       "L 123.020395 120.588725 \n",
       "L 123.351974 120.409469 \n",
       "L 125.341447 119.873059 \n",
       "L 127.330921 119.031101 \n",
       "L 129.320395 119.592406 \n",
       "L 131.309868 119.66257 \n",
       "L 133.299342 120.251941 \n",
       "L 133.630921 120.327989 \n",
       "L 135.620395 117.557672 \n",
       "L 137.609868 118.610121 \n",
       "L 139.599342 119.873059 \n",
       "L 141.588816 119.504702 \n",
       "L 143.578289 119.788864 \n",
       "L 143.909868 119.75763 \n",
       "L 145.899342 115.242285 \n",
       "L 147.888816 116.189489 \n",
       "L 149.878289 117.206856 \n",
       "L 151.867763 116.557846 \n",
       "L 153.857237 116.799909 \n",
       "L 154.188816 116.661394 \n",
       "L 156.178289 114.821306 \n",
       "L 158.167763 115.55802 \n",
       "L 160.157237 115.522938 \n",
       "L 162.146711 115.137041 \n",
       "L 164.136184 115.705363 \n",
       "L 164.467763 115.561415 \n",
       "L 166.457237 115.873755 \n",
       "L 168.446711 115.137041 \n",
       "L 170.436184 114.68098 \n",
       "L 172.425658 114.295082 \n",
       "L 174.415132 114.358229 \n",
       "L 174.746711 114.339217 \n",
       "L 176.736184 115.873755 \n",
       "L 178.725658 114.295082 \n",
       "L 180.715132 115.031796 \n",
       "L 182.704605 115.084418 \n",
       "L 184.694079 115.663265 \n",
       "L 185.025658 115.520675 \n",
       "L 187.015132 112.716409 \n",
       "L 189.004605 112.821654 \n",
       "L 190.994079 112.225266 \n",
       "L 192.983553 112.716409 \n",
       "L 194.973026 113.179486 \n",
       "L 195.304605 113.239238 \n",
       "L 197.294079 113.347878 \n",
       "L 199.283553 112.08494 \n",
       "L 201.273026 112.225266 \n",
       "L 203.2625 112.08494 \n",
       "L 205.251974 112.463821 \n",
       "L 205.583553 112.628139 \n",
       "L 207.573026 110.401022 \n",
       "L 209.5625 111.242981 \n",
       "L 211.551974 111.593797 \n",
       "L 213.541447 111.348226 \n",
       "L 215.530921 111.074589 \n",
       "L 215.8625 110.957801 \n",
       "\" clip-path=\"url(#p76247765d3)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_19\"/>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 20.5625 143.1 \n",
       "L 20.5625 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 215.8625 143.1 \n",
       "L 215.8625 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 20.5625 143.1 \n",
       "L 215.8625 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 20.5625 7.2 \n",
       "L 215.8625 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 131.09375 44.55625 \n",
       "L 208.8625 44.55625 \n",
       "Q 210.8625 44.55625 210.8625 42.55625 \n",
       "L 210.8625 14.2 \n",
       "Q 210.8625 12.2 208.8625 12.2 \n",
       "L 131.09375 12.2 \n",
       "Q 129.09375 12.2 129.09375 14.2 \n",
       "L 129.09375 42.55625 \n",
       "Q 129.09375 44.55625 131.09375 44.55625 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_20\">\n",
       "     <path d=\"M 133.09375 20.298437 \n",
       "L 143.09375 20.298437 \n",
       "L 153.09375 20.298437 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- train loss -->\n",
       "     <g transform=\"translate(161.09375 23.798437)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"232.763672\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"264.550781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"292.333984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"353.515625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"405.615234\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_21\">\n",
       "     <path d=\"M 133.09375 34.976562 \n",
       "L 143.09375 34.976562 \n",
       "L 153.09375 34.976562 \n",
       "\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- train acc -->\n",
       "     <g transform=\"translate(161.09375 38.476562)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"232.763672\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"264.550781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"325.830078\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"380.810547\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p76247765d3\">\n",
       "   <rect x=\"20.5625\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "net, preds = get_net(devices), []\n",
    "net.hybridize()\n",
    "train(net, train_valid_iter, None, num_epochs, lr, wd, devices, lr_period,\n",
    "      lr_decay)\n",
    "\n",
    "for X, _ in test_iter:\n",
    "    y_hat = net(X.as_in_ctx(devices[0]))\n",
    "    preds.extend(y_hat.argmax(axis=1).astype(int).asnumpy())\n",
    "sorted_ids = list(range(1, len(test_ds) + 1))\n",
    "sorted_ids.sort(key=lambda x: str(x))\n",
    "df = pd.DataFrame({'id': sorted_ids, 'label': preds})\n",
    "df['label'] = df['label'].apply(lambda x: train_valid_ds.synsets[x])\n",
    "df.to_csv('submission.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 45
   },
   "source": [
    "The above code\n",
    "will generate a `submission.csv` file,\n",
    "whose format\n",
    "meets the requirement of the Kaggle competition.\n",
    "The method\n",
    "for submitting results to Kaggle\n",
    "is similar to that in :numref:`sec_kaggle_house`.\n",
    "\n",
    "## Summary\n",
    "\n",
    "* We can read datasets containing raw image files after organizing them into the required format.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 46,
    "tab": [
     "mxnet"
    ]
   },
   "source": [
    "* We can use convolutional neural networks, image augmentation, and hybrid programing in an image classification competition.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 48
   },
   "source": [
    "## Exercises\n",
    "\n",
    "1. Use the complete CIFAR-10 dataset for this Kaggle competition. Set hyperparameters as `batch_size = 128`, `num_epochs = 100`, `lr = 0.1`, `lr_period = 50`, and `lr_decay = 0.1`.  See what accuracy and ranking you can achieve in this competition. Can you further improve them?\n",
    "1. What accuracy can you get when not using image augmentation?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 49,
    "tab": [
     "mxnet"
    ]
   },
   "source": [
    "[Discussions](https://discuss.d2l.ai/t/379)\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}