You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/11/07 19:23:57 UTC
[GitHub] sandeep-krishnamurthy closed pull request #12880: Fix variational
autoencoder example
sandeep-krishnamurthy closed pull request #12880: Fix variational autoencoder example
URL: https://github.com/apache/incubator-mxnet/pull/12880
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/example/vae/README.md b/example/autoencoder/variational_autoencoder/README.md
similarity index 100%
rename from example/vae/README.md
rename to example/autoencoder/variational_autoencoder/README.md
diff --git a/example/vae/VAE.py b/example/autoencoder/variational_autoencoder/VAE.py
similarity index 100%
rename from example/vae/VAE.py
rename to example/autoencoder/variational_autoencoder/VAE.py
diff --git a/example/autoencoder/variational_autoencoder/VAE_example.ipynb b/example/autoencoder/variational_autoencoder/VAE_example.ipynb
new file mode 100755
index 00000000000..964e13725c6
--- /dev/null
+++ b/example/autoencoder/variational_autoencoder/VAE_example.ipynb
@@ -0,0 +1,1204 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import mxnet as mx\n",
+ "import numpy as np\n",
+ "import os\n",
+ "import logging\n",
+ "import matplotlib.pyplot as plt\n",
+ "import matplotlib.cm as cm"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Building a Variational Autoencoder in MXNet\n",
+ "\n",
+ "#### Xiaoyu Lu, July 5th, 2017\n",
+ "\n",
+ "This tutorial guides you through the process of building a variational encoder in MXNet. In this notebook we'll focus on an example using the MNIST handwritten digit recognition dataset. Refer to [Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114/) for more details on the model description.\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "To complete this tutorial, we need following python packages:\n",
+ "\n",
+ "- numpy, matplotlib "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 1. Loading the Data\n",
+ "\n",
+ "We first load the MNIST dataset, which contains 60000 training and 10000 test examples. The following code imports required modules and loads the data. These images are stored in a 4-D matrix with shape (`batch_size, num_channels, width, height`). For the MNIST dataset, there is only one color channel, and both width and height are 28, so we reshape each image as a 28x28 array. See below for a visualization:\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "60000 784\n"
+ ]
+ }
+ ],
+ "source": [
+ "mnist = mx.test_utils.get_mnist()\n",
+ "image = np.reshape(mnist['train_data'],(60000,28*28))\n",
+ "label = image\n",
+ "image_test = np.reshape(mnist['test_data'],(10000,28*28))\n",
+ "label_test = image_test\n",
+ "[N,features] = np.shape(image) #number of examples and features\n",
+ "print(N,features)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAACWCAYAAAA7UIUvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAFI5JREFUeJzt3X+wVfO/x/H3u9+JfikVnb5FaYTEnLmqe5mLDDKJwfGVaYzfVJRBP+5FgzHjd6YoMkyFcbvUJDToxnW7GDql6YpSFwnpVKRfqDM+94+2O+f9Waf94+xfa+3P8zHTnF6rtfd6n/a73afde62lzjkBAAAAQtSs3AUAAAAA5cJiGAAAAMFiMQwAAIBgsRgGAABAsFgMAwAAIFgshgEAABAsFsMAAAAIFothAAAABCuvxbCqnq+q61V1o6pOLlRRAAAAQCloU+9Ap6rNReQrETlXRL4XkRUicqVz7otDPaZLly6ud+/eTToe4uXbb7+V7du3azGemz6pLCtXrtzunOtajOemVyoH7ynIFu8pyEYu7ykt8jjOP4jIRufc1yIiqvpvIjJSRA65GO7du7fU1tbmcUjERXV1ddGemz6pLKq6qVjPTa9UDt5TkC3eU5CNXN5T8hmTOEZENjfI36e2Gap6o6rWqmrttm3b8jgcKhl9gmzRK8gGfYJs0Sso+gl0zrnZzrlq51x1165F+V8NVAD6BNmiV5AN+gTZoleQz2L4BxGpapB7prYBAAAAiZDPYniFiPRT1T6q2kpE/i4iiwtTFgAAAFB8TT6BzjlXr6rjROQdEWkuIi8459YWrDIAAACgyPK5moQ455aIyJIC1QIAAACUFHegAwAAQLBYDAMAACBYLIYBAAAQLBbDAAAACBaLYQAAAASLxTAAAACCxWIYAAAAwWIxDAAAgGCxGAYAAECw8roDHYDiue2220yeMWNGZJ8OHTqYvG7dOpO7d+9e+MIAAKggfDIMAACAYLEYBgAAQLBYDAMAACBYzAwXwdatW00eNGiQyUOHDjV5wYIFRa8J8bdq1SqTn376aZObNYv+23X37t0m19XVmczMMJBcf/zxh8nz5883+Zprrkn7+OXLl5s8ePBgk/ft2xd5TJs2bUw+cOCAyf77UOvWrdPWACQBnwwDAAAgWCyGAQAAECwWwwAAAAgWM8NFMHv2bJP9GeJFixaVshzEVH19vcnz5s3L+TmOP/54k3v16pVXTQDKY+/evZFtU6dONfnJJ580WVXTPuftt99u8mmnnWbyc889F3nMZZddZvL7779vcu/evU32z4Hxz1O48cYbTe7UqdOhCwbKhE+GAQAAECwWwwAAAAgWi2EAAAAEi5nhEnDOlbsExNAnn3xi8owZM3J+jsmTJ5vcsWPHvGoCUB6XXnppZNvSpUtNzjQj7Fu5cqXJtbW1GZ/vtddeS/ucO3bsyOk53333XZOXLFkSeU6uVZzehg0bTO7fv39knxEjRpg8a9Ysk48++ujCF+bZuXOnyX6v+OdTZTJu3LjItqqqqtwLywKfDAMAACBYLIYBAAAQLBbDAAAACBYzw0Wwdu1ak/0ZqksuuaSU5SAm/vzzT5OnTJmS0+NXrFgR2XbyySfnVRMyu+WWW0z+4osvTL7qqqtMHjhwoMmDBw8uTmFItLlz55r8wQcflKmS4vr6669N/uOPPyL7MDOc3vz5801ubNb7jTfeMNk/J8WfGT7qqKNMHjVqVNoaPvvss8g2v2d//PFHk7dt22ayf/5Uphn48847L7KNmWEAAACgwFgMAwAAIFgshgEAABAsZoaLwJ/F8TV2P3hUPn/m6sMPP0y7/xFHHGFyY9eJbNmyZf6FwfCvs/rss8+m3X/58uUm+3NwLVpE32b9uWL/epp9+vRJ+xz79u0zediwYWlrRPn512CdOnWqyQcOHChlOQXjnwMzcuRIky+//HKT27RpU/Sakqa+vt7k1atXm/zEE0/k/Jz+vK6fff41rXOd7y2GIUOGlOxYfDIMAACAYLEYBgAAQLAyLoZV9QVVrVPVzxts66yqS1V1Q+prp+KWCQAAABReNjPDc0TkKRGZ12DbZBFZ5px7SFUnp/KkwpeXDPv37zd548aNZaoEcfbNN9/ktP8pp5xicvfu3QtZDg7Bn+dt27atyb/99lvax/uzdo3Ngq5cudLka665xuRmzeznFP68nn+Mww47zOQuXbpEjvnRRx+ZTD+V1po1a0z2/97wX1OR6LXJ/b7IpF27dia3b9/e5MauLeu/7/j8GeBWrVrlVBOidu3aZfLpp5+edv+amprItuHDh5vsn6OyaNEikzdt2pRLiSXhfw+NnW9RLBn/ZDnn/ktEfvY2jxSRv64YPldELi5wXQAAAEDRNXVmuJtzbkvq5z+JSLcC1QMAAACUTN4n0LmD/7cT/f+dFFW9UVVrVbU206U9EC76BNmiV5AN+gTZolfQ1MXwVlXtISKS+lp3qB2dc7Odc9XOuequXbs28XCodPQJskWvIBv0CbJFr6Cp08mLReRqEXko9fX1glWUQNu3bzf57bffNrmxEyMQnrfeeiun/R955JEiVYJ0jj/+eJP9P98zZ8402T/5xffMM89EtmX69Mk/cSqTPXv2pM0i0e9r4cKFJnPjjtKqq7OfITV2UwP/hDn/RMkrrrjC5NGjR5t87LHHmlxVVZVznSi8vXv3mnz33Xfn9Pjrr78+su2cc84x2e+FBx980OQtW7ZIvt59912T77zzTpMznWzsn5zsv1eW8qZS2Vxa7RUR+VhE+qvq96p6nRxcBJ+rqhtEZFgqAwAAAImS8ZNh59yVh/ilcw6xHQAAAEgE7kAHAACAYJXuisYV7OOPPza5sdkvhOfXX381edmyZTk9vrq6upDloIn8ubY77rgjp8dPmhS9H1GmmeBXXnnF5Ew38vEvqL9hw4bIPv4ccaZ5PhSWPyfaFL169TJ5xowZJvu9inj6/fffTZ4/f77J/nlGN998s8n+fHA2/N7w58kzWbduXWTbmDFjcnqODh06mPzhhx+afMwxx+T0fIXEJ8MAAAAIFothAAAABIvFMAAAAILFzHABnHrqqSb714Js1aqVyW3atCl6TSg//9qRP/zwQ9r9u3WzdzVn9rwy+O8H2bjhhhty2v/ee+81ee7cuZF9xo0bZ/LkyZNNHjFiRE7HRG78vyeOPPJIk3fs2JHxOdavX2/yrFmzTB45cqTJxx13XC4lokTat29v8oIFC0wePny4yY8//njRa/K9+eabJtfU1ET2yfR3lD+n7M8IDxgwoInVFR6fDAMAACBYLIYBAAAQLBbDAAAACBYzwwXw2Wefmbxv3z6T+/TpY3JTZggRf/X19Sb7830+f0b4pZdeMrlZM/6tiuy0a9fO5FGjRkX28WeGJ0yYUNSaYHXv3t3kfv36mZzNzLBv4sSJJk+dOtXkRx991GT/erUoj5YtW5o8ZMgQk9977z2TS3H96Hfeecfkiy66yORszmHxr8E+fvx4k8t5HeFM+NsWAAAAwWIxDAAAgGCxGAYAAECwmBkuAP/aef59xS+//PJSloMymT59usnLli1Lu/+JJ55o8tlnn13wmhCmZ555JuM+/gwrSuu1114zubq6OrKPf23yTOcR+OerjB071uS77rrL5BUrVkSeo2/fvia3aMEyodhat25t8uDBg4t+zO+++87kiy++OO3+Rx11VGTbLbfcYrI/w56keyrwyTAAAACCxWIYAAAAwWIxDAAAgGAxDFQA/uydfz2+QYMGlbIclEk2c5oNnXDCCUWqBKHZuXOnydOmTYvs48/vVVVVFbUmpOf/vTFv3rzIPqNHjzZ569ateR3Tnyk+6aSTIvv417g988wz8zom4sG/jrA/77t//36Tjz76aJM//fTTyHP6+yQZnwwDAAAgWCyGAQAAECwWwwAAAAgWM8MF4M/n+dcZXr16tckjRowoek2IP/oAheLPq2/bti2yj99vnMsQL41dZ9y/hr1/DeBiuOKKK0xesGCByUOHDi16Dcjfrl27TPZnwb/99tu0j9+7d6/J/nkJIswMAwAAABWBxTAAAACCxWIYAAAAwWIxDAAAgGBxAl0B+DfZ8DMqz6ZNmyLbfvzxx7SPad68ucmnn356QWtCOH7++WeTp0+fbnLPnj0jj3n55ZeLWhMKz78xyubNm01euHChyffff7/JO3bsyPmY/smXY8eONfmjjz4yuW3btjkfA8W3Zs0akx977DGTM61TZs2aZfKAAQMKU1hM8ckwAAAAgsViGAAAAMFiMQwAAIBgMTNcAP6Fp3/66SeTL7roolKWgxLwX2MRkd9++y3tY8aMGWNy+/btC1oTKtf+/ftNfvjhh032+/HCCy+MPMfhhx9e+MJQVP55Bj169DC5Y8eOJvs3WiiE1q1bp60J8VBbW2tyrjd1qqmpMfmCCy7Iu6Yk4ZNhAAAABCvjYlhVq1T1fVX9QlXXqur41PbOqrpUVTekvnYqfrkAAABA4WTzyXC9iNzhnBsgIoNFZKyqDhCRySKyzDnXT0SWpTIAAACQGBlnhp1zW0RkS+rnu1X1SxE5RkRGisg/p3abKyL/KSKTilJlzK1atcpk//p9nTrxoXmlefbZZ3N+zFNPPWXyTTfdZPIJJ5yQV02oXFu2bDH5kUceMdm/1usDDzxQ9JqQ3u7du02eOHGiye3atTO5W7dukedwzpn86quvmuz/3ZPJn3/+aXKzZpk/DzvssMNyOgaK7/XXX49su/baa03OND/euXNnk6dNm2Zyhw4dmlhdMuU0M6yqvUXkVBH5RES6pRbKIiI/iUj0TzIAAAAQY1kvhlX1cBFZICITnHPmnxzu4D9f3SEed6Oq1qpqrX9nG+Av9AmyRa8gG/QJskWvIKvFsKq2lIML4Zedc3/d/3GrqvZI/XoPEalr7LHOudnOuWrnXHXXrl0LUTMqEH2CbNEryAZ9gmzRK8g4M6wHB2CfF5EvnXNPNPilxSJytYg8lPoaHWIJRN++fU32r/nZq1evUpaDmPLn//w+YWYYh7Jz506T/fMSHnvsMZMHDRpU9JqQ3ubNm01+7rnncn4O/z3Df939nIk/I9zY4/33IX8+vVWrVjkdE/nbtGmTyffcc09kn19++cXkTL0xbNgwk0M/tymbm278o4iMFpH/UdXVqW3/IgcXwf+uqteJyCYRqTnE4wEAAIBYyuZqEv8tIof6J8Y5hS0HAAAAKB3uQAcAAIBgZTMmgQz8a4D6szr79u0zmes2Jl+PHj1yfow/a3fWWWcVqhxUmPr6epOnTp1qcsuWLU0eOnRo0WtCbvzruF533XUmP//886Usp1Ht27ePbFuyZInJVVVVpSoHKXV19noEp512msn+fLBIdL7cd8YZZ5j84osvmtyiRdjLQT4ZBgAAQLBYDAMAACBYLIYBAAAQrLCHRApkz549JvvzpMwIV55bb701sm3mzJkm+/d2nz59elFrQuVYu3atyYsXLzb5wgsvNJnrCsdP9+7dTX7wwQdNfvPNN03eunVr0WsaP368yaNGjYrsw4xw6flriP79+5u8a5e56W9W15e+7777TJ4wYYLJoc8I+/hkGAAAAMFiMQwAAIBgsRgGAABAsBgaKQD/PuH+rBgqT2OvcWPXfgSa4oEHHjDZn+/z508Rfx07djT5q6++MnnMmDGRx7z00ks5HaO6utrkadOmmTxkyJCcng/FsXfvXpMvu+wyk/0Z4Wz4M8KTJk0y2b/OPSw+GQYAAECwWAwDAAAgWCyGAQAAECwWwwAAAAgWJ9AVwP3331/uEgBUsDPOOMPkgQMHlqkSNJV/EqSf586dG3lMY9uQfO3atTO5rq4up8fX1NREtk2ZMsVkbqqRGz4ZBgAAQLBYDAMAACBYLIYBAAAQLIZKAAAAymTx4sUmjxs3zuS2bduaPGfOnMhzMCOcHz4ZBgAAQLBYDAMAACBYLIYBAAAQLIZMACBm/JnBhQsXlqkSAMXWs2dPkxctWlSmSsLFJ8MAAAAIFothAAAABIvFMAAAAIKlzrnSHUx1m4hsEpEuIrK9ZAduGmpM72/Oua7FeOKE9YlIMuqkV8qPGtMrRZ+I8DoUSqX3Cq9B4ZSrzqz7pKSL4f8/qGqtc6665AfOATWWX1K+vyTUmYQa85GE748a4yEJ3yM1ll8Svr8k1CiSjDoZkwAAAECwWAwDAAAgWOVaDM8u03FzQY3ll5TvLwl1JqHGfCTh+6PGeEjC90iN5ZeE7y8JNYokoM6yzAwDAAAAccCYBAAAAIJV0sWwqp6vqutVdaOqTi7lsdNR1RdUtU5VP2+wrbOqLlXVDamvncpcY5Wqvq+qX6jqWlUdH8c6CyWOvUKfxE8c+0SEXokjeqXJ9QXVJyLx7JW490mqnsT2SskWw6raXESeFpELRGSAiFypqgNKdfwM5ojI+d62ySKyzDnXT0SWpXI51YvIHc65ASIyWETGpn7/4lZn3mLcK3OEPomNGPeJCL0SK/RKXoLpE5FY98ociXefiCS5V5xzJfkhIkNE5J0GeYqITCnV8bOor7eIfN4grxeRHqmf9xCR9eWu0av3dRE5N+51Vlqv0Cfx+RHnPqFX4vWDXqFPKqFXktQnSeuVUo5JHCMimxvk71Pb4qqbc25L6uc/iUi3chbTkKr2FpFTReQTiXGdeUhSr8T2958+iZ3Yvgb0SuzE8jUIoE9EktUrsX0NktYrnECXBXfwnzOxuOyGqh4uIgtEZIJzblfDX4tTnSGK0+8/fRJvcXoN6JV4i8trQJ/EW5xegyT2SikXwz+ISFWD3DO1La62qmoPEZHU17oy1yOq2lIONtjLzrmFqc2xq7MAktQrsfv9p09iK3avAb0SW7F6DQLqE5Fk9UrsXoOk9kopF8MrRKSfqvZR1VYi8ncRWVzC4+dqsYhcnfr51XJw9qVsVFVF5HkR+dI590SDX4pVnQWSpF6J1e8/fRLbPhGJ2WtAr9Ar2QisT0SS1Suxeg0S3SslHqYeLiJficj/isi/lntgukFdr4jIFhE5IAfng64TkSPl4FmPG0TkP0Skc5lr/Cc5+F8La0RkderH8LjVWcm9Qp/E70cc+4ReiecPeoU+SXKvxL1Pkt4r3IEOAAAAweIEOgAAAASLxTAAAACCxWIYAAAAwWIxDAAAgGCxGAYAAECwWAwDAAAgWCyGAQAAECwWwwAAAAjW/wEgPmufEARJLAAAAABJRU5ErkJggg==\n",
+ "text/plain": [
+ "<Figure size 864x216 with 5 Axes>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "nsamples = 5\n",
+ "idx = np.random.choice(len(mnist['train_data']), nsamples)\n",
+ "_, axarr = plt.subplots(1, nsamples, sharex='col', sharey='row',figsize=(12,3))\n",
+ "\n",
+ "for i,j in enumerate(idx):\n",
+ " axarr[i].imshow(np.reshape(image[j,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
+ "\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can optionally save the parameters in the directory variable 'model_prefix'. We first create data iterators for MXNet, with each batch of data containing 100 images."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "model_prefix = None\n",
+ "\n",
+ "batch_size = 100\n",
+ "latent_dim = 5\n",
+ "nd_iter = mx.io.NDArrayIter(data={'data':image},label={'loss_label':label},\n",
+ " batch_size = batch_size)\n",
+ "nd_iter_test = mx.io.NDArrayIter(data={'data':image_test},label={'loss_label':label_test},\n",
+ " batch_size = batch_size)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 2. Building the Network Architecture\n",
+ "\n",
+ "### 2.1 Gaussian MLP as encoder\n",
+ "Next we constuct the neural network, as in the [paper](https://arxiv.org/abs/1312.6114/), we use *Multilayer Perceptron (MLP)* for both the encoder and decoder. For encoder, a Gaussian MLP is used as follows:\n",
+ "\n",
+ "\\begin{align}\n",
+ "\\log q_{\\phi}(z|x) &= \\log \\mathcal{N}(z:\\mu,\\sigma^2I) \\\\\n",
+ "\\textit{ where } \\mu &= W_2h+b_2, \\log \\sigma^2 = W_3h+b_3\\\\\n",
+ "h &= \\tanh(W_1x+b_1)\n",
+ "\\end{align}\n",
+ "\n",
+ "where $\\{W_1,W_2,W_3,b_1,b_2,b_3\\}$ are the weights and biases of the MLP.\n",
+ "Note below that `encoder_mu`(`mu`) and `encoder_logvar`(`logvar`) are symbols. So, we can use `get_internals()` to get the values of them, after which we can sample the latent variable $z$.\n",
+ "\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "## define data and loss labels as symbols \n",
+ "data = mx.sym.var('data')\n",
+ "loss_label = mx.sym.var('loss_label')\n",
+ "\n",
+ "## define fully connected and activation layers for the encoder, where we used tanh activation function.\n",
+ "encoder_h = mx.sym.FullyConnected(data=data, name=\"encoder_h\",num_hidden=400)\n",
+ "act_h = mx.sym.Activation(data=encoder_h, act_type=\"tanh\",name=\"activation_h\")\n",
+ "\n",
+ "## define mu and log variance which are the fully connected layers of the previous activation layer\n",
+ "mu = mx.sym.FullyConnected(data=act_h, name=\"mu\",num_hidden = latent_dim)\n",
+ "logvar = mx.sym.FullyConnected(data=act_h, name=\"logvar\",num_hidden = latent_dim)\n",
+ "\n",
+ "## sample the latent variables z according to Normal(mu,var)\n",
+ "z = mu + mx.symbol.broadcast_mul(mx.symbol.exp(0.5 * logvar), \n",
+ " mx.symbol.random_normal(loc=0, scale=1, shape=(batch_size, latent_dim)),\n",
+ " name=\"z\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 2.2 Bernoulli MLP as decoder\n",
+ "\n",
+ "In this case let $p_\\theta(x|z)$ be a multivariate Bernoulli whose probabilities are computed from $z$ with a feed forward neural network with a single hidden layer:\n",
+ "\n",
+ "\\begin{align}\n",
+ "\\log p(x|z) &= \\sum_{i=1}^D x_i\\log y_i + (1-x_i)\\log (1-y_i) \\\\\n",
+ "\\textit{ where } y &= f_\\sigma(W_5\\tanh (W_4z+b_4)+b_5)\n",
+ "\\end{align}\n",
+ "\n",
+ "where $f_\\sigma(\\dot)$ is the elementwise sigmoid activation function, $\\{W_4,W_5,b_4,b_5\\}$ are the weights and biases of the decoder MLP. A Bernouilli likelihood is suitable for this type of data but you can easily extend it to other likelihood types by parsing into the argument `likelihood` in the `VAE` class, see section 4 for details."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# define fully connected and tanh activation layers for the decoder\n",
+ "decoder_z = mx.sym.FullyConnected(data=z, name=\"decoder_z\",num_hidden=400)\n",
+ "act_z = mx.sym.Activation(data=decoder_z, act_type=\"tanh\",name=\"activation_z\")\n",
+ "\n",
+ "# define the output layer with sigmoid activation function, where the dimension is equal to the input dimension\n",
+ "decoder_x = mx.sym.FullyConnected(data=act_z, name=\"decoder_x\",num_hidden=features)\n",
+ "y = mx.sym.Activation(data=decoder_x, act_type=\"sigmoid\",name='activation_x')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 2.3 Joint Loss Function for the Encoder and the Decoder\n",
+ "\n",
+ "The variational lower bound also called evidence lower bound (ELBO) can be estimated as:\n",
+ "\n",
+ "\\begin{align}\n",
+ "\\mathcal{L}(\\theta,\\phi;x_{(i)}) \\approx \\frac{1}{2}\\left(1+\\log ((\\sigma_j^{(i)})^2)-(\\mu_j^{(i)})^2-(\\sigma_j^{(i)})^2\\right) + \\log p_\\theta(x^{(i)}|z^{(i)})\n",
+ "\\end{align}\n",
+ "\n",
+ "where the first term is the KL divergence of the approximate posterior from the prior, and the second term is an expected negative reconstruction error. We would like to maximize this lower bound, so we can define the loss to be $-\\mathcal{L}$(minus ELBO) for MXNet to minimize."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# define the objective loss function that needs to be minimized\n",
+ "KL = 0.5*mx.symbol.sum(1+logvar-pow( mu,2)-mx.symbol.exp(logvar),axis=1)\n",
+ "loss = -mx.symbol.sum(mx.symbol.broadcast_mul(loss_label,mx.symbol.log(y)) \n",
+ " + mx.symbol.broadcast_mul(1-loss_label,mx.symbol.log(1-y)),axis=1)-KL\n",
+ "output = mx.symbol.MakeLoss(sum(loss),name='loss')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 3. Training the model\n",
+ "\n",
+ "Now, we can define the model and train it. First we will initilize the weights and the biases to be Gaussian(0,0.01), and then use stochastic gradient descent for optimization. To warm start the training, one may also initilize with pre-trainined parameters `arg_params` using `init=mx.initializer.Load(arg_params)`. \n",
+ "\n",
+ "To save intermediate results, we can optionally use `epoch_end_callback = mx.callback.do_checkpoint(model_prefix, 1)` which saves the parameters to the path given by model_prefix, and with period every $1$ epoch. To assess the performance, we output $-\\mathcal{L}$(minus ELBO) after each epoch, with the command `eval_metric = 'Loss'` which is defined above. We will also plot the training loss for mini batches by accessing the log and saving it to a list, and then parsing it to the argument `batch_end_callback`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# set up the log\n",
+ "nd_iter.reset()\n",
+ "logging.getLogger().setLevel(logging.DEBUG) \n",
+ "\n",
+ "# define function to trave back training loss\n",
+ "def log_to_list(period, lst):\n",
+ " def _callback(param):\n",
+ " \"\"\"The checkpoint function.\"\"\"\n",
+ " if param.nbatch % period == 0:\n",
+ " name, value = param.eval_metric.get()\n",
+ " lst.append(value)\n",
+ " return _callback\n",
+ "\n",
+ "# define the model\n",
+ "model = mx.mod.Module(\n",
+ " symbol = output ,\n",
+ " data_names=['data'],\n",
+ " label_names = ['loss_label'])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "INFO:root:Epoch[0] Train-loss=373.547317\n",
+ "INFO:root:Epoch[0] Time cost=5.020\n",
+ "INFO:root:Epoch[1] Train-loss=212.232684\n",
+ "INFO:root:Epoch[1] Time cost=4.651\n",
+ "INFO:root:Epoch[2] Train-loss=207.448528\n",
+ "INFO:root:Epoch[2] Time cost=4.665\n",
+ "INFO:root:Epoch[3] Train-loss=205.369479\n",
+ "INFO:root:Epoch[3] Time cost=4.758\n",
+ "INFO:root:Epoch[4] Train-loss=203.651983\n",
+ "INFO:root:Epoch[4] Time cost=4.672\n",
+ "INFO:root:Epoch[5] Train-loss=202.061007\n",
+ "INFO:root:Epoch[5] Time cost=5.087\n",
+ "INFO:root:Epoch[6] Train-loss=199.348143\n",
+ "INFO:root:Epoch[6] Time cost=5.056\n",
+ "INFO:root:Epoch[7] Train-loss=196.266242\n",
+ "INFO:root:Epoch[7] Time cost=4.813\n",
+ "INFO:root:Epoch[8] Train-loss=194.694945\n",
+ "INFO:root:Epoch[8] Time cost=4.776\n",
+ "INFO:root:Epoch[9] Train-loss=193.699284\n",
+ "INFO:root:Epoch[9] Time cost=4.756\n",
+ "INFO:root:Epoch[10] Train-loss=193.036517\n",
+ "INFO:root:Epoch[10] Time cost=4.757\n",
+ "INFO:root:Epoch[11] Train-loss=192.555736\n",
+ "INFO:root:Epoch[11] Time cost=4.678\n",
+ "INFO:root:Epoch[12] Train-loss=192.020813\n",
+ "INFO:root:Epoch[12] Time cost=4.630\n",
+ "INFO:root:Epoch[13] Train-loss=191.648876\n",
+ "INFO:root:Epoch[13] Time cost=5.158\n",
+ "INFO:root:Epoch[14] Train-loss=191.057798\n",
+ "INFO:root:Epoch[14] Time cost=4.781\n",
+ "INFO:root:Epoch[15] Train-loss=190.315835\n",
+ "INFO:root:Epoch[15] Time cost=5.117\n",
+ "INFO:root:Epoch[16] Train-loss=189.311271\n",
+ "INFO:root:Epoch[16] Time cost=4.707\n",
+ "INFO:root:Epoch[17] Train-loss=187.285967\n",
+ "INFO:root:Epoch[17] Time cost=4.745\n",
+ "INFO:root:Epoch[18] Train-loss=185.271324\n",
+ "INFO:root:Epoch[18] Time cost=4.692\n",
+ "INFO:root:Epoch[19] Train-loss=183.510888\n",
+ "INFO:root:Epoch[19] Time cost=4.762\n",
+ "INFO:root:Epoch[20] Train-loss=181.756008\n",
+ "INFO:root:Epoch[20] Time cost=4.838\n",
+ "INFO:root:Epoch[21] Train-loss=180.546818\n",
+ "INFO:root:Epoch[21] Time cost=4.764\n",
+ "INFO:root:Epoch[22] Train-loss=179.479776\n",
+ "INFO:root:Epoch[22] Time cost=4.791\n",
+ "INFO:root:Epoch[23] Train-loss=178.352077\n",
+ "INFO:root:Epoch[23] Time cost=4.981\n",
+ "INFO:root:Epoch[24] Train-loss=177.385084\n",
+ "INFO:root:Epoch[24] Time cost=5.292\n",
+ "INFO:root:Epoch[25] Train-loss=175.920123\n",
+ "INFO:root:Epoch[25] Time cost=5.097\n",
+ "INFO:root:Epoch[26] Train-loss=174.377171\n",
+ "INFO:root:Epoch[26] Time cost=4.907\n",
+ "INFO:root:Epoch[27] Train-loss=172.590589\n",
+ "INFO:root:Epoch[27] Time cost=4.484\n",
+ "INFO:root:Epoch[28] Train-loss=170.933683\n",
+ "INFO:root:Epoch[28] Time cost=4.348\n",
+ "INFO:root:Epoch[29] Train-loss=169.866807\n",
+ "INFO:root:Epoch[29] Time cost=4.647\n",
+ "INFO:root:Epoch[30] Train-loss=169.182084\n",
+ "INFO:root:Epoch[30] Time cost=5.034\n",
+ "INFO:root:Epoch[31] Train-loss=168.121719\n",
+ "INFO:root:Epoch[31] Time cost=5.615\n",
+ "INFO:root:Epoch[32] Train-loss=167.389992\n",
+ "INFO:root:Epoch[32] Time cost=4.733\n",
+ "INFO:root:Epoch[33] Train-loss=166.189067\n",
+ "INFO:root:Epoch[33] Time cost=5.041\n",
+ "INFO:root:Epoch[34] Train-loss=163.783392\n",
+ "INFO:root:Epoch[34] Time cost=5.168\n",
+ "INFO:root:Epoch[35] Train-loss=162.167959\n",
+ "INFO:root:Epoch[35] Time cost=5.019\n",
+ "INFO:root:Epoch[36] Train-loss=161.192039\n",
+ "INFO:root:Epoch[36] Time cost=5.064\n",
+ "INFO:root:Epoch[37] Train-loss=160.307114\n",
+ "INFO:root:Epoch[37] Time cost=5.180\n",
+ "INFO:root:Epoch[38] Train-loss=159.591957\n",
+ "INFO:root:Epoch[38] Time cost=5.440\n",
+ "INFO:root:Epoch[39] Train-loss=159.109593\n",
+ "INFO:root:Epoch[39] Time cost=5.119\n",
+ "INFO:root:Epoch[40] Train-loss=158.463844\n",
+ "INFO:root:Epoch[40] Time cost=5.299\n",
+ "INFO:root:Epoch[41] Train-loss=158.037287\n",
+ "INFO:root:Epoch[41] Time cost=4.856\n",
+ "INFO:root:Epoch[42] Train-loss=157.598576\n",
+ "INFO:root:Epoch[42] Time cost=5.227\n",
+ "INFO:root:Epoch[43] Train-loss=157.097344\n",
+ "INFO:root:Epoch[43] Time cost=5.237\n",
+ "INFO:root:Epoch[44] Train-loss=156.594472\n",
+ "INFO:root:Epoch[44] Time cost=4.783\n",
+ "INFO:root:Epoch[45] Train-loss=156.177069\n",
+ "INFO:root:Epoch[45] Time cost=4.834\n",
+ "INFO:root:Epoch[46] Train-loss=155.825302\n",
+ "INFO:root:Epoch[46] Time cost=4.902\n",
+ "INFO:root:Epoch[47] Train-loss=155.318117\n",
+ "INFO:root:Epoch[47] Time cost=4.966\n",
+ "INFO:root:Epoch[48] Train-loss=154.890766\n",
+ "INFO:root:Epoch[48] Time cost=5.012\n",
+ "INFO:root:Epoch[49] Train-loss=154.504158\n",
+ "INFO:root:Epoch[49] Time cost=4.844\n",
+ "INFO:root:Epoch[50] Train-loss=154.035214\n",
+ "INFO:root:Epoch[50] Time cost=4.736\n",
+ "INFO:root:Epoch[51] Train-loss=153.692903\n",
+ "INFO:root:Epoch[51] Time cost=5.057\n",
+ "INFO:root:Epoch[52] Train-loss=153.257554\n",
+ "INFO:root:Epoch[52] Time cost=5.044\n",
+ "INFO:root:Epoch[53] Train-loss=152.849715\n",
+ "INFO:root:Epoch[53] Time cost=4.783\n",
+ "INFO:root:Epoch[54] Train-loss=152.483047\n",
+ "INFO:root:Epoch[54] Time cost=4.842\n",
+ "INFO:root:Epoch[55] Train-loss=152.091617\n",
+ "INFO:root:Epoch[55] Time cost=5.044\n",
+ "INFO:root:Epoch[56] Train-loss=151.715490\n",
+ "INFO:root:Epoch[56] Time cost=5.029\n",
+ "INFO:root:Epoch[57] Train-loss=151.362293\n",
+ "INFO:root:Epoch[57] Time cost=4.873\n",
+ "INFO:root:Epoch[58] Train-loss=151.003241\n",
+ "INFO:root:Epoch[58] Time cost=4.729\n",
+ "INFO:root:Epoch[59] Train-loss=150.619678\n",
+ "INFO:root:Epoch[59] Time cost=5.068\n",
+ "INFO:root:Epoch[60] Train-loss=150.296043\n",
+ "INFO:root:Epoch[60] Time cost=4.458\n",
+ "INFO:root:Epoch[61] Train-loss=149.964152\n",
+ "INFO:root:Epoch[61] Time cost=4.828\n",
+ "INFO:root:Epoch[62] Train-loss=149.694102\n",
+ "INFO:root:Epoch[62] Time cost=5.012\n",
+ "INFO:root:Epoch[63] Train-loss=149.290113\n",
+ "INFO:root:Epoch[63] Time cost=5.193\n",
+ "INFO:root:Epoch[64] Train-loss=148.934186\n",
+ "INFO:root:Epoch[64] Time cost=4.999\n",
+ "INFO:root:Epoch[65] Train-loss=148.657502\n",
+ "INFO:root:Epoch[65] Time cost=4.810\n",
+ "INFO:root:Epoch[66] Train-loss=148.331948\n",
+ "INFO:root:Epoch[66] Time cost=5.201\n",
+ "INFO:root:Epoch[67] Train-loss=148.018539\n",
+ "INFO:root:Epoch[67] Time cost=4.833\n",
+ "INFO:root:Epoch[68] Train-loss=147.746825\n",
+ "INFO:root:Epoch[68] Time cost=5.187\n",
+ "INFO:root:Epoch[69] Train-loss=147.406399\n",
+ "INFO:root:Epoch[69] Time cost=5.355\n",
+ "INFO:root:Epoch[70] Train-loss=147.181831\n",
+ "INFO:root:Epoch[70] Time cost=4.989\n",
+ "INFO:root:Epoch[71] Train-loss=146.860770\n",
+ "INFO:root:Epoch[71] Time cost=4.934\n",
+ "INFO:root:Epoch[72] Train-loss=146.604369\n",
+ "INFO:root:Epoch[72] Time cost=5.283\n",
+ "INFO:root:Epoch[73] Train-loss=146.351628\n",
+ "INFO:root:Epoch[73] Time cost=5.062\n",
+ "INFO:root:Epoch[74] Train-loss=146.102506\n",
+ "INFO:root:Epoch[74] Time cost=4.540\n",
+ "INFO:root:Epoch[75] Train-loss=145.828805\n",
+ "INFO:root:Epoch[75] Time cost=4.875\n",
+ "INFO:root:Epoch[76] Train-loss=145.571626\n",
+ "INFO:root:Epoch[76] Time cost=4.856\n",
+ "INFO:root:Epoch[77] Train-loss=145.365383\n",
+ "INFO:root:Epoch[77] Time cost=5.003\n",
+ "INFO:root:Epoch[78] Train-loss=145.101047\n",
+ "INFO:root:Epoch[78] Time cost=4.718\n",
+ "INFO:root:Epoch[79] Train-loss=144.810765\n",
+ "INFO:root:Epoch[79] Time cost=5.127\n",
+ "INFO:root:Epoch[80] Train-loss=144.619876\n",
+ "INFO:root:Epoch[80] Time cost=4.737\n",
+ "INFO:root:Epoch[81] Train-loss=144.399066\n",
+ "INFO:root:Epoch[81] Time cost=4.742\n",
+ "INFO:root:Epoch[82] Train-loss=144.220090\n",
+ "INFO:root:Epoch[82] Time cost=4.810\n",
+ "INFO:root:Epoch[83] Train-loss=143.904279\n",
+ "INFO:root:Epoch[83] Time cost=5.176\n",
+ "INFO:root:Epoch[84] Train-loss=143.734935\n",
+ "INFO:root:Epoch[84] Time cost=4.921\n",
+ "INFO:root:Epoch[85] Train-loss=143.499403\n",
+ "INFO:root:Epoch[85] Time cost=4.692\n",
+ "INFO:root:Epoch[86] Train-loss=143.304287\n",
+ "INFO:root:Epoch[86] Time cost=4.778\n",
+ "INFO:root:Epoch[87] Train-loss=143.096145\n",
+ "INFO:root:Epoch[87] Time cost=4.962\n",
+ "INFO:root:Epoch[88] Train-loss=142.877920\n",
+ "INFO:root:Epoch[88] Time cost=4.815\n",
+ "INFO:root:Epoch[89] Train-loss=142.677429\n",
+ "INFO:root:Epoch[89] Time cost=5.127\n",
+ "INFO:root:Epoch[90] Train-loss=142.499622\n",
+ "INFO:root:Epoch[90] Time cost=5.463\n",
+ "INFO:root:Epoch[91] Train-loss=142.300291\n",
+ "INFO:root:Epoch[91] Time cost=4.639\n",
+ "INFO:root:Epoch[92] Train-loss=142.111362\n",
+ "INFO:root:Epoch[92] Time cost=5.064\n",
+ "INFO:root:Epoch[93] Train-loss=141.912848\n",
+ "INFO:root:Epoch[93] Time cost=4.894\n",
+ "INFO:root:Epoch[94] Train-loss=141.723130\n",
+ "INFO:root:Epoch[94] Time cost=4.635\n",
+ "INFO:root:Epoch[95] Train-loss=141.516580\n",
+ "INFO:root:Epoch[95] Time cost=5.063\n",
+ "INFO:root:Epoch[96] Train-loss=141.362380\n",
+ "INFO:root:Epoch[96] Time cost=4.785\n",
+ "INFO:root:Epoch[97] Train-loss=141.178878\n",
+ "INFO:root:Epoch[97] Time cost=4.699\n",
+ "INFO:root:Epoch[98] Train-loss=141.004168\n",
+ "INFO:root:Epoch[98] Time cost=4.959\n",
+ "INFO:root:Epoch[99] Train-loss=140.865592\n",
+ "INFO:root:Epoch[99] Time cost=5.155\n"
+ ]
+ }
+ ],
+ "source": [
+ "# training the model, save training loss as a list.\n",
+ "training_loss=list()\n",
+ "\n",
+ "# initilize the parameters for training using Normal.\n",
+ "init = mx.init.Normal(0.01)\n",
+ "model.fit(nd_iter, # train data\n",
+ " initializer=init,\n",
+ " # if eval_data is supplied, test loss will also be reported\n",
+ " # eval_data = nd_iter_test,\n",
+ " optimizer='sgd', # use SGD to train\n",
+ " optimizer_params={'learning_rate':1e-3,'wd':1e-2}, \n",
+ " # save parameters for each epoch if model_prefix is supplied\n",
+ " epoch_end_callback = None if model_prefix==None else mx.callback.do_checkpoint(model_prefix, 1),\n",
+ " batch_end_callback = log_to_list(N/batch_size,training_loss), \n",
+ " num_epoch=100,\n",
+ " eval_metric = 'Loss')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "DEBUG:matplotlib.font_manager:findfont: Matching :family=sans-serif:style=normal:variant=normal:weight=normal:stretch=normal:size=12.0 to DejaVu Sans ('/usr/local/lib/python3.5/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf') with score of 0.050000\n"
+ ]
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEWCAYAAABIVsEJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmYXFWd//H3t6t676S7kw5kX4AAsgbMBMQNFQUZZ9AZxsFlwJVxfjjO4vxQ0BkUxVHHGZfRB0WG38AMi7tEh0cEFRjHYQmrbEIgW4ck3Ul3J7133brf3x/3VlPp1K0Kla6uTufzep56UnXurapz63bO957lnmPujoiIyEtVU+0MiIjIwUkBREREyqIAIiIiZVEAERGRsiiAiIhIWRRARESkLAogMqnM7Jtm9veTve9MYGaNZvYTM9ttZt+rwvdfbmbXHui+ZnammXVObu4S87HRzM6aiu+Sly5d7QzI9GFmG4EPuPud5X6Gu3+oEvvOEOcDhwNz3T2Y6i93989VYt+XwsyWAxuA2mr8BjK5VAOR/WZmh9QFRwWOdxnwTDkF56H228vBQQFEADCz/wCWAj8xswEzu9TMlpuZm9n7zWwz8Mt43++Z2fa4KeYeMzs+73P+3cw+Gz8/08w6zeyjZtZlZtvM7L1l7js3bv7ZY2YPmNlnzezXRY7nVWb2GzPrM7MtZvaeOP0uM/tA3n7vyf+c+HgvMbNngWfN7Goz+9KEz77VzP42fr7QzH5gZt1mtsHMPpKQn08D/wD8afz7vt/Maszsk2a2KT7mG8ysNd6/4G8/4TNzv9mleb/ZW83sXDN7xsx6zOzyvP0/ZWb/OeHzLzKzzWa208w+UWjfIr/x5fH7NprZu/LSf9/MHo7P1RYz+1Te2+6J/+2Lf4dXxO/5oJk9ZWb9ZvakmZ2a955VZvZY/Pf2HTNryPuut5jZI/F5/o2ZnZS37WNmtjX+zN+Z2RuKHY+Uwd310AN3B9gInJX3ejngwA1AM9AYp78PmAXUA18BHsl7z78Dn42fnwkEwJVALXAuMAS0l7HvLfGjCTgO2AL8OuE4lgH9wDviz5oLrIq33UXUTJfb9z35nxMf7x3AHKAReE38XRZvbweGgYVEF2APEgWGOuAI4Hng7IR8fQr4z7zX7wPWx+9rAX4I/Eex337C5+V+s3+Ij/ODQDdwU3x+jo/zumLi9+d9/rfj4zwZGAVeViivCd/7L/HfwGuBQeCYvO0nxr/PScAO4K0Tvjed93l/AmwFfg8w4ChgWd7f5P3x7z0HeAr4ULztFKALOA1IARfF+9cDx8TnbWHe9x5Z7f9jM+2hGojsj0+5+6C7DwO4+3Xu3u/uo0QFzcm5K+cCMsCV7p5x99uAAaL/3Pu9r5mlgD8GrnD3IXd/Eri+SH7fCdzp7jfHn7XL3R95Ccf7j+7eEx/vfxMVeK+Ot50P/K+7v0BU4M1z9yvdfczdnycqkC/Yz+95F/Av7v68uw8AlwEXTGiu2uu3LyADXOXuGaIA2wF8NT4/TwBPEgWHJJ9292F3fxR4tMS+E/29u4+6+93AfwFvB3D3u9z9t+4euvtjwM1EQSbJB4AvuvsDHlnv7pvytn/N3V9w9x7gJ8CqOP1i4Fvufp+7Z939eqIgeDqQJQokx5lZrbtvdPfnXsKxyX5QAJH9sSX3xMxSZvZ5M3vOzPYQXfFBVHAVssv3bvMfIrrafin7ziMa8LElb1v+84mWAAdSWIx/trs7UcH8jjjpncCN8fNlwMK4+aTPzPqAy4k6yvfHQiC/oNxEdJz57y92nBD9Ztn4eS7I7MjbPkzy7w2wPe95sXMzUa+7D+a93kR0PJjZaWb2q7hZbzfwIZL/PqD0+UrK4zLgoxN+/yVEtY71wF8TXeB0mdktZrZwP49N9pMCiORLmpo5P/2dwHnAWUArUdMARE0PldJN1GSyOC9tSZH9twBHJmwbJGoGy5lfYJ+Jv8PNwPlmtoyoueQHed+zwd3b8h6z3P3cInnL9wJRIZizlOg48wPAdJ0uu93MmvNeLyU6Hoia0NYCS9y9FfgmL/59FDqeYuermC1Eta/837/J3W8GcPeb3P1VRL+xA18o4zukCAUQybeDqD2+mFlEzQS7iAriigz3zBdfYf8Q+JSZNZnZscCFRd5yI3CWmb3dzNJxB3yu2eMR4I/izzkKeP9+fP/DwE7gWuB2d++LN90P9MedtY1x7ewEM/u9/Ty0m4G/MbMVZtZC9Ft+xw+e4a2fNrM6M3s18BYgd2/LLKDH3UfMbA3RRUdONxCy99/ZtcDfmdnLLXJUHKxL+TbwobjGY2bWHHfgzzKzY8zs9WZWD4wQ1cTCAzxemUABRPL9I/DJuDng7xL2uYGouWIrUfv6vVOUtw8T1Xi2A/9BVPiOFtrR3TcTdcJ/FOghChq5tv0vA2NEwfJ6XmyOKuUmolrXTXnfkyUqOFcR3duQCzJJ/UETXRcfyz3x+0eAv9zP91bbdqCXqNZxI1HH9tPxtv8DXGlm/UQd/N/Nvcndh4CrgP+J/85Od/fvxWk3EQ1++DFRh3lR7r6OaODA1+O8rCcaFAFR/8fnic7JduAwoj4mmUS5kSUiBxUz+wIw390vqnZeRA5VqoHIQcHMjjWzk+KmijVETU8/qna+RA5lurtVDhaziJqtFhI1P/0zcGtVcyRyiFMTloiIlEVNWCIiUpYZ3YTV0dHhy5cvr3Y2REQOKg8++OBOd59Xar8ZHUCWL1/OunXrqp0NEZGDipltKr2XmrBERKRMCiAiIlIWBRARESmLAoiIiJRFAURERMqiACIiImVRABERkbLM6PtAREQOBkNjATv7x+gbHqOjpZ7DZzeQqonW4BocDdi2e4TB0YCxbMhoJmTPSIaewTH6hsYIQqcuXUNdqoa6dA2pGiNdY8ybVc/rj93fxTHLowAiIhIbC0I6e4foHwnIZEPGsiFhCKE7oTtmRo1ByozhTJau/lG69ozSP5IhlTJSZoQOA6MZBkYCRjIhqVRUoNeYMZLJMpzJMjSWZc9wht3xY2gsu1c+alPG/NYG9gwH7B7OlHUspyxtUwAREUni7uwaHGPTriG69oxgFhXWZjAahAyPRQX2SCbL8FiWoUyWgZGAPSOZ8SCRDZ0gdLbtHmZr7zBhGfPLNtamXgwyGC0NaVrq0zTU1hCETpCNtjXWpmisS9FYm2LJnCZOaKyltbGWuS11dLTU09ZYS/fAKFt6hnmhb5jWxloWtDWwsLWRWQ1p6tMp6tI1zG5M095UR1tTLemaGjLZkNEgJMg7nlwNppIUQESkYsaCkL6hMXqGxkiZ0VyfprkuTf9ohm27R3ihb5gg6zTXp2iuTzM0lmXzriE29wyxbfcw3QNj7OyPrvBDj2oC7kRX9DVGJhvuc/VeTLrGmNWQZlZDLbMa0lGTj0WftWpJO29btYhlc5tpb44K5nTKqE3VEJXFUYEcupONm40On93AvJZ66tLV7U5O1aRoqE1N+fcqgIjIPsLQ6R0ao3coM15oB2HISCa6qh8aC+jPu5Lfndcc0zs4xq7BMXYOjNI/Ut7y7rMb0ixqb6KjpY4jO5qZ3VhLjRmpuJzOhpANQ1I1NSyZ08jSOU0saG2Mt0VX+w210ZV+Q11N9G9titqUxg1NJgUQkRlmJJMlCJ3muhRm0VXz0FjAroExegbHxgv6gdGA0UyWkSBk93CGLT1DdPZGTSe7BsfIvoS2nOa6FLPj5pg5zXWcsKiVuc11zG2uo725jvamOkJ3BkcDBkYDWurTLGhrZGFrA3XpGgZGAwZHszTU1rBsTjOtTbWV+nlkEimAiEwz7k7P4BizGmrHm0bC0NnaN8z6rgH6hscYHI1qAcNjISNB1L7f2TvMs139bO4Zwh1qDJrr0wRZZzhTvJmnNmUsbGtkcXsjZx4zj3mz6pnXUk97cx2puAO4xqChNkVTXZrG2hSzG9PMbqilpSGtK/tDlAKISAXlgkHv0BhjgZPJhgyOBWzri9r/dw6MjnfajmSyPNc9wLM7BugfjZp+OlrqmNNcx9beYQYT2vprU0Z9OsWC1gZOWNjK205ZRFNdiv6RqJkpVWN0tNQzt6WOOU11tDZFNYWokzdFQ20NDekUNVPQ6SoziwKISAm5kT4DIwFB3L6+s3+UzT1RZ2+und8MRjMhuwZH2TkwRnf/KF39I2SyyU1BsxvSpOOr91SNcURHM287NerI7R/JsGPPCDsHxjjjyA6OPnwWRx/ewtyWeprrX6wJTMVoG5FCFEDkkDQ4GjA4Go3TH8pEN2p19kbDOHcPZ8bb6l/oG2ZLz1Di1X9uVA+AA3WpGua21NPRUseKjmYOn93A4bPrmdNcR326htpU1KG7oK2RBa0NVRk5IzJZFEBkRtqxZ4SHNvXy9Pb+8fHxQ5ksG7oHWd89QHf/aMH31aaM1sba8eGmC9saOf2IuSyb20RbUy2pmmjYZ1tTbTzyp2G8BiFyqFEAkRlhcDTg1+t38qunu/j1+p109g4DUbNSbaqGdI1Rn65h2dxmXnv0PI6Y18zshtrxPoD5sxtY3N7EYbPq1Rcgsp8UQOSglMmGPNa5m/99bif/s34XD27qZSwb0lKf5pVHzeU9Zyzn5cvaOX5ha9Vv8hKZqRRAZNpzd17YPcKjW/p4ZEsfD2/u5bHO3YwGIQDHLZjNRWcs43XHHMbq5XMUMESmiAKITBuDowEPbOzh3ud72NI7xMBI1JG9adcQOweiPou6VA3HL5rNu09fxsuXtXP6EXOZ01xX5ZyLHJoUQKQqBkcDfvF0F09t28PmXUNs3DXI77b3E4RObcpY0t7ErIY0zfVpXnN0B6uWtHHS4jZetmAW9WmNXBKZDhRAZMqMBSG/eW4ntz7yAj97fDvDmSzpGmNxeyNL5zbzwdfM44wj5/LyZe001elPU2S60/9SqRh3p7N3mIe39HHnkzv41dNd9I8GzG5I89ZTFvG2UxZx6tI2DYMVOUgpgMikCkPnrme6uOm+LTy8uZddg2MAtDfVcs4J8zn7+Pm8amWHbqATmQEUQGS/BdmQrv5Rtu8ZYdfAGLsGRtkzkqEuVUNTXZqB0YAb79vEc92DzJ/dwOuOPYyTl7Rx8uJWjlswWzUNkRlGAUQS7R7KcNczXfziqS7WbexhR/9oySm+T1g0m69esIpzT1ygGVpFZjgFEBn34KZebvjfjWzbPcKOPdHcUNnQmdtcxxlHdbBsThML4zmcOlrqmdNSR2tjLWNByHAmSxg6i9sbx9egEJGZTQFExn3xZ0/z+NbdHL+wlZMWt3HeyQt57TGHsWpJW/EZX+unLo8iMn0ogBxCdg9n+PkT2/mv327jpMVt/O0bjx7ftnNglAc29vDh1x3F377pmCrmUkQOFgogM1AmG7Jh5yDP7Ohn066h8Rv1Ht7cx1g2pKG2ht+s38V7zlg+fhf3HU/uIHQ454QFVc69iBwsFECmscHRgK19w4xmQjJhyFgQsnNglK49o+wcGCWTDcmGkA1DeocydPWP0NU/ypaeob0WMepoqWfpnEbeffoy/uDkBTTVpTn7K/fww4c6+cCrjwDgZ49vZ9ncJl62YFa1DldEDjIKINPIWBDy40e28qOHtvJc9wBdCWtWQLR6XV2qJl6vGtqb65jXUs+x82dx9vHzOfrwFlYeNosj5jUXvKv71KVt3HT/Zt7/qhXsGQn4zXM7ed8rV6gDXET2mwJIFWzYOcj1v9nII1v6OOqwFo6dPwt3uO5/NrBt9wgrD2vhNUfPY0VHM0vmNNFYmyKdigLGnOY6Dp/dQFtj7QGtW3HBmqVc+v3HeGBjL529UY3lnBPmT+JRishMpwAyhe7f0MM3736OXz7dRW3KWLWkjbuf6eb7D3YCsGb5HD73Rydy5tHzKl4TeMtJC/jMT57klvs3MzAaMH92Aycvbqvod4rIzKIAMgUe6+zjSz9/hnue6aajpY6/esNK3nX6Ug6b1QBEI6B2D2c4cl7LlOWpqS7NH65aOB683rFmqVbiE5GXpCoBxMz+CfgDYAx4Dnivu/fF2y4D3g9kgY+4++1x+jnAV4EUcK27f74aed8f7s4zOwb41e+6+OXTXdy/oYf2plouP/dY/uz05TTW7T0PVEdLPR0tU38zxTvWLOXG+zYDcPbxar4SkZemWjWQO4DL3D0wsy8AlwEfM7PjgAuA44GFwJ1mlrtZ4RvAG4FO4AEzW+vuT1Yh7/vYM5Lh1oe38tDmPp7fOciG7gH2jARAtFre/z37GC58xTJmNdRWOad7O2FRKycuauWFvmHWrJhT7eyIyEGmKgHE3X+e9/Je4Pz4+XnALe4+Cmwws/XAmnjbend/HsDMbon3rVoAcXce2dLHzfdv5iePbmM4k2VhawNHzGvhvFWLOH7hbM485jDmtzZUK4v75V/fcQr9I0HxO81FRAqYDn0g7wO+Ez9fRBRQcjrjNIAtE9JPK/RhZnYxcDHA0qVLJzWjADv2jPD9Bzv5wUOdPN89SFNdireespB3rlnGiYtbJ/37Km15R3O1syAiB6mKBRAzuxMo1LD+CXe/Nd7nE0AA3DhZ3+vu1wDXAKxevbr41LEvUVf/CGd/5R76hjKsWTGHP3/NEZx74oJp1zQlIjIVKhZA3P2sYtvN7D3AW4A3uHuuoN8KLMnbbXGcRpH0KfPZnz7F0GiWn/7lqzhh0cFX2xARmUxVWbAhHlF1KfCH7j6Ut2ktcIGZ1ZvZCmAlcD/wALDSzFaYWR1RR/vaqczzPc90s/bRF/iLM49U8BARoXp9IF8nmgT8jviGuXvd/UPu/oSZfZeoczwALnH3LICZfRi4nWgY73Xu/sRUZXYkk+WTP36cIzqa+Yszj5yqrxURmdaqNQrrqCLbrgKuKpB+G3BbJfOV5F9/+Sybe4a46YOnaS1vEZGY1hwtIQydb//3Bv7g5IWccWRHtbMjIjJtKICUMJaNplE/dr6mORcRyacAUkIQRgPEalO60U5EJJ8CSAlBNgQgXaOfSkQkn0rFEnIr+6kGIiKyNwWQEoIwroGk9FOJiORTqVhCENdA0ppsUERkLwogJWRyfSBqwhIR2YsCSAm5UVjqRBcR2ZtKxRICdaKLiBSkAFLCeCe6aiAiIntRqVhCbhiv+kBERPamAFJC7kbCWg3jFRHZi0rFEl7sRFcNREQknwJICS8O49VPJSKST6ViCRqFJSJSmAJICRqFJSJSmErFEjSZoohIYQogJWgyRRGRwlQqlpDRZIoiIgUpgJTwYie6fioRkXwqFUt4sQlLNRARkXwKICWMd6JrFJaIyF5UKpYQaD0QEZGCFEBKyE1lklInuojIXhRASlAnuohIYSoVSwjCEDPVQEREJlIAKSGTdXWgi4gUoJKxhCAbqgNdRKQABZASgtB1F7qISAEKICVksqE60EVEClDJWEKQdTVhiYgUoABSQiYMtRaIiEgBKhlLCLKutUBERApQACkhCEOtBSIiUoBKxhIyWY3CEhEpRAGkhECjsEREClLJWEIQahSWiEghCiAlZLKhpjIRESmgKiWjmX3GzB4zs0fM7OdmtjBONzP7mpmtj7efmveei8zs2fhx0VTlVfeBiIgUVq1L639y95PcfRXwU+Af4vQ3Ayvjx8XA1QBmNge4AjgNWANcYWbtU5HRTOiaiVdEpICqBBB335P3shnw+Pl5wA0euRdoM7MFwNnAHe7e4+69wB3AOVORV3Wii4gUlq7WF5vZVcCFwG7gdXHyImBL3m6dcVpSesVlNZmiiEhBFbu0NrM7zezxAo/zANz9E+6+BLgR+PAkfu/FZrbOzNZ1d3cf8OdpMkURkcIqVgNx97P2c9cbgduI+ji2Akvyti2O07YCZ05Ivyvhe68BrgFYvXq1F9rnpdAwXhGRwqo1Cmtl3svzgKfj52uBC+PRWKcDu919G3A78CYza487z98Up1VckHVNpigiUkC1+kA+b2bHACGwCfhQnH4bcC6wHhgC3gvg7j1m9hnggXi/K929ZyoyGjVhqQYiIjJRVQKIu/9xQroDlyRsuw64rpL5KkRNWCIihaltpoRMVuuBiIgUopKxBK0HIiJSmAJICVoPRESksJIlo5mdYGY35O6tMLPrzeykqchctbk7maxTqxsJRUT2UTSAxDf9/Yjonov3xY+7gR/kbgicybJhdBuJaiAiIvsqNQrrSuCN7r4xL+0xM/slcGv8mLGC8QCiGoiIyESlLq3TE4IHAHFabSUyNJ1ksiGA1gMRESmgVMkYmNnSiYlmtgwIKpOl6SPIqgYiIpKkVBPWFcCdZvY54ME4bTXwceBjlczYdJAJoxqIZuMVEdlX0QDi7j82sw3AR4G/jJOfBN7u7o9WOnPV9mINRE1YIiITlZzKJA4UF05BXqad8VFYqoGIiOyj1DDeDjO7wsw+YmYtZnZ1vKbHrWZ21FRlslrGO9FVAxER2UepkvEmoJ5ojfL7gQ3A+UTrmF9b2axVn4bxiogkK9WEdbi7X25mBmxy9y/G6U+bWcFZc2eSXA1EkymKiOyrVMmYhfFp1ndO2BZWJEfTSK4TXZMpiojsq1QN5AgzWwtY3nPi1ysqmrNpIMgN41UfiIjIPkoFkPz5rr40YdvE1zNOJlcD0SgsEZF9lLoP5O6kbWb2HaKJFWcs3QciIpLsQErGV0xaLqap8TvR1QciIrIPXVoXMd6JrlFYIiL7KNqEZWanJm3iEJiNN8iqBiIikqRUJ/o/F9n29GRmZDrKhBrGKyKSpFQn+uumKiPTUaAbCUVEEpWaC+vSvOd/MmHb5yqVqelC64GIiCQrdWl9Qd7zyyZsO2eS8zLt5EZhaTJFEZF9lSoZLeF5odczTq4GktKNhCIi+ygVQDzheaHXM05uNl4N4xUR2VepUVgnm9keotpGY/yc+HVDRXM2DWgYr4hIslKjsFJTlZHpSOuBiIgkU9tMEeMrEqoJS0RkHyoZiwiyTo1BjTrRRUT2oQBSRCYMNROviEgClY5FBFnXWiAiIgkUQIoIsqqBiIgkUelYRCZ0TaQoIpJAAaSIIBtqIkURkQQqHYsIsq57QEREEiiAFBE1YeknEhEpRKVjEVETlmogIiKFKIAUkcm6RmGJiCSoauloZh81Mzezjvi1mdnXzGy9mT2Wvya7mV1kZs/Gj4umIn9BGGoUlohIglKz8VaMmS0B3gRszkt+M7AyfpwGXA2cZmZzgCuA1UTTyD9oZmvdvbeSeQyyrrVAREQSVLMG8mXgUvZeV+Q84AaP3Au0mdkC4GzgDnfviYPGHUzBioiZbKiJFEVEElSldDSz84Ct7v7ohE2LgC15rzvjtKT0Qp99sZmtM7N13d3dB5TPbKhhvCIiSSrWhGVmdwLzC2z6BHA5UfPVpHP3a4BrAFavXn1AqyZmQqdJnegiIgVVLIC4+1mF0s3sRGAF8KiZASwGHjKzNcBWYEne7ovjtK3AmRPS75r0TE8QZENNpigikmDKL6/d/bfufpi7L3f35UTNUae6+3ZgLXBhPBrrdGC3u28DbgfeZGbtZtZOVHu5vdJ51Z3oIiLJqjYKK8FtwLnAemAIeC+Au/eY2WeAB+L9rnT3nkpnRuuBiIgkq3oAiWshuecOXJKw33XAdVOULUDrgYiIFKPL6yK0HoiISDKVjkVoPRARkWQKIEVoPRARkWQqHYvQKCwRkWQKIEVkwlDrgYiIJFDpWESQda0HIiKSQAEkgbsThFoPREQkiUrHBEEYTaOl+0BERApTAEkQZKMAohqIiEhhKh0TZMIQQH0gIiIJFEASZMdrIAogIiKFKIAkGK+BqAlLRKQglY4Jcn0g6kQXESlMASSBOtFFRIpT6Zgg14SlyRRFRApTAEkwXgPRZIoiIgWpdEyQyeY60VUDEREpRAEkwfid6AogIiIFKYAkCHI1EDVhiYgUpNIxQUY3EoqIFKUAkiAYH4Wln0hEpBCVjgleHIWlGoiISCEKIAlyo7BUAxERKUylY4LcKCz1gYiIFKYAkiCjUVgiIkWpdEygPhARkeIUQBJk1YQlIlKUAkiCjIbxiogUpdIxgZqwRESKUwBJ8OJkivqJREQKUemYQJMpiogUpwCSQJMpiogUp9IxQW4yRdVAREQKUwBJEIQhqRrDTAFERKQQBZAEQdY1AktEpAgFkASZrOseEBGRIlRCJgjCUHehi4gUoQCSIJN1jcASESlCJWSCIBtqBJaISBFVCSBm9ikz22pmj8SPc/O2XWZm683sd2Z2dl76OXHaejP7eKXzGISuJiwRkSLSVfzuL7v7l/ITzOw44ALgeGAhcKeZHR1v/gbwRqATeMDM1rr7k5XKXCYbUqsmLBGRRNUMIIWcB9zi7qPABjNbD6yJt6139+cBzOyWeN+KBZAg66Q0jFdEJFE1L7E/bGaPmdl1ZtYepy0CtuTt0xmnJaXvw8wuNrN1Zrauu7u77MxFo7BUAxERSVKxEtLM7jSzxws8zgOuBo4EVgHbgH+erO9192vcfbW7r543b17ZnxOErk50EZEiKtaE5e5n7c9+ZvZt4Kfxy63AkrzNi+M0iqRXhO5EFxEprlqjsBbkvXwb8Hj8fC1wgZnVm9kKYCVwP/AAsNLMVphZHVFH+9pK5jGTVROWiEgx1epE/6KZrQIc2Aj8OYC7P2Fm3yXqHA+AS9w9C2BmHwZuB1LAde7+RCUzGIROQ60CiIhIkqoEEHf/syLbrgKuKpB+G3BbJfOVL8iGpOun2yA1EZHpQ5fYCaLJFNUHIiKSRAEkQRCGmgtLRKQIlZAJgqymMhERKUYBJEEmDLUeiIhIESohE+g+EBGR4hRAEmSyrvtARESKUAmZIAi1HoiISDEKIAkCrUgoIlKUSsgEGa1IKCJSlAJIAq1IKCJSnAJIAe5ONnRSasISEUmkErKAIHQAajWMV0QkkQJIAUE2CiAaxisikkwlZAGZMARQJ7qISBEKIAWM10DUhCUikkgBpIBUjfH7Jy5gxbyWamdFRGTa0opJBbQ21vKNd51a7WyIiExrqoGIiEhZFEBERKQsCiAiIlIWBRARESmLAoiIiJRFAURERMqiACIiImVRABERkbKYu1c7DxVjZt3ApgP4iA5g5yRl52BxKB4zHJrHfSgeMxyax/0mnpXeAAAFlUlEQVRSj3mZu88rtdOMDiAHyszWufvqaudjKh2KxwyH5nEfiscMh+ZxV+qY1YQlIiJlUQAREZGyKIAUd021M1AFh+Ixw6F53IfiMcOhedwVOWb1gYiISFlUAxERkbIogIiISFkUQAows3PM7Hdmtt7MPl7t/FSKmS0xs1+Z2ZNm9oSZ/VWcPsfM7jCzZ+N/26ud18lmZikze9jMfhq/XmFm98Xn/DtmVlftPE42M2szs++b2dNm9pSZvWKmn2sz+5v4b/txM7vZzBpm4rk2s+vMrMvMHs9LK3huLfK1+PgfM7OyV89TAJnAzFLAN4A3A8cB7zCz46qbq4oJgI+6+3HA6cAl8bF+HPiFu68EfhG/nmn+Cngq7/UXgC+7+1FAL/D+quSqsr4K/MzdjwVOJjr+GXuuzWwR8BFgtbufAKSAC5iZ5/rfgXMmpCWd2zcDK+PHxcDV5X6pAsi+1gDr3f15dx8DbgHOq3KeKsLdt7n7Q/HzfqICZRHR8V4f73Y98Nbq5LAyzGwx8PvAtfFrA14PfD/eZSYecyvwGuDfANx9zN37mOHnmmjZ7kYzSwNNwDZm4Ll293uAngnJSef2POAGj9wLtJnZgnK+VwFkX4uALXmvO+O0Gc3MlgOnAPcBh7v7tnjTduDwKmWrUr4CXAqE8eu5QJ+7B/HrmXjOVwDdwP+Lm+6uNbNmZvC5dvetwJeAzUSBYzfwIDP/XOckndtJK+MUQAQzawF+APy1u+/J3+bROO8ZM9bbzN4CdLn7g9XOyxRLA6cCV7v7KcAgE5qrZuC5bie62l4BLASa2beZ55BQqXOrALKvrcCSvNeL47QZycxqiYLHje7+wzh5R65KG//bVa38VcArgT80s41EzZOvJ+obaIubOWBmnvNOoNPd74tff58ooMzkc30WsMHdu909A/yQ6PzP9HOdk3RuJ62MUwDZ1wPAynikRh1Rp9vaKuepIuK2/38DnnL3f8nbtBa4KH5+EXDrVOetUtz9Mndf7O7Lic7tL939XcCvgPPj3WbUMQO4+3Zgi5kdEye9AXiSGXyuiZquTjezpvhvPXfMM/pc50k6t2uBC+PRWKcDu/Oaul4S3YlegJmdS9ROngKuc/erqpylijCzVwH/DfyWF/sDLifqB/kusJRoOvy3u/vEDrqDnpmdCfydu7/FzI4gqpHMAR4G3u3uo9XM32Qzs1VEAwfqgOeB9xJdRM7Yc21mnwb+lGjE4cPAB4ja+2fUuTazm4EziaZt3wFcAfyYAuc2DqZfJ2rOGwLe6+7ryvpeBRARESmHmrBERKQsCiAiIlIWBRARESmLAoiIiJRFAURERMqiACIyTZnZmbnZgkWmIwUQEREpiwKIyAEys3eb2f1m9oiZfStea2TAzL4cr0XxCzObF++7yszujddh+FHeGg1HmdmdZvaomT1kZkfGH9+St4bHjfFNYCLTggKIyAEws5cR3en8SndfBWSBdxFN3LfO3Y8H7ia6MxjgBuBj7n4S0QwAufQbgW+4+8nAGUSzx0I0Q/JfE61NcwTRXE4i00K69C4iUsQbgJcDD8SVg0aiSetC4DvxPv8J/DBek6PN3e+O068Hvmdms4BF7v4jAHcfAYg/735374xfPwIsB35d+cMSKU0BROTAGHC9u1+2V6LZ30/Yr9w5g/LnaMqi/7MyjagJS+TA/AI438wOg/F1qJcR/d/Kzfj6TuDX7r4b6DWzV8fpfwbcHa8G2Wlmb40/o97Mmqb0KETKoKsZkQPg7k+a2SeBn5tZDZABLiFasGlNvK2LqJ8Eomm1vxkHiNyMuBAFk2+Z2ZXxZ/zJFB6GSFk0G69IBZjZgLu3VDsfIpWkJiwRESmLaiAiIlIW1UBERKQsCiAiIlIWBRARESmLAoiIiJRFAURERMry/wGCuVFXyLXDyQAAAABJRU5ErkJggg==\n",
+ "text/plain": [
+ "<Figure size 432x288 with 1 Axes>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "ELBO = [-training_loss[i] for i in range(len(training_loss))]\n",
+ "plt.plot(ELBO)\n",
+ "plt.ylabel('ELBO');plt.xlabel('epoch');plt.title(\"training curve for mini batches\")\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "As expected, the ELBO is monotonically increasing over epoch, and we reproduced the results given in the paper [Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114/). Now we can extract/load the parameters and then feed the network forward to calculate $y$ which is the reconstructed image, and we can also calculate the ELBO for the test set. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "arg_params = model.get_params()[0]\n",
+ "nd_iter_test.reset()\n",
+ "test_batch = nd_iter_test.next()\n",
+ "\n",
+ "# if saved the parameters, can load them using `load_checkpoint` method at e.g. 100th epoch\n",
+ "# sym, arg_params, aux_params = mx.model.load_checkpoint(model_prefix, 100)\n",
+ "# assert sym.tojson() == output.tojson()\n",
+ "\n",
+ "e = y.bind(mx.cpu(), {'data': test_batch.data[0],\n",
+ " 'encoder_h_weight': arg_params['encoder_h_weight'],\n",
+ " 'encoder_h_bias': arg_params['encoder_h_bias'],\n",
+ " 'mu_weight': arg_params['mu_weight'],\n",
+ " 'mu_bias': arg_params['mu_bias'],\n",
+ " 'logvar_weight':arg_params['logvar_weight'],\n",
+ " 'logvar_bias':arg_params['logvar_bias'],\n",
+ " 'decoder_z_weight':arg_params['decoder_z_weight'],\n",
+ " 'decoder_z_bias':arg_params['decoder_z_bias'],\n",
+ " 'decoder_x_weight':arg_params['decoder_x_weight'],\n",
+ " 'decoder_x_bias':arg_params['decoder_x_bias'], \n",
+ " 'loss_label':label})\n",
+ "\n",
+ "x_fit = e.forward()\n",
+ "x_construction = x_fit[0].asnumpy()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAADACAYAAADhh27FAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmUXWWZ7/HfAxnIPJC6SQgJCRAwYQzUihG4t72SjggyZNnQAkKiMngXAi7TKtpqR1qQq40IF5cNNpNCqyBCEIIKNAi0QFPYQYaQhEwmUCEVMlUChCS894+zo4e8z0l21Rn32d/PWrVy6tnTu8956q03u/azXwshCAAAAMijPerdAAAAAKBeGAwDAAAgtxgMAwAAILcYDAMAACC3GAwDAAAgtxgMAwAAILcYDDcAM/s3M/tavdsBlMPMxppZMLMeJZa/ZGYfrnGzABf9LrKOPrdyjOcMS2a2qejbvpK2SNqefH9hCOGO2rcKeWZmyySdF0J4uN5tScvMxkpaKqlnCGFbfVuDRke/i0ZCn5tv7v8m8iaE0H/H6zQ/EGbWg8RD1pHHqCf6XeQNOdy4uE0iBTP7tpn9wsx+Zmadkj5lZreb2eyidaYmHfqO7/c1s3vMrMPMlprZRbvY/1/2tWM/ZvbVZNvXzexkM/u4mS0ys7Vm9uWibT9kZk+b2Xozazez68ysZ9Hyj5nZQjPbYGb/z8z+08xmFi0/z8xeMbN1ZvagmY2u0NuGKklyYV7ymf/BzA4vWnaZmS02s04ze9nMphctm5l8/teY2ZuSZiexJ83sX5IcWGpmHyvaZpCZ3ZTk1mvJz8KeybI9k+3WmNkSSSftpt3LzGxq8nq2md2V5H6nmb1gZgcleb/azFaY2bSibT9tZvOTdZeY2YU77fvLSRtfT3I6mNmBybLeSTv/bGZvmNm/mlmf8j4FVBv9LhoFfW7z97kMhtObLunfJQ2S9ItdrWhme0i6X9KzkkZJ+ltJXzKz41Mea18VPpt9JP2zpJskfVLSJEkflnS5mY1J1t0m6VJJwyQdK+kESRcm7fgfku6U9KVk+VJJk4va+Ylk2amSWiQ9k5wjGpSZTZJ0swqf8d6SbpB0n5n1TlZZLOl/qpCn35J0u5mNLNrFByUtkTRc0hVFsQUq5Mh3Jd1kZpYsu1WFHDtQhfybJum8ZNn5kj6exFsl/V0XT+dkST+VNETSf0v6rQp5P0rS5cm57bA6OdZASZ+WdI2ZHZW8JydI+qKkqUk7P7zTca6SdJCkI5PloyR9s4ttRX3Q76Ku6HNz0ueGEPgq+pK0TNLUnWLflvQfO8VulzS76PupkpYlr4+VtGSn9b8h6ccljvmXfSX72SRpz+T7IZKCpKOL1n9e0sdL7OsfJN2VvP6MpCeKlpmkdkkzk+8fkjSjaHkPFe7bG1XvzyHvX14eJvEfSfrnnWILJP1Nif3Mk3Rq8nqmpD/vtHympFeLvu+b5NsIFTrvLZL6FC0/U9Kjyev/kPS5omXTkm177O6cJM2W9FDRspN3yvsByb4Gl9jXvZIuTV7fLOk7RcsOTLY9MMn5zZIOKFr+IUlL6/0Z87XrfKff5aveOZjE6XND8/e53DOc3oourLufpDFmtr4otqekx1JuvyaEsKOQ5O3k3zeKlr8tqb8kmdkHJF0t6WgVfqh6qHClQSpc4fhLu0MIwcxW7tTOH5rZtUWx91S4QvJayraitvaTNMPMLi6K9VLhs5aZnavC/9jHJsv6q3D1YQcvj1fteBFCeCu5QNFf0lBJPSW1//WihfYo2sf78kvS8i6ey8457eV9f0nrkz8j/pMKVxv2UCHXXyhqR1vRvorb1JKs+1zROZgKP49ofPS7qDf63Bz0uQyG09v5sRubVfjAdxhR9HqFpEUhhAlVb1XhzxpPS/r7EMImM/sHFf60IRWuRhTfA2Qq/LmiuJ3fCCHs8s+PaCgrJF0RQrhi5wVmtp+kH0s6XtJTIYTtZjZPhY5oh648PmaFClcphgW/6KNdUvG9jmOcdcqW/DnybknnSpoTQthqZvfqr+fVrsJAYofiNq1RoZM/JITAQCN76HdRb/S5OehzuWe4++ZJOsnMhiT3B11StOwpSe+a2Swz2yu56f0wMzu6Cu0YIGmDpM1mNkHJfWuJ+yUdZYVCkB4q3OPWUrT8XyX9Y7KdzGywmXX1HiRUT88kf3Z89VCh4/2cmX3QCvqZ2UlmNkBSPxU63g6pUAAh6dDuHjyE0C7pd5KuNrOBZraHmR1gZn+TrHKnpEusULQ0RNJlZZzrrvSS1FuF89qWXLGYVrT8TkmfNrMJZtZXhT+N7ziH91R4z65J7uWUmY0ys49Wqa2oLvpdVBN9bkHu+lwGw913q6T5KvyZ4jeSfr5jQfI/uhNVKJpYpsL/lG5Q4Ub0SpslaYakzuQYf7naEEJ4Q9LfS/q+pDclHaDCTfNbkuV3JcvuMrONkv4kqaETNmfmqvA/7B1fs0MIbSoUUVwvaZ2kV1W4B00hhJdV+NPtUyr8OewwSf9ZZhvOVaFjfDk53i8l7SgO+bEKBRjPS/qjpF+VeSxXCKFThUHPnUkbzpJ0X9HyByVdJ+lRFd6Pp5NFW5J/v7IjnuT5w5IOrkZbUXW3in4X1UOfq3z2uUy6kSNWeDzL65L+LoTwRL3bA1RDcsXtRUm9S/ypEagZ+l00u2boc7ky3OTM7ITkz3C9VfhTxlZJ/1XnZgEVZWbTrfBsyyGS/q+kX2e1U0b20e+i2TVbn8tguPkdp8IzDjtU+FPc9BDCll1vAmTOhSo8F3OxClP6/p/6Ngc5R7+LZtdUfS63SQAAACC3uDIMAACA3CrrOcNWmJLvWhUepvxvIYSrdrX+sGHDwtixY8s5JKDnnntuTQihZfdrVg65i3ItW7ZMa9assd2vWTnkLSqBPhdZlTZ3uz0YTipkf6jC/O8rJT1rZvcljxpxjR07Vm1tbaUWA6mYWVdn3SkbuYtytba21vyY5C0qgT4XWZU2d8u5TWKyCvNrLwkhvKvC8x5PLWN/AAAAQE2VMxgepffPR71S759yUpJkZheYWZuZtXV0dJRxOKC2yF1kEXmLrCJ3US9VL6ALIdwYQmgNIbS2tNT0liOgLOQusoi8RVaRu6iXcgbDr0kaXfT9vkkMAAAAyIRyBsPPShpvZuPMrJekT6po7moAAACg0XX7aRIhhG1m9nlJv1Xh0Wo3hxBeqljLAAAAgCor6znDIYS5kuZWqC0AAABATTEDHQAAAHKrrCvDAAAA+KsQQr2b8D5mNZ34MpO4MgwAAIDcYjAMAACA3GIwDAAAgNxiMAwAAIDcYjAMAACA3OJpEgAAAN1QjSdH8PSH2uPKMAAAAHKLwTAAAAByi8EwAAAAcovBMAAAAHKLAjoAAIBdqOUUy2mPRaFd5XBlGAAAALnFYBgAAAC5xWAYAAAAucVgGAAAALlVVgGdmS2T1Clpu6RtIYTWSjQKQP7UskCl0ihkAbIpbb9Taj0vvn379lSxUrz+pEePeLjmrefFSrX9vffeS7XuHnvE1029WFf6wUbrMyvxNIn/HUJYU4H9AAAAADXFbRIAAADIrXIHw0HS78zsOTO7wFvBzC4wszYza+vo6CjzcEDtkLvIIvIWWUXuol7KHQwfF0I4StLHJF1kZv9r5xVCCDeGEFpDCK0tLS1lHg6oHXIXWUTeIqvIXdRLWfcMhxBeS/5dbWb3SJos6fFKNAxoduUUjHWlmMMrkvBUo6DBa081iizqXbhR7nki5r2nb775ZhSbM2eOu/0ll1wSxd56661Ux+7Xr18Uu+6669x1zz333CjmFTuh/rycShsrVQC3devWKLZhw4ZUsVL9eO/evVPFevXqFcW8fqfU7wAv3rNnzyjWt2/fVMf2iupKtanRdPvKsJn1M7MBO15LmibpxUo1DAAAAKi2cv77OlzSPcmIv4ekfw8h/KYirQIAAABqoNuD4RDCEklHVLAtAAAAQE3xaDUAAADkVm7u8n/66aej2LXXXuuuO2rUqCjWp0+fKDZjxowoNnToUHefpeLIB69QwYt5RRpbtmyJYp2dne5xNm3alGr7rhSIpG3TO++8E8XefffdKLbnnnu6x/EKMrxijr322iuKjRgxIooNGzbMPY63z7QzOaE6lixZEsW+/e1vR7HbbrutrOOUKvDZ2dtvvx3Fzj//fHfdhQsXRrErr7yy28dGZaQtjEvbN3v9myStWLEiij377LNRbOXKlVHM6/MkaeDAgVFs7733jmLeuMIr3vT6Ycnv271+c/To0VFs8ODBUSzLOZ7dlgMAAABlYjAMAACA3GIwDAAAgNxiMAwAAIDcYjAMAACA3MrN0yS8Jz8sWrSorH1eccUVUWzQoEHuulOmTCnrWLUwduzYKPbVr37VXXfMmDFVbk02lZr2ctu2bVHMm8Zz3bp1UeyVV16JYk8++aR7nD//+c9RbOPGjamOU2qq2s2bN6eKlTPVreRXUHtPfvB+xiZPnhzFZs6c6R7Hy12mz62NUtPPfulLX4pi9957b8WP7z3JxIuVqr73fO9734tixxxzTBQ75ZRTUu8T6XVlavq0T5Pwnpbj9a2SdOedd0axF154IYp5fZn3O1fyn9Tg9ZvePr0noZRqu/d7YMiQIVHMe+qF155S/WgWnszDlWEAAADkFoNhAAAA5BaDYQAAAOQWg2EAAADkVm6qRrxijHnz5rnrHnLIIVHspZdeimLPPPNMFJszZ467z9/+9rdRbNy4cVFs6dKl7vZpeTewjxw5Mop5U0h6St3g/5WvfKVL7cqLUsUc3rSXXsGZV+jwu9/9Loo9+uij7nHa29tTHSftVNCleNt75967d+8o5k2nLElr166NYhs2bIhiXju9IsHjjjvOPY433ToFdJXn5cMNN9zgrpu2WK5Pnz5R7NBDD3XXnTVrVhT7yEc+EsW8aW4vu+yyKOYVypVyxx13RLGPfvSjUcz7+UD1eP2WV9zsFZbNnTvX3ef9998fxbwitkmTJkWxAw880N2nl9NeUZ1X2Ob1owsWLHCP4z1EwOufW1paotg+++yTatus4MowAAAAcovBMAAAAHKLwTAAAAByi8EwAAAAcmu3VSNmdrOkj0taHUI4NIkNlfQLSWMlLZN0RgghvuO8gUyYMCFVrJTDDz88ip155plR7KqrrnK3X7ZsWRTzCuiWLFmSuk0e74Z6r4DOO3ZHR0cU+8AHPlBWe1CaV8yxxx7x/0+9Ipthw4a5+/SKlrziMK8YwyuSkPwCI2/WLm+WIW+2OG+GI8kvaH3ggQeimFcgkrZIsFqyMMNSrXnv/0UXXZR6ey/HvvnNb0axL3/5y11rWAqzZ8+OYt5MY5K0fPnyKPbLX/4yinkz0J199tldbxxSSTvbnDfboPf7+vHHH3eP4xU9jx49OopNnDgxinkzZ0p+/963b98o5vU7XSmEXr16dRTzZt8bMWJEFPMKAgcMGOAex/u91mh9Zporw7dKOmGn2GWSHgkhjJf0SPI9AAAAkCm7HQyHEB6XtPOlmFMl3Za8vk3SaRVuFwAAAFB13b1neHgIYccDTVdJGl5qRTO7wMzazKzN+zM80KjIXWQReYusIndRL2UX0IXCjTn+TAOF5TeGEFpDCK2l7kkEGhG5iywib5FV5C7qpbvTLr1hZiNDCO1mNlJSfBd2TpWagSVtIVpXivrS8mbKW7NmTRT74Ac/GMWmTZtW8fbkkVdA4BVEeLOjHX/88anWk6TNmzdHMa8Yw8vHUr98vHam5RXvebMzSdLWrVuj2G9+85so5r2Xw4fHf5zyij6k9MV/qK/LL788ilWjWM7j9eN33323u25ra2uqfV555ZVR7NRTT41i/fv3T7U/7Fran2mv3/EK2UvNDtuzZ88otv/++0exD33oQ1HM67ckPwe8fittsZxXFCdJ69evj2LerJ/eeKGzszOKlfod4v0e8Aoc69kPd/fK8H2SZiSvZ0jy5yAGAAAAGthuB8Nm9jNJT0k62MxWmtlnJV0l6W/NbJGkqcn3AAAAQKbs9jaJEEL8MN2C+G+3AAAAQIYwAx0AAAByi8EwAAAAcqu7T5NAg/KeJjB9+vQo5k1L+YMf/CCK9enTpzINywnvSQeSXwnsTZ3tTX3sPc1hzJgx7nG86uKBAwdGMW/aTK/iV0pf4Zt2+tNSlc0rV66MYl4+e+/HUUcdFcW8acgl/zx5mkTlvfjii6nX9abovvjiiyvZnC7xcvSss84qa5+vvPJKFPvud78bxbynaKB6vD7Km/J927Zt7vZeP+7lsxfr3bu3u89Sv0fStGnjxo1RzHtqhCStW7cuinlT23uxd955J4qVerqF97vBi3lq1TdzZRgAAAC5xWAYAAAAucVgGAAAALnFYBgAAAC5RQFdk7n11luj2KpVq6KYV6i13377VaNJuVLqZv+0BRFeoZ0XK1XY6BUleAVj5U5JnLb4wSuoKDWt6R/+8Ico9u6770axww8/PIqdeOKJUWzw4MHucSiWq42XX3459br1LGr0iotmzpwZxRYuXFjxY3v99ezZs9110/YhedOVPPHW9fpCr6jOW0/y+7hNmzZFMe/3sFfcLPn9u9cmb5rk+fPnR7H29nb3OB0dHVHM69u9Yjmvby5VZOi9R+X+fFe6f+CnCwAAALnFYBgAAAC5xWAYAAAAucVgGAAAALlFAV1GLV682I1/8YtfTLX9U089FcVGjBhRVptQmnezf9qCGG+9UsUcaY9dDV6Bx4YNG6LYnDlz3O2XLFkSxbwCE69Y7uCDD45iPXv2dI+D2mhtbU29rlfI85Of/CSKfe5zn0u9T68QyCs4+vrXvx7Ffv3rX6c+TjnOOOOMKEaBZ/V4763XTwwbNiyKlSp282are+mll6LYgw8+GMVeffVVd5977bVXFNu6dWsU6+zsjGLez9Ly5cvd43jFo9579Pbbb0cxb3bQUrOLejPteb/XvBgz0AEAAABVxmAYAAAAucVgGAAAALnFYBgAAAC5tdsCOjO7WdLHJa0OIRyaxGZLOl/Sjju1vxZCmFutRiJWqsDDu8n+9NNPj2L7779/xduErimnMKBUAV3ameEqva3kzz70wgsvRLHHHnvM3d6bpWjy5MlR7OSTT45igwYNimKlChQpUKqNcvuYW265JYqNHz8+ih1wwAHu9tdff30Uu+aaa8pqUzm8HD3nnHOiGPlZPV6f0KtXryh26KGHpopJfgGdV6jp9XsLFixw9zlgwIAo5hWh9e3bN4p551NqZjhvZjkv/7y+2Suq88Yfkl9cXe7vm0pLc2X4VkknOPFrQghHJl8MhAEAAJA5ux0MhxAelxT/1wcAAADIuHLuGf68mf3JzG42syGlVjKzC8yszczavOffAY2K3EUWkbfIKnIX9dLdwfCPJB0g6UhJ7ZKuLrViCOHGEEJrCKG1paWlm4cDao/cRRaRt8gqchf10q0Z6EIIb+x4bWY/lnR/xVqEiHdT+j333OOu691k/53vfCeKdWUGM5TPK0rwCgjSFs+UKj5Ie5y025ba3iuIWLVqVRS7/fbbo9iKFSvc44wcOTKKfeITn4hi48aNi2LeLFIUItWXV6x09dX+dZNZs2ZFsba2tig2bdq08hvWTUcccYQbf/7551NtP3Xq1NT7RPnSzvrZo0c8DBo7dmwUO+2009zjeL9zX3nllSjmFat5hWml2uT952D06NGp2vPmm2+6x0k745s3I16fPn2iWJbHFd26Mmxmxb+1pkt6sTLNAQAAAGonzaPVfibpw5KGmdlKSf8k6cNmdqSkIGmZpAur2EYAAACgKnY7GA4hnOmEb6pCWwAAAICaYgY6AAAA5BaDYQAAAORWt54mgdq66ab4rpQnnnjCXfess86KYky93JjKedpBV578UM56pdZ96623otgdd9wRxR5++OHUxzn++OOjmFeB701BmrYqGrXjvf+XXHKJu6431e3Pf/7zVDFvWthSxx8+fHgU86b87spTUA455BA3vrNLL7001XqoHi8nvCcgDBw4MIpNmTLF3aeXU4sWLYpi3hMdSvVRw4YNi2LeU3S89To7O6PY0qVL3eP069cvinnvx5Ah8VQS3nvkPdWn1D4brX/myjAAAAByi8EwAAAAcovBMAAAAHKLwTAAAAByiwK6BjNv3rwodvHFF0exwYMHu9tffvnlFW8Tmk9Xpm3etm1bFHvuueeimFdAt379+ih29NFHu8c555xzopg3BWkWijHg8wodJb9Q0ot94xvfiGKvvfaau09vSluvWC6tMWPGuPEDDzwwim3YsCGKHXbYYd0+NqrHy0mvEGzQoEHu9hMnToxi3nTOW7ZsiWLetPaSP/1x2sJhbz2vAE7yC+i86Zy9ftgroPO2LdXORit65sowAAAAcovBMAAAAHKLwTAAAAByi8EwAAAAcosCujryZk4688wzo9j27duj2Nlnn+3uk9nm8i1tAYJXLFeqmGPVqlVR7Prrr49i3ixHXiHI9OnT3eMcdNBBUcwrgvLOkQK6fNhvv/1SxaqhVIGp1z97RUxewREak9efeH2RlL4Ar1T/mvb43nHS7tMr3it1HC9PvVifPn2iWLkz0NWzb+fKMAAAAHKLwTAAAAByi8EwAAAAcovBMAAAAHJrtwV0ZjZa0k8kDZcUJN0YQrjWzIZK+oWksZKWSTojhLCuek3NNu9G95NOOimKLViwIIpNmDAhin3rW9+qTMOQS14xkFfQKUl33XVXFPv9738fxbZu3RrFvNnmTjnlFPc43uxF5RQEUlSHSlq8eLEb9wpHb7jhhmo3BzVWqj9JO5NaVwro0vL6vc2bN0exdev8oZlX8Ja2WC5tcXOpeKP1z2muDG+TNCuEMFHSFEkXmdlESZdJeiSEMF7SI8n3AAAAQGbsdjAcQmgPIfwxed0pab6kUZJOlXRbstptkk6rViMBAACAaujSPcNmNlbSJEnPSBoeQmhPFq1S4TYKb5sLzKzNzNo6OjrKaCpQW+Qusoi8RVaRu6iX1INhM+sv6W5JXwghbCxeFgo3rrhPJA8h3BhCaA0htLa0tJTVWKCWyF1kEXmLrCJ3US+pZqAzs54qDITvCCH8Kgm/YWYjQwjtZjZS0upqNbIZrF27Noo99thjqbb96U9/GsWGDh1abpOQE16RhVfsNn/+fHf7uXPnRrFNmzZFscGDB0exT33qU1Fsn332cY+TdpYioB66UhT3wAMPRLHzzjuvks1Bg0jbR3mFdl3hFeB5sXfeeSeKlWqjV7Ts9cOetIWDu4p3d71q2O2nY4XW3SRpfgjh+0WL7pM0I3k9Q9KcyjcPAAAAqJ40V4aPlXSOpBfMbF4S+5qkqyTdaWaflbRc0hnVaSIAAABQHbsdDIcQnpRU6tr18ZVtDgAAAFA7zEAHAACA3GIwDAAAgNxK9TQJpLdhwwY3PmXKlFTb33777VFs0qRJZbUJ+eE9OWL79u1RzMvTxx9/3N3nihUrolj//v2j2LRp06KYN/Vyr1693ONUupLYey9K4akVqKQnn3wyiq1fvz6KeU9gQfaV25+UM7289zSIUk+I8J5G4U2z7D05oiv9aznnUytcGQYAAEBuMRgGAABAbjEYBgAAQG4xGAYAAEBuUUBXYbfccosbX7JkSartjzvuuCjWaDeao/5KFS94BRFbtmyJYsuXL49iCxcudPfpTdl52GGHRbHTTz89ig0ZMiSKlTstaTWKMbJQ4IHsWLt2bRTzfr4mT55ci+agAZTbn3h9u9dvbd26NYqV6nP33nvvKDZw4MAo5hXgecfuSlFdo+HKMAAAAHKLwTAAAAByi8EwAAAAcovBMAAAAHKLAroyLFq0KIrNnj279g1BJqUtNuhKoYJXPOHNfPX8889HsXXr1rn79IrgjjjiiCh28MEHR7GuzFzkFZhQ2AYA6fXs2TOKjRkzxl1348aNUaylpSWKjRs3Lop5hdWlCvWy0GdzZRgAAAC5xWAYAAAAucVgGAAAALnFYBgAAAC5tdsCOjMbLeknkoZLCpJuDCFca2azJZ0vqSNZ9WshhLnVamgjeuKJJ6KYd0N6KRMmTIhiffr0KatNyI60BWNebNu2be4+N2/eHMVWrFgRxRYvXhzFNm3a5O6zb9++Ucybucgrnkg7a1K9ZaHAA/U1ffp0N37PPfdEsbfeeiuK7bvvvhVvE/LD66N69IiHcMOHD49iU6dOdfd51FFHRbH+/ftHsdGjR0exAQMGRDFvpjrJb3uj9blpniaxTdKsEMIfzWyApOfM7KFk2TUhhH+pXvMAAACA6tntYDiE0C6pPXndaWbzJY2qdsMAAACAauvSPcNmNlbSJEnPJKHPm9mfzOxmM4sfRlrY5gIzazOzto6ODm8VoCGRu8gi8hZZRe6iXlIPhs2sv6S7JX0hhLBR0o8kHSDpSBWuHF/tbRdCuDGE0BpCaPUe5gw0KnIXWUTeIqvIXdRLqsGwmfVUYSB8RwjhV5IUQngjhLA9hPCepB9Lmly9ZgIAAACVl+ZpEibpJknzQwjfL4qPTO4nlqTpkl6sThObwzHHHBPFHnrooSjG0ySwM6/qttS0l95UnN50ygcddFDq4/fr1y+KHXvssVFs8ODBUaxUOz1pp14ud4rmRqtiRjaUqsh//fXXa9wS5JHXb3lPb/Ce/rP//vu7+/SmWfaeUOEdx4uV6luz0OemeZrEsZLOkfSCmc1LYl+TdKaZHanC49aWSbqwKi0EAAAAqiTN0ySelOQN63P1TGEAAAA0H2agAwAAQG4xGAYAAEBupblnGCV85jOfSRUD0ko7bWWpggRvikyvWG78+PFRzJs6WUo/fXK5bS93ewBoVmkL6EpNiVzpYzcbrgwDAAAgtxgMAwAAILcYDAMAACC3GAwDAAAgtyxtcUxFDmbWIWl58u0wSWtqdvDq43xqZ78QQk0nri/K3UZ+X7qD86mdeuat1NjvTXdwPrVD7lYW51M7qXK3poPh9x3YrC2E0FqXg1cB55MPzfa+cD750WzvDeeTH8323nA+jYfbJAAAAJBbDIYBAACQW/UcDN9Yx2NXA+eTD832vnA++dFs7w3nkx/N9t5wPg2mbvcMAwAAAPXGbRIAAADILQbDAAAAyK2aD4Zu3yMbAAACn0lEQVTN7AQzW2Bmr5rZZbU+frnM7GYzW21mLxbFhprZQ2a2KPl3SD3b2BVmNtrMHjWzl83sJTO7NIln9pyqhdxtLORuOlnPW6m5cpe8TS/rudtMeSs1d+7WdDBsZntK+qGkj0maKOlMM5tYyzZUwK2STtgpdpmkR0II4yU9knyfFdskzQohTJQ0RdJFyWeS5XOqOHK3IZG7u9EkeSs1V+6Styk0Se7equbJW6mJc7fWV4YnS3o1hLAkhPCupJ9LOrXGbShLCOFxSWt3Cp8q6bbk9W2STqtpo8oQQmgPIfwxed0pab6kUcrwOVUJudtgyN1UMp+3UnPlLnmbWuZzt5nyVmru3K31YHiUpBVF369MYlk3PITQnrxeJWl4PRvTXWY2VtIkSc+oSc6pgsjdBkbultSseSs1wedM3u5Ss+ZuU3zOzZa7FNBVWCg8qy5zz6szs/6S7pb0hRDCxuJlWT0ndE1WP2dyF1n8nMlbZPVzbsbcrfVg+DVJo4u+3zeJZd0bZjZSkpJ/V9e5PV1iZj1VSOw7Qgi/SsKZPqcqIHcbELm7W82at1KGP2fyNpVmzd1Mf87Nmru1Hgw/K2m8mY0zs16SPinpvhq3oRrukzQjeT1D0pw6tqVLzMwk3SRpfgjh+0WLMntOVULuNhhyN5VmzVspo58zeZtas+ZuZj/nps7dEEJNvySdKGmhpMWS/rHWx69A+38mqV3SVhXuYfqspL1VqKBcJOlhSUPr3c4unM9xKvxJ40+S5iVfJ2b5nKr4XpG7DfRF7qZ+nzKdt8k5NE3ukrddeq8ynbvNlLfJ+TRt7jIdMwAAAHKLAjoAAADkFoNhAAAA5BaDYQAAAOQWg2EAAADkFoNhAAAA5BaDYQAAAOQWg2EAAADk1v8HLgAFn4S3bUIAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ "<Figure size 864x216 with 4 Axes>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "# learning images on the test set\n",
+ "f, ((ax1, ax2, ax3, ax4)) = plt.subplots(1,4, sharex='col', sharey='row',figsize=(12,3))\n",
+ "ax1.imshow(np.reshape(image_test[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
+ "ax1.set_title('True image')\n",
+ "ax2.imshow(np.reshape(x_construction[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
+ "ax2.set_title('Learned image')\n",
+ "ax3.imshow(np.reshape(image_test[99,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
+ "ax3.set_title('True image')\n",
+ "ax4.imshow(np.reshape(x_construction[99,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
+ "ax4.set_title('Learned image')\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[('loss', 140.17346005859375)]"
+ ]
+ },
+ "execution_count": 13,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# calculate the ELBO which is minus the loss for test set\n",
+ "metric = mx.metric.Loss()\n",
+ "model.score(nd_iter_test, metric)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 4. All together: MXNet-based class VAE"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from VAE import VAE"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "One can directly call the class `VAE` to do the training:\n",
+ "\n",
+ "```VAE(n_latent=5,num_hidden_ecoder=400,num_hidden_decoder=400,x_train=None,x_valid=None,\n",
+ "batch_size=100,learning_rate=0.001,weight_decay=0.01,num_epoch=100,optimizer='sgd',model_prefix=None,\n",
+ "initializer = mx.init.Normal(0.01),likelihood=Bernoulli)```\n",
+ "\n",
+ "The outputs are the learned model and training loss."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "INFO:root:Epoch[0] Train-loss=383.478870\n",
+ "INFO:root:Epoch[0] Time cost=5.075\n",
+ "INFO:root:Epoch[1] Train-loss=211.923867\n",
+ "INFO:root:Epoch[1] Time cost=4.741\n",
+ "INFO:root:Epoch[2] Train-loss=206.789445\n",
+ "INFO:root:Epoch[2] Time cost=4.601\n",
+ "INFO:root:Epoch[3] Train-loss=204.428186\n",
+ "INFO:root:Epoch[3] Time cost=4.865\n",
+ "INFO:root:Epoch[4] Train-loss=202.417322\n",
+ "INFO:root:Epoch[4] Time cost=4.606\n",
+ "INFO:root:Epoch[5] Train-loss=200.635136\n",
+ "INFO:root:Epoch[5] Time cost=4.711\n",
+ "INFO:root:Epoch[6] Train-loss=199.009614\n",
+ "INFO:root:Epoch[6] Time cost=5.159\n",
+ "INFO:root:Epoch[7] Train-loss=197.565788\n",
+ "INFO:root:Epoch[7] Time cost=4.588\n",
+ "INFO:root:Epoch[8] Train-loss=196.524507\n",
+ "INFO:root:Epoch[8] Time cost=4.905\n",
+ "INFO:root:Epoch[9] Train-loss=195.725745\n",
+ "INFO:root:Epoch[9] Time cost=4.426\n",
+ "INFO:root:Epoch[10] Train-loss=194.902025\n",
+ "INFO:root:Epoch[10] Time cost=4.685\n",
+ "INFO:root:Epoch[11] Train-loss=194.026873\n",
+ "INFO:root:Epoch[11] Time cost=4.622\n",
+ "INFO:root:Epoch[12] Train-loss=193.350646\n",
+ "INFO:root:Epoch[12] Time cost=4.712\n",
+ "INFO:root:Epoch[13] Train-loss=192.737502\n",
+ "INFO:root:Epoch[13] Time cost=4.618\n",
+ "INFO:root:Epoch[14] Train-loss=192.338165\n",
+ "INFO:root:Epoch[14] Time cost=4.763\n",
+ "INFO:root:Epoch[15] Train-loss=191.888625\n",
+ "INFO:root:Epoch[15] Time cost=5.168\n",
+ "INFO:root:Epoch[16] Train-loss=191.170650\n",
+ "INFO:root:Epoch[16] Time cost=4.809\n",
+ "INFO:root:Epoch[17] Train-loss=190.307264\n",
+ "INFO:root:Epoch[17] Time cost=4.622\n",
+ "INFO:root:Epoch[18] Train-loss=188.988063\n",
+ "INFO:root:Epoch[18] Time cost=4.543\n",
+ "INFO:root:Epoch[19] Train-loss=187.616311\n",
+ "INFO:root:Epoch[19] Time cost=5.154\n",
+ "INFO:root:Epoch[20] Train-loss=186.352783\n",
+ "INFO:root:Epoch[20] Time cost=4.661\n",
+ "INFO:root:Epoch[21] Train-loss=185.428020\n",
+ "INFO:root:Epoch[21] Time cost=5.193\n",
+ "INFO:root:Epoch[22] Train-loss=184.543097\n",
+ "INFO:root:Epoch[22] Time cost=4.519\n",
+ "INFO:root:Epoch[23] Train-loss=184.029907\n",
+ "INFO:root:Epoch[23] Time cost=4.732\n",
+ "INFO:root:Epoch[24] Train-loss=183.643270\n",
+ "INFO:root:Epoch[24] Time cost=5.011\n",
+ "INFO:root:Epoch[25] Train-loss=183.246912\n",
+ "INFO:root:Epoch[25] Time cost=4.706\n",
+ "INFO:root:Epoch[26] Train-loss=183.065233\n",
+ "INFO:root:Epoch[26] Time cost=4.673\n",
+ "INFO:root:Epoch[27] Train-loss=182.680542\n",
+ "INFO:root:Epoch[27] Time cost=4.628\n",
+ "INFO:root:Epoch[28] Train-loss=182.428677\n",
+ "INFO:root:Epoch[28] Time cost=4.772\n",
+ "INFO:root:Epoch[29] Train-loss=182.219946\n",
+ "INFO:root:Epoch[29] Time cost=4.571\n",
+ "INFO:root:Epoch[30] Train-loss=182.070927\n",
+ "INFO:root:Epoch[30] Time cost=4.603\n",
+ "INFO:root:Epoch[31] Train-loss=181.837968\n",
+ "INFO:root:Epoch[31] Time cost=4.559\n",
+ "INFO:root:Epoch[32] Train-loss=181.624303\n",
+ "INFO:root:Epoch[32] Time cost=5.069\n",
+ "INFO:root:Epoch[33] Train-loss=181.534547\n",
+ "INFO:root:Epoch[33] Time cost=4.654\n",
+ "INFO:root:Epoch[34] Train-loss=181.239556\n",
+ "INFO:root:Epoch[34] Time cost=4.776\n",
+ "INFO:root:Epoch[35] Train-loss=181.098188\n",
+ "INFO:root:Epoch[35] Time cost=4.571\n",
+ "INFO:root:Epoch[36] Train-loss=180.820560\n",
+ "INFO:root:Epoch[36] Time cost=4.815\n",
+ "INFO:root:Epoch[37] Train-loss=180.828095\n",
+ "INFO:root:Epoch[37] Time cost=4.455\n",
+ "INFO:root:Epoch[38] Train-loss=180.495569\n",
+ "INFO:root:Epoch[38] Time cost=5.096\n",
+ "INFO:root:Epoch[39] Train-loss=180.389106\n",
+ "INFO:root:Epoch[39] Time cost=4.797\n",
+ "INFO:root:Epoch[40] Train-loss=180.200965\n",
+ "INFO:root:Epoch[40] Time cost=5.054\n",
+ "INFO:root:Epoch[41] Train-loss=179.851014\n",
+ "INFO:root:Epoch[41] Time cost=4.642\n",
+ "INFO:root:Epoch[42] Train-loss=179.719933\n",
+ "INFO:root:Epoch[42] Time cost=4.603\n",
+ "INFO:root:Epoch[43] Train-loss=179.431740\n",
+ "INFO:root:Epoch[43] Time cost=4.341\n",
+ "INFO:root:Epoch[44] Train-loss=179.235384\n",
+ "INFO:root:Epoch[44] Time cost=4.638\n",
+ "INFO:root:Epoch[45] Train-loss=179.108771\n",
+ "INFO:root:Epoch[45] Time cost=4.754\n",
+ "INFO:root:Epoch[46] Train-loss=178.714163\n",
+ "INFO:root:Epoch[46] Time cost=4.457\n",
+ "INFO:root:Epoch[47] Train-loss=178.508338\n",
+ "INFO:root:Epoch[47] Time cost=4.960\n",
+ "INFO:root:Epoch[48] Train-loss=178.288002\n",
+ "INFO:root:Epoch[48] Time cost=4.562\n",
+ "INFO:root:Epoch[49] Train-loss=178.083288\n",
+ "INFO:root:Epoch[49] Time cost=4.619\n",
+ "INFO:root:Epoch[50] Train-loss=177.791330\n",
+ "INFO:root:Epoch[50] Time cost=4.580\n",
+ "INFO:root:Epoch[51] Train-loss=177.570741\n",
+ "INFO:root:Epoch[51] Time cost=4.704\n",
+ "INFO:root:Epoch[52] Train-loss=177.287114\n",
+ "INFO:root:Epoch[52] Time cost=5.172\n",
+ "INFO:root:Epoch[53] Train-loss=177.122645\n",
+ "INFO:root:Epoch[53] Time cost=4.678\n",
+ "INFO:root:Epoch[54] Train-loss=176.816022\n",
+ "INFO:root:Epoch[54] Time cost=4.819\n",
+ "INFO:root:Epoch[55] Train-loss=176.670484\n",
+ "INFO:root:Epoch[55] Time cost=4.568\n",
+ "INFO:root:Epoch[56] Train-loss=176.459671\n",
+ "INFO:root:Epoch[56] Time cost=4.450\n",
+ "INFO:root:Epoch[57] Train-loss=176.174175\n",
+ "INFO:root:Epoch[57] Time cost=4.579\n",
+ "INFO:root:Epoch[58] Train-loss=175.935856\n",
+ "INFO:root:Epoch[58] Time cost=4.552\n",
+ "INFO:root:Epoch[59] Train-loss=175.739928\n",
+ "INFO:root:Epoch[59] Time cost=4.385\n",
+ "INFO:root:Epoch[60] Train-loss=175.579695\n",
+ "INFO:root:Epoch[60] Time cost=4.496\n",
+ "INFO:root:Epoch[61] Train-loss=175.403871\n",
+ "INFO:root:Epoch[61] Time cost=5.088\n",
+ "INFO:root:Epoch[62] Train-loss=175.157114\n",
+ "INFO:root:Epoch[62] Time cost=4.628\n",
+ "INFO:root:Epoch[63] Train-loss=174.953950\n",
+ "INFO:root:Epoch[63] Time cost=4.826\n",
+ "INFO:root:Epoch[64] Train-loss=174.743393\n",
+ "INFO:root:Epoch[64] Time cost=4.832\n",
+ "INFO:root:Epoch[65] Train-loss=174.554056\n",
+ "INFO:root:Epoch[65] Time cost=4.375\n",
+ "INFO:root:Epoch[66] Train-loss=174.366719\n",
+ "INFO:root:Epoch[66] Time cost=4.583\n",
+ "INFO:root:Epoch[67] Train-loss=174.160622\n",
+ "INFO:root:Epoch[67] Time cost=4.586\n",
+ "INFO:root:Epoch[68] Train-loss=173.981699\n",
+ "INFO:root:Epoch[68] Time cost=5.149\n",
+ "INFO:root:Epoch[69] Train-loss=173.751617\n",
+ "INFO:root:Epoch[69] Time cost=4.495\n",
+ "INFO:root:Epoch[70] Train-loss=173.548732\n",
+ "INFO:root:Epoch[70] Time cost=4.588\n",
+ "INFO:root:Epoch[71] Train-loss=173.380950\n",
+ "INFO:root:Epoch[71] Time cost=5.042\n",
+ "INFO:root:Epoch[72] Train-loss=173.158519\n",
+ "INFO:root:Epoch[72] Time cost=4.817\n",
+ "INFO:root:Epoch[73] Train-loss=172.970726\n",
+ "INFO:root:Epoch[73] Time cost=4.791\n",
+ "INFO:root:Epoch[74] Train-loss=172.782357\n",
+ "INFO:root:Epoch[74] Time cost=4.377\n",
+ "INFO:root:Epoch[75] Train-loss=172.581992\n",
+ "INFO:root:Epoch[75] Time cost=4.518\n",
+ "INFO:root:Epoch[76] Train-loss=172.385020\n",
+ "INFO:root:Epoch[76] Time cost=4.863\n",
+ "INFO:root:Epoch[77] Train-loss=172.198309\n",
+ "INFO:root:Epoch[77] Time cost=5.104\n",
+ "INFO:root:Epoch[78] Train-loss=172.022333\n",
+ "INFO:root:Epoch[78] Time cost=4.571\n",
+ "INFO:root:Epoch[79] Train-loss=171.816585\n",
+ "INFO:root:Epoch[79] Time cost=4.557\n",
+ "INFO:root:Epoch[80] Train-loss=171.643714\n",
+ "INFO:root:Epoch[80] Time cost=4.567\n",
+ "INFO:root:Epoch[81] Train-loss=171.460581\n",
+ "INFO:root:Epoch[81] Time cost=4.735\n",
+ "INFO:root:Epoch[82] Train-loss=171.284854\n",
+ "INFO:root:Epoch[82] Time cost=5.012\n",
+ "INFO:root:Epoch[83] Train-loss=171.113129\n",
+ "INFO:root:Epoch[83] Time cost=4.877\n",
+ "INFO:root:Epoch[84] Train-loss=170.947790\n",
+ "INFO:root:Epoch[84] Time cost=4.487\n",
+ "INFO:root:Epoch[85] Train-loss=170.766223\n",
+ "INFO:root:Epoch[85] Time cost=4.723\n",
+ "INFO:root:Epoch[86] Train-loss=170.602559\n",
+ "INFO:root:Epoch[86] Time cost=4.803\n",
+ "INFO:root:Epoch[87] Train-loss=170.448713\n",
+ "INFO:root:Epoch[87] Time cost=4.636\n",
+ "INFO:root:Epoch[88] Train-loss=170.273053\n",
+ "INFO:root:Epoch[88] Time cost=4.562\n",
+ "INFO:root:Epoch[89] Train-loss=170.099485\n",
+ "INFO:root:Epoch[89] Time cost=4.567\n",
+ "INFO:root:Epoch[90] Train-loss=169.934289\n",
+ "INFO:root:Epoch[90] Time cost=4.905\n",
+ "INFO:root:Epoch[91] Train-loss=169.768920\n",
+ "INFO:root:Epoch[91] Time cost=4.636\n",
+ "INFO:root:Epoch[92] Train-loss=169.620803\n",
+ "INFO:root:Epoch[92] Time cost=4.429\n",
+ "INFO:root:Epoch[93] Train-loss=169.448189\n",
+ "INFO:root:Epoch[93] Time cost=4.985\n",
+ "INFO:root:Epoch[94] Train-loss=169.295794\n",
+ "INFO:root:Epoch[94] Time cost=4.649\n",
+ "INFO:root:Epoch[95] Train-loss=169.143627\n",
+ "INFO:root:Epoch[95] Time cost=4.602\n",
+ "INFO:root:Epoch[96] Train-loss=168.989410\n",
+ "INFO:root:Epoch[96] Time cost=4.904\n",
+ "INFO:root:Epoch[97] Train-loss=168.841089\n",
+ "INFO:root:Epoch[97] Time cost=4.602\n",
+ "INFO:root:Epoch[98] Train-loss=168.694906\n",
+ "INFO:root:Epoch[98] Time cost=4.589\n",
+ "INFO:root:Epoch[99] Train-loss=168.527604\n",
+ "INFO:root:Epoch[99] Time cost=4.560\n",
+ "INFO:root:Epoch[100] Train-loss=168.385596\n",
+ "INFO:root:Epoch[100] Time cost=4.835\n",
+ "INFO:root:Epoch[101] Train-loss=168.246526\n",
+ "INFO:root:Epoch[101] Time cost=4.558\n",
+ "INFO:root:Epoch[102] Train-loss=168.093663\n",
+ "INFO:root:Epoch[102] Time cost=4.609\n",
+ "INFO:root:Epoch[103] Train-loss=167.938807\n",
+ "INFO:root:Epoch[103] Time cost=4.599\n",
+ "INFO:root:Epoch[104] Train-loss=167.814916\n",
+ "INFO:root:Epoch[104] Time cost=4.394\n",
+ "INFO:root:Epoch[105] Train-loss=167.676473\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "INFO:root:Epoch[105] Time cost=4.724\n",
+ "INFO:root:Epoch[106] Train-loss=167.560241\n",
+ "INFO:root:Epoch[106] Time cost=4.316\n",
+ "INFO:root:Epoch[107] Train-loss=167.424132\n",
+ "INFO:root:Epoch[107] Time cost=4.646\n",
+ "INFO:root:Epoch[108] Train-loss=167.284482\n",
+ "INFO:root:Epoch[108] Time cost=4.472\n",
+ "INFO:root:Epoch[109] Train-loss=167.184511\n",
+ "INFO:root:Epoch[109] Time cost=4.768\n",
+ "INFO:root:Epoch[110] Train-loss=167.037793\n",
+ "INFO:root:Epoch[110] Time cost=4.717\n",
+ "INFO:root:Epoch[111] Train-loss=166.916652\n",
+ "INFO:root:Epoch[111] Time cost=4.803\n",
+ "INFO:root:Epoch[112] Train-loss=166.796803\n",
+ "INFO:root:Epoch[112] Time cost=4.617\n",
+ "INFO:root:Epoch[113] Train-loss=166.655028\n",
+ "INFO:root:Epoch[113] Time cost=4.420\n",
+ "INFO:root:Epoch[114] Train-loss=166.561129\n",
+ "INFO:root:Epoch[114] Time cost=4.333\n",
+ "INFO:root:Epoch[115] Train-loss=166.434593\n",
+ "INFO:root:Epoch[115] Time cost=4.526\n",
+ "INFO:root:Epoch[116] Train-loss=166.322805\n",
+ "INFO:root:Epoch[116] Time cost=4.310\n",
+ "INFO:root:Epoch[117] Train-loss=166.195452\n",
+ "INFO:root:Epoch[117] Time cost=4.458\n",
+ "INFO:root:Epoch[118] Train-loss=166.073792\n",
+ "INFO:root:Epoch[118] Time cost=4.333\n",
+ "INFO:root:Epoch[119] Train-loss=165.967437\n",
+ "INFO:root:Epoch[119] Time cost=4.459\n",
+ "INFO:root:Epoch[120] Train-loss=165.876094\n",
+ "INFO:root:Epoch[120] Time cost=5.070\n",
+ "INFO:root:Epoch[121] Train-loss=165.748064\n",
+ "INFO:root:Epoch[121] Time cost=4.782\n",
+ "INFO:root:Epoch[122] Train-loss=165.656283\n",
+ "INFO:root:Epoch[122] Time cost=4.640\n",
+ "INFO:root:Epoch[123] Train-loss=165.540462\n",
+ "INFO:root:Epoch[123] Time cost=4.522\n",
+ "INFO:root:Epoch[124] Train-loss=165.448734\n",
+ "INFO:root:Epoch[124] Time cost=4.858\n",
+ "INFO:root:Epoch[125] Train-loss=165.347751\n",
+ "INFO:root:Epoch[125] Time cost=4.842\n",
+ "INFO:root:Epoch[126] Train-loss=165.230048\n",
+ "INFO:root:Epoch[126] Time cost=4.495\n",
+ "INFO:root:Epoch[127] Train-loss=165.147932\n",
+ "INFO:root:Epoch[127] Time cost=4.766\n",
+ "INFO:root:Epoch[128] Train-loss=165.036021\n",
+ "INFO:root:Epoch[128] Time cost=4.526\n",
+ "INFO:root:Epoch[129] Train-loss=164.977613\n",
+ "INFO:root:Epoch[129] Time cost=5.091\n",
+ "INFO:root:Epoch[130] Train-loss=164.881467\n",
+ "INFO:root:Epoch[130] Time cost=5.223\n",
+ "INFO:root:Epoch[131] Train-loss=164.785627\n",
+ "INFO:root:Epoch[131] Time cost=4.165\n",
+ "INFO:root:Epoch[132] Train-loss=164.707629\n",
+ "INFO:root:Epoch[132] Time cost=4.527\n",
+ "INFO:root:Epoch[133] Train-loss=164.598039\n",
+ "INFO:root:Epoch[133] Time cost=4.167\n",
+ "INFO:root:Epoch[134] Train-loss=164.502932\n",
+ "INFO:root:Epoch[134] Time cost=4.354\n",
+ "INFO:root:Epoch[135] Train-loss=164.422286\n",
+ "INFO:root:Epoch[135] Time cost=4.387\n",
+ "INFO:root:Epoch[136] Train-loss=164.344749\n",
+ "INFO:root:Epoch[136] Time cost=4.662\n",
+ "INFO:root:Epoch[137] Train-loss=164.264898\n",
+ "INFO:root:Epoch[137] Time cost=4.671\n",
+ "INFO:root:Epoch[138] Train-loss=164.178707\n",
+ "INFO:root:Epoch[138] Time cost=4.776\n",
+ "INFO:root:Epoch[139] Train-loss=164.109071\n",
+ "INFO:root:Epoch[139] Time cost=4.787\n",
+ "INFO:root:Epoch[140] Train-loss=163.993291\n",
+ "INFO:root:Epoch[140] Time cost=4.726\n",
+ "INFO:root:Epoch[141] Train-loss=163.956234\n",
+ "INFO:root:Epoch[141] Time cost=4.337\n",
+ "INFO:root:Epoch[142] Train-loss=163.845638\n",
+ "INFO:root:Epoch[142] Time cost=4.787\n",
+ "INFO:root:Epoch[143] Train-loss=163.790882\n",
+ "INFO:root:Epoch[143] Time cost=5.563\n",
+ "INFO:root:Epoch[144] Train-loss=163.723495\n",
+ "INFO:root:Epoch[144] Time cost=4.529\n",
+ "INFO:root:Epoch[145] Train-loss=163.634262\n",
+ "INFO:root:Epoch[145] Time cost=5.028\n",
+ "INFO:root:Epoch[146] Train-loss=163.552854\n",
+ "INFO:root:Epoch[146] Time cost=4.933\n",
+ "INFO:root:Epoch[147] Train-loss=163.501429\n",
+ "INFO:root:Epoch[147] Time cost=4.912\n",
+ "INFO:root:Epoch[148] Train-loss=163.444245\n",
+ "INFO:root:Epoch[148] Time cost=5.034\n",
+ "INFO:root:Epoch[149] Train-loss=163.348476\n",
+ "INFO:root:Epoch[149] Time cost=4.600\n",
+ "INFO:root:Epoch[150] Train-loss=163.256955\n",
+ "INFO:root:Epoch[150] Time cost=4.704\n",
+ "INFO:root:Epoch[151] Train-loss=163.216139\n",
+ "INFO:root:Epoch[151] Time cost=4.670\n",
+ "INFO:root:Epoch[152] Train-loss=163.144691\n",
+ "INFO:root:Epoch[152] Time cost=4.678\n",
+ "INFO:root:Epoch[153] Train-loss=163.050236\n",
+ "INFO:root:Epoch[153] Time cost=4.595\n",
+ "INFO:root:Epoch[154] Train-loss=162.991225\n",
+ "INFO:root:Epoch[154] Time cost=5.307\n",
+ "INFO:root:Epoch[155] Train-loss=162.907200\n",
+ "INFO:root:Epoch[155] Time cost=4.684\n",
+ "INFO:root:Epoch[156] Train-loss=162.838075\n",
+ "INFO:root:Epoch[156] Time cost=4.686\n",
+ "INFO:root:Epoch[157] Train-loss=162.759286\n",
+ "INFO:root:Epoch[157] Time cost=4.750\n",
+ "INFO:root:Epoch[158] Train-loss=162.725998\n",
+ "INFO:root:Epoch[158] Time cost=4.637\n",
+ "INFO:root:Epoch[159] Train-loss=162.635852\n",
+ "INFO:root:Epoch[159] Time cost=4.498\n",
+ "INFO:root:Epoch[160] Train-loss=162.563777\n",
+ "INFO:root:Epoch[160] Time cost=5.048\n",
+ "INFO:root:Epoch[161] Train-loss=162.527387\n",
+ "INFO:root:Epoch[161] Time cost=5.040\n",
+ "INFO:root:Epoch[162] Train-loss=162.395881\n",
+ "INFO:root:Epoch[162] Time cost=4.764\n",
+ "INFO:root:Epoch[163] Train-loss=162.353654\n",
+ "INFO:root:Epoch[163] Time cost=4.561\n",
+ "INFO:root:Epoch[164] Train-loss=162.285584\n",
+ "INFO:root:Epoch[164] Time cost=5.051\n",
+ "INFO:root:Epoch[165] Train-loss=162.204332\n",
+ "INFO:root:Epoch[165] Time cost=4.455\n",
+ "INFO:root:Epoch[166] Train-loss=162.147100\n",
+ "INFO:root:Epoch[166] Time cost=5.021\n",
+ "INFO:root:Epoch[167] Train-loss=162.051296\n",
+ "INFO:root:Epoch[167] Time cost=4.551\n",
+ "INFO:root:Epoch[168] Train-loss=161.978708\n",
+ "INFO:root:Epoch[168] Time cost=4.744\n",
+ "INFO:root:Epoch[169] Train-loss=161.927990\n",
+ "INFO:root:Epoch[169] Time cost=4.821\n",
+ "INFO:root:Epoch[170] Train-loss=161.883088\n",
+ "INFO:root:Epoch[170] Time cost=4.365\n",
+ "INFO:root:Epoch[171] Train-loss=161.785367\n",
+ "INFO:root:Epoch[171] Time cost=4.448\n",
+ "INFO:root:Epoch[172] Train-loss=161.716386\n",
+ "INFO:root:Epoch[172] Time cost=4.622\n",
+ "INFO:root:Epoch[173] Train-loss=161.656391\n",
+ "INFO:root:Epoch[173] Time cost=4.500\n",
+ "INFO:root:Epoch[174] Train-loss=161.598127\n",
+ "INFO:root:Epoch[174] Time cost=4.677\n",
+ "INFO:root:Epoch[175] Train-loss=161.518613\n",
+ "INFO:root:Epoch[175] Time cost=4.958\n",
+ "INFO:root:Epoch[176] Train-loss=161.418783\n",
+ "INFO:root:Epoch[176] Time cost=4.607\n",
+ "INFO:root:Epoch[177] Train-loss=161.407767\n",
+ "INFO:root:Epoch[177] Time cost=4.427\n",
+ "INFO:root:Epoch[178] Train-loss=161.319552\n",
+ "INFO:root:Epoch[178] Time cost=4.930\n",
+ "INFO:root:Epoch[179] Train-loss=161.234087\n",
+ "INFO:root:Epoch[179] Time cost=4.240\n",
+ "INFO:root:Epoch[180] Train-loss=161.187404\n",
+ "INFO:root:Epoch[180] Time cost=4.484\n",
+ "INFO:root:Epoch[181] Train-loss=161.123118\n",
+ "INFO:root:Epoch[181] Time cost=4.937\n",
+ "INFO:root:Epoch[182] Train-loss=160.999420\n",
+ "INFO:root:Epoch[182] Time cost=4.489\n",
+ "INFO:root:Epoch[183] Train-loss=160.955369\n",
+ "INFO:root:Epoch[183] Time cost=4.894\n",
+ "INFO:root:Epoch[184] Train-loss=160.908542\n",
+ "INFO:root:Epoch[184] Time cost=4.269\n",
+ "INFO:root:Epoch[185] Train-loss=160.846908\n",
+ "INFO:root:Epoch[185] Time cost=4.998\n",
+ "INFO:root:Epoch[186] Train-loss=160.765964\n",
+ "INFO:root:Epoch[186] Time cost=4.467\n",
+ "INFO:root:Epoch[187] Train-loss=160.687773\n",
+ "INFO:root:Epoch[187] Time cost=4.609\n",
+ "INFO:root:Epoch[188] Train-loss=160.652674\n",
+ "INFO:root:Epoch[188] Time cost=5.327\n",
+ "INFO:root:Epoch[189] Train-loss=160.551175\n",
+ "INFO:root:Epoch[189] Time cost=4.267\n",
+ "INFO:root:Epoch[190] Train-loss=160.477424\n",
+ "INFO:root:Epoch[190] Time cost=4.798\n",
+ "INFO:root:Epoch[191] Train-loss=160.501221\n",
+ "INFO:root:Epoch[191] Time cost=4.695\n",
+ "INFO:root:Epoch[192] Train-loss=160.370335\n",
+ "INFO:root:Epoch[192] Time cost=4.640\n",
+ "INFO:root:Epoch[193] Train-loss=160.279749\n",
+ "INFO:root:Epoch[193] Time cost=4.653\n",
+ "INFO:root:Epoch[194] Train-loss=160.242415\n",
+ "INFO:root:Epoch[194] Time cost=5.044\n",
+ "INFO:root:Epoch[195] Train-loss=160.197063\n",
+ "INFO:root:Epoch[195] Time cost=4.684\n",
+ "INFO:root:Epoch[196] Train-loss=160.132983\n",
+ "INFO:root:Epoch[196] Time cost=4.460\n",
+ "INFO:root:Epoch[197] Train-loss=160.083149\n",
+ "INFO:root:Epoch[197] Time cost=4.713\n",
+ "INFO:root:Epoch[198] Train-loss=160.025012\n",
+ "INFO:root:Epoch[198] Time cost=4.779\n",
+ "INFO:root:Epoch[199] Train-loss=159.945513\n",
+ "INFO:root:Epoch[199] Time cost=4.659\n"
+ ]
+ }
+ ],
+ "source": [
+ "# can initilize weights and biases with the learned parameters as follows: \n",
+ "# init = mx.initializer.Load(params)\n",
+ "\n",
+ "# call the VAE, output model contains the learned model and training loss\n",
+ "out = VAE(n_latent=2, x_train=image, x_valid=None, num_epoch=200) "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# encode test images to obtain mu and logvar which are used for sampling\n",
+ "[mu,logvar] = VAE.encoder(out,image_test)\n",
+ "# sample in the latent space\n",
+ "z = VAE.sampler(mu,logvar)\n",
+ "# decode from the latent space to obtain reconstructed images\n",
+ "x_construction = VAE.decoder(out,z)\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAADACAYAAADhh27FAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xm0XFWZ9/HfQxJCyETm4ZLkQgiahOYNcknC8NpR0jSDLMAWFbUNk9AuURHpBmmlozRL+qVB8JWlQAeChrZBZQgsWhtBRpkueYMEAoQkF5NwM0IGQshA9vtHnWh591Pk3NSt4dT5fta6K3WfOsM+Vfvu2jl1nvNYCEEAAABAHu1V6wYAAAAAtcJkGAAAALnFZBgAAAC5xWQYAAAAucVkGAAAALnFZBgAAAC5xWS4DpjZf5jZZbVuB1AOM2s2s2Bm3Us8/5KZTatyswAX4y6yjjG36xj3GZbM7J2iX/eVtFXS+8nv54cQbq9+q5BnZtYm6dwQwm9r3Za0zKxZ0lJJPUIIO2rbGtQ7xl3UE8bcfHP/N5E3IYQ+ux6n+YMws+50PGQd/Ri1xLiLvKEP1y8uk0jBzP7VzO4ws5+b2SZJXzCzOWY2s2iZ6cmAvuv3/c3sbjNbY2ZLzewrH7D9P21r13bM7FvJum+a2clm9gkzW2Rmb5nZPxWte6SZPW1m682s3cx+aGY9ip4/wcxeM7MNZvZ/zexJMzuz6PlzzewVM3vbzP7bzEZ10cuGCkn6wvzkPf+9mR1a9NylZrbYzDaZ2ctmdlrRc2cm7/8PzGydpJlJ7Akz+/ekDyw1sxOK1ulvZrOSvrUi+VvoljzXLVlvrZktkXTSbtrdZmbTk8czzewXSd/fZGYvmtnBSb9fbWbLzOy4onXPMrOFybJLzOz8Dtv+p6SNbyZ9OpjZQclzPZN2/tHMVpnZT8ysV3nvAiqNcRf1gjG38cdcJsPpnSbpPyX1l3THBy1oZntJul/Sc5KaJP2NpH80s2NT7mt/Fd6bkZKukDRL0mclHSZpmqTvmdnoZNkdkr4uabCkoyUdL+n8pB1DJd0p6R+T55dKmlzUzr9LnjtF0hBJzyTHiDplZodJukWF93iQpBslzTWznskiiyX9bxX66XclzTGzEUWbmCJpiaRhkq4sir2qQh/5P5JmmZklz81WoY8dpEL/O07SuclzX5L0iSTeIulTnTyckyX9TNIASf9P0m9U6PdNkr6XHNsuq5N99ZN0lqQfmNlHktfkeEkXSZqetHNah/1cJelgSZOS55skXd7JtqI2GHdRU4y5ORlzQwj8FP1IapM0vUPsXyU93CE2R9LMot+nS2pLHh8taUmH5b8j6eYS+/zTtpLtvCOpW/L7AElB0uFFy78g6RMltnWxpF8kj8+W9HjRcyapXdKZye8PSppR9Hx3Fa7ba6r1+5D3H68fJvEfS7qiQ+xVSX9dYjvzJZ2SPD5T0h87PH+mpNeLft836W/DVRi8t0rqVfT8GZJ+lzx+WNI/FD13XLJu990dk6SZkh4seu7kDv2+b7Kt/Ups6x5JX08e3yLp+0XPHZSse1DS5zdLGlv0/JGSltb6Pebng/s74y4/te6DSZwxNzT+mMs1w+kt68SyYySNNrP1RbFukh5Juf7aEMKuRJItyb+rip7fIqmPJJnZhyVdI+lwFf6ouqtwpkEqnOH4U7tDCMHMlndo5w1mdn1RbKcKZ0hWpGwrqmuMpBlm9tWi2N4qvNcysy+q8D/25uS5PiqcfdjF68crdz0IIbybnKDoI2mgpB6S2v980kJ7FW3jL/qXpDc6eSwd+7TX7/tIWp98jfgvKpxt2EuFvv5iUTtai7ZV3KYhybLPFx2DqfD3iPrHuItaY8zNwZjLZDi9jrfd2KzCG77L8KLHyyQtCiGMr3irCl9rPC3pMyGEd8zsYhW+2pAKZyOKrwEyFb6uKG7nd0IIH/j1I+rKMklXhhCu7PiEmY2RdLOkYyU9FUJ438zmqzAQ7dKZ28csU+EsxeDgJ320Syq+1nG0s0zZkq8jfyXpi5LuDSFsN7N79OfjaldhIrFLcZvWqjDITwwhMNHIHsZd1Bpjbg7GXK4Z3nPzJZ1kZgOS64O+VvTcU5K2mdk3zWyf5KL3vzKzwyvQjr6SNkjabGbjlVy3lrhf0keskAjSXYVr3IYUPf8TSf+crCcz28/MOnsNEiqnR9J/dv10V2Hg/Qczm2IFvc3sJDPrK6m3CgPvGqmQACHpkD3deQihXdL/SLrGzPqZ2V5mNtbM/jpZ5E5JX7NC0tIASZeWcawfZG9JPVU4rh3JGYvjip6/U9JZZjbezPZV4avxXcewU4XX7AfJtZwysyYz+9sKtRWVxbiLSmLMLcjdmMtkeM/NlrRQha8pfi3pv3Y9kfyP7kQVkibaVPif0o0qXIje1b4paYakTck+/nS2IYSwStJnJF0raZ2ksSpcNL81ef4XyXO/MLONkv4gqa47bM48oML/sHf9zAwhtKqQRPEjSW9Lel2Fa9AUQnhZha9un1Lh67C/kvRkmW34ogoD48vJ/n4paVdyyM0qJGC8IGmepLvK3JcrhLBJhUnPnUkbPidpbtHz/y3ph5J+p8Lr8XTy1Nbk30t2xZN+/ltJH6pEW1Fxs8W4i8phzFU+x1yKbuSIFW7P8qakT4UQHq91e4BKSM64LZDUs8RXjUDVMO6i0TXCmMuZ4QZnZscnX8P1VOGrjO2Snq1xs4AuZWanWeHelgMk/Zuk+7I6KCP7GHfR6BptzGUy3PiOUeEeh2tU+CrutBDC1g9eBcic81W4L+ZiFUr6frm2zUHOMe6i0TXUmMtlEgAAAMgtzgwDAAAgt8q6z7AVSvJdr8LNlP8jhHDVBy0/ePDg0NzcXM4uAT3//PNrQwhDdr9k16HvolxtbW1au3at7X7JrkO/RVdgzEVWpe27ezwZTjJkb1Ch/vtySc+Z2dzkViOu5uZmtba2lnoaSMXMOlt1p2z0XZSrpaWl6vuk36IrMOYiq9L23XIuk5isQn3tJSGEbSrc7/GUMrYHAAAAVFU5k+Em/WU96uX6y5KTkiQzO8/MWs2sdc2aNWXsDqgu+i6yiH6LrKLvolYqnkAXQrgphNASQmgZMqSqlxwBZaHvIovot8gq+i5qpZzJ8ApJo4p+3z+JAQAAAJlQzmT4OUnjzOwAM9tb0mdVVLsaAAAAqHd7fDeJEMIOM7tA0m9UuLXaLSGEl7qsZQAAAECFlXWf4RDCA5Ie6KK2AAAAAFVFBToAAADkVllnhgEAAFB9IYQu36ZZVYtk1g3ODAMAACC3mAwDAAAgt5gMAwAAILeYDAMAACC3mAwDAAAgt7ibBAAAQJ1Ie5cIbzkvVq07RGT5ThScGQYAAEBuMRkGAABAbjEZBgAAQG4xGQYAAEBukUAHIFMqUYLUk+VkEADVkXY82rlzZxR7//333WV37NiRallvm9641b27P9Xba6/4fGi3bt1SbTNtrJRy1+9qnBkGAABAbjEZBgAAQG4xGQYAAEBuMRkGAABAbpWVQGdmbZI2SXpf0o4QQktXNApA/pRTdamc5aT0iRsk1WXbhg0botjNN98cxebMmRPFXnjhBXebvXv3jmILFy6MYqNGjUrTRNSBtJXdJD+JzUuAe/fdd6NYe3u7u82XX345inl9av369VGsR48eUWzw4MHufpqbm6PYmDFjUq3ft2/fKNarVy93P14CX9rkPW85qevH4q64m8THQghru2A7AAAAQFVxmQQAAAByq9zJcJD0P2b2vJmd5y1gZueZWauZta5Zs6bM3QHVQ99FFtFvkVX0XdRKuZPhY0IIH5F0gqSvmNlHOy4QQrgphNASQmgZMmRImbsDqoe+iyyi3yKr6LuolbKuGQ4hrEj+XW1md0uaLOmxrmgYkEflVlerRHJZ2mQSL5Ek7XKd2Y+XOOElWZRKsPCWTRsrhcS62inVn7yEt/POi7/AnDdvXqr9lOoPW7ZsiWLLli2LYiTQ1ae0406panHbt2+PYhs3boxiixYtimIPPPCAu81HHnkkii1dujTVvvv06RPF+vfv7+5n7NixUezggw+OYhMnToxiH/rQh6LYyJEj3f3069cvinmJfp5SY2tXj7l7fGbYzHqbWd9djyUdJ2lBVzUMAAAAqLRyzgwPk3R3MjvvLuk/Qwi/7pJWAQAAAFWwx5PhEMISSf+rC9sCAAAAVBW3VgMAAEBudUXRjUx4+umno9j111/vLtvU1BTFvMoqM2bMiGIDBw50t1kqjnwoJwnNq2a0bds2dz9bt26NYu+9916q2ObNm1Nv00sa8ioseTFve6W26SWIeIkTw4cPj2Jecojk/33vu+++UcxL8CBRrra8222deOKJ7rJpE+PSKpVA5/3Ntra2RrGjjjoqinl/h9ddd527n1tvvTWKvfTSS1HMq/aFgnISjEslanpj1Lp166LY888/H8UWLPDTrFavXh3FvDmEVy3OS6Ar9Xnh9Wmvqt2bb74ZxYYOHRrFBg0a5O6n1GuXRrlJ5WlxZhgAAAC5xWQYAAAAucVkGAAAALnFZBgAAAC5xWQYAAAAuZWbtFPvzg9eecTOuPLKK6NYqbKHU6dOLWtf1dDc3BzFvvWtb7nLjh49usKtyaZSma9py3t6Wb8bNmyIYl5WveRnIb/++utR7I033ohiK1eudLe5atWqKOZlHHsZwz179oxi++yzj7sf7zXy9uM56KCDotjRRx/tLjt9+vQo5pXKTVsuFJWxadOmKHbEEUdEMa/0cSn77bdfFDvnnHOi2LRp06KYV35Wkm644YYo1tLSEsUWL14cxc4666wo9uSTT7r78Tz22GNR7OMf/3jq9VE+bxz3+u4777wTxXr37u1u0+vnH/7wh6NYqT7ZkTfeS/5dL7w75nh3KPHGx1J3MunWrVsU8+5kUcu79XBmGAAAALnFZBgAAAC5xWQYAAAAucVkGAAAALmVmwS6e+65J4rNnz/fXXbixIlRzCt7+cwzz0Sxe++9193mb37zmyh2wAEHRLGlS5e666flXcA+YsSIKJY26cRLqpOkSy65pFPtyru05T290p5vv/12FHvllVfc/XjlPRcuXBjFXnvttSjmJd9JfvlkLwmuX79+UWzYsGFRrG/fvu5+vLLTS5YsiWJeUp33uh144IHufkqVJu2I0svV4/19XHHFFVHMG7e85BxJOvPMM6PYhRdeGMUmTJiQooWlXX311VHMG+8vuOCCKPbCCy+k3s/pp58exbxEP5Tm/U17fc9brtR4kDYR2tuPV2JZ8hN6jzzyyCg2ZMiQKOYl6nljeKm497nk3RjAG++9svaSPy+pt6Q6zgwDAAAgt5gMAwAAILeYDAMAACC3mAwDAAAgt3abQGdmt0j6hKTVIYRDkthASXdIapbUJunTIYQ4y6eOjB8/PlWslEMPPTSKnXHGGVHsqquuctdva2uLYl4CnZcw1Bl77713FPMS6Lx9e1XNvKo3qBwvWcBLKvCSzSTp3XffjWJegseAAQOi2ODBg91teklwI0eOjGJesuXQoUOjWKkKdF6lPK8qnlcRz0ugK7UfL8nDS+boTBINyvPtb387il1zzTVRzHufvGQ1SfrYxz5WfsOKeIlFknTrrbdGMS9Rb8uWLVHM+5u78cYb3f2cfPLJUcwbG9A5lfib9vqKl0DXp08fd33vM9tLYvOqwHmfDRs3bnT3431e9OrVK4oNHz48ig0aNCjVupKfQJc2Wa6eEuhmSzq+Q+xSSQ+FEMZJeij5HQAAAMiU3U6GQwiPSXqrQ/gUSbclj2+TdGoXtwsAAACouD39jmVYCKE9ebxSUvw9asLMzjOzVjNr9b6GB+oVfRdZRL9FVtF3UStlX3AUChfCxBfD/Pn5m0IILSGEFu8G0UC9ou8ii+i3yCr6LmplTyvQrTKzESGEdjMbIckvXZVDpZJ20iaidSapLy2vUt7atWuj2JQpU6LYcccd1+XtyaO0iXFeokHv3r2jmJdgIUlNTU2ptuklkXnrStLBBx+calmvIpGX8ORVSCoV9xI/vOQUL5lj3Lhx7n68RBTvNUL1/PKXv0y1nFdVrqsT5SS/6uPZZ5/tLjt37txU2/Qmd88991wU86qPofZKJXKlraTmxbykOknatGlTFHvrrY5Xq/oJcIsWLYpib7zxhrsfbyz1EqG9PulVz+vZs6e7H298TZsYV08JdJ65kmYkj2dI8msQAwAAAHVst5NhM/u5pKckfcjMlpvZOZKukvQ3ZrZI0vTkdwAAACBTdvvdYAghvpluwbFd3BYAAACgqrhjNwAAAHKLyTAAAAByixTqBrN58+Yodtppp0UxL4v0uuuui2Klyiuic7yMWC8L2cu69d4D744IkjRmzJgoNnr06CjmZQd7pZMlv3Szd9cU7xi9UtBeprQkzZs3L4otX748inklxw855JAoNmHCBHc/3t050pZjRm194Qtf6PJteiW/r7jiiiiW9q4RknTCCSdEsR/+8IdRjDtHZEdn7ibhjeOdubPOggULoph3hxNvLFy2bFkUK1WOeezYsVHMG0u9uwd5n0ul7sqThdL2nBkGAABAbjEZBgAAQG4xGQYAAEBuMRkGAABAbpFA12Bmz54dxbwEEa98rZd8hc4plRRQquxmR2lLNPft29dd30vI6dOnTxTz3n8vsazU/r3j9I5x27ZtUWz+/Pnufu65554o5iWYeCXLp06dGsX2339/dz9e0gmy4Y477ohiRx55pLtsjx49opiX0HnJJZdEsTlz5qRuk5dIdPnll0exAw88MPU2UX86k0Dnlbv3xlcv4V2SlixZEsW8MsveWOaNmV4pcMkvqTxy5Mgo5n2GlFNiuR5xZhgAAAC5xWQYAAAAucVkGAAAALnFZBgAAAC5RQJdRi1evNiNX3TRRanWf+qpp6LY8OHDy2oTOidtsoGXoOElNEhSz549o5iX4JO2gpzkJx15y+7YsSOKrVixIordfvvt7n7a2tqimJcgMm3atCjmJVF5SSySn6SI2jrggAOi2Ouvvx7FfvKTn0SxF1980d3mT3/60yh29dVXR7G0yXKlKjQ+++yzUYzKcvnhjc9egrOXxFYqmddLglu/fn0US/sZUqqSrDcWep8h3nJZTpbz8KkAAACA3GIyDAAAgNxiMgwAAIDcYjIMAACA3NptAp2Z3SLpE5JWhxAOSWIzJX1J0ppksctCCA9UqpGI3XfffW58+/btUez000+PYlRDqq60Fds8XqWfUslh3jbTJj/s3LnT3WbaZTdt2hTFHn744Sj26KOPuvvxtulVm/vUpz4VxUaMGBHFvApkUuMlfjSCO++8M4oNGDAg1bpPPvmkGx87duwet+fcc8+NYldddZW7bNp2IttKjRve+OolOHtjlFcJVPIrw3m8MdNLZN6yZYu7/rp161It632ueLEsj61pzgzPlnS8E/9BCGFS8sNEGAAAAJmz28lwCOExSW9VoS0AAABAVZVzzfAFZvYHM7vFzEp+T2Rm55lZq5m1rlmzptRiQN2h7yKL6LfIKvouamVPJ8M/ljRW0iRJ7ZKuKbVgCOGmEEJLCKHFu+k0UK/ou8gi+i2yir6LWtmjCnQhhFW7HpvZzZLu77IWIeIlxd19993usl71mO9///tRzKuag9rzkjG896pU5aK0SXmeziQ/bNu2LYotXbo0is2dOzeKedWVJL8CopfIdPjhh0cxr8JSZyrNZTnxoxF4Fbu8KpmXXHJJFHviiSfcbZZKCE3j1FNPjWIkyuVbqbHVGzu8Cp/9+/ePYqWqGo4bNy6KeZ8DXnXQP/7xj1HMq14nSStXroxi3hn5ziQoZ9UenRk2s+JX5jRJC7qmOQAAAED1pLm12s8lTZM02MyWS/oXSdPMbJKkIKlN0vkVbCMAAABQEbudDIcQznDCsyrQFgAAAKCqqEAHAACA3GIyDAAAgNzao7tJoLpmzYqvSnn88cfdZT/3uc9FMUov1ycvC9mLeeWYS90poZwM+lK8jGUv49i7c8S8efOiWKlS0p/85Cej2Gc+85ko5pU65e4o2eb1+8mTJ0ex2bNnR7GJEye62yxVgjaNyy+/PIq1tLS4y3ILsGxLW2q4lHLG8f3228/dpnc3iZEjR0Yxb2x+7rnnotjChQvd/bz33ntRbMOGDVGsEp8r9YYzwwAAAMgtJsMAAADILSbDAAAAyC0mwwAAAMgtEujqzPz586PYV7/61ShW6sL7733ve13eJlSPlxjnJXOUKh9cTiKZl4wh+aU8vVK59913X6r9TJkyxY1/+ctfjmJeCVyS5fLLKx9bKlHu5JNPjmJeCdm77rorinmJn4cccoi7H6/8bc+ePd1l0XjSJuB546uXDCxJTU1NUWzUqFFRbMeOHVFs7dq1UWz16tXufrzPm+3bt0exziQUZhVnhgEAAJBbTIYBAACQW0yGAQAAkFtMhgEAAJBbJNDVkJf4ccYZZ0Qx78L7z3/+8+42qTbXeNJWOOoMLyGiVCLSq6++GsXuvffeKOYlN40ZMyaKnX766e5+vL5Lslx+vfPOO1Hs7LPPjmK9e/d21/cS4zzf+MY3otiPfvSjKOYlJkl+da9Jkyal2jfqU2eq0qVNlvOS3QYOHOhu06tq6CUTb968OYrtvffeqWKlpD32Rkuq48wwAAAAcovJMAAAAHKLyTAAAAByi8kwAAAAcmu3CXRmNkrSTyUNkxQk3RRCuN7MBkq6Q1KzpDZJnw4hvF25pmbbzp07o9hJJ50UxbxkpfHjx0ex7373u13TMOSSlyzX1tbmLnv//fdHsaeffjqKeQkeH/3oR6PYtGnT3P10JskDjc9LoHvllVeiWKkEOq+6lufaa6+NYocffngUO+uss9z1L7rooijmVWMs1U5UTznJYd5nuOQnxm3YsCGKvfvuu1GsVKXCffbZJ4p5SXlvvfVWFGtvb0/VHslP4POqNJabsJ0FaUaLHZK+GUKYIGmqpK+Y2QRJl0p6KIQwTtJDye8AAABAZux2MhxCaA8hzEseb5K0UFKTpFMk3ZYsdpukUyvVSAAAAKASOnXNsJk1SzpM0jOShoUQdp2PX6nCZRTeOueZWauZta5Zs6aMpgLVRd9FFtFvkVX0XdRK6smwmfWR9CtJF4YQNhY/FwoX2Lh3YA4h3BRCaAkhtHg3kgbqFX0XWUS/RVbRd1ErqSrQmVkPFSbCt4cQdpX1WWVmI0II7WY2QtLqSjWyEXgXuj/yyCOp1v3Zz34WxUpVrgE62rp1axTzqsX9/ve/d9f34l6y22GHHRbFTjnllChW6kMubcIT8sFLOBs9enQUW7Vqlbu+d2bR63telcNhw9wvOl0vv/xyqm0iO7xkue3bt7vLrl+/PoqtWLEiVaxU0rC3fy8Bz0tkfuqpp6KYN/+QpKFDh0axfv36RbE8jM27PUIrpBHOkrQwhFCcdjtX0ozk8QxJcX1WAAAAoI6lOTN8tKS/l/Simc1PYpdJukrSnWZ2jqQ3JH26Mk0EAAAAKmO3k+EQwhOSSt1k7tiubQ4AAABQPY1/IQgAAABQApNhAAAA5Faqu0kgvVJlD6dOnZpq/Tlz5kQxL0sf8HilQd9+O66S7mXAP/vss+42N27cGMUOPPDAKDZ58uQo5pUS98p9Svko+Yn0+vbtG8UuvvjiKPa1r33NXX/kyJFRzMuKP/roo6PYY489lqaJkvw7VHjldJFtpe4msW7duij22muvRbEFCxakWleSNm/eHMWWL18exZYuXZpq3TFjxrj7aWpqimLenVTKLdGchbGdM8MAAADILSbDAAAAyC0mwwAAAMgtJsMAAADILRLoutitt97qxpcsWZJq/WOOOSaKZeHic1SXV65T8pMnli1bFsXmz58fxdrb291tDhgwIIp5CXRTpkyJYl4SVGdKe9L3UWzGjBlRbN68ee6ys2fPjmLe382jjz5aVpt+/etfl7U+sqHUWOSVVPYSzrZu3RrFvERmSVq8eHEU8xKhvbHUK1k+bdo0dz/Tp0+PYl4CXffu8VTR23eWx2vODAMAACC3mAwDAAAgt5gMAwAAILeYDAMAACC3SKArw6JFi6LYzJkzq98QNDQv6cdLxpCk9evXR7GVK1dGMa/y0b777utu06uwNXHixCjmVTPyEi86I4QQxbKcpIHy9OnTJ4rdeOON7rLNzc1RbNasWVHMSzD1+vLDDz/s7mfEiBFuHPUn7djhJYd5iXKSNHTo0Ch2xBFHRLHBgwdHMa9KouQn0HljvrfNQw89NIp5yc2Sn2znfQ5069YtijXaOMyZYQAAAOQWk2EAAADkFpNhAAAA5BaTYQAAAOTWbrNbzGyUpJ9KGiYpSLophHC9mc2U9CVJa5JFLwshPFCphtajxx9/PIpt3Lgx9frjx4+PYr169SqrTcg2L2HMS6DbsWOHu74X9xI/vMSNnj17utscOHBgFJs0aVIU86rNeUkW3jGWindm/TTrovGUStL8zne+kyqGfPPGCS9hzItJ/rjZv3//KHbQQQdFsWOPPdbdpjfmpx33vOS/Um3PQ2JcWmlSvXdI+mYIYZ6Z9ZX0vJk9mDz3gxDCv1eueQAAAEDl7HYyHEJol9SePN5kZgslxfedAQAAADKmU9cMm1mzpMMkPZOELjCzP5jZLWY2oMQ655lZq5m1rlmzxlsEqEv0XWQR/RZZRd9FraSeDJtZH0m/knRhCGGjpB9LGitpkgpnjq/x1gsh3BRCaAkhtHg37wfqFX0XWUS/RVbRd1ErqSbDZtZDhYnw7SGEuyQphLAqhPB+CGGnpJslTa5cMwEAAICul+ZuEiZplqSFIYRri+IjkuuJJek0SQsq08TGcNRRR0WxBx98MIpxNwl05GX39ujRw1120KBBqZb1SoiWuhOKly3trb/PPvu461dDXjOgAXS9zownjD2NIc3dJI6W9PeSXjSz+UnsMklnmNkkFW631ibp/Iq0EAAAAKiQNHeTeEKS91+fXN1TGAAAAI2HCnQAAADILSbDAAAAyK001wyjhLPPPjtVDEirWqVB999//yiWttxnZ3jHU27CCQkrAICuxJlhAAAA5BaTYQAAAOQWk2EAAADkFpNhAAAA5JZVImmm5M7M1kh6I/l1sKS1Vdt55XGxTVv2AAADPUlEQVQ81TMmhFDVwvVFfbeeX5c9wfFUTy37rVTfr82e4Hiqh77btTie6knVd6s6Gf6LHZu1hhBaarLzCuB48qHRXheOJz8a7bXhePKj0V4bjqf+cJkEAAAAcovJMAAAAHKrlpPhm2q470rgePKh0V4Xjic/Gu214Xjyo9FeG46nztTsmmEAAACg1rhMAgAAALnFZBgAAAC5VfXJsJkdb2avmtnrZnZptfdfLjO7xcxWm9mCothAM3vQzBYl/w6oZRs7w8xGmdnvzOxlM3vJzL6exDN7TJVC360v9N10st5vpcbqu/Tb9LLedxup30qN3XerOhk2s26SbpB0gqQJks4wswnVbEMXmC3p+A6xSyU9FEIYJ+mh5Pes2CHpmyGECZKmSvpK8p5k+Zi6HH23LtF3d6NB+q3UWH2XfptCg/Td2Wqcfis1cN+t9pnhyZJeDyEsCSFsk/Rfkk6pchvKEkJ4TNJbHcKnSLoteXybpFOr2qgyhBDaQwjzksebJC2U1KQMH1OF0HfrDH03lcz3W6mx+i79NrXM991G6rdSY/fdak+GmyQtK/p9eRLLumEhhPbk8UpJw2rZmD1lZs2SDpP0jBrkmLoQfbeO0XdLatR+KzXA+0y//UCN2ncb4n1utL5LAl0XC4V71WXufnVm1kfSryRdGELYWPxcVo8JnZPV95m+iyy+z/RbZPV9bsS+W+3J8ApJo4p+3z+JZd0qMxshScm/q2vcnk4xsx4qdOzbQwh3JeFMH1MF0HfrEH13txq130oZfp/pt6k0at/N9PvcqH232pPh5ySNM7MDzGxvSZ+VNLfKbaiEuZJmJI9nSLq3hm3pFDMzSbMkLQwhXFv0VGaPqULou3WGvptKo/ZbKaPvM/02tUbtu5l9nxu674YQqvoj6URJr0laLOmfq73/Lmj/zyW1S9quwjVM50gapEIG5SJJv5U0sNbt7MTxHKPCVxp/kDQ/+Tkxy8dUwdeKvltHP/Td1K9TpvttcgwN03fpt516rTLddxup3ybH07B9l3LMAAAAyC0S6AAAAJBbTIYBAACQW0yGAQAAkFtMhgEAAJBbTIYBAACQW0yGAQAAkFtMhgEAAJBb/x851DqC1HpbSAAAAABJRU5ErkJggg==\n",
+ "text/plain": [
+ "<Figure size 864x216 with 4 Axes>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "f, ((ax1, ax2, ax3, ax4)) = plt.subplots(1,4, sharex='col', sharey='row',figsize=(12,3))\n",
+ "ax1.imshow(np.reshape(image_test[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
+ "ax1.set_title('True image')\n",
+ "ax2.imshow(np.reshape(x_construction[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
+ "ax2.set_title('Learned image')\n",
+ "ax3.imshow(np.reshape(image_test[146,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
+ "ax3.set_title('True image')\n",
+ "ax4.imshow(np.reshape(x_construction[146,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
+ "ax4.set_title('Learned image')\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "DEBUG:matplotlib.font_manager:findfont: Matching :family=sans-serif:style=normal:variant=normal:weight=normal:stretch=normal:size=15.0 to DejaVu Sans ('/usr/local/lib/python3.5/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf') with score of 0.050000\n"
+ ]
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEICAYAAABLdt/UAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztnXuQJFd15r9T1VXqqepBgmx5tUbT2RIWbLQxL7URhLyWlsK2GMkrO0wIyy0Y1oTbKtiNMQFhHh1reTeiYQkcYFgZRIM1zKpqWWQEy2MF2EiyMCC09IDQogeyzHSPhBCaaUkITQ2a0fTZP7KylZWdj5uZNx+VeX4RJ2a6KjPrZlXmd0+ee+65xMwQBEEQykMt7wYIgiAIehFhFwRBKBki7IIgCCVDhF0QBKFkiLALgiCUDBF2QRCEkiHCLuQKEa0R0avzbocglAkRdmFsICImol/RdKwLiehBHccShKIhwi4IglAyRNiFwkBELyei24jocSL6CRFdTUTN4XtfH272fSJ6koheN3z9EiK6Y7jPt4joRY7jrRHR24noTiL6GRF9mogmiagN4MsAfnl4rCeJ6Jc92rObiO4mop8T0Y+J6O3D1y8kogeJ6N1EdGT4OQuO/S4mou8R0RNE9AAR/aXruL8xbOvjw/ffOHz9FCL6KyI6REQ/JaJriGiH1i9ZqAQi7EKROAngrQCmAbwSQAfAmwGAmX9zuM2LmXmKmT9NRC8FcC2APwVgAPgYgC8Q0SmOY14G4CIAZwF4EYA3MvNRAK8B8NDwWFPM/JBHe/4WwJ8y804ALwRws+O9M4btfC6APQBWiOgFw/eOAngDgNMAXAygS0S/BwBEZMLqVP47gNMBvATAHcP9/huA5w9f+5Xhsf9C8bsThC1E2IXCwMwHmPnbzPw0M6/BEuoLAnZZBPAxZr6dmU8y834ATwF4hWObDzPzQ8z8KIAvwhJNVU4AmCOiZzHzY8z8Xdf7/5mZn2LmWwH8H1idCJj5H5n5/zHzJjPfCeBTjvP4IwBfY+ZPMfMJZt5g5juIiIbn81ZmfpSZfw7gPQD+MEJ7BQGACLtQIIjo+UT0JSJ6mIiegCVs0wG7mADeNgxpPE5EjwPYBcAZVnnY8f8BgKkITfoDALsBrBPRrUT0Ssd7jw09f5t1+3OJ6DwiuoWIDhPRzwBc6TiPXQD+xeOzTgfQAnDAcS5fGb4uCJEQYReKxEcB3AvgHGZ+FoB3A6CA7R8AsMzMpzmsxcyfUvis0LKmzPwdZr4UwC8B+N8Arne8/exhrN5mBoAdzvmfAL4AYBcznwrgGsd5PADgeR4fdwTAMQC/6jiXU5k5SkckCABE2IVisRPAEwCeJKJ/A6Drev+nAM52/P1xAFcOPWQiovZw4HKnwmf9FIBBRKd6vUlETSJaIKJTmfnEsF2brs3+y3C7fwvgEgB/5ziPR5n5F0T0cljhF5s+gFcT0WVENEFEBhG9hJk3h+fzQSL6pWEbnktEv6NwLoIwggi7UCTeDksEfw5L5D7tev8vAewfhiouY+ZVAH8C4GoAjwG4H8AbVT6Ime+FFfv+0fB427JiALwewNowLHQlgAXHew8PP/MhWGJ95fCYgDXg+1+J6OewBj+3PH1mPgQrvPM2AI/CGjh98fDtdwzP4dvDz/waAHtAVhCUIVloQxCiQUQXAugx85l5t0UQvBCPXRAEoWSIsAuCIJQMCcUIgiCUDPHYBUEQSsaEjoMQ0WkAPgFr2jUD+GNmvs1v++npaZ6dndXx0YIgCJXhwIEDR5g5dNKaFmEH8CEAX2Hm1w6LNrWCNp6dncXq6qqmjxYEQagGRLSusl1iYR9O8PhNDPOHmfk4gONJjysIgiDEQ0eM/SwAhwHsG5Yq/YRrqjUAgIgWiWiViFYPHz6s4WMFQRAEL3QI+wSAlwH4KDO/FFbJ0ne6N2LmFWaeZ+b500+XukaCIAhpoUPYHwTwIDPfPvz7M7CEXhAEQciBxMLOzA8DeMCxyEAHwN1JjysIgiDEQ1ce+38C0CeiO2EtZPAeTccVhBH6/T5mZ2dRq9UwOzuLfr+fd5MEoXBoSXdk5jsAzOs4liD40e/3sbi4iMFgAABYX1/H4uIiAGBhYSFoV0GoFDLzVBgblpaWtkTdZjAYYGlpKacWCUIxEWEXCo0z9LK+7j0349ChQxm3ShCKjQi7UFjs0Mv6+jqCitXNzMxsbS/x92Ijv1FGMHPmdu6557IghGGaJsOqPeRrrVaLe70e93o9brVanu8J+uj1emyaJhMRm6YZ6fuV3yg5AFZZQWNF2IXCQkS+gu4WFr9OwDTNfE+iRCQVZvmNkqMq7LnUY5+fn2cpAiaEMTs76xlXN00Ta2trI6/VajXPcA0RYXPTvQa1EIcov4cX8hslh4gOMHNoBqLE2IXCsry8jFZrtFBoq9XC8vLytm3tOLvq60J0/AapVQev5TfKDhF2obAsLCxgZWUFpmmCiGCaJlZWVjxz1qN0AoJF1IHMpMIsv1GGqMRrdJvE2IU0SDKwVzX84uXdbtf3O/Tap9FosGEYyt+5/EbJgMTYBUHwwy9eTkQjcfBWqzXylNTv97F3715sbGx4Hte9vaAXibELguCLX1zc7eh5zew9duyY73EHgwH27t0rueo5I8IuCBUkyoDl+vr6ljh7lXVws7GxsTWpzK7nI+KeLSLsgpAiecy0dH7m9PQ0pqent33+7t27QUQj+7n/dmKLs19ZhyAGgwH27Nkj4p4lKoF43SaDp0Ke9Ho9NgxjawDQMIxUBvHSnGnpNwjp9Zlum5yc9JzwNTc3F7ifPUgatE2QOc9dBlHjAZl5KpSdOOLQ6/W40WhsE51ms6ldXNKaaRnUYaiUYfCzer0ee19Vs38nKS0QDxF2odR4CXSj0QgVhyDhs0XH7iwMw/BN5VPpVPy8WyLadi5ROqigDiNtYU5q9jmm0eFVARF2oRCk9cjtDKU4zTCMwP3CQglBYYyoBcf8BMwwjJHOw91BhYUskoRDwkQ3bWG3z8Xv84VgRNiF3NH9yK0aaggiaH+VUETQNm6P029CT7PZVBJAv++v3W5rF9xGo8ETExOpinpYuEg89nBE2IXc0XkDqwwKuoXdKSK2IPt5+jrMK8TiHqRV/fygkEUa7Y7SrrifEzTAKzF2NUTYhdzR+citKnJ2KCZKR2Bb0sFDZxjI6/OjiGJQyEK3dTqdzMIwdskC5/ctWTHqiLALuaPTY1cRHmdmS1xvN4nAOYU9ibdte69TU1OZCHueJp56NFSFXSYoCamhs5pf2EzJer2Oa6+9dqtGSdx1UJl5a6KOYRgwDANEhHq9HrrvxsbG1kSgKBN5ms0m2u321t87duwAABw9ejRi68cPWYw8JVTUX7eJx14ddGXFBIVWnF5ft9tVCqmEeeb2U0XS3HAV63Q6kcNGZTLJhlEHEooRyoaXaDs7i263q/z474z1BgluFrHnWq2Wu7jmaZINo46qsEsoRhgL+v0+9u/fj5MnT269Zod17PDLysqK0rH27NmD888/P3S7m266yfJ+UqbMy8IF1Z8BZKGN1FBRf90mHnt1iRuaURmI9XrfywzDqHToI2tz/t7OhTyCZvYK3kBCMULRSJK/rJI6mVV6oNh28xvX8AuzSC57PDIXdgB1AN8D8KWwbUXYq0mS9EeVfU855ZTcBa6qNjc3t61jDRJqmX0aD1Vh1xlj3wvgHo3HE0pGklXuw1In+/0+nnrqqeSNFGJxzz33jIxHEBH27NkDANvq0QfVdY+bpiq4UFH/MANwJoCbALwK4rELPiT10oLi8+NQ2bBqVqvVPCtwBtXKEY89GGTssf81gD8H4Du8T0SLRLRKRKuHDx/W9LHCOJF0wtLCwgLW1tawubmJtbW1kQWT46zsI6TL5uYmTpw4MfLaiRMncPz4cc/tJUNGH4mFnYguAfAIMx8I2o6ZV5h5npnnTz/99KQfK4whCwsLWFlZgWmaICKYpqltRXuVmaFCsdF1LQgAsSMuFusARO8F8HoATwOYBPAsAJ9l5iv89pmfn+fV1dVEnysITsLypYV0ISIk0RLTNLG2tqavQSWFiA4w83zYdok9dmZ+FzOfycyzAP4QwM1Boi4Iuun3+yLsOaMq6o1GA81mc+Q1CcHoR2aeCmPP0tJSIm9R8Ea1+Jl7HyfNZnOrkJppmti3bx+uvfbaVMJxwjMkDsXEQUIxgk5qtZoIu2ZM08Ty8jL27t2LjY2NyPseOnQIMzMzIyUfhOSohmImsmiMIKTJzMyMZMVopNVqYffu3VhcXMRgMIi0r8TKi4GEYgQAVpzaPZFkXPBKoxTiYU8suvHGGz1FPWgsQ2LlxUGEXUC/38fi4iLW19fBzFhfX8fi4uJYibu9OIWQDGbGjTfe6DsD1C/kFRQrH2enYWxRmcWk22TmabHwm7VZr9cLX3kvztqmYuEWZdHvoIUypNiXXiDVHQVVVKoi6rwZda2qxCylBNKyZrOpvABIUBkA1TISOq+JMiPCLnjidQOpiqOOOh66PTgp1ZudTUxMbKv9EvbbqZRb9rsmnLXbRewtRNgFZh4VcsMwthVgsm8g1XBGWh523E5DPPZszTCMbWIbpzib8/f22yZKGeCqIMIuKMef7ZtRNa6ahoftF6cNe0RXPceqryuqy9y/U9gTmMoTWpSnrqpXfxRhrwhu4XM+vvqtauNlUbz2JDdYFI9dNWzT6/WUzjXK9yHm/7s7rzmVlZPCOucoT11BA7VVACLs5UdnRkicWLWuNvs9AYR1AmFhJqfV63Vut9u5C+M4mVcoRNUBiCLAXteE3/UoHrsIe+nJO74cJRzjFmGVRYyDOhuvsJGItz7zG7xMMtAe5Ll7PXlKmuR2IMJeLrxuirwzQgzDUG57nKwH1UE1d5tUQzNi/qIcp7N1/rZe4TKvJ6put8vMzN1ud+s3q9fr3O12JQXSA4iwlwc/YYwyiSQt82uv84b0a2dY1kOUR3R3m/Lu9MbRGo1GqHiGeex2x+om6FrtdDqer9uiLzwDRNjLg9/NNDk56XuzvATgpwE+nLIYtNvtkbBK1EFYt7kf4d2dhMoxgkIGdrpe3iJaRHM/gXl5zGHjOkQ0IshRwjduq9frWdxeYwVE2PND9yNkVO+zXqvxNwB+mCh1YddtYYNuKk8ptVrNM9Zuf4/2b5P3uRbNVCYNOfPWg45l/05Jv2cJv4wCEfZ8SKM2RlSP5wqAD05M8Ad27Bg7YQ/LeihC+GlczTCMkVh20HevkpaaRecoA6ajQIQ9H6LkaavS6/W40WiwCVg/mYddMPycKYB/DPClAF+F9EMxOs1+jA962hFPO7q5s5BUMk6Cjhd2rafRfsECisIuZXs141fu1O91VYgIPwHwCpf9HYBjAB4YbvcXAO4B8PmQYxURZsb+/ft9ywf3+33UanLJRsEwDBw7dgwbGxtb3+n+/fuxZ88eGIaxtZ277LHf9+xcKm95eTnRtTQxobbOz8bGhpT6jYqK+us28dhH8Ur1UjnmxQCfBPgNw7+fD/BRgF84/PsqFNNj9wun+IUIbI8z73aPmwVlThmGERhDDzquTa/X46mpqUzOpeoTk2wgoZh8iBpj73a7nhdyp9MJFLNzAH4M4Ksdr30Z4L9x/H0ViifstVrN95zF9FncuQ6maQaGWJyzfr2yY9KaIGYP7Lo7E3cWTtmBCHt+RMmKiTORZgrguwD+BsATw9cuAvgJgJ8H8KlDey/AR4b/b6YsJFfBO/b/O67tOp2O9tisePPeIsisPw5uX8t+37kdw8/6nKempnzvM3eBO79c+3EAIuzFIEzkwy7YywE+CCvkcnD49w0APwTwGY7t9sJbWG1bSvnGugrWE8R5LnuWazt7EE/nZwfl81fRnGELnTOU7UHMsFBN2PtpmZf37tfJNJvNsRR3iLDnT9Djqi30QeVkLwf4SYwK9FMAnwD4/OHFaW/7XFiZMU7bB/Djw//PpHxTXQX1sI942OnZxMREZOdBxZzhxLBwS56zoolopGRwUKc2jnF7iLDnj8pjcFAo5iC8ve8nsN0z3umx/1UIF1vVUJDp0xaG1XGofJbz5svjpi+7OSdgxS3f7Gd2+CKKN57X72wLdljnMo4lgCHCnj+qF3a73d66+YiIJyYmGLDCL6xoF3gc9yoEiy2z/6Oq25rY3plcD/AA4LOHn3Vi+HnHAf4uwL+fw01dNbPDe7Kg9zNme+0q3924ARH2/FEduHKO+Dtv0IPwFvGDmm6ATqcTWxDcqZYLAL8V4AsB/l2AvzRsq4h7upakFktZTWW1LImxi7DHRtUb9itM5RVjf3L4ep43jleqpZd9C+DvFeBGL6u1221mjjYbVzx7S9THdaFsZCXsAHYBuAXA3QDuArA3bJ+yCrtqudoo5pUVk+dN4ZVq6WdvH7a7VoCbuYxme52qHrvUqrdsYmKCG43GyGvjkg+PDIX9XwN42fD/OwHcB2AuaJ+iC3vQOqLu3t15Y1VhUNAr1dLP3gardLAIe3rmHJsJ29YW9rzbXHQrsgePvEIxsMqU/FbQNkUWdpWBKOfU6zI/2rqfFq6HlW55vuL+3wL4QAHOoypWBcciKytqVUkoCjtZ2+qBiGYBfB3AC5n5Cdd7iwAWAWBmZubc9fV1bZ+rk9nZWai0zTAMTE1NKW07jlwO4OMA2o7XGMA/AniXa9u7AXwRwA0A7h3u8ycALgLwe8P3hGwgIui8p6uMaZpYW1vLuxkjENEBZp4P3U7XRUBEUwBuBbDMzJ8N2nZ+fp5XV1e1fK5uarWa3BgADgKYVdz2QgCvB/DvYMXlNgF8F8B7AHwlhbYJQhYQETY3N/Nuxgiqwq5WNzP8wxqwHLZ+mKgXnZmZmdJ64VGY8Xl9E0Dd4/VbU2yLkC31eh0nT57MuxmpMTk5iV/84heh2z3nOc/JoDXpkLi4NVkFmf8WwD3M/IHkTcqX5eVltFqtXNtQhHrpftXjk1WVF4pOq9XC/v37R+qulw0VUR93dKxacD6sJ/FXEdEdQ9ut4bip0O/3MTs7i1qthtnZ2W0F/BcWFrCysgLTNEFEME0T7Xbb52jp4BcKIqLM2vJuAEddrx0dvi6Ul8FggKWlJSwuLubdlNx59NFH825CfFRGWHVbXlkxKgv0+qU0qsxmy8I6nU5mn1W0HHqxbMyekl+Uaz4vK2LJAcjM01GCJmYErSZjU4SLPI2St2JibgtbbKMKNu7pjpUQ9rj55s7VYvK+0AApdyuWvtneehUdCGd1zCKKOrMI+whxvQ/7Ii/KJKQq3mxi2ZlzWn2Ue8a5LsA4W7fbjbT6WR5AhP0ZggQxaFEAeSQVK7NNTk5uiZi9YLizjEbe7cvaarXatk6qaDVkIML+DH7iXK/XfUsD2DG2cfOSbS9DwjZiYWYLutd7rVZra12AqptzVaa8gQj7MwQJt3Mbr0cwv06hCIOpfhdhULvFxFRt3JyaNK1erxei1C+qKOxe4uysvmhnxUT5Ufw6hbB1H/Mye8FhuSnFxNK1PDJnUDVhVx3kjPJj+HUKRY4/Rq3RLSYmFt+yznVH1YQ9ipCp/BhB4ZtxEE3DMAobLhIbbytLFoyfnQvwPoDvhTU5b1/AtlkviA1FYddRUqAQHDqkXsUkaFu75MAVV1yBwWAw8t5gMMCePXvGokjYxsZG4SrTCeNNvV5Ht9vFU089hV6vl3dzUuN8AL8B4DsAHg7ZdmbGr1xeCN/8JnDeecDkJHDWWcCHPxzvOH6oqL9uK4rH7rVSkko4R+LXYlW1RqMRmlgw7kaO/38H/h67agLGNv75n5nbbebXvY75ppuY3/te5nqd+eMfD9U5VC0Uo5qa2Gq1uNvtJk4HFHEXq6rZA/RFmZEd1UzAkj4Pu8C1rZ+w12q1LeH2Sy/2Hc9bXGQ+5xzmEyeeea3bZT7zTObNzUCdQ9WEfXjSoRekqleudIGU1GMREwuzcRlr8rImwOe57HqABwCf7drWT9idnZtbT0z4dxx8yy3Mu3Yxv+Mdo+J1663W+3feGaZx1RP2sIlIQdtENTucU6QMmbIPaomJpWEXwxokfYPHe0GhGD89cXYcrwCYb7uN+bWvZZ6cZP7BDyzZ3bdvVLweecR6/frrAzUOVRs8BYDdu73LwC8uLmJhYQEAtAx8tlotLC8vAwA+8pGPJD6eLt70pjfl3QRBGCvOAdAD8FEA/yPivv1+3zMR4ziA24f2E9MEjhwBbrgBuOYa4LTTrI3sf22e/Wzr38cei9gKH1TUX7dl7bE7B0vjxMYNwwgcFCnKI2lR2iE2XqZjzCmKERF3Op3cr9cpgO8C+BsAT/hsE+Sx+5UBt63VavHn3/9+5lNPZX7zmy2xePBByzP/3OdGxevECev1j30sUONQxVBMkGgTUegP4bdfWBGgoFrvYmJFNjtMmbXI2gOLWd437oVjbgf4IYDPCNgnSNjDbArg+xoNfuScc5iPH7fE4sknLdn95CdHRURzKKZUwp7WxakyW1WKbomNq+ksSx3lPsha1J8ELMkb2ibAfxmyXxRh9+s4zt6xY1Q/du1ifuc7RwXk61+32iWDp9tJu3Z6UG6qpD+KiWErZJl3O9x2EKOibttDGM2O2QnwNMB/MLT7Ab7Z8bff8cM6jpHZ7ouLzC94AfPTTz/z2lveYgm+pDt645wkkMYF4ue95x0vFBMrihXxXjgJb2F32wVD83vf7/gHfbZ3dhx8223MP/vZMxOULr+c+eabmd/3PuaJCZmgpEpQ+qM9EDo1NRXrwrUZx3xeSYsUS9M6nU5hVh2z7SC8hfegpuOrdhx8yy2WcPzTPzH/+q8zn3IKs2kyf+hDSpoGEXa1OuxxPHu78H5RS/eKiemyRqMRecENe0A277Y7zStU8uTwdR3HP+g6tm0HHdvoACLsFmH1G+J424ZhcKPRyP1ijdPuvNsgNj5Wq9W40+nEutaZOff2u809uKlL1O1jh3UcOmq3Q4RdjagDrkFrpBbZ5ubmCjmoJVYc8yrzHPeJdtzCkzosrOOwyxAkASLs6ji9eq9Ffd0evwikWNksjsPiNz4la6X6W1Igwp4eVfRGxMprdkw8isNie5/dbncrH71er8dKRqiSJV0zFSLs3ijXTA45hsw0FSuLxVkA3S+sIE+z6hZnzVRkKewALgLwQwD3A3hn2PZ5CbtKlowquuLs4xivFyuX2VleUceb/BaNF1O3qGumIithB1AH8C8AzgbQBPB9AHNB++Ql7GFFwsLQPfnJMAyJ2YsVwuyVkeIIdLPZHMsssSJY1DVTkaGwvxLAVx1/vwvAu4L2yUvY/QRU5cvVXa7A+aRQpJruYtU123NnlnEk3eY356XIHvtrAXzC8ffrAVztsd0igFUAqzMzM5FORhdJPHadF7rtqTuRkIxYUUyuRf3Wbre1hIFRNGF32rjF2KPMogvrAPwGncKeCBqNRiFKATSbTe50Orm3Qyxdk/CgftORuAEJxXgT9uW634+yRqpTtL3EL6wTCcqnd8Y/7fe9JpRkYXLTV9ckGyy+6QAZCvsEgB8BOAvPDJ7+atA+Rc1j9/Kao4hYs9n0zS7wW7AjSS8usXkxsfEwHbNOmTMUduuzsBvAfbCyY5bCti+qsOuIowd5NO5YftL0SxngEhMrvtVqNS11YpgzFvaoVlRhTzvEYGcd2B66XyegMpirEvefm5vL/aIOu+AlTU6szOaVKJEEiLBHx88D1iX4qkLmTL/0CtWopF7aC3DnfWGrXPgSsxcrm8Wd+BgGRNij4xcasQuBAdkMHDpF2f15KsWa7Fj/uAjmuLRTTCzMkmS8qAAR9ngEDWZm4QEnTWt0PvqNg8cuJlYmSxuIsOsnzLOM63naS/UlTWFUGZwVExNLx6LOIo0DFIW9BkGZmZkZ3/darRauvPJKmKYJIoJpmjAMI/SYrVYL+/fvx3XXXYdjx45hc3MzdJ92uw0i2nac5eXlrb/7/T6WlpYwGAxQr9cBAKZpotPpbP0tCIIe3Pdf7qiov24bN4/dWRjJyyv3G/n28pibzea2iUfM6mETv1DNKaecsvX/dru9bRt7rEA8eDExfZZ2TN0NJBSjh6BJSyo/qOoEJJUwjixkoM8WAf57gB8G+HGAvwHwbxWgXWLjY1mEXtxAhF0PSUv9Jv0csXRsHeCPAXwpwK8G+JOw1qr83QK0Taz4Zmee2eioA6MCRNj1kKTUbxRkoDNbMzxe+ybANxegbWLFtlqtNlIeROcCPmFAhF0PWXnszNYFIiVT9ZgJWJe3h13gs8/VAN9bgLYX3ewxJXeN8SotYu0U7iw1AiLsekirN/Z7dFMNyeRV2XFcrAnweS67HuABwGf77HMA4C8UoO1FNr9BePdEviqYLdxZPdUzi7D7EicWFmUflW2DZrjmfbGW1S6GFUN/g8/7/wGWN39hAdpaZAt6orRnTBPR2Dkecdobtgi4eOwpoFpjxcv79tvXS7DdNdLdtWDcx+/1elLTOmM7B+DHYIVavN5/GcBPAvzBArS1zGZ7tvb1X4T7wA4rBWlDmHBLjD0jYff7ov28DWfP6rWvX+64V7541AtALJ51Af4OwI8CfBTgO4evubebAvguWOmMEx7vnwXwTwD+PMC1ApxX2cyeVW0OF61JEn6MY41GYyREZHcmXk/TQc5bmHBLVkwGwh71QnHGwtK4yMIe2cSi27uHdgnArwJ4GVaoZWP470GALwf4BoAfAvgMj2OcDvB9AH8b4B0FOKcymn3tB4lj2ES+uJ69zpK5WQl3GKiysEet2eL02NOoNBg2yCKW3C4H+ARgXdJDe2r42vke27dhefz3ATxdgPaPu/ld2/bKQSrhDD/h7PV6nuWu7ZRDdzaZ7hroRQJlE/YoPabfRWQYRugjlW6vWiUtSszjhseoSDvtAo/tD/ps+wS2Z8fsBPirsIT/co/38z73Mlm73WZm9cwRv/Etd8jTPUGoKqBMwh51cCLssS+og4gSYw8z9zqnEmNXN9V0xTos7/sk/DsCt10Q8n7e5142C3JobI/ebw6H6thYVUCZhD1OOlGSmJif1+B1gTUajcAUMHcbJaUxnnmlK/4rhIv4wQK0Xczf7MHNOA5PGnniRQdlEvYsJwCE4ddhqLZRwjG5FRJlAAAR4UlEQVTRzS9dsQ7wubA88M8AvIlRUX8SVqgl7/aLBVvcnHfx2Mdc2LOcAOCFivev2sY8bhx3RcharcadTmfknLrd7kjpXxXLovxBWLqi066H5dU7s2LyEisxfaYyNlYVUCZhz3ICgMpn2+GXqJOfirQO6cTEhGdd+F6vp7R/rVZLtISfqgWlK7rtYlie+vMK8P2K6THVsbGqgDIJO3N+eaQqoRP3xQd4T4QochgmSvZOs9lMpYO6HJanbXvc18PKXPFKV/Sy9wB8DJKTXhYrc9piXFA2Yc8LVQELml1qi2beN0qSc0jbLocVE2eHbcIqo+uVrvh/AX4LrFrqrwH4AwAfB/i9BfgexZKbnS0jjAIRdj2oetkqBYGKUBsjzGyCamSkYQcxKupBdgHAKwD/EFY5gcMAfwvghQJ8f2L6TNgORNj1oOppF2V2qdcMPVUjom2Pvlmdj18e+skCCIxY9lav17fdhxJjF2H3Jc4FEpb9oRqfTmu9UvcgaBJP253Fk5XXfhDewn6wACIjlo8579kk6w6XCWQh7ADeD+BeAHcC+ByA01T2y0vY42bXRLmwgjx8ItIeu/ZK+UwSI7e9drvzc6+Sk5Z5xdglD7265vTYw5yLKqU+IiNh/20AE8P/vw/A+1T2y0vYk+TDq9Rjt18PughVPGrTNJVyxMPKKsQRZ6+c4azMnRUjol5ts+8nlXBgVSYrIetQDIDfB9BX2TYvYdc9g9XvCcBvJp3TCwm6SP2O7TT7M9yLFng9mkYJp0xOTuZ+Q4uJ2RZUK8ZpVSkvgByE/YsArlDZdhw99ijH8/OQnQXB/C5WZ5qX07uPMojp9OTHIc1STCzIVJ4ixWOPKOwAvgbgBx52qWObJVgxdgo4ziKAVQCrMzMzsU8saXEvnTNYg54Aut3uiDftFHW7Le4MlkajEalcgcoNMQ4plmJiQeYc97H/dr4vMfYUPHYAbwRwG4CW6j5xPXYdwqwzbSrpE4BqW/JOoRQTy8L8nBD3/VTl1EdkNHh6EYC7AZweZb+4wq47lJKUtGvYZD1JSEwsb3M7Me41DaoOMhL2+wE8AOCOoV2jsl9cYS9S+V6btLyHLKb1y5OAWNYWdM35vVelUEsYUBR2srbNlvn5eV5dXY283+zsLNbX17e9bpom1tbWNLSsOPidKwDUajXs2LEDR48eTfQZzWYTjUYj8XEEQQUiQq1Ww8mTJyPvW8Z7PA5EdICZ58O2q2XRGF0sLy+j1WqNvNZqtbC8vJxTi9Lj0KFDvu9tbm5CR4d8/PhxTE9PwzTNxMcShDCYOZaoA8H3g7CdsRL2hYUFrKyswDRNEBFM08TKygoWFhbybpp2ZmZmAt8fDAbKx2o2m77vHTp0CLt371Y+liDkQdj9ILhQidfotnEqApYXXiuzxzE77h93+TExsSKY1Ga3gGKMfaw89iqxsLCAnTt3JjpGq9XC7t27sbS0hM3NTU0tE4Ts2djYwBVXXIHp6Wn0+/28m1N4RNgLQr/fx+zsLGq1GmZnZ9Hv97GxsRH5OHa8vF6vYzAY4JprrvEdhBWEIkJEvu9tbGxgcXFxm7h73T+VRsWt120SihnFLx8+ajqiYRiZLDAtJpa3Oeeu5LkmctagjOmOZWV6ejqWd+6k2WyCmXHixAlNrRKE4kJEW+HFKqVBlzLdsYzEDbk0Gg0YhrGVHbRz504RdWHsqdfrSts5s2T8Qo1VTpEUYc+ZpaUl3/cMw9iWt2+/vm/fPhw5cgTXXXcdACh3Do1GA+12O15jBSFlTp48GZiea3PkyBFMT08HxuMrnSKpEq/RbeMcY9ddQgABccRerxf4eaplB9yrPWVRfybJ2qti1TXDMLZdO7VaLfKykl7r95YByJqn+kljkMavol2tVgvdV0WgvfJ/064RY3cgUjK4mhZ3ScWgRTXiXEtlBCLs+kmjumSSCzNIoIOeJtL02JvN5sgiH3ktsxckHlmt41plsx0e1e1rtZrWjK6yLrwBEXb9pFFdMkln4bdvvV4PDBWpriMZZu70Snt2q2ma3O12t9pXFM/dfnrJu8OJGlYYV8sr9basqY7MIuypkIbHniS8oyJQfsfqdrvKN4qXh9tsNtkwDCYiz7ioVzt03OhEFLmjqNfr276DXq+Xi/BURdSzNvc4UlmBCLt+0poIkXS5P3tf1RVonPuGiZthGJ41a+LUnlHpAILMPo+oHnfQE5WquOuo2yOWjlWpjgxE2NMhqghnuYxXklCRX6el28OMK5B2Bxonq8cOTdmhI+dvoRKScoaVxNRtcnIyk88pc+jFDUTY8yfrqc5prMGq8waMW2HSbku329Wa0dNoNEKP1263mVnWnS26lXWw1A1E2PMn6zVa0+hIdN14Ojx1VXG1t9MxaGsYBjOzxMbHwKrgtUOEPX/yWKNVd+gnSgzaGeawwxf230kGKk3TjBwKsT87qVjYE13yFi2xcKtCSAYi7PmTtseeRfy+1+ttG/BsNBrbhNsr68T5ftKbNo5I6/DY43QqYslscnIydjpq2UMyEGHPnzRj7FnG7+MMGKtmvxiGoSScUUM5umLinU4nd6GrmnW73dhPSWk+DRcBiLAXg7S86qzj91FQDbs44+c6a8tkPdCpa8KXnzWbzUjzDopojUYj8nUR1OH7DcQX4fpPE4iwl5s84vdeRM2k8erkdNaVUQ2b6FoDtt1upx6qabfbY72AirP9qh2gfX34zaHodruVWVzDCUTYy00RPHa/cFDQDWsLuFPYdQmj3amFHU9SF/VaGpO37N/SPYnOORmp2+1uXU/1ep273W5m135eQIS93BRhOTA/AVX1hu32Rk1j9DP7xh7HLJZxLXNsC63usYgwB6UI138eQIS9/ITVak87YyZIaFVDK6qpkK1WayQTp91ub3UgtVqN2+32SKplEbzyiYkJ5e9gHDsjpwDrDEcRUaj3XYQn1jyACHt1ycqbCbqZ7bx2lRs5zMO3i3g5OytnZUlVEW80Gtpi6zpFTOX7TNOSfifM0Wfm2tlQROQ5+Svsei3KGFPWQFHYZWm8ErK0tITBYDDy2mAw8F2Gr9/vY3Z2FrVaDbOzs+j3+0qfs7y87Ll0HwAcP34cAAKXLgOAWq22tSixH/v37wcALC4uYn19HcyMjY2NreUAres9nBMnToR+Vhj1eh1EpLw2ZxjO5dt2794d+n2lQavVQqPRSHwMVYgIl1122dbfx44d27bNYDDAnj17fK9Fv2XvKr0cnhMV9Q8zAG+D1WNOq2wvHnu6RPFmonr37hBP2ml4uhdgSGLO70VHqKfRaBR6URJVi3INEBF3Oh3lMQW/a1Fi7CmHYgDsAvBVAOsQYS8EUeKPUbbNqgJkUc0pGjrCJs7VpsZ1dmutVouUqtrtdiN31H5x8ywrpxYFZCjsnwHwYgBrEGEvBFG8mSjevZ/4JPFe2+32WGSE2MXAgr7jOGaPH+ga7C3KalV+FqcDK3vcPArIQtgBXArgQ8P/ryFA2AEsAlgFsDozM5PBV1BtVLyZoIlBXl5SGpkm9qzKooRb/MxrMQddqzDpWl1qHCzONVT2TJcoQJewA/gagB942KUAbgdwKisIu9PEY8+fII/Tz7v387aSeon2jRslVuteb9XP5ubmtImS+3vRGRc3DGNsY+xRf+sonVgV4uZRQNoeO4BfA/AILEFfA/A0gEMAzgjbV4Q9f4Ieif2WGvML8XhN745idt6yijfnvtGz9nSd3qPuuHgVVmqy01ZVZqtWJW4eBWSdxw7x2AuDShgmTESDshH86r3EFSW7jnvUG1138TAVc8Z7dYemnEXRyui9O8cpwq4XCb94AxH2aqI6cKoiwnFuLlVvzNk2Fa87SkZPFtbpdFL5fPs8k3SUft9z3mMZkraYHMjM02qimr6o4hWGZSN4ee9Ra7GrZIREzejJyubm5rR71u7vPMmx7EW8nb+NfX2kMQO33W4H/tZRriPBGxH2ihJ1clLcx2E/T0vFI3TH8MM8U7+6ITo82qSdg87BWa/vPO7AtH1eTlFPowqjiokHrg8R9ooSpzhSnMfhOKLqV1o17OkhaILKOOTAq5p73dgohcGctXncnVWj0cjt6cbO0xf0IMJeUeLGLKM+DscViqBBWb99/EJCUbzQoPYWoTBYrVbb1kmpPgE5xVNnXF7H9yKTi/Qiwl5hsohZJhEQPw886tNGlDaE5ci7OwidYm8/qcQJq6hkDDnFU1eb3QuWB7U9KDddslv0IsIupEpY+CRICII88ChPG1EW6AiaJWrn0Ts7Q93ZIyrfWVDbgs7VKZ46Swo4jxv220h2SzaIsAupExYDjlKuwHlM1aeNKB67fSw/gXS3SWdMul6vb52bs8NQeSpwpj96dTZu8dTZGbk74LDfRrJb0keEXciEqCGZqBOfgojiAdsipSpiqudlL2Yd1hHEnXTkHmwOW+dTZ4xdwijFQ4RdyAQVwbLzqe36Ll6zVuM+xrs7hLBYb5Q8f5WMG2eH4PfZ9mpBScVV5Xvy2sYr28a9GpXXwK143MVDhF3IjLB8eDvG7SdKUQdNg7x7nbFgleqNzjYGCXvc0I7K0nlenVKcpx8JoxQfEXYhc4KEJ+i9KJOqvAYSvbxW3bFglQ4h6Dz8zj/Mm3fO2KzqOp/CM4iwC5kTJH5xRM/LE1Ud/AxrZxzvNGy/oPNQeZLwysl3Lp8X9clGKB8i7EIu+IlfEtGzCQv3qLYvrbQ8FfEO6hjCxgckpVAQYRcKRVgIRcWLVs3lDiJtrzdJrFol1CKx8Gojwi4UBi9P0554EwU/UbYHZ1UocpxaQi1CGKrCXoMgpMzS0hIGg8HIa8yMG2+8MdJxlpeX0Wq1Rl4jIlx55ZVYWFhQOsbMzEyk17PE6/xarRaWl5dzapEwtqiov24Tj71a6PSSk4YiksbB0ybvzxeKDSQUIxSFooUY/MRTBieFoqMq7GRtmy3z8/O8urqa+ecK+dDv97G4uDgSjmm1WlhZWVEOoWTB7Ows1tfXt71umibW1tayb5AguCCiA8w8H7adxNiF1FlYWMDKygpM0wQRwTTNwok6ABw6dCjS64JQVMRjF4Qh4rELRUc8dkGIiGSlCGVBhF0QhoxLyEgQwpBQjCAIwpggoRhBEISKIsIuCIJQMkTYBUEQSoYIuyAIQskQYRcEQSgZuWTFENFhANtngpSLaQBH8m5Ejsj5y/nL+evHZObTwzbKRdirABGtqqQllRU5fzl/Of/8zl9CMYIgCCVDhF0QBKFkiLCnx0reDcgZOf9qI+efIxJjFwRBKBnisQuCIJQMEXZBEISSIcKeMkT0NiJiIprOuy1ZQkTvJ6J7iehOIvocEZ2Wd5uygIguIqIfEtH9RPTOvNuTJUS0i4huIaK7ieguItqbd5vygIjqRPQ9IvpSXm0QYU8RItoF4LcBVHFttX8A8EJmfhGA+wC8K+f2pA4R1QH8DYDXAJgDcDkRzeXbqkx5GsDbmHkOwCsAvKVi52+zF8A9eTZAhD1dPgjgz2GteF8pmPnvmfnp4Z/fBnBmnu3JiJcDuJ+Zf8TMxwH8LwCX5tymzGDmnzDzd4f//zkscXtuvq3KFiI6E8DFAD6RZztE2FOCiC4F8GNm/n7ebSkAfwzgy3k3IgOeC+ABx98PomLCZkNEswBeCuD2fFuSOX8Ny5nbzLMRE3l++LhDRF8DcIbHW0sA3g0rDFNags6fmT8/3GYJ1iN6P8u2CflBRFMAbgDwZ8z8RN7tyQoiugTAI8x8gIguzLMtIuwJYOZXe71ORL8G4CwA3yciwApDfJeIXs7MD2fYxFTxO38bInojgEsAdLgaEyZ+DGCX4+8zh69VBiJqwBL1PjN/Nu/2ZMz5AP49Ee0GMAngWUTUY+Yrsm6ITFDKACJaAzDPzJWpdkdEFwH4AIALmPlw3u3JAiKagDVQ3IEl6N8B8EfMfFeuDcsIsryY/QAeZeY/y7s9eTL02N/OzJfk8fkSYxfS4moAOwH8AxHdQUTX5N2gtBkOFv9HAF+FNXB4fVVEfcj5AF4P4FXD3/yOofcqZIx47IIgCCVDPHZBEISSIcIuCIJQMkTYBUEQSoYIuyAIQskQYRcEQSgZIuyCIAglQ4RdEAShZPx/7EEwhkbXQPkAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ "<Figure size 432x288 with 1 Axes>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAACPCAYAAAAfidZ8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztnXmsXdWV5r8FMYHEmHjCGGP8jHHwQAzGDjghjkgYhEMqgNLEga4qSk11ulsVpao7ippUq1v1T3VXSqpqtVrd6oqqUpCESlIRICBBIsSYyQNgYwYP2MZ4wMYDZjSEEAi7/3jXO99e9e72uffd6Zzz/SSLdd8+7979znf2uZu7vruWhRAghBBCCCFEHTmu3xMQQgghhBCiX2gzLIQQQgghaos2w0IIIYQQorZoMyyEEEIIIWqLNsNCCCGEEKK2aDMshBBCCCFqizbDQgghhBCitmgzXBAb5jtm9krj33fMzPo9L9E6ZvY5M1tpZm+Y2a5+z0e0j5l9y8w2mtkRM9tpZt/q95xEe5jZfzSzF8zsTTN7ycz+p5l9qN/zEu1hZieY2RYz29vvuYj2MLO/MLP3zOwt+ndWv+fVDbQZLs7XAFwD4DwACwD8HoB/19cZiXZ5G8D3AGjjVH4MwB8CGA/gSgBfN7Ov9ndKok3uBnBBCGEcgHMxfK/9Rn+nJEbBtwC83O9JiFHzkxDCWPr3Qr8n1A20GXaY2XL3f0HvmtmDAG4E8DchhL0hhH0A/gbAH/VzriJPMy1DCI+HEH4AoJKLuopktPzrEMKTIYT3QwhbAdwF4OJ+z1c0J6PljhDC60cPA/ABgLP7OFWRIfNeCTObCeD3AfyPvk5SFCKnZV3QZtgRQoj/FwTgdAxvmH4EYD6Ap+nQpxs/EwNKRktRMopo2bAtLQWwqQ9TFAXJaWlmN5jZmwAOY/iT4b/r30xFjmOsyf8N4M8BvNOv+YniHEPL3zOzV81sk5n9h/7NsrtoM9wEMzsOwD8BeDCE8HcAxgJ4gw55A8BY+YYHnxG0FCXlGFr+BYbvaf/Y63mJ1hlJyxDCPzVsEh8H8P8AHOzjFEUBvI5mdi2A40MId/Z5aqJFRliT/wxgLoDJAP4tgP9mZtf3cYpdQ5vh5vwlgJPxO8/aWwDG0fg4AG+FEEKvJyZaxmspysuIWprZ1zHsHb4qhPBuPyYmWqbpugwhbMfwJ/z/t9eTEi0TdTSzjwL4a+heW1aSNRlC2BxCeCmE8NsQwmoA/wvAv+rnBLuFvqk7Ao0v4FwP4JMhhPcaP96E4bTd443H50Hp2IGniZaihDTT0sz+DYCbAXw2hKBvrpeAguvyQwBm9W5WolW8jmY2H8AQgEcaSdMTAJxiZgcALAkh7OrXXEWegmsyYNjPXzn0ybDDzBZi2O90TQiBvwn7fQD/ycymmdnpAL4J4JY+TFEUpJmWZnacmZ0IYMzwQzvRzE7o1zzFsclo+a8B/HcAl1f1W85VI6PlH5vZqY14HoBvA1jRn1mKY9FEx40ApgM4v/HvjzFsdTkfwIv9mKc4Npk1ebWZjW+Ulr0Qw58Y39WveXYTfTL8L7kaw2WaHiU78CMAvgDgLADPNn7299CXOwadZlp+B8BKOu4dAA8BuKSXkxMt0UzLOQAmAniCfv7DEMK/7/kMRVGaaXkAwF+a2VgMl+T6KYD/2pcZiiKMqGMIYdnRB2b2KoAPQggH+jA/UZxma/J1DJch/TCAvQC+E0K4tS8z7DImy6sQQgghhKgrskkIIYQQQojaos2wEEIIIYSoLdoMCyGEEEKI2jKqzbCZXWlmW83seTO7uVOTEkIIIYQQohe0/QU6MzsewDYAl2P4W4ZPALg+hLC5c9MTQgghhBCie4ymtNqFAJ4/WtvTzH6M4fIcTTfDkyZNCkNDQ6N4SdEuu3btwuHDhztWLFta9o9Oaikd+8v69esPhxAmd+K5pGV/kZbVQVpWg1beK0ezGZ6GtIj2XgAX5X5haGgI69atG8VLinZZvHhxR59PWvaPTmopHfuLme3u1HNJy/4iLauDtKwGrbxXdv0LdGb2NTNbZ2brXn755WP/ghhYpGU1kI7VQVpWB2lZHaRl+RjNZngfhtsuHuWMxs8SQgjfDSEsDiEsnjy5I1kH0SekZTWQjtVBWlYHaVkdpGX5GM1m+AkAs81sppmdAOCrAO7uzLSEEEIIIYToPm17hkMI75vZ1wHcB+B4AN8LIWzq2MyEEEIIIYToMqP5Ah1CCPcCuLdDcxFCCCGEEKKnjGozLIQQnSBX79yP8WOOzdIKOscdd1zTMdFZiurnj/vggw9G/B3Wzj+Wlp2l3V4D7T5/M809Xmd+nLsGdH38jk5r67XLre2cDkXXcy+1VDtmIYQQQghRW7QZFkIIIYQQtUU2CSFEX8il2PixT80VTa0PSvqtihS1rgCpXrmxdvWSlsUomjLvRGq9nfXrydkkBjHN3i86rVdu7Le//W0yxo9zNomcha1Z7Om2lvpkWAghhBBC1BZthoUQQgghRG3RZlgIIYQQQtQWeYZR3O/2m9/8JsaHDh1Kjtu7d2+MP/zhDydjs2bNivHYsWOTseOPPz7GKh0zeor6UN97770Yv/baa8lxe/bsifH777+fjM2YMSPGEydOTMbGjBkTY3nahimqR27d5TyGufOc88G141uso47NNGpFy6Lll3LnupXSe0Wog5Y5HTrxnP752D/67rvvxvj1119Pjnv11Vdj/NZbbyVj/H74kY98JBmbNGlSjE855ZQY+/dbfg5PWbXtxPXfyv2Q3/fefvvtGPt9z4EDB2Ls30dZl/Hjxydjp59+eoz5ffTEE09MjvvQh5pvUTutpT4ZFkIIIYQQtUWbYSGEEEIIUVtkk2gBTq1v2bIlGbvvvvti7FMCy5cvj7FP/eRSOs3IpTvKmgZql1ZSf81sEjt27EiOu/POO2PM1hgA+NKXvhTjCy+8MBnjlE5RjaqgZdGyaLmyPLkxb1Xh88zWFL+WciWBeF6+nA8/D8e50m2tlHUbZIraVXLlslrpUtXs/LaS0uVji2pU1TJ8ne44lrse/Lp84403YvzCCy/EeO3atclxzz333Ii/41+PrRAAMG/evBh/9rOfjfHs2bOT4z760Y/GuEzaebptjcjdY1955ZUYb9q0KcYPPfRQchy/d3o7DK+xU089NRlbsGBBjJcuXRrjc845JzmObaW5+3sndNYnw0IIIYQQorZoMyyEEEIIIWqLNsNCCCGEEKK21NIz3K7P9Ne//nWMn3zyyeQ49kF534sYDJp5hrdu3Zoct23bthh731rR9rFVJ9dulf1nHHv/NfsF2aMGAC+//HKMubQPAIwbN27E2Pvx2U/steIx/3v+cTOKelwH/TrJeaubacnrB0hLZPlyWfzYexNPOumkGPN5z5VYyrV29WW2TjjhBBShaCnEMpErY5c7Lre2uWTaSy+9lIyxn3TNmjUx3rx5c3Icr+133nknGePX89cA36e5xNd1112XHMfeYn89lJVOeMH92ub7sddyxYoVMX744Ydj7Pc9fA/39wSe88knn5yM7d+/P8Z8DXi9eC/VzverWkGfDAshhBBCiNqizbAQQgghhKgtlbJJdLqsjH9OTtdyKh0ADh48GGMuGwKkKYJcSZ9OpOeKpsbKRLu2llz6jy0v3ibBXXW46xEATJkyJca+O06VtWylxBKnyzgFxp2nAGD79u0x5vI9QKoBWxqAVIPTTjstxj6tymk1rxXr6sv+cGo91wGprOQ6h/lUJ6fFjxw5EmNOdQOplrt3707G3nzzzRh72wLrwLp6exL/nk+XciktX9ayWaeyKtwXgfbtOUU7yf3qV79KxjZu3Bjj22+/PRl74oknYsxpcJ+ez9laOHXv7TZ8jfFzslUKSLub8TVVB4p2BQTSsmh33HFHMvbAAw/EeOfOnSM+H5DanHx3XX5f8K/9/PPPx5jX6LRp05Lj+LG/VtSBTgghhBBCiA6hzbAQQgghhKgtpc8B9rLjDqd+nnnmmeQ4tlBwmgZI0wfeJiFGphO65rTklL3vJsjpOZ9m45S6T9c2S9v4n5e58sBRiqZZeV3s3bs3Oe7ZZ5+NMadAgdSecNZZZyVjs2bNivGECRNi7KtVcIWKXKcr/01nTrvmrC9l0Qoorpe3SbA1gr9xzilxIK0Y8NprryVjfP8788wzkzE+95xy9Vryc/oxtkn46gecWs11K6wiRTuY+dQ33/8ee+yxZOz73/9+0zG2VLDmvF6B1BrjK7ewrcpbEfft2xdj7nC3evXq5LiLL754xNcCqm978tc/rxXfafXWW2+N8b333puMcbUO1ogrdQDpeuZ7MZBq6V97165dI475yiMXXXRRjL0FqtNaamcmhBBCCCFqizbDQgghhBCitmgzLIQQQgghakvpDTTs2+tGaTX2U23YsCHG3v/I3qT58+cnY1wWqNu+wzL5GHsNa8mlXXxpNT6HH//4x5OxnP+7ndJqueMGWcuiZZvYN+ZLbvEa8l63mTNnxvjTn/50MnbGGWfEmL2f7G/1r82lD4HUSzd16tSm889RFT8x43Vgzzd32Xz66aeT49jP6b19Z599dowvuOCCZIw9+Ozp9WW1uBQil90D0nXty+tNnDgxxr70UxVo5fsIPMZlr3yHxwcffDDG7BEGUp8wawKknlH2euZ8pr4s2uuvvx5jvy5XrlwZY76XeJ/p2rVrY7x48eJkrCqe4WY+Ye+n37NnT4x/+MMfJmNcTo09wkBa7ozvv4sWLUqOmzt3boz9uufvaaxfvz4Z4/s2+4f9vorv2957zs/RifvtMT8ZNrPvmdkhM9tIP5tgZveb2fbGf8fnnkMIIYQQQohBpIhN4hYAV7qf3QxgRQhhNoAVjcdCCCGEEEKUimPmDEIID5vZkPvx1QAuacS3AngQwH/u4LzaouhH5a3YKbjU0OOPPx5jn8Y755xzYuxLQdWhjE+nKZqGbwXWctWqVTH23bQ4HeO7CeYsL0Upawq9GblygWxd8Ckw7kzmSyCde+65MZ4xY0Yyxim8XIqQux5xWTAgTZdyGt8/J1M13UbC/+2sH5ez8lYFLos2Z86cZGzJkiUxnj59ejLGZdF4XftrivXyKV0uk8jPBwBDQ0MxbqZrVfH3SbZGcBk0thUAwG233RZjXz4tV3Zy2bJlI8b+/ZA18jrz83st2YrDNgm2VgCp7c2/T/N1Wqb1nCsTyu9rfl3+5Cc/ibHvGMj3Y29xuPTSS2N85ZW/+yzUvx+yNcZbUPjc+5KNL774Yoz5vsLla/0c/XP4jnSjpd0v0E0JIRyd9QEA9ep5KIQQQgghKsGoq0mE4f9lafpRnZl9zczWmdk6/wmcKBfSshpIx+ogLauDtKwO0rJ8tLsZPmhmUwGg8d9DzQ4MIXw3hLA4hLB48uTJbb6cGASkZTWQjtVBWlYHaVkdpGX5aLfOyN0AbgTwV43/3tWxGfUZ781hXyO3IPUtLNknx+V8gNQXJZ/p6ClaTs+PsbdwzZo1MWZvKZB63KZNm5aMsZa50mo5qqBlrqwY+9m4ja73g3HpM9/Kk8sq+ZJY7MFnjdkHCaQle3bu3JmMcYtR36q5bt7S3PXIvj/+hMvf/1g/X1qSS+F5nXkNsafVw9eR939z+SUu1Qak1xjr2o0ynIOGv475PrdxYywOhR/84AfJcewT5vMHAKeffnqMly9fnoxdc801MWavtvd25u7fvC75OYDUo8z3AD9Hvh68Z7gqG1Nef+yZvueee5LjfvrTn8aYfbpA2gr9iiuuSMa+8pWvxPj888+PsS+Fxz5hryXr7PdErDP/Lb6VO++//D2n0xQprfYjAGsAnGNme83sJgxvgi83s+0ALms8FkIIIYQQolQUqSZxfZOhS5v8XAghhBBCiFJQjXYsHcR/1M8pOS7zwSW2AOAzn/lMjH0XpKI2iSqkzwcJn1bZsWNHjLds2RJjb3dYuHBhjDmVBBTvelNnLbkEzuHDh2Psv0jCtgZ/nrmskj+XnE5niwN3FQRSjbdt25aMcQrWd9Li66Yq6fSi1iKfWue0Jevlj/vYxz4WY06lA6nNxZeZ5LlwaTxvXeFOWl5nThP7jpGcQs/pyo9z6d4y4e9/zbqRsWUMSHX2tpMbbrhhxBhIdffvj83InWu+poA0Rc/XkbfXcEc9X46rrPjzxJYXLvnKXeWA9D3P70u4fNr116efeXLnPj7vuTKa/p7AFgq/hnitc5x7/m6vw1FXkxBCCCGEEKKsaDMshBBCCCFqizbDQgghhBCittTSM+y9J+zH8b6XdevWxZi9SL5F7KJFi2Ls2xLmylCJ7uG9ZI888kiM2ZPoPY7s//bet06UySsjrbTHZj8b+4RfeeWV5Dhea74EEvtAvY+MfaDsiVu9enVy3DPPPBNjLrMGpCW+fEk29rCx79J7MMu6rnNz9X8j68D3P1+OkH/Pt8X2jxl+Hm6zzH5vIPUJc/tWIP17eI7+MXvDfbmv3Lrm63vQdeY1xaUkgbQd7y9/+csY+3bG3O78qquuSsa4nBqXzAOAMWPGxLjoefL3DtbBv4/ytcL3du9DZ28sz6nM+PcyLhXJ5dP4ngek54b3KEBaPu2iiy5Kxvg7HDkfb25t8Jhfl7zW+d7vvzvC16LXudPok2EhhBBCCFFbtBkWQgghhBC1pZY2iRw+/bdq1aoY88f+S5YsSY7jTmXtdiYTncWn/1hLTutyKTUAmDVrVoxzlpc6k7MWcUqPY58u57Jdzz33XDLGKTFfEoh15a5KW7duTY7j1Lov1cWdjXJdj7hTkl/XVSyZ6G0Szbo++fPJOnCpJyA9n3498bnn5/AdAzn9y+X6gFQjn45la86RI0difNJJJyXH8fWWSwsPOrzenn766WTs3nvvjTGXDPVWggULFsT42muvTcbYGtFtO6C3L3Fqna9Lb3lh25vXuayWF15DAHD//ffHeOXKlTH21z/bOb/4xS8mYxdffHGMfZfPduyAuRKY3trE92bec40fPz45bvr06TGWTUIIIYQQQoguoc2wEEIIIYSoLbW0SfhvsHI64uDBg8kYp/w4ncQVB4A0VTfo6Zcqw+kz36lq8+bNMWYtP/WpTyXHcapGWrYOnzNOv/kU2P79+2O8YcOGZIxT5j4dy6lATqX6tCqn6fya5/S8t0nwt/D52828xkd6zrLCenmLwMSJE2PMWvq/ndeWrwyycePGGHstuTII/x5bGoDUluFtOVwlwl8DbKngVLPvbuatOGXB68BrgytGAKmNiM8ZawykFkDf0a8TFSNyY5wy3759ezK2e/fuGHPKfMqUKclxbHvjTpaDjj8X3D2P/3YAeOCBB2LMexZfjYGtENxxDkjXQCesQX5dsp1t06ZNyRhXGeK1N2/evOQ4trx0276kT4aFEEIIIURt0WZYCCGEEELUFm2GhRBCCCFEbamlZ9jDPlNfjob9g+x59B1b2vFSidHjfVZcvmvFihXJGPsQJ02aFGPv/5aWo4NLHbHn0PvB2C/K/mEg9Xr68kj8/OwX9CWWWEfvuWN8OUV+nPPTVuXa4L/Dd1zkkpGsn/cFsz9/7969yRh7sr2HkzXi2PtA2f+9b9++ZCzX/Y6vMcaXaSpzOTWGS8n5bmTeh32U2bNnJ4/5OxS5kltFyX1Hh73LQFpi8ec//3kyxvcI9jmzLxYAli5dGmN/PZcJPjePPfZYMsba8vk888wzk+MuvPDCGPtOq7wG/L2sWZk87wvmtefLLT766KMxXrt2bTLG99izzz47xqwdkL5Pq7SaEEIIIYQQXUKbYSGEEEIIUVtqY5PgVE0utc7dXIA0PccpX+7EA1QnZVoGct3POE3oUzPMueeeG+OZM2cmY6ylv1ak8zA5+wCnVvncLlu2LDmOU2C7du1KxjiN5kuajRs3LsannHJKjP21wL/nS25xWSH//BMmTBjxON+pq6ypdX8N89/hS4zxfY5TmHzegbRTlO/8yM/vSz/xuefn9OltLs3kSybmOtyxfny9eetNOx23BgF/zbO9yNtV+Fj+++fMmZMcxxYVf40XLSfIx3GJMCC1zaxZsyYZu/POO2O8fv36ZIx1Wbx4cYy//OUvJ8dx1zWfWh9kbf25ZduBL03GlhdeQ/y+BgBz586NsV/bufe5Zjp7GxLbSLncGwD8+Mc/jrG/v/O6vOyyy2LsO/vyddpt7cp5NxdCCCGEEKIDaDMshBBCCCFqizbDQgghhBCitlTWM9xKC0j2MHFpFyD1oC1YsCDGZWrzWEaK6seebgDYsmVLjPfs2ZOMsWdq0aJFMfY+RtEa3pfH5/nUU0+NsT/P7E3k9stAWrqLPcJA6pHj9Xno0KHkOH5O9gH7eXH5MD8v/lty/sNcaaJBJ1dajc8b+/d86bPzzjsvxtxqFUjXq9eSy+GxluwDBoADBw7E2Hsf2ZPKvmAg/Z4H/y05/3eZtMy18PXw9cta+vcyfg5fdpBfz58XHuPf8x7ve+65J8a/+MUvkjEugeh1Zj/p8uXLY8z3cv97ZfL151pr+5bx/HfxdzR8a21eU/57E7n7F5dM43l47+99990X45/97GfJGOvu7/1XXHFFjK+77roYn3baaU3nL8+wEEIIIYQQXUKbYSGEEEIIUVsqa5PIwSkAANi5c2eMfScsTifNnz8/xr6ED6NyXO3RTtken8Z7/PHHY+y7T3E5l/PPPz/GOS3FyBTtzMapWZ+C53QmawOkJXx8+Sgm18WO072+Ox2n032ZRE4bd7vr0SDQTC8g1ZatBb4cHZ9Pn47NlUJs1o3Md7jzncoYtl74Lltsh+H5567ZQb9f5+6T/H7FfzuQ2sb4vL/00kvJcc8++2yMfSktTnf7MS6px2XRHn744eS4J598MsY+/c9rz3d5vfHGG2PMpdV8Cj7XWW3QyJV85cf+/sX3Tj7OlzXctm1b09dme4VfX2xL4m53/P4KABs2bGj62tyx9+qrr07GbrrpphhzBzr/d/bS5nLMVzKz6Wa20sw2m9kmM/vTxs8nmNn9Zra98d/xx3ouIYQQQgghBoki2+73AXwzhDAPwBIAf2Jm8wDcDGBFCGE2gBWNx0IIIYQQQpSGY26GQwj7QwhPNuIjALYAmAbgagC3Ng67FcA13ZqkEEIIIYQQ3aAls6SZDQFYCOAxAFNCCEdNegcATGnyawNBzmfKHiY/xqVKuNVhrqzMoPuUyob3UrHnm9svA6n/2+tw5plnxnhoaKjpcaI1cucv5y1m2F8G5Ms7sSeVvW7cvhRIr5NcaTXvV27HJ1yVayhXViznLWbfvfeGsw6+FCJry7p63zEfx15HIC3z5v3f7K2sikaM/5v4uuZSoACwffv2GPNaWbVqVXIcn3u+TwKp79r7urmkHntVfcnDXCm8z33uczFmXymQfmeHvdFlarncCnydex24dTl/V2L16tXJcayRb2/NeO82t/LmEpX+Hsvnmr2/AHDNNb/7fPSGG25Ixnidsk+4FS9/p3Uu7E42s7EAbgfwZyGEpAhkGN6tjOjqN7Ovmdk6M1vn60+KciEtq4F0rA7SsjpIy+ogLctHoc2wmY3B8Eb4thDCHY0fHzSzqY3xqQAOjfS7IYTvhhAWhxAWT548uRNzFn1CWlYD6VgdpGV1kJbVQVqWj2PaJGz4s+h/ALAlhPC3NHQ3gBsB/FXjv3d1ZYbHoGg5Lk7V+RIg3PXGl9nij/59Skf0Bq8xp1p9WSDuXOXLP7GWnJbPpVty11dV0nGdptl58eeSU2J+3eXSY5xm5RSpL8uT05g7HfluZLk5N5tjHeC/N6elPy885tOgXOKLrwGvJdtcuKucH/N2m9ycmx1XJl39XPnvv+yyy5IxTnevWbMmxocPH06Oe+ihh2LMpbOAf3lPZbjUGp9r3xVt9uzZMf785z+fjC1btizGU6dOTcbYfpOzMpW1M6SfG9u3uPseAOzYsSPGK1eujLF/P+QSaV47Xm/evsT7JT7vvlvnJz/5yRhfe+21Tcf8uuTXzt07ekkRz/DFAP4AwLNm9lTjZ3+O4U3wP5vZTQB2A/hKd6YohBBCCCFEdzjmZjiE8CiAZtv1Szs7HSGEEEIIIXpHZVtv+U5H/LH/q6++moy9/fbbMfbfLJ8xY0aMuTuOUuujJ3eecl2rOB3HaSD/e96rNWvWrBizlr3sclM3iq6FVr4Rzt+y5vSeT+HxN6L983GVAU+zOWvtNqdoajqnA997fSe5s846K8ZshQLS9K+32/B9P9ftq0zkrB9cZcF3cOMqEXzOuOMckJ7f3DXP91AAmD59eozZkrZo0aLkOJ5XrvpHK9apZscNOrm/ibU877zzkjGuEsE2L18xgi0wbC8D0vc97uIIpBYVfu1LLrkkOY619R0P2V7RbsfHXmqpXYAQQgghhKgt2gwLIYQQQojaos2wEEIIIYSoLaXzDBf1mebGfNke7kzmPSoXXHBBjLmET87jWKZyLoNEM09fTldfLoa9TtyZCkg7Mp188skxlpadpdN+zKIeVO9L43XO/jsgX6aJHw9K2Z9BoF1di66nXBc79jT6Mk2sl/eZNnutnIexTPh587nwJc2uuuqqGM+dOzfGTz31VHIcd/Hk72cA6f3W31/nzJkTY/6ujdcrt/Zy662sGhXF/318LfuyrpdffnmMuVQdd9MF0hJsvmMg+7N9GbuFCxfGmP36/jtVOS07/T7abf31ybAQQgghhKgt2gwLIYQQQojaUjqbRFH8R+pcfmRoaCgZ+8Y3vhFjX8aLU61FS4WI9mh2Dv255pT30qVLkzHueuN/j1N80rL/FO0OloPXtS/1xKlatsUAaRkoX4KP1zyn/uqWtgU6Y3kp+hx8Pn1XQO4YeOTIkWSMNfKluthewWnnqmiX+zu8ZYTXAFvG5s+fnxzH5ej8+yFrWbTTYCvp805TJp1zc/Xrga/rT3ziEzGeN29echyXnvTrkB/798Bm+rVbIq1dVFpNCCGEEEKIHqDNsBBCCCGEqC3aDAshhBBCiNpSKc9wzl/CvhfvYfKl1tp5ftFZcq16+bEvwcSteos+v+gsRc9tUf+hp5nf218LvrQUw9eJL7vG/ry6l1Zr52/23sScN5zPL5933x6Wy1/6MlN8//ZZ5+vrAAAESElEQVRlvJq1aq7q9wSK/h25MnPenyr6Qzttiv17pb8niubok2EhhBBCCFFbtBkWQgghhBC1pXQ2iVzqoBNlgKqSLisDOtf1pmgaMGeh4NS3tztxKancc7STjhTN8ecsV8Kpmc6+syR3y/K0U8ZLugohGH0yLIQQQgghaos2w0IIIYQQorZoMyyEEEIIIWpL6TzDOeQDE6J61LENcpVoxxvuvb/tPr8QQhRBnwwLIYQQQojaos2wEEIIIYSoLdaJcmSFX8zsZQC7AUwCcLhnL9ycOs1jRghhcqeerKHl26jP+StKqbTUmmxKr+YhLbuPtBw9dZtHp7UclPdKoF5aFtaxp5vh+KJm60IIi3v+wppHRxmUeQ/KPIDBmksrDMq8NY/RMyhz1zxGz6DMXfMYHYM070GZy6DM4yiySQghhBBCiNqizbAQQgghhKgt/doMf7dPr+vRPEbHoMx7UOYBDNZcWmFQ5q15jJ5BmbvmMXoGZe6ax+gYpHkPylwGZR4A+uQZFkIIIYQQYhCQTUIIIYQQQtSWnm6GzexKM9tqZs+b2c09fu3vmdkhM9tIP5tgZveb2fbGf8d3eQ7TzWylmW02s01m9qf9mEcn6JeWg6Bj4zWl5ehfV1p2kLrfXxuvKS1H/9p917IqOgK6v5ZFy55ths3seAD/B8AyAPMAXG9m83r1+gBuAXCl+9nNAFaEEGYDWNF43E3eB/DNEMI8AEsA/EnjHPR6HqOiz1regv7rCEjLTnALpGVH0P01Ii1Hzy3ov5al1xHou5a3oP86AmXRMoTQk38APgXgPnr8bQDf7tXrN15zCMBGerwVwNRGPBXA1h7P5y4Al/d7HmXTctB0lJbSst//+q2jtJSW0nHwtBw0HQdZy17aJKYBeJEe7238rJ9MCSHsb8QHAEzp1Qub2RCAhQAe6+c82mTQtOzr+ZOWHUVatseg6QhIy3aRlkSJdQQGT0utySboC3QNwvD/nvSktIaZjQVwO4A/CyG82a95VJFenz9p2T2kZXWQltVB75XVQGsypZeb4X0AptPjMxo/6ycHzWwqADT+e6jbL2hmYzB8QdwWQrijX/MYJYOmZV/On7TsCtKyPQZNR0Batou0RCV0BAZPS63JJvRyM/wEgNlmNtPMTgDwVQB39/D1R+JuADc24hsx7GXpGmZmAP4BwJYQwt/2ax4dYNC07Pn5k5ZdQ1q2x6DpCEjLdqm9lhXRERg8LbUmm9Fj4/QXAGwDsAPAf+nxa/8IwH4A72HYt3MTgIkY/hbjdgC/BDChy3P4DIZTAc8AeKrx7wu9nkeZtRwEHaWltBzEf3W/v0rL6mhZFR37qeUg6FgmLdWBTgghhBBC1BZ9gU4IIYQQQtQWbYaFEEIIIURt0WZYCCGEEELUFm2GhRBCCCFEbdFmWAghhBBC1BZthoUQQgghRG3RZlgIIYQQQtQWbYaFEEIIIURt+f+UtoefRFpU0wAAAABJRU5ErkJggg==\n",
+ "text/plain": [
+ "<Figure size 864x180 with 6 Axes>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "z1 = z[:,0]\n",
+ "z2 = z[:,1]\n",
+ "\n",
+ "fig = plt.figure()\n",
+ "ax = fig.add_subplot(111)\n",
+ "ax.plot(z1,z2,'ko')\n",
+ "plt.title(\"latent space\")\n",
+ "\n",
+ "#np.where((z1>3) & (z2<2) & (z2>0))\n",
+ "#select the points from the latent space\n",
+ "a_vec = [2,5,7,789,25,9993]\n",
+ "for i in range(len(a_vec)):\n",
+ " ax.plot(z1[a_vec[i]],z2[a_vec[i]],'ro') \n",
+ " ax.annotate('z%d' %i, xy=(z1[a_vec[i]],z2[a_vec[i]]), \n",
+ " xytext=(z1[a_vec[i]],z2[a_vec[i]]),color = 'r',fontsize=15)\n",
+ "\n",
+ "\n",
+ "f, ((ax0, ax1, ax2, ax3, ax4,ax5)) = plt.subplots(1,6, sharex='col', sharey='row',figsize=(12,2.5))\n",
+ "for i in range(len(a_vec)):\n",
+ " eval('ax%d' %(i)).imshow(np.reshape(x_construction[a_vec[i],:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
+ " eval('ax%d' %(i)).set_title('z%d'%i)\n",
+ "\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Above is a plot of points in the 2D latent space and their corresponding decoded images, it can be seen that points that are close in the latent space get mapped to the same digit from the decoder, and we can see how it evolves from left to right."
+ ]
+ }
+ ],
+ "metadata": {
+ "anaconda-cloud": {},
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.5.2"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/example/vae/VAE_example.ipynb b/example/vae/VAE_example.ipynb
deleted file mode 100755
index e7ec03afdb7..00000000000
--- a/example/vae/VAE_example.ipynb
+++ /dev/null
@@ -1,1204 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "import mxnet as mx\n",
- "import numpy as np\n",
- "import os\n",
- "import logging\n",
- "import matplotlib.pyplot as plt\n",
- "import matplotlib.cm as cm"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Building a Variational Autoencoder in MXNet\n",
- "\n",
- "#### Xiaoyu Lu, July 5th, 2017\n",
- "\n",
- "This tutorial guides you through the process of building a variational encoder in MXNet. In this notebook we'll focus on an example using the MNIST handwritten digit recognition dataset. Refer to [Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114/) for more details on the model description.\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Prerequisites\n",
- "\n",
- "To complete this tutorial, we need following python packages:\n",
- "\n",
- "- numpy, matplotlib "
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 1. Loading the Data\n",
- "\n",
- "We first load the MNIST dataset, which contains 60000 training and 10000 test examples. The following code imports required modules and loads the data. These images are stored in a 4-D matrix with shape (`batch_size, num_channels, width, height`). For the MNIST dataset, there is only one color channel, and both width and height are 28, so we reshape each image as a 28x28 array. See below for a visualization:\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "mnist = mx.test_utils.get_mnist()\n",
- "image = np.reshape(mnist['train_data'],(60000,28*28))\n",
- "label = image\n",
- "image_test = np.reshape(mnist['test_data'],(10000,28*28))\n",
- "label_test = image_test\n",
- "[N,features] = np.shape(image) #number of examples and features"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {
- "collapsed": false
- },
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAADFCAYAAACxSv92AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFzNJREFUeJzt3Xuw1XW5x/HPg8rdGMEtoYfj9gJqOZyNbdBGOYOCRFp5\noRwtHUpHgkzFQHFwTNMsrGMKURYIgdnxWIi3xjlSdPIymYlhBwEvpRBX2YxQQolcnvMH68wQ6/nF\nWnvd9lrf92umYe/P/u7fen6sZ+8efq7fd5m7CwAAAEhRp1oXAAAAANQKwzAAAACSxTAMAACAZDEM\nAwAAIFkMwwAAAEgWwzAAAACSxTAMAACAZDEMAwAAIFkMwwAAAEjWwaV8s5mNljRd0kGS7nP3af9s\n/eGHH+7Nzc2lPCSgl156abO7N1XzMeldlGrVqlXavHmzVfMx6VuUA79zUa8K7d12D8NmdpCk70k6\nW9JaSS+a2ePuviLre5qbm7VkyZL2PiQgSTKz1dV+THoXpWptba36Y9K3KAd+56JeFdq7pbxMYqik\nP7r7m+7+vqT/knReCccDAAAAqqqUYfgoSWv2+XxtLvsHZjbOzJaY2ZK2trYSHg6oLnoX9Yi+Rb2i\nd1ErFb+Bzt1nuXuru7c2NVX1JUdASehd1CP6FvWK3kWtlDIMr5PUf5/P/yWXAQAAAHWhlGH4RUkD\nzOwYM+ss6WJJj5enLAAAAKDy2r2bhLvvMrMvS3pKe7dWm+vuy8tWGQAAAFBhJe0z7O5PSnqyTLUA\nAAAAVcU70AEAACBZDMMAAABIFsMwAAAAksUwDAAAgGQxDAMAACBZDMMAAABIFsMwAAAAksUwDAAA\ngGQxDAMAACBZDMMAAABIFsMwAAAAksUwDAAAgGQxDAMAACBZB9e6AACNbc2aNWE+ffr0ML/77rvz\nsuuuuy5ce+2114Z5//79C6wOAJA6rgwDAAAgWQzDAAAASBbDMAAAAJLFMAwAAIBkMQwDAAAgWewm\nUWF79uwJ8x07dpTl+PPnzw/z7du352UrVqwI195zzz1hPnXq1DCfOXNmmHfr1i3M77rrrrxswoQJ\n4VrUr3Xr1oX54MGDw3zr1q1hbmZ5WVaPZvV/W1tbmAMd2cqVK8N85MiRYf7yyy+HeVNTU9lqQppm\nz54d5uPHjw/zrFnntddeC/OBAwe2r7AKKWkYNrNVkt6VtFvSLndvLUdRAAAAQDWU48rwme6+uQzH\nAQAAAKqK1wwDAAAgWaUOwy7pl2b2kpmNixaY2TgzW2JmS3gdH+oJvYt6RN+iXtG7qJVSh+Ez3L1F\n0sclXWVm/77/Anef5e6t7t7Ki/pRT+hd1CP6FvWK3kWtlPSaYXdfl/tzk5k9ImmopGfKUVg1/eUv\nfwnz3bt3h/kf/vCHvGzRokXh2qw75mfNmlVgdeXT3Nwc5pMmTQrzOXPmhHmvXr3CfNiwYWF+1lln\nHbg41I3Vq1eH+fDhw8N8y5YtYR7tGiHF/dWlS5dw7aZNm8L8zTffDPOjjz46zA866KAwT9kbb7yR\nl2U9l0OHDq10OUl44YUXwnzEiBFVrgQpWbx4cV72la98JVzbqVNx11Czfs93NO2+MmxmPczs0P//\nWNIoSa+UqzAAAACg0kq5MtxX0iO5qf9gSf/p7v9dlqoAAACAKmj3MOzub0r6tzLWAgAAAFQVW6sB\nAAAgWQzDAAAASFY53oGubqxduzbMW1pawjzrzul6EN3xmbU7RLdu3cL8iiuuCPMjjjgizHv27Bnm\nbJHTse3cuTPMs3aNGD16dJivWbOmLPVEP4933HFHuPaMM84I8wEDBoR51i4uWb2esugO81dffTVc\ny24SxXH3MI928JCk119/vZLlIHFRf7333ns1qKR2uDIMAACAZDEMAwAAIFkMwwAAAEgWwzAAAACS\nxTAMAACAZCW1m0SfPn3CvG/fvmFei90kRo0aFeZZtS9cuDDMu3TpkpcNHz683XWhcV1//fVhPnPm\nzCpXstfTTz+dl23fvj1ce8EFF4R51s/F0qVL219YYmbMmJGXZf1+QnG2bdsW5t/85jfD/Nprrw1z\ndupBMVasWBHmt956a8HHOOWUU8J80aJFYd6jR4+Cj11LXBkGAABAshiGAQAAkCyGYQAAACSLYRgA\nAADJYhgGAABAspLaTaJbt25hPm/evDBfsGBBmH/0ox/Ny8aMGVNULWeccUaYP/bYY2HeuXPnMN+4\ncWOYT58+vah6kIY1a9bkZQ888EC41t2LOnbWzg5ZPxuXXnppmPfv3z8vO+mkk8K1U6ZMCfOsn91i\nzyllu3fvrnUJDWv8+PFFrc/qfyDyxz/+MczPOeecMH/nnXcKPva0adPCvFevXgUfoyPiyjAAAACS\nxTAMAACAZDEMAwAAIFkMwwAAAEgWwzAAAACSdcDdJMxsrqRPSNrk7ifnst6SHpLULGmVpIvcfUvl\nyqysIUOGhPmgQYPCPNrZ4YYbbgjXfutb3wrz22+/veBj/zMf/OAHwzzrPe6RhnXr1oX54MGD87Kt\nW7eGa80szD/3uc+F+ezZs8N8xYoVRa2/+OKL87Lu3buHa4888sgw79Qp/nf+j3/84zC/8cYbwzza\n2aLRrF+/PsyzegilK+bufUk6++yzK1QJGtF9990X5tFuQlkuvPDCMD/zzDPbVVNHV8iV4XmSRu+X\n3ShpsbsPkLQ49zkAAABQVw44DLv7M5L2/2fseZLm5z6eL+n8MtcFAAAAVFx7XzPc19035D7eKKlv\n1kIzG2dmS8xsSVtbWzsfDqg+ehf1iL5FvaJ3USsl30Dne9/SKfNtndx9lru3untrU1NTqQ8HVA29\ni3pE36Je0buolfa+HfPbZtbP3TeYWT9Jm8pZVEfRpUuXgtcedthhRR17xowZYT5s2LAwz7qZCWnb\nvHlzmN95551hvmVL/n2uffvG/2HnmGOOCfMJEyaEedbNny0tLUXllfS3v/0tzL/97W+HedbPaSNZ\ntGhRmGf9XaFw27dvD/Nly5YVdZw+ffqUoxw0mGJ/n2XdWBz1V9ZN/o2qvVeGH5c0NvfxWEmPlacc\nAAAAoHoOOAyb2YOSnpd0gpmtNbMrJE2TdLaZvSFpZO5zAAAAoK4c8GUS7n5JxpdGlLkWAAAAoKp4\nBzoAAAAki2EYAAAAyWrvbhLYz8SJE8P8d7/7XZg/8sgjYb58+fIwP/nkk9tXGBrCrl27wnzy5Mlh\n/sADD4R5r1698rKnnnoqXHv88ceH+c6dO8O8nr311lu1LqFmXnnllYLX1mIHkHp20003hXnWW2AP\nGjQozLN2akEatm7dGubnnXdeWY5/66235mUnnnhiWY5dL7gyDAAAgGQxDAMAACBZDMMAAABIFsMw\nAAAAksUwDAAAgGSxm0SZZN3tO2vWrDBfvHhxmGfdHXr++eeH+emnnx7mF1xwQV5mZuFadHx//vOf\nwzxr14gsv/3tb/OygQMHFnWMbt26FbUejePUU0+tdQlVs2PHjjB/6aWXwjz6Xf/QQw8V9ZgzZswI\n865duxZ1HDSWZ599Nsx/85vfFHWcz3zmM2H++c9/vtiSGg5XhgEAAJAshmEAAAAki2EYAAAAyWIY\nBgAAQLIYhgEAAJAsdpOosN69e4f5U089FeajR48O83vuuaeofO7cuXnZmDFjwrU9e/YMc3QcV111\nVZi7e5hHu4lIxe8cUa/27NkT5p06xf/+z/p7xD/aunVrRY+/fv36MI+ez6effjpc+9Zbb4X5+++/\nH+bf/e53w3z37t1h3qNHjzAfNWpUXpa1C8TOnTvD/KSTTgpzpOHFF18M87FjxxZ1nE9+8pNhPnv2\n7DBntxKuDAMAACBhDMMAAABIFsMwAAAAksUwDAAAgGQxDAMAACBZB9xNwszmSvqEpE3ufnIuu1XS\nlZLacsumuvuTlSqyEQ0dOjTMly9fHubXXXddmP/sZz8L88svvzwv+9Of/hSuvf7668P80EMPDXNU\nztKlS8P8mWeeCXMzC/Os96BPRdauEVl/X62trZUsp0Pr3r17mEd/V5/61KfCtSeccEJZann++efD\nPNrt4+CD4//7ytod59RTTw3zyZMnh/mwYcPCvKWlJcyjXSb69+8frt2+fXuYNzU1hTkaS9auLKed\ndlpZjn/88ceHedZOKCjsyvA8SdF+X3e7e0vufwzCAAAAqDsHHIbd/RlJ71ShFgAAAKCqSnnN8NVm\n9r9mNtfMDstaZGbjzGyJmS1pa2vLWgZ0OPQu6hF9i3pF76JW2jsM3yvpWEktkjZIuitrobvPcvdW\nd2/l9VCoJ/Qu6hF9i3pF76JW2jUMu/vb7r7b3fdImi0pvhsMAAAA6MAOuJtExMz6ufuG3KcXSHql\nfCWlrV+/fmE+b968MB8/fnyYjxw5Mi+74447wrWvvfZamD/00ENhjsp57733wnzHjh1hfuSRR4b5\nueeeW7aaOoJdu3aF+YwZM4o6zqc//ekwnzp1atE1NYrbbrstzI877ri87Ne//nVFaxkwYECYf/az\nn83Lsu6YP+aYY8paU6GefDL/PvKNGzeGa0888cRKl4MO7K674v+YnrULTrGmTJlSluOkpJCt1R6U\nNFzS4Wa2VtItkoabWYskl7RK0hcrWCMAAABQEQccht39kiCeU4FaAAAAgKriHegAAACQLIZhAAAA\nJIthGAAAAMlq124SqL6uXbuG+fDhw8P8oIMOysuy7sh/9NFHwzxrl4kTTjghzFF9WX3Rs2fPKldS\nPlGf3nvvveHaG264Icybm5vD/Kabbgrzzp07F1ZcQsaOHVtQhr1+/vOfF7z28ssvr2Al6EjWrVuX\nly1YsKAsx/7CF74Q5uzRXDyuDAMAACBZDMMAAABIFsMwAAAAksUwDAAAgGRxA10Hs379+jBfuHBh\nmD///PNhnnWzXGTIkCFhPnDgwIKPgdq47LLLal1Cu0U3lkjSnXfemZd9//vfD9dm3UAye/bs9hcG\nVNiFF15Y6xJQJa2trXnZ5s2bizrGxz72sTCfOXNmu2pCPq4MAwAAIFkMwwAAAEgWwzAAAACSxTAM\nAACAZDEMAwAAIFnsJlFhbW1tYf69730vzH/0ox+F+dq1a0uuJXqLZin7rWvNrOTHRHHcvah83rx5\nYX7zzTeXq6SSPfjgg2F+9dVXh/mWLVvysmuuuSZce/fdd7e/MACosE2bNuVlnToVdx1yypQpYc7b\nyJcPV4YBAACQLIZhAAAAJIthGAAAAMliGAYAAECyGIYBAACQrAPuJmFm/SXdL6mvJJc0y92nm1lv\nSQ9Japa0StJF7p5/G3gD2rZtW172xBNPhGtvu+22MH/99dfLWtP+zjrrrLxs2rRp4dqPfOQjFa0F\nhcvawSMrz9plJKvvrrjiirzs0EMPDdcuX748zH/4wx+G+bPPPhvmq1atCvPjjjsuzC+++OK8LGs3\nCaAjy9oFZvXq1WF+7LHHVrIcVNDkyZPDfM+ePSUfe9CgQSUfA/9cIVeGd0ma5O4fknSapKvM7EOS\nbpS02N0HSFqc+xwAAACoGwccht19g7v/Pvfxu5JWSjpK0nmS5ueWzZd0fqWKBAAAACqhqNcMm1mz\npMGSXpDU19035L60UXtfRhF9zzgzW2JmS7LegALoiOhd1CP6FvWK3kWtFDwMm1lPSQ9Lmujuf933\na773hVHhi6PcfZa7t7p7a1NTU0nFAtVE76Ie0beoV/QuaqWgYdjMDtHeQfgn7r4wF79tZv1yX+8n\nKf89BwEAAIAOrJDdJEzSHEkr3f07+3zpcUljJU3L/flYRSqsgu3bt4f5mjVrwvzSSy/Ny5YuXVrW\nmvY3atSoMP/a174W5kOGDMnLsnYkQP3avXt3mGftJjFnzpy8rHfv3uHaZcuWtb+wfXz84x8P89Gj\nR4f5l7/85bI8LlBrWb9zy7HDAGpj3bp1Yb5gwYIw79Qp/5pjly5dwrW33HJLmPfo0aPA6tBeBxyG\nJZ0u6TJJy8zs5Vw2VXuH4J+a2RWSVku6qDIlAgAAAJVxwGHY3Z+TlHVJcUR5ywEAAACqh3egAwAA\nQLIYhgEAAJAshmEAAAAkq5Ab6OrO3//+9zCfOHFimD/33HNh/uqrr5atpv2dc845Yf7Vr341zFta\nWsL8kEMOKVtNqL0Pf/jDYT5y5Mgw/+Uvf1nU8deuXZuXZd0dneWII44I8wkTJoT5zTffXNTxgUb3\nq1/9KsxHjOA2nI5u27ZtYV7M79Hm5uYwnzJlSntKQhlwZRgAAADJYhgGAABAshiGAQAAkCyGYQAA\nACSLYRgAAADJqpvdJFatWpWXfeMb3wjXZt1hv3r16nKW9A+6d+8e5rfffnuYf+lLXwrzzp07l60m\n1J8PfOADYZ71vvf3339/mF9zzTUl1/L1r389zK+88sow79OnT8mPCTQSd691CQAKwJVhAAAAJIth\nGAAAAMliGAYAAECyGIYBAACQLIZhAAAAJKtudpN4+OGH87I5c+aU5dinnHJKmF9yySVhfvDB+X9t\n48aNC9d27dq1/YUBOT179gzzrF1JsnIAlTFmzJi87Ac/+EENKkElHXXUUWF+7rnnhvkTTzxRyXJQ\nJlwZBgAAQLIYhgEAAJAshmEAAAAki2EYAAAAyTrgDXRm1l/S/ZL6SnJJs9x9upndKulKSW25pVPd\n/clKFTpp0qSCMgAAqm3EiBF52Z49e2pQCSop62bmRx99tMqVoJwK2U1il6RJ7v57MztU0ktm9ovc\n1+529/+oXHkAAABA5RxwGHb3DZI25D5+18xWSor3FgEAAADqSFGvGTazZkmDJb2Qi642s/81s7lm\ndljG94wzsyVmtqStrS1aAnRI9C7qEX2LekXvolYKHobNrKekhyVNdPe/SrpX0rGSWrT3yvFd0fe5\n+yx3b3X31qampjKUDFQHvYt6RN+iXtG7qJWChmEzO0R7B+GfuPtCSXL3t919t7vvkTRb0tDKlQkA\nAACU3wGHYTMzSXMkrXT37+yT99tn2QWSXil/eQAAAEDlFLKbxOmSLpO0zMxezmVTJV1iZi3au93a\nKklfrEiFAAAAQIUUspvEc5Is+FLF9hQGAAAAqoF3oAMAAECyGIYBAACQLIZhAAAAJIthGAAAAMli\nGAYAAECyGIYBAACQLIZhAAAAJIthGAAAAMliGAYAAECyzN2r92BmbZJWV+0BK+9wSZtrXUSVdKRz\nPdrdm6r5gPRu3epI50nflkdHek4rqSOdJ71buo70fFZaRzrXgnq3qsNwozGzJe7eWus6qiGlc01B\nKs9nKueZklSe01TOMxUpPZ/1eK68TAIAAADJYhgGAABAshiGSzOr1gVUUUrnmoJUns9UzjMlqTyn\nqZxnKlJ6PuvuXHnNMAAAAJLFlWEAAAAki2EYAAAAyWIYLpCZzTWzTWb2yj5ZbzP7hZm9kfvzsFrW\nWA5m1t/M/sfMVpjZcjO7Npc33Lmmgt5tvHNNAX3beOeaCnq3/s6VYbhw8ySN3i+7UdJidx8gaXHu\n83q3S9Ikd/+QpNMkXWVmH1Jjnmsq5onebbRzTcE80beNdq6pmCd6t67OlWG4QO7+jKR39ovPkzQ/\n9/F8SedXtagKcPcN7v773MfvSlop6Sg14Lmmgt5tvHNNAX3beOeaCnq3/s6VYbg0fd19Q+7jjZL6\n1rKYcjOzZkmDJb2gBj/XBDX080nvNqyGfi7p24bW0M9nvfcuw3CZ+N496hpmnzoz6ynpYUkT3f2v\n+36t0c41dY32fNK7aWi055K+TUejPZ+N0LsMw6V528z6SVLuz001rqcszOwQ7W3sn7j7wlzckOea\nsIZ8PundhteQzyV9m4SGfD4bpXcZhkvzuKSxuY/HSnqshrWUhZmZpDmSVrr7d/b5UsOda+Ia7vmk\nd5PQcM8lfZuMhns+G6l3eQe6ApnZg5KGSzpc0tuSbpH0qKSfSvpXSaslXeTu+79ovq6Y2RmSnpW0\nTNKeXDxVe18H1FDnmgp6l96tR/QtfVuv6N36612GYQAAACSLl0kAAAAgWQzDAAAASBbDMAAAAJLF\nMAwAAIBkMQwDAAAgWQzDAAAASBbDMAAAAJL1f1iM7ql26a6bAAAAAElFTkSuQmCC\n",
- "text/plain": [
- "<matplotlib.figure.Figure at 0x11234e9e8>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "f, (ax1, ax2, ax3, ax4) = plt.subplots(1,4, sharex='col', sharey='row',figsize=(12,3))\n",
- "ax1.imshow(np.reshape(image[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
- "ax2.imshow(np.reshape(image[1,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
- "ax3.imshow(np.reshape(image[2,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
- "ax4.imshow(np.reshape(image[3,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can optionally save the parameters in the directory variable 'model_prefix'. We first create data iterators for MXNet, with each batch of data containing 100 images."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "model_prefix = None\n",
- "\n",
- "batch_size = 100\n",
- "nd_iter = mx.io.NDArrayIter(data={'data':image},label={'loss_label':label},\n",
- " batch_size = batch_size)\n",
- "nd_iter_test = mx.io.NDArrayIter(data={'data':image_test},label={'loss_label':label_test},\n",
- " batch_size = batch_size)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 2. Building the Network Architecture\n",
- "\n",
- "### 2.1 Gaussian MLP as encoder\n",
- "Next we constuct the neural network, as in the [paper](https://arxiv.org/abs/1312.6114/), we use *Multilayer Perceptron (MLP)* for both the encoder and decoder. For encoder, a Gaussian MLP is used as follows:\n",
- "\n",
- "\\begin{align}\n",
- "\\log q_{\\phi}(z|x) &= \\log \\mathcal{N}(z:\\mu,\\sigma^2I) \\\\\n",
- "\\textit{ where } \\mu &= W_2h+b_2, \\log \\sigma^2 = W_3h+b_3\\\\\n",
- "h &= \\tanh(W_1x+b_1)\n",
- "\\end{align}\n",
- "\n",
- "where $\\{W_1,W_2,W_3,b_1,b_2,b_3\\}$ are the weights and biases of the MLP.\n",
- "Note below that `encoder_mu`(`mu`) and `encoder_logvar`(`logvar`) are symbols. So, we can use `get_internals()` to get the values of them, after which we can sample the latent variable $z$.\n",
- "\n",
- "\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "## define data and loss labels as symbols \n",
- "data = mx.sym.var('data')\n",
- "loss_label = mx.sym.var('loss_label')\n",
- "\n",
- "## define fully connected and activation layers for the encoder, where we used tanh activation function.\n",
- "encoder_h = mx.sym.FullyConnected(data=data, name=\"encoder_h\",num_hidden=400)\n",
- "act_h = mx.sym.Activation(data=encoder_h, act_type=\"tanh\",name=\"activation_h\")\n",
- "\n",
- "## define mu and log variance which are the fully connected layers of the previous activation layer\n",
- "mu = mx.sym.FullyConnected(data=act_h, name=\"mu\",num_hidden = 5)\n",
- "logvar = mx.sym.FullyConnected(data=act_h, name=\"logvar\",num_hidden = 5)\n",
- "\n",
- "## sample the latent variables z according to Normal(mu,var)\n",
- "z = mu + np.multiply(mx.symbol.exp(0.5 * logvar), \n",
- " mx.symbol.random_normal(loc=0, scale=1, shape=np.shape(logvar.get_internals()[\"logvar_output\"])))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 2.2 Bernoulli MLP as decoder\n",
- "\n",
- "In this case let $p_\\theta(x|z)$ be a multivariate Bernoulli whose probabilities are computed from $z$ with a feed forward neural network with a single hidden layer:\n",
- "\n",
- "\\begin{align}\n",
- "\\log p(x|z) &= \\sum_{i=1}^D x_i\\log y_i + (1-x_i)\\log (1-y_i) \\\\\n",
- "\\textit{ where } y &= f_\\sigma(W_5\\tanh (W_4z+b_4)+b_5)\n",
- "\\end{align}\n",
- "\n",
- "where $f_\\sigma(\\dot)$ is the elementwise sigmoid activation function, $\\{W_4,W_5,b_4,b_5\\}$ are the weights and biases of the decoder MLP. A Bernouilli likelihood is suitable for this type of data but you can easily extend it to other likelihood types by parsing into the argument `likelihood` in the `VAE` class, see section 4 for details."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "# define fully connected and tanh activation layers for the decoder\n",
- "decoder_z = mx.sym.FullyConnected(data=z, name=\"decoder_z\",num_hidden=400)\n",
- "act_z = mx.sym.Activation(data=decoder_z, act_type=\"tanh\",name=\"activation_z\")\n",
- "\n",
- "# define the output layer with sigmoid activation function, where the dimension is equal to the input dimension\n",
- "decoder_x = mx.sym.FullyConnected(data=act_z, name=\"decoder_x\",num_hidden=features)\n",
- "y = mx.sym.Activation(data=decoder_x, act_type=\"sigmoid\",name='activation_x')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 2.3 Joint Loss Function for the Encoder and the Decoder\n",
- "\n",
- "The variational lower bound also called evidence lower bound (ELBO) can be estimated as:\n",
- "\n",
- "\\begin{align}\n",
- "\\mathcal{L}(\\theta,\\phi;x_{(i)}) \\approx \\frac{1}{2}\\left(1+\\log ((\\sigma_j^{(i)})^2)-(\\mu_j^{(i)})^2-(\\sigma_j^{(i)})^2\\right) + \\log p_\\theta(x^{(i)}|z^{(i)})\n",
- "\\end{align}\n",
- "\n",
- "where the first term is the KL divergence of the approximate posterior from the prior, and the second term is an expected negative reconstruction error. We would like to maximize this lower bound, so we can define the loss to be $-\\mathcal{L}$(minus ELBO) for MXNet to minimize."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "# define the objective loss function that needs to be minimized\n",
- "KL = 0.5*mx.symbol.sum(1+logvar-pow( mu,2)-mx.symbol.exp(logvar),axis=1)\n",
- "loss = -mx.symbol.sum(mx.symbol.broadcast_mul(loss_label,mx.symbol.log(y)) \n",
- " + mx.symbol.broadcast_mul(1-loss_label,mx.symbol.log(1-y)),axis=1)-KL\n",
- "output = mx.symbol.MakeLoss(sum(loss),name='loss')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 3. Training the model\n",
- "\n",
- "Now, we can define the model and train it. First we will initilize the weights and the biases to be Gaussian(0,0.01), and then use stochastic gradient descent for optimization. To warm start the training, one may also initilize with pre-trainined parameters `arg_params` using `init=mx.initializer.Load(arg_params)`. \n",
- "\n",
- "To save intermediate results, we can optionally use `epoch_end_callback = mx.callback.do_checkpoint(model_prefix, 1)` which saves the parameters to the path given by model_prefix, and with period every $1$ epoch. To assess the performance, we output $-\\mathcal{L}$(minus ELBO) after each epoch, with the command `eval_metric = 'Loss'` which is defined above. We will also plot the training loss for mini batches by accessing the log and saving it to a list, and then parsing it to the argument `batch_end_callback`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "# set up the log\n",
- "nd_iter.reset()\n",
- "logging.getLogger().setLevel(logging.DEBUG) \n",
- "\n",
- "# define function to trave back training loss\n",
- "def log_to_list(period, lst):\n",
- " def _callback(param):\n",
- " \"\"\"The checkpoint function.\"\"\"\n",
- " if param.nbatch % period == 0:\n",
- " name, value = param.eval_metric.get()\n",
- " lst.append(value)\n",
- " return _callback\n",
- "\n",
- "# define the model\n",
- "model = mx.mod.Module(\n",
- " symbol = output ,\n",
- " data_names=['data'],\n",
- " label_names = ['loss_label'])"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {
- "collapsed": false
- },
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "INFO:root:Epoch[0] Train-loss=375.023381\n",
- "INFO:root:Epoch[0] Time cost=6.127\n",
- "INFO:root:Epoch[1] Train-loss=212.780315\n",
- "INFO:root:Epoch[1] Time cost=6.409\n",
- "INFO:root:Epoch[2] Train-loss=208.209400\n",
- "INFO:root:Epoch[2] Time cost=6.619\n",
- "INFO:root:Epoch[3] Train-loss=206.146854\n",
- "INFO:root:Epoch[3] Time cost=6.648\n",
- "INFO:root:Epoch[4] Train-loss=204.530598\n",
- "INFO:root:Epoch[4] Time cost=7.000\n",
- "INFO:root:Epoch[5] Train-loss=202.799992\n",
- "INFO:root:Epoch[5] Time cost=6.778\n",
- "INFO:root:Epoch[6] Train-loss=200.333474\n",
- "INFO:root:Epoch[6] Time cost=7.187\n",
- "INFO:root:Epoch[7] Train-loss=197.506393\n",
- "INFO:root:Epoch[7] Time cost=6.712\n",
- "INFO:root:Epoch[8] Train-loss=195.969775\n",
- "INFO:root:Epoch[8] Time cost=6.896\n",
- "INFO:root:Epoch[9] Train-loss=195.418288\n",
- "INFO:root:Epoch[9] Time cost=6.887\n",
- "INFO:root:Epoch[10] Train-loss=194.739763\n",
- "INFO:root:Epoch[10] Time cost=6.745\n",
- "INFO:root:Epoch[11] Train-loss=194.380536\n",
- "INFO:root:Epoch[11] Time cost=6.706\n",
- "INFO:root:Epoch[12] Train-loss=193.955462\n",
- "INFO:root:Epoch[12] Time cost=6.592\n",
- "INFO:root:Epoch[13] Train-loss=193.493671\n",
- "INFO:root:Epoch[13] Time cost=6.775\n",
- "INFO:root:Epoch[14] Train-loss=192.958739\n",
- "INFO:root:Epoch[14] Time cost=6.600\n",
- "INFO:root:Epoch[15] Train-loss=191.928542\n",
- "INFO:root:Epoch[15] Time cost=6.586\n",
- "INFO:root:Epoch[16] Train-loss=189.797939\n",
- "INFO:root:Epoch[16] Time cost=6.700\n",
- "INFO:root:Epoch[17] Train-loss=186.672446\n",
- "INFO:root:Epoch[17] Time cost=6.869\n",
- "INFO:root:Epoch[18] Train-loss=184.616599\n",
- "INFO:root:Epoch[18] Time cost=7.144\n",
- "INFO:root:Epoch[19] Train-loss=183.305978\n",
- "INFO:root:Epoch[19] Time cost=6.997\n",
- "INFO:root:Epoch[20] Train-loss=181.944634\n",
- "INFO:root:Epoch[20] Time cost=6.481\n",
- "INFO:root:Epoch[21] Train-loss=181.005329\n",
- "INFO:root:Epoch[21] Time cost=6.754\n",
- "INFO:root:Epoch[22] Train-loss=178.363118\n",
- "INFO:root:Epoch[22] Time cost=7.000\n",
- "INFO:root:Epoch[23] Train-loss=176.363421\n",
- "INFO:root:Epoch[23] Time cost=6.923\n",
- "INFO:root:Epoch[24] Train-loss=174.573954\n",
- "INFO:root:Epoch[24] Time cost=6.510\n",
- "INFO:root:Epoch[25] Train-loss=173.245940\n",
- "INFO:root:Epoch[25] Time cost=6.926\n",
- "INFO:root:Epoch[26] Train-loss=172.082522\n",
- "INFO:root:Epoch[26] Time cost=6.733\n",
- "INFO:root:Epoch[27] Train-loss=171.123084\n",
- "INFO:root:Epoch[27] Time cost=6.616\n",
- "INFO:root:Epoch[28] Train-loss=170.239300\n",
- "INFO:root:Epoch[28] Time cost=7.004\n",
- "INFO:root:Epoch[29] Train-loss=169.538416\n",
- "INFO:root:Epoch[29] Time cost=6.341\n",
- "INFO:root:Epoch[30] Train-loss=168.952901\n",
- "INFO:root:Epoch[30] Time cost=6.736\n",
- "INFO:root:Epoch[31] Train-loss=168.169076\n",
- "INFO:root:Epoch[31] Time cost=6.616\n",
- "INFO:root:Epoch[32] Train-loss=167.208973\n",
- "INFO:root:Epoch[32] Time cost=6.446\n",
- "INFO:root:Epoch[33] Train-loss=165.732213\n",
- "INFO:root:Epoch[33] Time cost=6.405\n",
- "INFO:root:Epoch[34] Train-loss=163.606801\n",
- "INFO:root:Epoch[34] Time cost=6.139\n",
- "INFO:root:Epoch[35] Train-loss=161.985880\n",
- "INFO:root:Epoch[35] Time cost=6.678\n",
- "INFO:root:Epoch[36] Train-loss=160.763072\n",
- "INFO:root:Epoch[36] Time cost=8.749\n",
- "INFO:root:Epoch[37] Train-loss=160.025193\n",
- "INFO:root:Epoch[37] Time cost=6.519\n",
- "INFO:root:Epoch[38] Train-loss=159.319723\n",
- "INFO:root:Epoch[38] Time cost=7.584\n",
- "INFO:root:Epoch[39] Train-loss=158.670701\n",
- "INFO:root:Epoch[39] Time cost=6.874\n",
- "INFO:root:Epoch[40] Train-loss=158.225733\n",
- "INFO:root:Epoch[40] Time cost=6.402\n",
- "INFO:root:Epoch[41] Train-loss=157.741337\n",
- "INFO:root:Epoch[41] Time cost=8.617\n",
- "INFO:root:Epoch[42] Train-loss=157.301411\n",
- "INFO:root:Epoch[42] Time cost=6.515\n",
- "INFO:root:Epoch[43] Train-loss=156.765170\n",
- "INFO:root:Epoch[43] Time cost=6.447\n",
- "INFO:root:Epoch[44] Train-loss=156.389668\n",
- "INFO:root:Epoch[44] Time cost=6.130\n",
- "INFO:root:Epoch[45] Train-loss=155.815434\n",
- "INFO:root:Epoch[45] Time cost=6.155\n",
- "INFO:root:Epoch[46] Train-loss=155.432254\n",
- "INFO:root:Epoch[46] Time cost=6.158\n",
- "INFO:root:Epoch[47] Train-loss=155.114027\n",
- "INFO:root:Epoch[47] Time cost=6.749\n",
- "INFO:root:Epoch[48] Train-loss=154.612441\n",
- "INFO:root:Epoch[48] Time cost=6.255\n",
- "INFO:root:Epoch[49] Train-loss=154.137659\n",
- "INFO:root:Epoch[49] Time cost=7.813\n",
- "INFO:root:Epoch[50] Train-loss=153.634072\n",
- "INFO:root:Epoch[50] Time cost=7.408\n",
- "INFO:root:Epoch[51] Train-loss=153.417397\n",
- "INFO:root:Epoch[51] Time cost=7.747\n",
- "INFO:root:Epoch[52] Train-loss=152.851887\n",
- "INFO:root:Epoch[52] Time cost=8.587\n",
- "INFO:root:Epoch[53] Train-loss=152.575068\n",
- "INFO:root:Epoch[53] Time cost=7.554\n",
- "INFO:root:Epoch[54] Train-loss=152.084419\n",
- "INFO:root:Epoch[54] Time cost=6.628\n",
- "INFO:root:Epoch[55] Train-loss=151.724836\n",
- "INFO:root:Epoch[55] Time cost=6.535\n",
- "INFO:root:Epoch[56] Train-loss=151.302525\n",
- "INFO:root:Epoch[56] Time cost=7.148\n",
- "INFO:root:Epoch[57] Train-loss=150.960916\n",
- "INFO:root:Epoch[57] Time cost=7.195\n",
- "INFO:root:Epoch[58] Train-loss=150.603895\n",
- "INFO:root:Epoch[58] Time cost=6.649\n",
- "INFO:root:Epoch[59] Train-loss=150.237795\n",
- "INFO:root:Epoch[59] Time cost=6.222\n",
- "INFO:root:Epoch[60] Train-loss=149.936080\n",
- "INFO:root:Epoch[60] Time cost=8.450\n",
- "INFO:root:Epoch[61] Train-loss=149.514617\n",
- "INFO:root:Epoch[61] Time cost=6.113\n",
- "INFO:root:Epoch[62] Train-loss=149.229345\n",
- "INFO:root:Epoch[62] Time cost=6.088\n",
- "INFO:root:Epoch[63] Train-loss=148.893769\n",
- "INFO:root:Epoch[63] Time cost=6.558\n",
- "INFO:root:Epoch[64] Train-loss=148.526837\n",
- "INFO:root:Epoch[64] Time cost=7.590\n",
- "INFO:root:Epoch[65] Train-loss=148.249951\n",
- "INFO:root:Epoch[65] Time cost=6.180\n",
- "INFO:root:Epoch[66] Train-loss=147.940414\n",
- "INFO:root:Epoch[66] Time cost=6.242\n",
- "INFO:root:Epoch[67] Train-loss=147.621304\n",
- "INFO:root:Epoch[67] Time cost=8.501\n",
- "INFO:root:Epoch[68] Train-loss=147.294314\n",
- "INFO:root:Epoch[68] Time cost=7.645\n",
- "INFO:root:Epoch[69] Train-loss=147.074479\n",
- "INFO:root:Epoch[69] Time cost=7.092\n",
- "INFO:root:Epoch[70] Train-loss=146.796387\n",
- "INFO:root:Epoch[70] Time cost=6.914\n",
- "INFO:root:Epoch[71] Train-loss=146.508842\n",
- "INFO:root:Epoch[71] Time cost=6.606\n",
- "INFO:root:Epoch[72] Train-loss=146.230444\n",
- "INFO:root:Epoch[72] Time cost=7.755\n",
- "INFO:root:Epoch[73] Train-loss=145.970296\n",
- "INFO:root:Epoch[73] Time cost=6.409\n",
- "INFO:root:Epoch[74] Train-loss=145.711610\n",
- "INFO:root:Epoch[74] Time cost=6.334\n",
- "INFO:root:Epoch[75] Train-loss=145.460053\n",
- "INFO:root:Epoch[75] Time cost=7.269\n",
- "INFO:root:Epoch[76] Train-loss=145.156451\n",
- "INFO:root:Epoch[76] Time cost=6.744\n",
- "INFO:root:Epoch[77] Train-loss=144.957674\n",
- "INFO:root:Epoch[77] Time cost=7.100\n",
- "INFO:root:Epoch[78] Train-loss=144.729749\n",
- "INFO:root:Epoch[78] Time cost=6.242\n",
- "INFO:root:Epoch[79] Train-loss=144.481728\n",
- "INFO:root:Epoch[79] Time cost=6.865\n",
- "INFO:root:Epoch[80] Train-loss=144.236061\n",
- "INFO:root:Epoch[80] Time cost=6.632\n",
- "INFO:root:Epoch[81] Train-loss=144.030473\n",
- "INFO:root:Epoch[81] Time cost=6.764\n",
- "INFO:root:Epoch[82] Train-loss=143.776374\n",
- "INFO:root:Epoch[82] Time cost=6.564\n",
- "INFO:root:Epoch[83] Train-loss=143.538847\n",
- "INFO:root:Epoch[83] Time cost=6.181\n",
- "INFO:root:Epoch[84] Train-loss=143.326444\n",
- "INFO:root:Epoch[84] Time cost=6.220\n",
- "INFO:root:Epoch[85] Train-loss=143.078987\n",
- "INFO:root:Epoch[85] Time cost=6.823\n",
- "INFO:root:Epoch[86] Train-loss=142.877117\n",
- "INFO:root:Epoch[86] Time cost=7.755\n",
- "INFO:root:Epoch[87] Train-loss=142.667316\n",
- "INFO:root:Epoch[87] Time cost=6.068\n",
- "INFO:root:Epoch[88] Train-loss=142.461755\n",
- "INFO:root:Epoch[88] Time cost=6.111\n",
- "INFO:root:Epoch[89] Train-loss=142.270438\n",
- "INFO:root:Epoch[89] Time cost=6.221\n",
- "INFO:root:Epoch[90] Train-loss=142.047086\n",
- "INFO:root:Epoch[90] Time cost=8.061\n",
- "INFO:root:Epoch[91] Train-loss=141.855774\n",
- "INFO:root:Epoch[91] Time cost=6.433\n",
- "INFO:root:Epoch[92] Train-loss=141.688955\n",
- "INFO:root:Epoch[92] Time cost=7.153\n",
- "INFO:root:Epoch[93] Train-loss=141.442910\n",
- "INFO:root:Epoch[93] Time cost=7.113\n",
- "INFO:root:Epoch[94] Train-loss=141.279274\n",
- "INFO:root:Epoch[94] Time cost=7.152\n",
- "INFO:root:Epoch[95] Train-loss=141.086522\n",
- "INFO:root:Epoch[95] Time cost=6.472\n",
- "INFO:root:Epoch[96] Train-loss=140.901925\n",
- "INFO:root:Epoch[96] Time cost=6.767\n",
- "INFO:root:Epoch[97] Train-loss=140.722496\n",
- "INFO:root:Epoch[97] Time cost=7.044\n",
- "INFO:root:Epoch[98] Train-loss=140.579295\n",
- "INFO:root:Epoch[98] Time cost=7.040\n",
- "INFO:root:Epoch[99] Train-loss=140.386067\n",
- "INFO:root:Epoch[99] Time cost=6.669\n"
- ]
- }
- ],
- "source": [
- "# training the model, save training loss as a list.\n",
- "training_loss=list()\n",
- "\n",
- "# initilize the parameters for training using Normal.\n",
- "init = mx.init.Normal(0.01)\n",
- "model.fit(nd_iter, # train data\n",
- " initializer=init,\n",
- " # if eval_data is supplied, test loss will also be reported\n",
- " # eval_data = nd_iter_test,\n",
- " optimizer='sgd', # use SGD to train\n",
- " optimizer_params={'learning_rate':1e-3,'wd':1e-2}, \n",
- " # save parameters for each epoch if model_prefix is supplied\n",
- " epoch_end_callback = None if model_prefix==None else mx.callback.do_checkpoint(model_prefix, 1),\n",
- " batch_end_callback = log_to_list(N/batch_size,training_loss), \n",
- " num_epoch=100,\n",
- " eval_metric = 'Loss')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 23,
- "metadata": {
- "collapsed": false
- },
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEWCAYAAACnlKo3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcZGV97/HPt6v3bfZ9BmaEEdlRRwSjUSO5EGIC90oM\nMYkYt/iSxJhrouISlStJNIkmvhKNxusVIooaF9C4gUaMMYADso6AA8MwM8zSs/Z0Ty+1/O4f53RT\n9PSpGmqmunq6v+/Xq15z6jmnqp7ndM3zq2c5z1FEYGZmVoumRmfAzMyOXw4iZmZWMwcRMzOrmYOI\nmZnVzEHEzMxq5iBiZmY1cxCxY0rSP0t677E+diaQ1CHpG5IOSPpyAz7/XZI+fbTHSnqJpK3HNneZ\n+XhM0gVT8VlWm+ZGZ8CmD0mPAa+PiFtqfY+IeFM9jp0hLgOWAAsiojDVHx4Rf1mPY58OSauBTUBL\nI86BHXtuidgRkzSrfnTUobwnAg/XUnnOtnNvxw8HEQNA0r8CJwDfkDQg6e2SVksKSa+T9Djwg/TY\nL0vakXbL/EjS6WXv81lJH0y3XyJpq6S3SdolabukP6jx2AVpV1C/pJ9K+qCkH1cozwsl/UTSfklb\nJL0mTf+hpNeXHfea8vdJy3ulpF8Av5D0CUl/O+G9b5T0v9Pt5ZK+IqlP0iZJb8nIzweAvwB+Oz2/\nr5PUJOk9kjanZb5O0pz0+EnP/YT3HDtnby87Z5dKuljSw5L2SnpX2fHvl/S5Ce9/haTHJe2W9O7J\njq1wjt+Vvu4xSb9blv7rkn6W/q22SHp/2ct+lP67Pz0P56eveYOkn0s6KGmDpOeUveYcSfem37cv\nSmov+6yXS7o7/Tv/RNJZZfveIWlb+p4PSXpZpfJYjSLCDz+ICIDHgAvKnq8GArgO6AI60vTXAj1A\nG/D3wN1lr/ks8MF0+yVAAbgaaAEuBg4B82o49ob00QmcBmwBfpxRjhOBg8DvpO+1ADgn3fdDki67\nsWNfU/4+aXlvBuYDHcAvp5+ldP88YAhYTvIj7E6S4NAKPAN4FLgwI1/vBz5X9vy1wMb0dd3AV4F/\nrXTuJ7zf2Dn7i7ScbwD6gM+nf5/T07yumfj5Ze//L2k5zwZGgFMny2vG534k/Q68GBgETinbf2Z6\nfs4CdgKXTvjc5rL3+y1gG/A8QMDJwIll38k70vM9H/g58KZ037OBXcDzgRxwRXp8G3BK+ndbXva5\nJzX6/9hMfLglYkfi/RExGBFDABHxmYg4GBEjJJXN2WO/oCeRB66OiHxEfAsYIPkPfsTHSsoBrwDe\nFxGHImIDcG2F/L4KuCUivpC+156IuPtplPevImJvWt7/JKn0XpTuuwz474h4gqTSWxQRV0fEaEQ8\nSlIpX36En/O7wEci4tGIGACuAi6f0HX1lHM/iTxwTUTkSYLsQuAf0r/PA8AGkgCR5QMRMRQR9wD3\nVDl2ovdGxEhE3Ar8O/BKgIj4YUTcFxGliLgX+AJJoMnyeuDDEfHTSGyMiM1l+z8WEU9ExF7gG8A5\nafobgU9GxO0RUYyIa0kC4XlAkSSYnCapJSIei4hHnkbZ7Ag5iNiR2DK2ISkn6a8lPSKpn+SXHySV\n12T2xFPHAA6R/Op+OscuIpkEsqVsX/n2RKuAo6kwxt87IoKkcv6dNOlVwPXp9onA8rQrZb+k/cC7\nSAbPj8RyoLyy3ExSzvLXVyonJOesmG6PBZqdZfuHyD7fADvKtiv9bSbaFxGDZc83k5QHSc+X9B9p\nF98B4E1kfz+g+t8rK48nAm+bcP5XkbQ+NgJvJfmRs0vSDZKWH2HZ7GlwELFyWUs6l6e/CrgEuACY\nQ9JNAEk3RL30kXSfrCxLW1Xh+C3ASRn7Bkm6xMYsneSYiefhC8Blkk4k6Tr5StnnbIqIuWWPnoi4\nuELeyj1BUhGOOYGknOVBYLousz1PUlfZ8xNIygNJd9pNwKqImAP8M09+PyYrT6W/VyVbSFph5ee/\nMyK+ABARn4+IF5Kc4wA+VMNnWBUOIlZuJ0n/fCU9JF0Ge0gq47pMBS2X/tL+KvB+SZ2SngW8usJL\nrgcukPRKSc3poPxYF8jdwP9K3+dk4HVH8Pk/A3YDnwa+GxH70113AAfTAdyOtJV2hqTnHWHRvgD8\nqaQ1krpJzuUX4/iZ+voBSa2SXgS8HBi79qUH2BsRw5LOJfnhMaYPKPHU79mngT+T9FwlTk4DdjX/\nArwpbflIUlc6qN8j6RRJvyKpDRgmaZGVjrK8NgkHESv3V8B70q6BP8s45jqSrottJP3tt01R3v6I\npOWzA/hXkgp4ZLIDI+JxkoH5twF7SQLHWF//R4FRkoB5LU92TVXzeZLW1+fLPqdIUnmeQ3Ltw1ig\nyRofmugzaVl+lL5+GPjjI3xto+0A9pG0Pq4nGex+MN33ZuBqSQdJBv2/NPaiiDgEXAP8V/o9Oy8i\nvpymfZ5kQsTXSQbRK4qI9SSTCf4xzctGkokSkIyH/DXJ32QHsJhkzMmOsbEZJ2bHFUkfApZGxBWN\nzovZbOaWiB0XJD1L0llpt8W5JN1QX2t0vsxmO18Fa8eLHpIurOUkXVF/B9zY0ByZmbuzzMysdu7O\nMjOzms347qyFCxfG6tWrG50NM7Pjyp133rk7IhZVO27GB5HVq1ezfv36RmfDzOy4Imlz9aPcnWVm\nZkfBQcTMzGrmIGJmZjVzEDEzs5o5iJiZWc0cRMzMrGYOImZmVrMZf52Imdl0FxEM5YsMjBTIF4NS\nKShNWJKqFFAsxfijFEEEyb/pewznS/QNjNB3cIQ9AyP8+YWnINXzfnEOImZmNRkaLbJp9yAb+wZ4\nYv8QAnJNoiXXRE97M3M7W+hsbWbf4Cg7+4fZdXCEwZECh0aLHMoX2TswOl7hHxzOUzrGyxi25pp4\n80tPprutvtW8g4iZHbcigpFCiUOjxfEKenC0wKGRIvliiXyxRKEU5IslRgslRoslhkaTX/wDwwX6\nh/PsP5TnwFCeUgQdrc10tuTIF0vs6B9mZ/8w/UMFJGhKf9EXI2kpFJ5mrZ9rEl2tOTpbm+lszTG/\nq5W1i7t5wUkLmNPRQldbM12tOVqbm5BETmKsEREBTU2Qa2qiuUk0pflpSo+RQIi25iYW9bSxqKeN\nOR0tdW+FgIOImTVARDA4WmTPwAgDIwVyTUmlOVossat/hO0Hkgp8d/pL/cBQnuacaM01kWsSuwdG\n2dU/TN/ACPlibT/hO1py9HY0M7ejlTkdLTQ3NXFgKM+OA0PkmppY2tvGWSvnMKejlSDpOooImtK8\ndrbmWLOwm5MWd7FyXiciCTCjhRIHhwscGMozOFJgbmcLS3rbmd/ZSlNT/Sv1qeYgYmZV5Ysltuw9\nxKHRIqVI+uQPjRY5MJSnfyhPrkn0tLfQ097MwEiB7fuH2H5geDwY7CrrsilFMDRaZKRQ/Zbn87ta\nWdjdytyOVobzJfqHChRKwcLuVk5atJBFPW30tDeP/8Lvamumsy1HZ0uOtpYczU0i1yRam5tozTXR\n2txER2uOrtZmcnWs0Bd2t9XtvacbBxGzWSZfTLp/BkYK7DgwzI4Dw/QdHGak8GSXz6HRIodGC/QP\nFXikb4BH+wYZLVav9Mu1NjextLedpb3tnL68l96OFnJKumLaWnIs6GplQXcb3W258UHj5iaxuLed\npXPaWdzTRkvOE0inOwcRs+NERFAKGBgpsHdwlL2DoxwYGqV/KOnb3z0wyo4DSQtg98Ao/Wkr4VC+\niEj60IOo2v0jQVfab9/d1syJCzp58SmLeObiHrrbm5NA0JQc09vRQm9HC6VScHC4wMHhPJ2tzSyb\n286CrtYp6ZO3xnIQMWuAQ6MFdvaPsKt/mEP5IqOFEiOFEvsGR9k9MMLugWRc4In9Q2zfP8zgaKHq\n7B0JFve0sbS3nRVz2zl1WQ9zOlrobM0l/fnpcZ0tOTrSALFkTjvL5rSzqLuNjtYcLblk4NaVvx0p\nBxGzGo0WSmw/METfwRGG8kWGRosM5Yv0DxfoH8ozMFJI+/6LDI4U6Ts4wq6Dw+zqH+HgSCHzfZsE\n87vaWDqnjdULunjBSQvpbmseH9Dtaktm9szvamVuZzIo3NvenAwOu/vHppiDiM1qhWKJx/ceon+4\nMH4B19iA8dig8Vg3zYGhZDro/qHRNCCMEBVaBy050d6So70lR2drjkXdbZyytIcXrV3Ekt52lvS2\nsbinnc62HG3NTbQ1NzG3s5V5na11HfQ1O5YcRGzGyhdLbN5ziI27DvKLnQM80jfAUL4IJPPut+4b\nYmPfAKNVZgm15ERve9L3P7ezhUXdbTxraS8r5nawYl4HS3rb6WpNgkVHa46e9mZ621tob8lNRTHN\nGspBxGaEgZEC927dzz1bDnDftv08vHOAzXsGnzKIvGJuBz3tyVc+ApbOaeeFaxeydnE3C3va0ou3\nkusH5nS0JN1EHS20pRd/mdnhHETsuFQqBRu29/PDh3bxw4f6uOvxfeMDzyfM7+SUpT386mlLOHlR\nN2uXdHPy4m46W/11NzvW/L/Kjhtb9x3iBw/u4icb93D7pj3sO5QH4MwVc3jzS05m3ep5nL1yLvO6\nWhucU7PZw0HEpp3hfJHNew6ND25v3DXAd+7fzj1bDwBJt9TLTl3C+c9YwIueuZDFPe0NzrHZ7OUg\nYg1RLAV3b9nPY7sHKaaL2W0/MMTtm/Zy95b9hw12n7VyDu+46FlcdMZS1izsalCuzWwiBxGrq4jg\nwFCeXQdHxpfYWL95Lz94cBe7B0afcmyuSZyxvJcrzj+RM1fOZX5nK70dzel0WLc2zKYjBxE7pvYf\nGuXf79vON+/ZzqO7B9gzMHrYktk97c289JTFXHDaEs5aMYeW5uQq6Z72Zg9+mx1n/D/Wjtqh0QI3\nb9jJN+7Zzq0P7yJfDE5a1MWLn7mIhd1tLOhO7m8wthjfsrntXljPbIZwELGnpVgKHukb4OGdB9m4\na4Cfb+/nRw/vZihfZElvG1ecv5pLn72C05f3+toKs1nAQcQqigju3XqA7z+4i7s27+PuLfsZSNd9\nkpJrMl7x3BX8xlnLed7q+TPypjtmls1BxIAkWPxi1wC3b9rL4EiB4XyRXQdH+MHPd7Gjf5gmwanL\nern02cs5Z9U8Tl3Ww0mLur20h9ks5yAyi/UdHOG+bfu5/dG9fG/DTjbtHnzK/u62Zl5w0gL+/PRT\neNmpi5nb6Yv4zOypHERmmYPDeT5566N85a6tbD8wDEBzkzj/pAW87oVrePEzF7Ggu5X25py7psys\nKgeRWWK0UOKGnz7OP9zyC/YMjnLBqUt43QvXcNbKuZy+vJeuNn8VzOzpc80xQ5VKwc6Dw6x/bB83\nb9jJfzy4i4MjBZ6/Zj6fufhUzl41t9FZNLMZwEHkOBAR4/fV3j3w5O1TB0cKjOST26oeHM6z91Ce\nfYOjbD8wxJZ9Q+NLhyzoauXXzlzKb5y9nBeevNBTb83smHEQaaBiKXhszyAbnujnkb4BettbWDan\nnYU9bTy88yA/3bSXux7fz44Dw4wWs2+c1KRkEHxBdxvzOltYu7iHC05dwsr5nZy6tIdnnzDPd8oz\ns7pwEJliEcFPH9vHl9dv4dv37xi/5mIyi3raWHfiPC4+cxnzu1qY19nKwu625NHTSk97csOk5ia5\ndWFmDeEgMkW27R/ia3dt5St3bWPT7kG625q5+MylnLtmAacu6+Hkxd0MjhR5Yv8QfQMjPGNhFyfM\n73RwMLNpzUGkjvYOjnLzhh18457t/Ncju4mAc1fP58qXnszFZy49bLHBtuYc831DJTM7jjQkiEj6\nG+A3gFHgEeAPImJ/uu8q4HVAEXhLRHw3TX8u8FmgA/gW8CcREYe/e2Pt6h/mext28p37d/Dfj+6h\nWApWze/gLb+yllc8ZyUnLOhsdBbNzI6ZRrVEbgauioiCpA8BVwHvkHQacDlwOrAcuEXSMyOiCHwC\neANwO0kQuQj4dkNyP8Hjew7x3Qd28N0HdnDn4/uIgDULu3jTi5/Br52xzIsRmtmM1ZAgEhHfK3t6\nG3BZun0JcENEjACbJG0EzpX0GNAbEbcBSLoOuJQGBpEtew/x9Z9t49/v286DOw4CcNqyXv70gmdy\n0RlLWbu424HDzGa86TAm8lrgi+n2CpKgMmZrmpZPtyemT0rSG4E3ApxwwgnHMq/8ZONuPnLzw6zf\nvA+A562ex3t+/VQuPH0pq+a7q8rMZpe6BRFJtwBLJ9n17oi4MT3m3UABuP5YfnZEfAr4FMC6deuO\n2bjJjgPD/OHn7mRORwt/fuEpXHLOclbOc+Aws9mrbkEkIi6otF/Sa4CXAy8rGyDfBqwqO2xlmrYt\n3Z6YPmUignd+9V7yxRKfe93zWb2wayo/3sxsWmrIPUolXQS8HfjNiDhUtusm4HJJbZLWAGuBOyJi\nO9Av6TwlAw2vBm6cyjz/251b+eFDfbzjomc5gJiZpRo1JvKPQBtwczr4fFtEvCkiHpD0JWADSTfX\nlenMLIA38+QU328zhYPqOw4Mc/U3N3Du6vlccf7qqfpYM7Npr1Gzs06usO8a4JpJ0tcDZ9QzX1k+\n9J0HyRdLfPiys3yPDTOzMg3pzjrebN4zyLoT57sby8xsAgeRI1AoBc05t0DMzCZyEDkC+WLQ3ORT\nZWY2kWvGI1AolmhxS8TM7DAOIkcg6c7yqTIzm8g14xHIF0u0eFaWmdlhHESOQKHogXUzs8k4iByB\nQqnk7iwzs0m4ZjwC+WK4O8vMbBIOIkegUHRLxMxsMq4Zj0DeFxuamU3KQeQIFIolWnyxoZnZYVwz\nVlEqBaXALREzs0k4iFSRL5UAaPGYiJnZYVwzVlEoJjddbPbsLDOzwziIVDEeRNwSMTM7jGvGKsa6\ns9wSMTM7nINIFU+2RBxEzMwmchCpIl9MB9Y9xdfM7DCuGasoltwSMTPL4iBSRWFsTMQD62Zmh3HN\nWEU+HRPxAoxmZodzEKnCU3zNzLK5ZqxifIqvx0TMzA7jIFJFYbw7y6fKzGwi14xVFIpuiZiZZXEQ\nqSKfTvFtcRAxMzuMg0gV4y0Rd2eZmR3GNWMVeS97YmaWyUGkioLvJ2Jmlsk1YxW+n4iZWTYHkSrG\nF2B0S8TM7DCuGasoeAFGM7NMDiJVeHaWmVk214xVjC/A6JaImdlhHESq8FLwZmbZXDNWkffsLDOz\nTA4iVXiKr5lZNgeRKsa6s3IOImZmh2lIEJH0fyTdK+luSd+TtLxs31WSNkp6SNKFZenPlXRfuu9j\nkqakVi+UgpacmKKPMzM7rjSqJfI3EXFWRJwDfBP4CwBJpwGXA6cDFwEfl5RLX/MJ4A3A2vRx0VRk\ntFAseXqvmVmGhtSOEdFf9rQLiHT7EuCGiBiJiE3ARuBcScuA3oi4LSICuA64dCrymi+GLzQ0M8vQ\n3KgPlnQN8GrgAPDSNHkFcFvZYVvTtHy6PTE9673fCLwR4IQTTjiqfBZKJS95YmaWoW61o6RbJN0/\nyeMSgIh4d0SsAq4H/uhYfnZEfCoi1kXEukWLFh3VexWK4ZlZZmYZ6tYSiYgLjvDQ64FvAe8DtgGr\nyvatTNO2pdsT0+suXwy3RMzMMjRqdtbasqeXAA+m2zcBl0tqk7SGZAD9jojYDvRLOi+dlfVq4Map\nyGuhVPKYiJlZhkaNify1pFOAErAZeBNARDwg6UvABqAAXBkRxfQ1bwY+C3QA304fdefuLDOzbA0J\nIhHxigr7rgGumSR9PXBGPfM1mXzRA+tmZllcO1ZRKHmKr5lZFgeRKvK+2NDMLFPV2lHSGZKuk7Q+\nfVwr6aypyNx0UCiG7yViZpahYhBJr+n4GvBD4LXp41bgK2PXe8x0hZJbImZmWaoNrF8N/GpEPFaW\ndq+kH5BMsZ2SabaNlC8G7S1uiZiZTabaT+zmCQEEgDStpR4Zmm687ImZWbZqtWNB0mGLT0k6keQ6\njhnP14mYmWWr1p31PuAWSX8J3JmmrQPeCbyjnhmbLvJFX7FuZpalYhCJiK9L2gS8DfjjNHkD8MqI\nuKfemZsOCqXwwLqZWYaqV6ynweLVU5CXaang+4mYmWWqNsV3oaT3SXqLpG5Jn0iXc79R0slTlclG\nKpRKtLglYmY2qWq14+eBNtLVdIFNwGUkt7T9dH2zNj24JWJmlq1ad9aSiHhXuvz65oj4cJr+oKQr\n65y3acELMJqZZatWOxYB0vua756wr1SXHE0zycC6WyJmZpOp1hJ5hqSbAJVtkz5fU9ecTRNJd5Zb\nImZmk6kWRMrXx/rbCfsmPp+R8qWSF2A0M8tQ7TqRW7P2SfoiyWKMM1axFETg60TMzDIcTe14/jHL\nxTSVLybDPp6dZWY2Of/ErqBQCgB3Z5mZZajYnSXpOVm7mAWr+BbGWiLuzjIzm1S1gfW/q7DvwWOZ\nkekoX3RLxMyskmoD6y+dqoxMR4XS2JiIWyJmZpOptnbW28u2f2vCvr+sV6ami0LaEvHFhmZmk6v2\nE/vysu2rJuy76BjnZdoZm53lZU/MzCZXrXZUxvZkz2ecsdlZnuJrZja5akEkMrYnez7j5D07y8ys\nomqzs86W1E/S6uhIt0mft9c1Z9NAwbOzzMwqqjY7KzdVGZmOxmZn5TywbmY2KffTVPDkdSI+TWZm\nk3HtWEGx5Cm+ZmaVOIhU8OQCjD5NZmaTce1YgQfWzcwqcxCpYHzZE0/xNTOblGvHCrwAo5lZZQ4i\nFXgBRjOzylw7VpD3AoxmZhU5iFRQ8HUiZmYVuXas4MnuLLdEzMwm4yBSwfjAumdnmZlNqqG1o6S3\nSQpJC8vSrpK0UdJDki4sS3+upPvSfR+TVPfmwfg91t0SMTObVMOCiKRVwP8AHi9LO43kRlink9z0\n6uOSxhaB/ATwBmBt+qj7TbF8PxEzs8oa2RL5KPB2nnpfkkuAGyJiJCI2ARuBcyUtA3oj4raICOA6\n4NJ6Z3D8zobuzjIzm1RDakdJlwDbIuKeCbtWAFvKnm9N01ak2xPTs97/jZLWS1rf19dXcz4LxaBJ\n0OQpvmZmk6p2U6qaSboFWDrJrncD7yLpyqqLiPgU8CmAdevW1XwHxnyp5AsNzcwqqFsQiYgLJkuX\ndCawBrgnHRtfCdwl6VxgG7Cq7PCVadq2dHtiel0VikGLWyFmZpmm/Gd2RNwXEYsjYnVErCbpmnpO\nROwAbgIul9QmaQ3JAPodEbEd6Jd0Xjor69XAjfXOa6HoloiZWSV1a4nUIiIekPQlYANQAK6MiGK6\n+83AZ4EO4Nvpo67ypfCSJ2ZmFTQ8iKStkfLn1wDXTHLceuCMKcoWMNYScRAxM8vivpoKCqXwvUTM\nzCpwDVlBoRi+l4iZWQUOIhUUPMXXzKwi15AV5IseWDczq8RBpIJCseR7iZiZVeAasoJCKTw7y8ys\nAgeRCvLFkhdfNDOrwDVkBYWiWyJmZpU4iFSQL4VnZ5mZVeAasoJCseQFGM3MKnAQqcDdWWZmlTmI\nVOD7iZiZVeYasgLfT8TMrDIHkQp8PxEzs8pcQ1aQL3kBRjOzShxEKigUS14K3sysAteQFXh2lplZ\nZQ4iFeRLXoDRzKwS15AVFIpBzrOzzMwyOYhkiAgKJU/xNTOrxEEkQ7EUAJ7ia2ZWgWvIDIXxIOKW\niJlZFgeRDPliCcD3EzEzq8A1ZIZC0S0RM7NqHEQy5EtJS8RjImZm2VxDZhhriXh2lplZNgeRDE92\nZ/kUmZllcQ2ZYaw7ywswmpllcxDJMN4S8ewsM7NMriEzjE3x9ewsM7NsDiIZxi42dHeWmVk2B5EM\nhbGWiLuzzMwyuYbMkPfFhmZmVTmIZCiMz87yKTIzy+IaMsOTs7PcEjEzy+IgkmF8AUa3RMzMMrmG\nzOCl4M3MqnMQyTB+nYi7s8zMMjmIZPAV62Zm1TWkhpT0fknbJN2dPi4u23eVpI2SHpJ0YVn6cyXd\nl+77mKS6NhGK7s4yM6uqkT+zPxoR56SPbwFIOg24HDgduAj4uKRcevwngDcAa9PHRfXMXN5TfM3M\nqppuNeQlwA0RMRIRm4CNwLmSlgG9EXFbRARwHXBpPTPiKb5mZtU1Moj8saR7JX1G0rw0bQWwpeyY\nrWnainR7YvqkJL1R0npJ6/v6+mrK3JMLME63OGtmNn3UrYaUdIuk+yd5XELSNfUM4BxgO/B3x/Kz\nI+JTEbEuItYtWrSopvfwAoxmZtU11+uNI+KCIzlO0r8A30yfbgNWle1emaZtS7cnpteNF2A0M6uu\nUbOzlpU9/Z/A/en2TcDlktokrSEZQL8jIrYD/ZLOS2dlvRq4sZ55HFuA0S0RM7NsdWuJVPFhSecA\nATwG/CFARDwg6UvABqAAXBkRxfQ1bwY+C3QA304fdVMolcg1iTrPJDYzO641JIhExO9X2HcNcM0k\n6euBM+qZr3KFYnhmlplZFe7wz5Avhq8RMTOrwrVkhkKp5KvVzcyqcBDJkC+GZ2aZmVXhWjJDoVjy\nzCwzsyocRDIUSuHuLDOzKhxEMuSLJVrcnWVmVpFryQyFolsiZmbVOIhkKJRKHlg3M6vCtWSGvFsi\nZmZVOYhkSFoiDiJmZpU4iGRIxkR8eszMKnEtmaFQCl8nYmZWhYNIhkLRA+tmZtW4lsyQLMDoloiZ\nWSUOIhk8xdfMrLpG3ZRq2nvR2kUsm9Pe6GyYmU1rDiIZ3vvy0xqdBTOzac/9NWZmVjMHETMzq5mD\niJmZ1cxBxMzMauYgYmZmNXMQMTOzmjmImJlZzRxEzMysZoqIRuehriT1AZtrfPlCYPcxzM7xYDaW\nGWZnuWdjmWF2lruWMp8YEYuqHTTjg8jRkLQ+ItY1Oh9TaTaWGWZnuWdjmWF2lrueZXZ3lpmZ1cxB\nxMzMauYgUtmnGp2BBpiNZYbZWe7ZWGaYneWuW5k9JmJmZjVzS8TMzGrmIGJmZjVzEJmEpIskPSRp\no6R3Njo/9SJplaT/kLRB0gOS/iRNny/pZkm/SP+d1+i8HmuScpJ+Jumb6fPZUOa5kv5N0oOSfi7p\n/Jlebkl/mn6375f0BUntM7HMkj4jaZek+8vSMssp6aq0fntI0oVH89kOIhNIygH/BPwacBrwO5Jm\n6m0OC8CgrOS/AAAEb0lEQVTbIuI04DzgyrSs7wS+HxFrge+nz2eaPwF+XvZ8NpT5H4DvRMSzgLNJ\nyj9jyy1pBfAWYF1EnAHkgMuZmWX+LHDRhLRJy5n+H78cOD19zcfTeq8mDiKHOxfYGBGPRsQocANw\nSYPzVBcRsT0i7kq3D5JUKitIyntteti1wKWNyWF9SFoJ/Drw6bLkmV7mOcAvA/8XICJGI2I/M7zc\nJLcA75DUDHQCTzADyxwRPwL2TkjOKuclwA0RMRIRm4CNJPVeTRxEDrcC2FL2fGuaNqNJWg08G7gd\nWBIR29NdO4AlDcpWvfw98HagVJY208u8BugD/l/ajfdpSV3M4HJHxDbgb4HHge3AgYj4HjO4zBNk\nlfOY1nEOIoakbuArwFsjor98XyRzwGfMPHBJLwd2RcSdWcfMtDKnmoHnAJ+IiGcDg0zoxplp5U7H\nAC4hCaDLgS5Jv1d+zEwrc5Z6ltNB5HDbgFVlz1emaTOSpBaSAHJ9RHw1Td4paVm6fxmwq1H5q4Nf\nAn5T0mMkXZW/IulzzOwyQ/Jrc2tE3J4+/zeSoDKTy30BsCki+iIiD3wVeAEzu8zlssp5TOs4B5HD\n/RRYK2mNpFaSAaibGpynupAkkj7yn0fER8p23QRckW5fAdw41Xmrl4i4KiJWRsRqkr/tDyLi95jB\nZQaIiB3AFkmnpEkvAzYws8v9OHCepM70u/4yknG/mVzmclnlvAm4XFKbpDXAWuCOWj/EV6xPQtLF\nJP3mOeAzEXFNg7NUF5JeCPwncB9Pjg+8i2Rc5EvACSTL6L8yIiYO2h33JL0E+LOIeLmkBczwMks6\nh2QyQSvwKPAHJD8kZ2y5JX0A+G2SmYg/A14PdDPDyizpC8BLSJZ83wm8D/g6GeWU9G7gtSTn5a0R\n8e2aP9tBxMzMauXuLDMzq5mDiJmZ1cxBxMzMauYgYmZmNXMQMTOzmjmImE1Tkl4ytsqw2XTlIGJm\nZjVzEDE7SpJ+T9Idku6W9Mn0XiUDkj6a3svi+5IWpceeI+k2SfdK+trYPR4knSzpFkn3SLpL0knp\n23eX3QPk+vTKa7Npw0HE7ChIOpXkiuhfiohzgCLwu0AXsD4iTgduJbmCGOA64B0RcRbJSgFj6dcD\n/xQRZ5Os7zS2+uqzgbeS3NvmGSRrf5lNG82NzoDZce5lwHOBn6aNhA6She5KwBfTYz4HfDW9p8fc\niLg1Tb8W+LKkHmBFRHwNICKGAdL3uyMitqbP7wZWAz+uf7HMjoyDiNnREXBtRFz1lETpvROOq3V9\noZGy7SL+P2vTjLuzzI7O94HLJC2G8ftan0jyf+uy9JhXAT+OiAPAPkkvStN/H7g1vavkVkmXpu/R\nJqlzSkthViP/qjE7ChGxQdJ7gO9JagLywJUkN306N923i2TcBJIluf85DRJjK+lCElA+Kenq9D1+\nawqLYVYzr+JrVgeSBiKiu9H5MKs3d2eZmVnN3BIxM7OauSViZmY1cxAxM7OaOYiYmVnNHETMzKxm\nDiJmZlaz/w/UuBmpwDVyRQAAAABJRU5ErkJggg==\n",
- "text/plain": [
- "<matplotlib.figure.Figure at 0x145e16898>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "ELBO = [-training_loss[i] for i in range(len(training_loss))]\n",
- "plt.plot(ELBO)\n",
- "plt.ylabel('ELBO');plt.xlabel('epoch');plt.title(\"training curve for mini batches\")\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "As expected, the ELBO is monotonically increasing over epoch, and we reproduced the results given in the paper [Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114/). Now we can extract/load the parameters and then feed the network forward to calculate $y$ which is the reconstructed image, and we can also calculate the ELBO for the test set. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 80,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "arg_params = model.get_params()[0]\n",
- "\n",
- "# if saved the parameters, can load them using `load_checkpoint` method at e.g. 100th epoch\n",
- "# sym, arg_params, aux_params = mx.model.load_checkpoint(model_prefix, 100)\n",
- "# assert sym.tojson() == output.tojson()\n",
- "\n",
- "e = y.bind(mx.cpu(), {'data': nd_iter_test.data[0][1],\n",
- " 'encoder_h_weight': arg_params['encoder_h_weight'],\n",
- " 'encoder_h_bias': arg_params['encoder_h_bias'],\n",
- " 'mu_weight': arg_params['mu_weight'],\n",
- " 'mu_bias': arg_params['mu_bias'],\n",
- " 'logvar_weight':arg_params['logvar_weight'],\n",
- " 'logvar_bias':arg_params['logvar_bias'],\n",
- " 'decoder_z_weight':arg_params['decoder_z_weight'],\n",
- " 'decoder_z_bias':arg_params['decoder_z_bias'],\n",
- " 'decoder_x_weight':arg_params['decoder_x_weight'],\n",
- " 'decoder_x_bias':arg_params['decoder_x_bias'], \n",
- " 'loss_label':label})\n",
- "\n",
- "x_fit = e.forward()\n",
- "x_construction = x_fit[0].asnumpy()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 78,
- "metadata": {
- "collapsed": false,
- "scrolled": true
- },
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAADSCAYAAACvmc1VAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuUXGWZ7/HfkyvkSkKHJARICDAkxMkBaUm4CZrAwQku\nRAdnogzxNjhneRhdOjOiZ80ZdIYj5yxvzNHlDB4QFEcFuSsjIANyRxIMQhJCQi6E0Em6SUI693R4\nzx+1w5Sp56F7d1d1d9X+ftbKSvev3q7au/dT1W9X7+fdllISAAAAUEQD+noDAAAAgL7CZBgAAACF\nxWQYAAAAhcVkGAAAAIXFZBgAAACFxWQYAAAAhcVkuB8ws/9nZl/u6+0AesLMpphZMrNBwe1LzOzc\nXt4soFPULuoRdVs9TIYlmdn2sn9vmtmuss8/WuvHTyl9KqX0v2r9OKgfZrbGzOb29XZUU0ppRkrp\n4b7eDtQWtYt6RN0Wm/vbRNGklEYc+NjM1kj6VErp19F4MxuUUurojW0DaoU6Rr2idlGPqNv+i3eG\nu8DM/snMfmZmPzGzdkmXmtnNZnZV2Zi52UT6wOdHmdkdZtZqZqvN7DNvc/9v3deB+zGzL2Vf+5qZ\nvd/MLjSzFWa22cz+ruxrTzezp8xsq5m1mNk/m9ngstvfZ2YvmdkbZvZ/zexxM/tY2e2fMrMXzWyL\nmf27mR1dpW8baiSrhcXZMX/CzGaW3Xalmb1sZu1mttTMLi677WPZ8f+Wmb0u6aose8zMvp7VwGoz\ne1/Z14w2s+uz2lqfPRcGZrcNzL6uzcxWSZrXyXa/9c6LmV1lZrdmtd9uZs+b2R9ldb/JzNaZ2fll\nX/txM1uWjV1lZp8+6L7/LtvG17KaTmZ2fHbb0Gw7XzGzjWb2L2Z2aM+OArqD2qV26xF12/h1y2S4\n6y6W9G+SRkv62dsNNLMBkn4h6RlJkySdJ+lvzWxOFx/rKJWOzZGS/lHS9ZL+XNIpks6V9FUzOyYb\n2yHps5KaJJ0p6QJJn8624whJt0j62+z21ZJOK9vOD2W3XSRpnKSns31EP2Vmp0i6QaVjfLikf5V0\nt5kNzYa8LOlsler0K5JuNrOJZXcxS9IqSeMlXV2WLVepRv6PpOvNzLLbblSpxo5Xqf7Ol/Sp7La/\nlHRhljdL+tOcu/N+ST+SNEbS7yTdp1LdT5L01WzfDtiUPdYoSR+X9C0ze2f2PblA0uclzc2289yD\nHucaSX8k6eTs9kmS/mfObUUPUbvUbj2ibgtStykl/pX9k7RG0tyDsn+S9B8HZTdLuqrs87mS1mQf\nnylp1UHj/17S94PHfOu+svvZLmlg9vkYSUnSqWXjn5N0YXBffyPp1uzjT0h6tOw2k9Qi6WPZ5w9I\nWlB2+yBJeyRN6uvjUPR/Xh1m+fck/eNB2XJJ5wT3s1jSRdnHH5P0ykG3f0zSyrLPh2X1NkGlF+89\nkg4tu32+pIeyj/9D0l+V3XZ+9rWDOtsnSVdJeqDstvcfVPcjs/s6LLivOyV9Nvv4BklfK7vt+Oxr\nj89qfoek48puP13S6r4+xo36j9qlduvxH3Vb7LrlnOGuW5dj7GRJx5jZ1rJsoKSHu/j1bSml/dnH\nu7L/N5bdvkvSCEkys2mSviHpVJWeVINUeodXKr2z/NZ2p5SSmb160HZ+18yuLcveVOmd6fVd3Fb0\nrsmSFpjZFWXZEJWOtczsMpV+Y5+S3TZCpXcfDvDqeMOBD1JKO7M3KEZIGitpsKSW/3zTQgPK7uMP\n6kvS2pz7cnBNe3U/QtLW7M+I/6DSuw0DVKr158u2Y2HZfZVv07hs7KKyfTCVno/oXdQutVuPqNsC\n1C2T4a5LB32+Q6UDfsCEso/XSVqRUppe860q/VnjKUl/llLabmZ/o9KfNqTSu8Dl5wCZSn+uKN/O\nv08pve1pH+hX1km6OqV09cE3mNlkSd+XNEfSkyml/Wa2WKUXogMOruPOHmuPpKbkN320SCo/x/wY\nZ0yPZX+OvE3SZZLuSintM7M79Z/71aLSL3AHlG9Tm0ov8jNSSvyC17eoXWq3HlG3BahbzhnuvsWS\n5pnZmOz8oL8uu+1JSXvN7Atmdkh20vsfm9mpNdiOkZLekLTDzKYrO1848wtJ77RSA94glc4tHld2\n+79I+h/Z18nMDjOzvOcgoXYGZ/Vz4N8glV54/8rMZlnJcDObZ2YjJQ1X6YW3VSo1QEh6R3cfPKXU\nIul+Sd8ws1FmNsDMjjOzc7Iht0j6ays1i46RdGUP9vXtDJE0VKX96sjesTi/7PZbJH3czKab2TCV\nTkk6sA9vqvQ9+1Z2Dr3MbJKZ/dcabStKqN0Sare+ULclhatbJsPdd6OkZSr9meJXkn564IbsN7o/\nUalZbY1Kvyn9q0onolfbFyQtkNSePcZb7/KmlDZK+jNJ35T0uqTjVDppfk92+63Zbbea2TZJv5fU\nrwu2YO5V6TfsA/+uSiktVKmJ4juStkhaqdI5aEopLVXplJknVfpz2B9LeryH23CZSi+MS7PH+7mk\nA80h31epAeM5Sc9Kur2Hj+VKKbWr9MvmLdk2fETS3WW3/7ukf5b0kErfj6eym/Zk/3/xQJ7V+a8l\nnViLbcVbqF1Ru3WIulUx69ayk5tRAFZanuU1SX+aUnq0r7cHqIXsLx0vSBoa/KkR6JeoXdSjRqhb\n3hlucGZ2QXb6w1CV/pSxT9Jv+3izgKoys4uttLblGEn/W9I99fqijGKhdlGPGq1umQw3vrNUWuOw\nVaVTIC5OKe15+y8B6s6nVVoX82VJ+yX9t77dHKDLqF3Uo4aqW06TAAAAQGHxzjAAAAAKq0eT4ex8\n1OVmttLMarXEBwAAAFAT3T5NIluZ4CVJ50l6VdIzkuZnS424mpqa0pQpU7r1eMABixYtakspjet8\nZPVQu+ipNWvWqK2tzTofWT3ULaqB11zUq67Wbk+uQHeaStfXXiVJZvZTSReptDaea8qUKVq4cGF0\nM9AlZpb3EpQ9Ru2ip5qbm3v9MalbVAOvuahXXa3dnpwmMUl/eD3qV/WHl/oFAAAA+rWaN9CZ2eVm\nttDMFra2ttb64YCqoXZRj6hb1CtqF32lJ5Ph9ZKOLvv8qCz7Ayml61JKzSml5nHjevWUI6BHqF3U\nI+oW9YraRV/pyTnDz0g6wcyOVWkS/OcqXb8aAAAAVdAX14Mw69Ve3z7X7clwSqnDzP67pPskDZR0\nQ0ppSdW2DAAAAKixnrwzrJTSvZLurdK2AAAAAL2KK9ABAACgsJgMAwAAoLCYDAMAAKCwenTOMAAA\nQNFFKz54+ZtvvumO7ejocPPdu3e7+bZt29x88+bNbv7GG2+4+cCBAyuyCRMmuGPHjh3r5sOGDXPz\nwYMHu/mAAf57sX21igXvDAMAAKCwmAwDAACgsJgMAwAAoLCYDAMAAKCwmAwDAACgsFhNAgAAoEy0\nOkQkWiFi//79Fdm+ffvcsVu2bHHzVatWufmDDz7o5kuWLMl1/+PHj6/IZs6c6Y6dO3eum0+dOtXN\nq7HKRG+sMME7wwAAACgsJsMAAAAoLCbDAAAAKCwmwwAAACgsGugAAHUnb4NTXn11WVj0b3nrzhsf\nXXY5urzyokWL3Pzpp5928xUrVrh51MzmXXo5uuzykCFD3LyWou95NZ+jvDMMAACAwmIyDAAAgMJi\nMgwAAIDCYjIMAACAwmIyDAAAgMLq0WoSZrZGUruk/ZI6UkrN1dgoAMVV61UCelsRViWoRof92+Xe\nJW2jy99GvMu8vl0eHbe8216Nx8ybo+eq9b3Ncz/t7e1u/tprr7l5W1ubm0erRkybNs3NZ82aVZGd\neuqp7thx48a5ebTKRFTrefRGnVdjabX3pJT8IwIAAAD0Y5wmAQAAgMLq6WQ4Sfq1mS0ys8ursUEA\nAABAb+npZPislNLJkt4n6TNm9u6DB5jZ5Wa20MwWtra29vDhgN5D7aIeUbeoV9Qu+kqPJsMppfXZ\n/5sk3SHpNGfMdSml5pRSc3TiNdAfUbuoR9Qt6hW1i77S7QY6MxsuaUBKqT37+HxJX63algENIk+X\ned5O9aijPk+nfX/rVM/zuNXa9lrua72ujlGNuo3qsKOjw82jbvrNmzdXZNu3b3fH7tu3z82jYxx1\n3kf5oEH+j00vP/TQQ92xUR515Odd2YJVJmon77Hw7Nq1y81feuklN3/llVfcPKrRGTNmuPmcOXPc\nvLm5ciGwsWPHumOHDx/u5tHzIu/KKX2lJ6tJjJd0R7ZDgyT9W0rpV1XZKgAAAKAXdHsynFJaJem/\nVHFbAAAAgF7F0moAAAAoLCbDAAAAKCwmwwAAACisalyOud956qmn3Pzaa69180mTJrl51PG7YMGC\niizqvIxy1K9qrPiwd+9ed+y2bdvcPFpzc+vWrW4eddR79u/f7+YDBw5087wrB+S9f68refDgwe7Y\nqLN5woQJbj5mzBg3j+4/6oRuJHnqOe+x37Nnj5u3tLS4+SOPPOLmzz//fEX2+uuvu2Oj59bQoUPd\nfPz48W4eLe01atQoN/dq68QTT3THHnPMMW5e7x35qOStnLJ27Vp37GOPPebmmzZtcvPode7MM890\n89mzZ7t5U1NTRZb3NTHvyj79rXYb/5UeAAAACDAZBgAAQGExGQYAAEBhMRkGAABAYTEZBgAAQGE1\n5GoS3moPkrRixYqq3P/VV19dkY0ePdodG3Vv1oMpU6a4+Ze+9CU3jzqkG03UUe91DUt+R/2rr77q\njn344Yfd/Omnn3bzDRs2uPmWLVvcfOfOnRVZtBJA1DUcrQ7h3ffb3f+QIUPc3OvWj1YC8LqgJens\ns89280svvdTNo5UDvI7n/tYFXStenUfHMqr9aMWHX/ziF25+//33u/mqVasqssMOO8wdG71uRcc4\nqqFI9HPEq4vo+3X44Ye7+YgRI3JtC3pf3hVV2traKrI777zTHfvcc8+5ebSy1fTp0938tNNOc/Po\nORC9vnry7n80vr/hnWEAAAAUFpNhAAAAFBaTYQAAABQWk2EAAAAUFpNhAAAAFFZDriYRdWouXrzY\nzWfMmOHmS5YscXOvs/+uu+5yx953331ufuyxx7r56tWr3Twv7xr3EydOdMeuW7cu131H3dpf/OIX\nc91PvYq6Y6NVFnbs2FGRrVy50h3729/+1s2ja9ZHq0Z4K1hIfl14mRRfmz7v/kf3f8ghh7i5t0/R\nfkai1Q3mzp3r5lF3/8CBA3M9bn8WHbc83d55V1JpaWlx86VLl7r5+vXr3fzII4+syM455xx3bN5O\n+qhTv7W11c1/+ctfurn38yLq0m9ubnbzeum8L7Jo1YRt27a5+e23316RRfMC72eFJE2bNs3N58yZ\n4+be80WK6zFaOcgTvc5HeXTfUe6tyhI9L6q5sg/vDAMAAKCwmAwDAACgsJgMAwAAoLCYDAMAAKCw\nOp0Mm9kNZrbJzF4oy8aa2QNmtiL7f0xtNxMAAACovq6sJnGjpO9I+mFZdqWkB1NK15jZldnn/WYp\ngeh63VEemTlzppvPnz+/IrvmmmvcsWvWrHHzaDWJVatWdW3jOjFkyJCKLFpNItqWqJs66mxtdJ11\nekedrV7X7OjRo92xRxxxhJtHtbt9+3Y3j+7f6zKeMGGCOzZaYSFaZWLnzp25xkfdxIsWLarI7rnn\nHnfs5s2bc21L3q7kanQrV7Pjub+KOuzzHocTTzzRzWfPnl2RXXDBBe7YqG6jVSPyruAQrZzx4osv\nVmQjRoxwx0arBrCaRP8RHYvdu3e7+RNPPOHmt956a0W2YcMGd2w055g3b56bH3/88W4+bNgwN8+z\ngkP0nI7qf9euXW4erRoUvS56qw/1xmtop+8Mp5QekXTwT5yLJN2UfXyTpA9UebsAAACAmuvuOcPj\nU0oHFpDcIGl8lbYHAAAA6DU9bqBLpb8lhH/bMbPLzWyhmS2M/uwO9EfULuoRdYt6Re2ir3R3MrzR\nzCZKUvb/pmhgSum6lFJzSqk5ugIQ0B9Ru6hH1C3qFbWLvtLdyzHfLWmBpGuy//1rERdIdJJ43maz\nvE1+eXiXkZaktrY2N581a5abn3/++VXbpnpy4CT+PI1ykt/MMHXqVHfsBz7gn35/xhlnuHnU5BE1\nS3rNcsOHD3fHRo1vkaixIhI1/3n5vffe646NtnHSpEluPnbsWDfPcznSelWNJpTo+xRdtjo6Pk1N\nTW4e1e2ZZ55ZkUXNplGjXLSN+/btc/OoPqNLRntNUVu3bnXH5tUbl6Mtquh7G9XF6tWr3fynP/2p\nm3tN9FHtfuhDH3Jzr/4ladSoUW4ePU+jffUupRw1xG3a5L/3Gc0joktARz8DvX3ymuqkfA2BnenK\n0mo/kfSkpBPN7FUz+6RKk+DzzGyFpLnZ5wAAAEBd6fSd4ZRS5TpiJXOqvC0AAABAr2r8vw0CAAAA\nASbDAAAAKCwmwwAAACis7q4mgX7Ou9znxRdf7I6NLrv47W9/282jbu2iiDpVo25179LY0SVjR44c\n6ebRZTcj0THyuvur1ZEb1VHeS/W+8sorFVnU2Rx1U0fd19H3vZpdyY0sb+1H9Tx+vH+dpqjz3Luf\n6DEjXse8JG3bts3Nly5d6ubLly938z179lRk0eol0epD6H3RCgvt7e1u/qtf/crNf/e737m595p7\n7rnnumPnzPFbsaLXuUh0yei9e/e6uff6unbtWnfsM8884+be67YUrygTfQ9OP/30iizaf+9na3fx\nzjAAAAAKi8kwAAAACovJMAAAAAqLyTAAAAAKi8kwAAAACovVJBrUjTfeWJFt2LDBHRt12E+ePLma\nm9Tw8nTaR6sXRNdgz9t9HnXa13J1hOi+Ozo63HzlypVu/vjjj1dkXqe+5HceS9LZZ5/t5tFqBdHx\nKLI8tRJ9/6LVJI444gg3jzr4t2zZUpFFxzLqMI9qKKrDhQsXunlra6ube9+v0aNHu2Or1QUfrYTA\nKiiVou9VtMJCtDrEww8/7ObR6jhTp06tyObNm+eOHTNmjJtHK6Fs3brVzTdv3uzm0fOrpaWlIluy\nZIk79qWXXnLzaPWJaDWh6Pl45JFHVmTTpk1zx7KaBAAAAFAFTIYBAABQWEyGAQAAUFhMhgEAAFBY\nTIYBAABQWKwmUedefvllN//85z/f5ft48skn3XzChAnd2qZGV41O7aj7Pup4jlaHqEY3eXQfeUVd\nw95KAJJ02223ufnq1asrsqgr/4ILLnDzo446ys2j1TqKLKoVry6isXlXk2hqanJzr6tdkp577rmK\nbN26de7Y6BhHdbh8+XI3f+GFF9w86tT3RN+vaHWA6LkYPbei14Wi876P0fd806ZNbn7rrbe6eVQX\n0TGaMWNGRRa9Pu3bt8/No5qLanfjxo1uHm2j972Jvl/bt2/P9ZjRqhGjRo1yc29ViuOOO84dW83V\nVHhnGAAAAIXFZBgAAACFxWQYAAAAhcVkGAAAAIXV6WTYzG4ws01m9kJZdpWZrTezxdm/P6ntZgIA\nAADV15XW6hslfUfSDw/Kv5VS+nrVtwi53HPPPW7udaVecskl7ljv2umonjydrXm7YPOsBFCt+4jy\nvXv3uvkjjzzi5vfff7+bd3R0VGSzZs1yx77nPe9x82HDhrl5tOoBKnl1kXe1k+g4RCvVrFy50s1f\nffXVisxbdUSKj/HOnTvdfMOGDW6+detWN4868r189+7d7tjouRJ18EcrZFTjed6IvO9LtKrBQw89\n5OZPPPGEm0d1MXbsWDc/9thjK7KoRl9//XU3X7ZsmZu/9NJLbu69hkrxz/qjjz66IotWfIlWcdm1\na1euvL293c29ba/Gz7nOdPqTIaX0iKSuryUDAAAA1ImevE1yhZn9PjuNYkzVtggAAADoJd2dDH9P\n0lRJJ0tqkfSNaKCZXW5mC81sYWtrazcfDuh91C7qEXWLekXtoq90azKcUtqYUtqfUnpT0vclnfY2\nY69LKTWnlJrHjRvX3e0Eeh21i3pE3aJeUbvoK92aDJvZxLJPL5bkX58QAAAA6Mc6XU3CzH4i6VxJ\nTWb2qqR/kHSumZ0sKUlaI+nTNdxGKL5m+R133OHmQ4cOrci+9rWvuWO5vn119EW3dt5u2mp030Yd\n71Fn8/XXX+/mLS0tbn7kkUdWZB/84Afdscccc4ybDx482M3RM3lrfMiQIW7uHWNJOuWUU9zce43a\ntGlTrm2JVraIVmqIVh9YtWqVm3td8HlWnqi16Llfz6tMRPvkHYstW7a4Y5999lk3j07TiFYCiV5z\nvOfAtm3b3LFtbW1uHtVctFJD9Lo4Y8YMNz/ssMMqsqhGo1VZtm/f7ubRMRo1apSbT5w4sSKLnqPV\nrN1OJ8MppflO7P90AwAAAOoIi24CAACgsJgMAwAAoLCYDAMAAKCwmAwDAACgsDptoEP/EHXkP/ro\no27+kY98pCKLrksOeKJu4s2b/auzf/e733Xzp556ys2jlQYuvPDCiuy9732vO/bQQw918wED/N/z\n67lzvh5FK9WMHDnSzU844QQ3Hzt2bEUWrQ6Qd1ui7vj77rvPzZ955hk397rmo475qPajuqWe8/Fe\nu6IVHKKVmqJ6ifJo1QRv1ZxodYhoNYnXX3/dzaPXv2iN5mi8tyrFY4895o6NXs+j55H33JWkmTNn\nuvmUKVMqst5YHYh3hgEAAFBYTIYBAABQWEyGAQAAUFhMhgEAAFBYNND1M4sXL3bzK664ws29yyhK\n0le/+tWqbRPqT57GmqhRLrrsaNRYdPfdd7v57t273fz000938wULFlRkUUNIdJlO9K6o3qLGr6gh\nJmqs8xp/JkyYUJVtaW9vd/OVK1e6edQQ5DXFRdsYXRo6as6qRqNckZrtvH2NmhabmprcPHrNiV4X\nowY9rxEt76XEd+zY4ebRPkWv6dHjrl69uiL7+c9/7o7duHGjm0c1PXv2bDe/5JJL3Nw7Hr1xOWbe\nGQYAAEBhMRkGAABAYTEZBgAAQGExGQYAAEBhMRkGAABAYdGK3Ue8yx9K0vz58918//79bv7Rj37U\nzbn0cjFE3bTRpUE9UefxunXr3PyGG25w8+iSoVH3/WWXXebmxx13XEXWG93EqL68KztE3fHe8c9T\n4283vqOjw82jFS+iSyx7+dFHH+2OPeSQQ9y8WvVc9OeFtypHtGpEc3Ozmy9btszNo8vRR69/ixYt\nqsiWLl3qjo1WE4lqNO8KGVHtepc237p1qzt20qRJbh6tGnH55Ze7+YwZM9x86NChFVn0elFNvDMM\nAACAwmIyDAAAgMJiMgwAAIDCYjIMAACAwmIyDAAAgMLqdDUJMzta0g8ljZeUJF2XUrrWzMZK+pmk\nKZLWSPpwSqmyJbHgok79efPmufny5cvdfPr06W7+la98pXsbhkLyOup37tzpjr3lllvcfPHixW4e\ndfyed955uXKvQ7ro3fGNJu/x9Gor72oS0Wtx1Km/Y8cONx82bJibH3744RXZmDFjurh16I6ojrxV\nGYYPH+6OnTVrlpuvX7/ezbdv3+7mzz77rJt7KzVEr7l5azpaTSK6H2+lBkk64YQTKrJzzjnHHTtz\n5kw3j8ZPnjw517Z4z/XeeP3vyjvDHZK+kFI6SdJsSZ8xs5MkXSnpwZTSCZIezD4HAAAA6kank+GU\nUktK6dns43ZJyyRNknSRpJuyYTdJ+kCtNhIAAACohVznDJvZFEmnSHpa0viUUkt20waVTqPwvuZy\nM1toZgtbW1t7sKlA76J2UY+oW9Qrahd9pcuTYTMbIek2SZ9LKW0rvy2VTlBxT1JJKV2XUmpOKTWP\nGzeuRxsL9CZqF/WIukW9onbRV7o0GTazwSpNhH+cUro9izea2cTs9omSNtVmEwEAAIDa6MpqEibp\neknLUkrfLLvpbkkLJF2T/X9XTbawzkXXMX/44Ydz3c+PfvQjNx87dmzeTUIBRN3E+/btq8hefPFF\nd+xvfvMbN4+68qdMmeLmn/zkJ9086rSv5XXoo+8Lq1X0b3mOT3SMo3zPnj1uHq0yEdXtoYceWpHt\n3r3bHes9D6X8qwnA59XLoEH+dKepqcnNL730Ujd/5zvf6eYPPfSQm7/wwgsV2Zo1a9yx0XwhqosJ\nEya4+SmnnOLmZ5xxhptPmzatIhs9erQ7dtSoUW7u1b8Uf9+j53RfvRZ3OhmWdKakv5D0vJkdWFPp\nyypNgm8xs09KWivpw7XZRAAAAKA2Op0Mp5QekxRN1edUd3MAAACA3sMV6AAAAFBYTIYBAABQWEyG\nAQAAUFhdaaBDF7zxxhtuPnv27Fz3c/PNN7t51B2KYou6jPfv3+/mW7Zsqcgef/xxd2zU2Tx+vHt9\nHc2fP9/No+7rwYMHu7nXTRx1GOftvq/G/bDyRO3kOT55j0Pe50q0asrOnTvdfO/evRVZW1ubOzZa\nwSLvShjUYtdF36uBAwe6+YgRI9z8Xe96l5tHP6O9lUOi1USiPKrRaKWGaGWHaF+9FXyiVX2i72Pe\n8f0N7wwDAACgsJgMAwAAoLCYDAMAAKCwmAwDAACgsJgMAwAAoLBYTaJKfvCDH7j5qlWrct3PWWed\n5eb10pGJ2oi6yaOO96hb/bXXXqvI1q9f745tampy85kzZ7r5vHnz3DzqbM5T03lXjch7Pzy/+jfv\n+ORdeSHqyG9vb3fz1tZWN9+2bZubjxw5ssuPGeXVqnP0XPSaEOVDhgzJlXvq+fjX+2so7wwDAACg\nsJgMAwAAoLCYDAMAAKCwmAwDAACgsJgMAwAAoLBYTaIbVqxYUZFdddVVvb8hqAt5OoTzrhrR0dHh\n5lHH+8qVKyuyXbt2uWOPOOIIN49Wk5gwYYKbR13GfbGyQ713PKNzUV1Fz5V9+/a5+dChQ9382GOP\ndfMxY8ZUZFOnTnXHjhgxws0HDPDfn6Jui4Hj3Hd4ZxgAAACFxWQYAAAAhcVkGAAAAIXFZBgAAACF\n1WkDnZkdLemHksZLSpKuSylda2ZXSfpLSQeuWfnllNK9tdrQ/uTRRx+tyKKGpcj06dPdPLp0LepX\nNS4lGzX/7Ny5082jSyyvXbu2Itu9e7c7dtiwYW4eNcoNGtT7/bi1bjihoaVxRPUZ1XN0efF3v/vd\nbu41xU2u/Qu7AAAEuElEQVSaNMkdO3r0aDePtjHvpYEB5NOVn14dkr6QUnrWzEZKWmRmD2S3fSul\n9PXabR4AAABQO51OhlNKLZJaso/bzWyZJP/XXQAAAKCO5Dpn2MymSDpF0tNZdIWZ/d7MbjCzykUW\nS19zuZktNLOFra2t3hCgX6J2UY+oW9Qrahd9pcuTYTMbIek2SZ9LKW2T9D1JUyWdrNI7x9/wvi6l\ndF1KqTml1Dxu3LgqbDLQO6hd1CPqFvWK2kVf6dJk2MwGqzQR/nFK6XZJSiltTCntTym9Ken7kk6r\n3WYCAAAA1deV1SRM0vWSlqWUvlmWT8zOJ5akiyW9UJtNrG9nnHGGmz/wwANuzmoSxRZ1h0eXaR08\neLCbH3744W5+0kknVWTRpWG9y8tK0jve8Q43HzlypJvn7Xj3VtSo9SWd6cpvHNFzZciQIW4e1Xm0\n4kNUcwMHDuxS9nY5q0YAfaMrq0mcKekvJD1vZouz7MuS5pvZySott7ZG0qdrsoUAAABAjXRlNYnH\nJHm/lhZiTWEAAAA0Lq5ABwAAgMJiMgwAAIDCYjIMAACAwupKAx0O8olPfKJLGRDJ2zUerRoR5cOH\nD3fzyZMnV2RRd3yU5xV19+fpkKfLHgfLWxNRHq3sUK3tAdD/8c4wAAAACovJMAAAAAqLyTAAAAAK\ni8kwAAAACovJMAAAAArLqtUx3qUHM2uVtLbXHrD2miS19fVG9JL+tK+TU0rjevMBqd261Z/2k7qt\njv50TGupP+0ntdtz/el41lp/2tcu1W6vToYbjZktTCk19/V29IYi7WsRFOV4FmU/i6Qox7Qo+1kU\nRTqe9bivnCYBAACAwmIyDAAAgMJiMtwz1/X1BvSiIu1rERTleBZlP4ukKMe0KPtZFEU6nnW3r5wz\nDAAAgMLinWEAAAAUFpPhLjKzG8xsk5m9UJaNNbMHzGxF9v+YvtzGajCzo83sITNbamZLzOyzWd5w\n+1oU1G7j7WsRULeNt69FQe3W374yGe66GyVdcFB2paQHU0onSHow+7zedUj6QkrpJEmzJX3GzE5S\nY+5rUdwoarfR9rUIbhR122j7WhQ3itqtq31lMtxFKaVHJG0+KL5I0k3ZxzdJ+kCvblQNpJRaUkrP\nZh+3S1omaZIacF+LgtptvH0tAuq28fa1KKjd+ttXJsM9Mz6l1JJ9vEHS+L7cmGozsymSTpH0tBp8\nXwuooY8ntduwGvpYUrcNraGPZ73XLpPhKkmlZTkaZmkOMxsh6TZJn0spbSu/rdH2tega7XhSu8XQ\naMeSui2ORjuejVC7TIZ7ZqOZTZSk7P9Nfbw9VWFmg1Uq7B+nlG7P4obc1wJryONJ7Ta8hjyW1G0h\nNOTxbJTaZTLcM3dLWpB9vEDSXX24LVVhZibpeknLUkrfLLup4fa14BrueFK7hdBwx5K6LYyGO56N\nVLtcdKOLzOwnks6V1CRpo6R/kHSnpFskHSNpraQPp5QOPmm+rpjZWZIelfS8pDez+MsqnQfUUPta\nFNQutVuPqFvqtl5Ru/VXu0yGAQAAUFicJgEAAIDCYjIMAACAwmIyDAAAgMJiMgwAAIDCYjIMAACA\nwmIyDAAAgMJiMgwAAIDCYjIMAACAwvr/nULp1p8Yt+AAAAAASUVORK5CYII=\n",
- "text/plain": [
- "<matplotlib.figure.Figure at 0x141020f98>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "# learning images on the test set\n",
- "f, ((ax1, ax2, ax3, ax4)) = plt.subplots(1,4, sharex='col', sharey='row',figsize=(12,3))\n",
- "ax1.imshow(np.reshape(image_test[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
- "ax1.set_title('True image')\n",
- "ax2.imshow(np.reshape(x_construction[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
- "ax2.set_title('Learned image')\n",
- "ax3.imshow(np.reshape(x_construction[999,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
- "ax3.set_title('Learned image')\n",
- "ax4.imshow(np.reshape(x_construction[9999,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
- "ax4.set_title('Learned image')\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 37,
- "metadata": {
- "collapsed": false
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "[('loss', 139.73684648437501)]"
- ]
- },
- "execution_count": 37,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "# calculate the ELBO which is minus the loss for test set\n",
- "metric = mx.metric.Loss()\n",
- "model.score(nd_iter_test, metric)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 4. All together: MXNet-based class VAE"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "from VAE import VAE"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "One can directly call the class `VAE` to do the training:\n",
- "\n",
- "```VAE(n_latent=5,num_hidden_ecoder=400,num_hidden_decoder=400,x_train=None,x_valid=None,\n",
- "batch_size=100,learning_rate=0.001,weight_decay=0.01,num_epoch=100,optimizer='sgd',model_prefix=None,\n",
- "initializer = mx.init.Normal(0.01),likelihood=Bernoulli)```\n",
- "\n",
- "The outputs are the learned model and training loss."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {
- "collapsed": false
- },
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "INFO:root:Epoch[0] Train-loss=377.146422\n",
- "INFO:root:Epoch[0] Time cost=5.989\n",
- "INFO:root:Epoch[1] Train-loss=211.998043\n",
- "INFO:root:Epoch[1] Time cost=6.303\n",
- "INFO:root:Epoch[2] Train-loss=207.103096\n",
- "INFO:root:Epoch[2] Time cost=7.368\n",
- "INFO:root:Epoch[3] Train-loss=204.958183\n",
- "INFO:root:Epoch[3] Time cost=7.530\n",
- "INFO:root:Epoch[4] Train-loss=203.342700\n",
- "INFO:root:Epoch[4] Time cost=8.887\n",
- "INFO:root:Epoch[5] Train-loss=201.649251\n",
- "INFO:root:Epoch[5] Time cost=9.147\n",
- "INFO:root:Epoch[6] Train-loss=199.782661\n",
- "INFO:root:Epoch[6] Time cost=8.924\n",
- "INFO:root:Epoch[7] Train-loss=198.044015\n",
- "INFO:root:Epoch[7] Time cost=8.920\n",
- "INFO:root:Epoch[8] Train-loss=195.732077\n",
- "INFO:root:Epoch[8] Time cost=8.857\n",
- "INFO:root:Epoch[9] Train-loss=194.070547\n",
- "INFO:root:Epoch[9] Time cost=9.216\n",
- "INFO:root:Epoch[10] Train-loss=193.186871\n",
- "INFO:root:Epoch[10] Time cost=8.966\n",
- "INFO:root:Epoch[11] Train-loss=192.700208\n",
- "INFO:root:Epoch[11] Time cost=8.843\n",
- "INFO:root:Epoch[12] Train-loss=192.191504\n",
- "INFO:root:Epoch[12] Time cost=8.152\n",
- "INFO:root:Epoch[13] Train-loss=191.842837\n",
- "INFO:root:Epoch[13] Time cost=6.180\n",
- "INFO:root:Epoch[14] Train-loss=191.310450\n",
- "INFO:root:Epoch[14] Time cost=6.067\n",
- "INFO:root:Epoch[15] Train-loss=190.520681\n",
- "INFO:root:Epoch[15] Time cost=6.058\n",
- "INFO:root:Epoch[16] Train-loss=189.784146\n",
- "INFO:root:Epoch[16] Time cost=6.046\n",
- "INFO:root:Epoch[17] Train-loss=188.515020\n",
- "INFO:root:Epoch[17] Time cost=6.062\n",
- "INFO:root:Epoch[18] Train-loss=187.530712\n",
- "INFO:root:Epoch[18] Time cost=6.088\n",
- "INFO:root:Epoch[19] Train-loss=186.194826\n",
- "INFO:root:Epoch[19] Time cost=6.491\n",
- "INFO:root:Epoch[20] Train-loss=185.492288\n",
- "INFO:root:Epoch[20] Time cost=6.182\n",
- "INFO:root:Epoch[21] Train-loss=184.922654\n",
- "INFO:root:Epoch[21] Time cost=6.058\n",
- "INFO:root:Epoch[22] Train-loss=184.677911\n",
- "INFO:root:Epoch[22] Time cost=6.042\n",
- "INFO:root:Epoch[23] Train-loss=183.921396\n",
- "INFO:root:Epoch[23] Time cost=5.994\n",
- "INFO:root:Epoch[24] Train-loss=183.600690\n",
- "INFO:root:Epoch[24] Time cost=6.038\n",
- "INFO:root:Epoch[25] Train-loss=183.388476\n",
- "INFO:root:Epoch[25] Time cost=6.025\n",
- "INFO:root:Epoch[26] Train-loss=182.972208\n",
- "INFO:root:Epoch[26] Time cost=6.014\n",
- "INFO:root:Epoch[27] Train-loss=182.561678\n",
- "INFO:root:Epoch[27] Time cost=6.064\n",
- "INFO:root:Epoch[28] Train-loss=182.475261\n",
- "INFO:root:Epoch[28] Time cost=5.983\n",
- "INFO:root:Epoch[29] Train-loss=182.308808\n",
- "INFO:root:Epoch[29] Time cost=6.371\n",
- "INFO:root:Epoch[30] Train-loss=182.135900\n",
- "INFO:root:Epoch[30] Time cost=6.038\n",
- "INFO:root:Epoch[31] Train-loss=181.978367\n",
- "INFO:root:Epoch[31] Time cost=6.924\n",
- "INFO:root:Epoch[32] Train-loss=181.677153\n",
- "INFO:root:Epoch[32] Time cost=8.205\n",
- "INFO:root:Epoch[33] Train-loss=181.677775\n",
- "INFO:root:Epoch[33] Time cost=6.017\n",
- "INFO:root:Epoch[34] Train-loss=181.257998\n",
- "INFO:root:Epoch[34] Time cost=6.056\n",
- "INFO:root:Epoch[35] Train-loss=181.125288\n",
- "INFO:root:Epoch[35] Time cost=6.020\n",
- "INFO:root:Epoch[36] Train-loss=181.018858\n",
- "INFO:root:Epoch[36] Time cost=6.035\n",
- "INFO:root:Epoch[37] Train-loss=180.785110\n",
- "INFO:root:Epoch[37] Time cost=6.049\n",
- "INFO:root:Epoch[38] Train-loss=180.452598\n",
- "INFO:root:Epoch[38] Time cost=6.083\n",
- "INFO:root:Epoch[39] Train-loss=180.362733\n",
- "INFO:root:Epoch[39] Time cost=6.198\n",
- "INFO:root:Epoch[40] Train-loss=180.060788\n",
- "INFO:root:Epoch[40] Time cost=6.049\n",
- "INFO:root:Epoch[41] Train-loss=180.022728\n",
- "INFO:root:Epoch[41] Time cost=6.135\n",
- "INFO:root:Epoch[42] Train-loss=179.648499\n",
- "INFO:root:Epoch[42] Time cost=6.055\n",
- "INFO:root:Epoch[43] Train-loss=179.507952\n",
- "INFO:root:Epoch[43] Time cost=6.108\n",
- "INFO:root:Epoch[44] Train-loss=179.303132\n",
- "INFO:root:Epoch[44] Time cost=6.020\n",
- "INFO:root:Epoch[45] Train-loss=178.945211\n",
- "INFO:root:Epoch[45] Time cost=6.004\n",
- "INFO:root:Epoch[46] Train-loss=178.808598\n",
- "INFO:root:Epoch[46] Time cost=6.016\n",
- "INFO:root:Epoch[47] Train-loss=178.550906\n",
- "INFO:root:Epoch[47] Time cost=6.050\n",
- "INFO:root:Epoch[48] Train-loss=178.403674\n",
- "INFO:root:Epoch[48] Time cost=6.115\n",
- "INFO:root:Epoch[49] Train-loss=178.237544\n",
- "INFO:root:Epoch[49] Time cost=6.004\n",
- "INFO:root:Epoch[50] Train-loss=178.033747\n",
- "INFO:root:Epoch[50] Time cost=6.051\n",
- "INFO:root:Epoch[51] Train-loss=177.802884\n",
- "INFO:root:Epoch[51] Time cost=6.028\n",
- "INFO:root:Epoch[52] Train-loss=177.533980\n",
- "INFO:root:Epoch[52] Time cost=6.052\n",
- "INFO:root:Epoch[53] Train-loss=177.490143\n",
- "INFO:root:Epoch[53] Time cost=6.019\n",
- "INFO:root:Epoch[54] Train-loss=177.136637\n",
- "INFO:root:Epoch[54] Time cost=6.014\n",
- "INFO:root:Epoch[55] Train-loss=177.062524\n",
- "INFO:root:Epoch[55] Time cost=6.024\n",
- "INFO:root:Epoch[56] Train-loss=176.869033\n",
- "INFO:root:Epoch[56] Time cost=6.065\n",
- "INFO:root:Epoch[57] Train-loss=176.704606\n",
- "INFO:root:Epoch[57] Time cost=6.037\n",
- "INFO:root:Epoch[58] Train-loss=176.470091\n",
- "INFO:root:Epoch[58] Time cost=6.012\n",
- "INFO:root:Epoch[59] Train-loss=176.261440\n",
- "INFO:root:Epoch[59] Time cost=6.215\n",
- "INFO:root:Epoch[60] Train-loss=176.133904\n",
- "INFO:root:Epoch[60] Time cost=6.042\n",
- "INFO:root:Epoch[61] Train-loss=175.941920\n",
- "INFO:root:Epoch[61] Time cost=6.000\n",
- "INFO:root:Epoch[62] Train-loss=175.731296\n",
- "INFO:root:Epoch[62] Time cost=6.025\n",
- "INFO:root:Epoch[63] Train-loss=175.613303\n",
- "INFO:root:Epoch[63] Time cost=6.002\n",
- "INFO:root:Epoch[64] Train-loss=175.438844\n",
- "INFO:root:Epoch[64] Time cost=5.982\n",
- "INFO:root:Epoch[65] Train-loss=175.254716\n",
- "INFO:root:Epoch[65] Time cost=6.016\n",
- "INFO:root:Epoch[66] Train-loss=175.090210\n",
- "INFO:root:Epoch[66] Time cost=6.008\n",
- "INFO:root:Epoch[67] Train-loss=174.895443\n",
- "INFO:root:Epoch[67] Time cost=6.008\n",
- "INFO:root:Epoch[68] Train-loss=174.701321\n",
- "INFO:root:Epoch[68] Time cost=6.418\n",
- "INFO:root:Epoch[69] Train-loss=174.553292\n",
- "INFO:root:Epoch[69] Time cost=6.072\n",
- "INFO:root:Epoch[70] Train-loss=174.349379\n",
- "INFO:root:Epoch[70] Time cost=6.048\n",
- "INFO:root:Epoch[71] Train-loss=174.174641\n",
- "INFO:root:Epoch[71] Time cost=6.036\n",
- "INFO:root:Epoch[72] Train-loss=173.966333\n",
- "INFO:root:Epoch[72] Time cost=6.017\n",
- "INFO:root:Epoch[73] Train-loss=173.798454\n",
- "INFO:root:Epoch[73] Time cost=6.018\n",
- "INFO:root:Epoch[74] Train-loss=173.635657\n",
- "INFO:root:Epoch[74] Time cost=5.985\n",
- "INFO:root:Epoch[75] Train-loss=173.423795\n",
- "INFO:root:Epoch[75] Time cost=6.016\n",
- "INFO:root:Epoch[76] Train-loss=173.273981\n",
- "INFO:root:Epoch[76] Time cost=6.018\n",
- "INFO:root:Epoch[77] Train-loss=173.073401\n",
- "INFO:root:Epoch[77] Time cost=5.996\n",
- "INFO:root:Epoch[78] Train-loss=172.888044\n",
- "INFO:root:Epoch[78] Time cost=6.035\n",
- "INFO:root:Epoch[79] Train-loss=172.694943\n",
- "INFO:root:Epoch[79] Time cost=8.492\n",
- "INFO:root:Epoch[80] Train-loss=172.504260\n",
- "INFO:root:Epoch[80] Time cost=7.380\n",
- "INFO:root:Epoch[81] Train-loss=172.323245\n",
- "INFO:root:Epoch[81] Time cost=6.063\n",
- "INFO:root:Epoch[82] Train-loss=172.131274\n",
- "INFO:root:Epoch[82] Time cost=6.209\n",
- "INFO:root:Epoch[83] Train-loss=171.932986\n",
- "INFO:root:Epoch[83] Time cost=6.060\n",
- "INFO:root:Epoch[84] Train-loss=171.755262\n",
- "INFO:root:Epoch[84] Time cost=6.068\n",
- "INFO:root:Epoch[85] Train-loss=171.556803\n",
- "INFO:root:Epoch[85] Time cost=6.004\n",
- "INFO:root:Epoch[86] Train-loss=171.384773\n",
- "INFO:root:Epoch[86] Time cost=6.059\n",
- "INFO:root:Epoch[87] Train-loss=171.185034\n",
- "INFO:root:Epoch[87] Time cost=6.001\n",
- "INFO:root:Epoch[88] Train-loss=170.995980\n",
- "INFO:root:Epoch[88] Time cost=6.143\n",
- "INFO:root:Epoch[89] Train-loss=170.818701\n",
- "INFO:root:Epoch[89] Time cost=6.690\n",
- "INFO:root:Epoch[90] Train-loss=170.629929\n",
- "INFO:root:Epoch[90] Time cost=6.869\n",
- "INFO:root:Epoch[91] Train-loss=170.450824\n",
- "INFO:root:Epoch[91] Time cost=7.156\n",
- "INFO:root:Epoch[92] Train-loss=170.261806\n",
- "INFO:root:Epoch[92] Time cost=6.972\n",
- "INFO:root:Epoch[93] Train-loss=170.070318\n",
- "INFO:root:Epoch[93] Time cost=6.595\n",
- "INFO:root:Epoch[94] Train-loss=169.906993\n",
- "INFO:root:Epoch[94] Time cost=6.561\n",
- "INFO:root:Epoch[95] Train-loss=169.734455\n",
- "INFO:root:Epoch[95] Time cost=6.744\n",
- "INFO:root:Epoch[96] Train-loss=169.564318\n",
- "INFO:root:Epoch[96] Time cost=6.601\n",
- "INFO:root:Epoch[97] Train-loss=169.373926\n",
- "INFO:root:Epoch[97] Time cost=6.725\n",
- "INFO:root:Epoch[98] Train-loss=169.215408\n",
- "INFO:root:Epoch[98] Time cost=6.391\n",
- "INFO:root:Epoch[99] Train-loss=169.039854\n",
- "INFO:root:Epoch[99] Time cost=6.677\n",
- "INFO:root:Epoch[100] Train-loss=168.869222\n",
- "INFO:root:Epoch[100] Time cost=6.370\n",
- "INFO:root:Epoch[101] Train-loss=168.703175\n",
- "INFO:root:Epoch[101] Time cost=6.607\n",
- "INFO:root:Epoch[102] Train-loss=168.523054\n",
- "INFO:root:Epoch[102] Time cost=6.368\n",
- "INFO:root:Epoch[103] Train-loss=168.365964\n",
- "INFO:root:Epoch[103] Time cost=10.267\n",
- "INFO:root:Epoch[104] Train-loss=168.181174\n",
- "INFO:root:Epoch[104] Time cost=11.132\n",
- "INFO:root:Epoch[105] Train-loss=168.021498\n",
- "INFO:root:Epoch[105] Time cost=10.187\n",
- "INFO:root:Epoch[106] Train-loss=167.858251\n",
- "INFO:root:Epoch[106] Time cost=10.676\n",
- "INFO:root:Epoch[107] Train-loss=167.690670\n",
- "INFO:root:Epoch[107] Time cost=10.973\n",
- "INFO:root:Epoch[108] Train-loss=167.535069\n",
- "INFO:root:Epoch[108] Time cost=10.108\n",
- "INFO:root:Epoch[109] Train-loss=167.373971\n",
- "INFO:root:Epoch[109] Time cost=11.013\n",
- "INFO:root:Epoch[110] Train-loss=167.207507\n",
- "INFO:root:Epoch[110] Time cost=11.427\n",
- "INFO:root:Epoch[111] Train-loss=167.043077\n",
- "INFO:root:Epoch[111] Time cost=10.349\n",
- "INFO:root:Epoch[112] Train-loss=166.884060\n",
- "INFO:root:Epoch[112] Time cost=13.129\n",
- "INFO:root:Epoch[113] Train-loss=166.746976\n",
- "INFO:root:Epoch[113] Time cost=11.255\n",
- "INFO:root:Epoch[114] Train-loss=166.572499\n",
- "INFO:root:Epoch[114] Time cost=10.037\n",
- "INFO:root:Epoch[115] Train-loss=166.445170\n",
- "INFO:root:Epoch[115] Time cost=10.406\n",
- "INFO:root:Epoch[116] Train-loss=166.284912\n",
- "INFO:root:Epoch[116] Time cost=10.170\n",
- "INFO:root:Epoch[117] Train-loss=166.171475\n",
- "INFO:root:Epoch[117] Time cost=10.034\n",
- "INFO:root:Epoch[118] Train-loss=166.015457\n",
- "INFO:root:Epoch[118] Time cost=10.047\n",
- "INFO:root:Epoch[119] Train-loss=165.882208\n",
- "INFO:root:Epoch[119] Time cost=10.008\n",
- "INFO:root:Epoch[120] Train-loss=165.753836\n",
- "INFO:root:Epoch[120] Time cost=10.056\n",
- "INFO:root:Epoch[121] Train-loss=165.626045\n",
- "INFO:root:Epoch[121] Time cost=10.704\n",
- "INFO:root:Epoch[122] Train-loss=165.492859\n",
- "INFO:root:Epoch[122] Time cost=10.609\n",
- "INFO:root:Epoch[123] Train-loss=165.361132\n",
- "INFO:root:Epoch[123] Time cost=10.027\n",
- "INFO:root:Epoch[124] Train-loss=165.256487\n",
- "INFO:root:Epoch[124] Time cost=11.225\n",
- "INFO:root:Epoch[125] Train-loss=165.119995\n",
- "INFO:root:Epoch[125] Time cost=11.266\n",
- "INFO:root:Epoch[126] Train-loss=165.012773\n",
- "INFO:root:Epoch[126] Time cost=10.547\n",
- "INFO:root:Epoch[127] Train-loss=164.898748\n",
- "INFO:root:Epoch[127] Time cost=10.339\n",
- "INFO:root:Epoch[128] Train-loss=164.775702\n",
- "INFO:root:Epoch[128] Time cost=10.875\n",
- "INFO:root:Epoch[129] Train-loss=164.692449\n",
- "INFO:root:Epoch[129] Time cost=8.412\n",
- "INFO:root:Epoch[130] Train-loss=164.564323\n",
- "INFO:root:Epoch[130] Time cost=7.239\n",
- "INFO:root:Epoch[131] Train-loss=164.468273\n",
- "INFO:root:Epoch[131] Time cost=10.096\n",
- "INFO:root:Epoch[132] Train-loss=164.328320\n",
- "INFO:root:Epoch[132] Time cost=9.680\n",
- "INFO:root:Epoch[133] Train-loss=164.256156\n",
- "INFO:root:Epoch[133] Time cost=10.707\n",
- "INFO:root:Epoch[134] Train-loss=164.151625\n",
- "INFO:root:Epoch[134] Time cost=13.835\n",
- "INFO:root:Epoch[135] Train-loss=164.046402\n",
- "INFO:root:Epoch[135] Time cost=10.049\n",
- "INFO:root:Epoch[136] Train-loss=163.960676\n",
- "INFO:root:Epoch[136] Time cost=9.625\n",
- "INFO:root:Epoch[137] Train-loss=163.873193\n",
- "INFO:root:Epoch[137] Time cost=9.845\n",
- "INFO:root:Epoch[138] Train-loss=163.783837\n",
- "INFO:root:Epoch[138] Time cost=9.618\n",
- "INFO:root:Epoch[139] Train-loss=163.658903\n",
- "INFO:root:Epoch[139] Time cost=10.411\n",
- "INFO:root:Epoch[140] Train-loss=163.588920\n",
- "INFO:root:Epoch[140] Time cost=9.633\n",
- "INFO:root:Epoch[141] Train-loss=163.493254\n",
- "INFO:root:Epoch[141] Time cost=10.668\n",
- "INFO:root:Epoch[142] Train-loss=163.401188\n",
- "INFO:root:Epoch[142] Time cost=10.644\n",
- "INFO:root:Epoch[143] Train-loss=163.334470\n",
- "INFO:root:Epoch[143] Time cost=9.665\n",
- "INFO:root:Epoch[144] Train-loss=163.235133\n",
- "INFO:root:Epoch[144] Time cost=9.612\n",
- "INFO:root:Epoch[145] Train-loss=163.168029\n",
- "INFO:root:Epoch[145] Time cost=9.578\n",
- "INFO:root:Epoch[146] Train-loss=163.092392\n",
- "INFO:root:Epoch[146] Time cost=10.215\n",
- "INFO:root:Epoch[147] Train-loss=163.014362\n",
- "INFO:root:Epoch[147] Time cost=12.296\n",
- "INFO:root:Epoch[148] Train-loss=162.891574\n",
- "INFO:root:Epoch[148] Time cost=9.578\n",
- "INFO:root:Epoch[149] Train-loss=162.831664\n",
- "INFO:root:Epoch[149] Time cost=9.536\n",
- "INFO:root:Epoch[150] Train-loss=162.768784\n",
- "INFO:root:Epoch[150] Time cost=9.607\n",
- "INFO:root:Epoch[151] Train-loss=162.695416\n",
- "INFO:root:Epoch[151] Time cost=9.681\n",
- "INFO:root:Epoch[152] Train-loss=162.620814\n",
- "INFO:root:Epoch[152] Time cost=9.464\n",
- "INFO:root:Epoch[153] Train-loss=162.527031\n",
- "INFO:root:Epoch[153] Time cost=9.518\n",
- "INFO:root:Epoch[154] Train-loss=162.466575\n",
- "INFO:root:Epoch[154] Time cost=9.562\n",
- "INFO:root:Epoch[155] Train-loss=162.409388\n",
- "INFO:root:Epoch[155] Time cost=9.483\n",
- "INFO:root:Epoch[156] Train-loss=162.308957\n",
- "INFO:root:Epoch[156] Time cost=9.545\n",
- "INFO:root:Epoch[157] Train-loss=162.211725\n",
- "INFO:root:Epoch[157] Time cost=9.542\n",
- "INFO:root:Epoch[158] Train-loss=162.141098\n",
- "INFO:root:Epoch[158] Time cost=9.768\n",
- "INFO:root:Epoch[159] Train-loss=162.124311\n",
- "INFO:root:Epoch[159] Time cost=7.155\n",
- "INFO:root:Epoch[160] Train-loss=162.013039\n",
- "INFO:root:Epoch[160] Time cost=6.147\n",
- "INFO:root:Epoch[161] Train-loss=161.954485\n",
- "INFO:root:Epoch[161] Time cost=9.121\n",
- "INFO:root:Epoch[162] Train-loss=161.913859\n",
- "INFO:root:Epoch[162] Time cost=9.936\n",
- "INFO:root:Epoch[163] Train-loss=161.830799\n",
- "INFO:root:Epoch[163] Time cost=8.612\n",
- "INFO:root:Epoch[164] Train-loss=161.768672\n",
- "INFO:root:Epoch[164] Time cost=9.722\n",
- "INFO:root:Epoch[165] Train-loss=161.689120\n",
- "INFO:root:Epoch[165] Time cost=9.478\n",
- "INFO:root:Epoch[166] Train-loss=161.598279\n",
- "INFO:root:Epoch[166] Time cost=9.466\n",
- "INFO:root:Epoch[167] Train-loss=161.551172\n",
- "INFO:root:Epoch[167] Time cost=9.419\n",
- "INFO:root:Epoch[168] Train-loss=161.488880\n",
- "INFO:root:Epoch[168] Time cost=9.457\n",
- "INFO:root:Epoch[169] Train-loss=161.410458\n",
- "INFO:root:Epoch[169] Time cost=9.504\n",
- "INFO:root:Epoch[170] Train-loss=161.340681\n",
- "INFO:root:Epoch[170] Time cost=9.866\n",
- "INFO:root:Epoch[171] Train-loss=161.281700\n",
- "INFO:root:Epoch[171] Time cost=9.526\n",
- "INFO:root:Epoch[172] Train-loss=161.215523\n",
- "INFO:root:Epoch[172] Time cost=9.511\n",
- "INFO:root:Epoch[173] Train-loss=161.152452\n",
- "INFO:root:Epoch[173] Time cost=9.498\n",
- "INFO:root:Epoch[174] Train-loss=161.058544\n",
- "INFO:root:Epoch[174] Time cost=9.561\n",
- "INFO:root:Epoch[175] Train-loss=161.036475\n",
- "INFO:root:Epoch[175] Time cost=9.463\n",
- "INFO:root:Epoch[176] Train-loss=161.009996\n",
- "INFO:root:Epoch[176] Time cost=9.629\n",
- "INFO:root:Epoch[177] Train-loss=160.853546\n",
- "INFO:root:Epoch[177] Time cost=9.518\n",
- "INFO:root:Epoch[178] Train-loss=160.860520\n",
- "INFO:root:Epoch[178] Time cost=9.395\n",
- "INFO:root:Epoch[179] Train-loss=160.810621\n",
- "INFO:root:Epoch[179] Time cost=9.452\n",
- "INFO:root:Epoch[180] Train-loss=160.683071\n",
- "INFO:root:Epoch[180] Time cost=9.411\n",
- "INFO:root:Epoch[181] Train-loss=160.674101\n",
- "INFO:root:Epoch[181] Time cost=8.784\n",
- "INFO:root:Epoch[182] Train-loss=160.554823\n",
- "INFO:root:Epoch[182] Time cost=7.265\n",
- "INFO:root:Epoch[183] Train-loss=160.536528\n",
- "INFO:root:Epoch[183] Time cost=6.108\n",
- "INFO:root:Epoch[184] Train-loss=160.525913\n",
- "INFO:root:Epoch[184] Time cost=6.349\n",
- "INFO:root:Epoch[185] Train-loss=160.399412\n",
- "INFO:root:Epoch[185] Time cost=7.364\n",
- "INFO:root:Epoch[186] Train-loss=160.380027\n",
- "INFO:root:Epoch[186] Time cost=7.651\n",
- "INFO:root:Epoch[187] Train-loss=160.272921\n",
- "INFO:root:Epoch[187] Time cost=7.309\n",
- "INFO:root:Epoch[188] Train-loss=160.243907\n",
- "INFO:root:Epoch[188] Time cost=7.162\n",
- "INFO:root:Epoch[189] Train-loss=160.194351\n",
- "INFO:root:Epoch[189] Time cost=8.941\n",
- "INFO:root:Epoch[190] Train-loss=160.130400\n",
- "INFO:root:Epoch[190] Time cost=10.242\n",
- "INFO:root:Epoch[191] Train-loss=160.073841\n",
- "INFO:root:Epoch[191] Time cost=10.528\n",
- "INFO:root:Epoch[192] Train-loss=160.021623\n",
- "INFO:root:Epoch[192] Time cost=9.482\n",
- "INFO:root:Epoch[193] Train-loss=159.938673\n",
- "INFO:root:Epoch[193] Time cost=9.465\n",
- "INFO:root:Epoch[194] Train-loss=159.885823\n",
- "INFO:root:Epoch[194] Time cost=9.523\n",
- "INFO:root:Epoch[195] Train-loss=159.886516\n",
- "INFO:root:Epoch[195] Time cost=9.599\n",
- "INFO:root:Epoch[196] Train-loss=159.797400\n",
- "INFO:root:Epoch[196] Time cost=8.675\n",
- "INFO:root:Epoch[197] Train-loss=159.705562\n",
- "INFO:root:Epoch[197] Time cost=9.551\n",
- "INFO:root:Epoch[198] Train-loss=159.738354\n",
- "INFO:root:Epoch[198] Time cost=9.919\n",
- "INFO:root:Epoch[199] Train-loss=159.619932\n",
- "INFO:root:Epoch[199] Time cost=10.121\n"
- ]
- }
- ],
- "source": [
- "# can initilize weights and biases with the learned parameters as follows: \n",
- "# init = mx.initializer.Load(params)\n",
- "\n",
- "# call the VAE, output model contains the learned model and training loss\n",
- "out = VAE(n_latent=2, x_train=image, x_valid=None, num_epoch=200) "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "# encode test images to obtain mu and logvar which are used for sampling\n",
- "[mu,logvar] = VAE.encoder(out,image_test)\n",
- "# sample in the latent space\n",
- "z = VAE.sampler(mu,logvar)\n",
- "# decode from the latent space to obtain reconstructed images\n",
- "x_construction = VAE.decoder(out,z)\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "metadata": {
- "collapsed": false
- },
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAADSCAYAAACvmc1VAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuU3GWd5/HPl9xJQsilE5JAEgIBQgQCtshNyQpBRVE5\nq86gO6COg7PHdeTozIjumRWdYWX3eJ11jw4uCOIFdcHb6OhGRA0YkA6JQIgxkgSTkPu1cyfJs3/U\nL0yR+n5J/7qrurvqeb/OyUn3p55UP7/+fav6SXU935+llAQAAADk6Li+ngAAAADQV1gMAwAAIFss\nhgEAAJAtFsMAAADIFothAAAAZIvFMAAAALLFYrgfMLP/Y2Yf6+t5AD1hZtPMLJnZwOD2JWY2p5en\nBRwTtYtmRN3WD4thSWa2q+rPYTPbW/X5Oxv99VNK700p/fdGfx00DzNbZWZX9vU86imlNCul9Mu+\nngcai9pFM6Ju8+b+byI3KaURRz42s1WS3ptS+nk03swGppQO9sbcgEahjtGsqF00I+q2/+KV4S4w\ns38ys2+b2bfMrFPSfzKzr5vZLVVjriwW0kc+P9nMvmdmm8xspZm9/yXu/4X7OnI/ZvbR4t8+Z2bX\nmNkbzWy5mW01s7+v+rcXm9kjZrbdzNaZ2T+b2aCq219vZn8wsx1m9r/M7GEze1fV7e81s9+b2TYz\n+zczO6VO3zY0SFELi4tz/hszO7fqtpvN7Bkz6zSzp83s2qrb3lWc/8+Z2RZJtxTZQ2b26aIGVprZ\n66v+zSgzu6OorbXFY2FAcduA4t9tNrMVkt5wjHm/8MqLmd1iZt8tar/TzJ40szOKut9oZqvN7Kqq\nf/tuM1tajF1hZu876r7/vpjjc0VNJzM7vbhtSDHPP5nZBjP7spkN69lZQHdQu9RuM6JuW79uWQx3\n3bWSvilplKRvv9RAMztO0r9KekzSZElzJf2dmV3Rxa91sirnZpKkf5R0h6Q/l3S+pDmSPmlmU4qx\nByV9UNI4SZdKep2k9xXzGC/pO5L+rrh9paQLq+b5H4vb3iypTdKjxTGinzKz8yXdqco5HivpXyT9\n0MyGFEOekfQqVer0E5K+bmYTq+7ilZJWSJog6daqbJkqNfI/Jd1hZlbcdpcqNXa6KvV3laT3Frf9\nlaQ3Fnm7pLeWPJxrJN0jabSkRZJ+pkrdT5b0yeLYjthYfK0TJL1b0ufM7ILie/I6SR+SdGUxzzlH\nfZ3bJJ0haXZx+2RJ/63kXNFD1C6124yo20zqNqXEn6o/klZJuvKo7J8k/eKo7OuSbqn6/EpJq4qP\nL5W04qjx/yDpK8HXfOG+ivvZJWlA8floSUnSy6vG/07SG4P7+ltJ3y0+fo+k+VW3maR1kt5VfD5P\n0g1Vtw+UtF/S5L4+D7n/8eqwyL8k6R+PypZJujy4n8WS3lx8/C5Jfzrq9ndJ+mPV58cX9XaSKk/e\n+yUNq7r9OkkPFh//QtJfV912VfFvBx7rmCTdImle1W3XHFX3I4v7OjG4r+9L+mDx8Z2SPlV12+nF\nvz29qPndkk6ruv1iSSv7+hy36h9ql9ptxj/Ubd51y3uGu251ibFTJU0xs+1V2QBJv+ziv9+cUjpU\nfLy3+HtD1e17JY2QJDM7S9JnJL1clQfVQFVe4ZUqryy/MO+UUjKzNUfN83+b2ReqssOqvDK9totz\nRe+aKukGM/tAVTZYlXMtM7telf+xTytuG6HKqw9HeHW8/sgHKaU9xQsUIySNkTRI0rp/f9FCx1Xd\nx4vqS9KzJY/l6Jr26n6EpO3FrxE/rsqrDcepUutPVs2jo+q+qufUVoxdWHUMpsrjEb2L2qV2mxF1\nm0HdshjuunTU57tVOeFHnFT18WpJy1NKMxs+q8qvNR6R9GcppV1m9req/GpDqrwKXP0eIFPl1xXV\n8/yHlNJLvu0D/cpqSbemlG49+gYzmyrpK5KukLQgpXTIzBar8kR0xNF1fKyvtV/SuORv+lgnqfo9\n5lOcMT1W/DryPknXS/pBSul5M/u+/v241qnyH7gjque0WZUn+VkpJf6D17eoXWq3GVG3GdQt7xnu\nvsWS3mBmo4v3B/1N1W0LJB0wsw+b2dDiTe/nmNnLGzCPkZJ2SNptZjNVvF+48K+SLrDKBryBqry3\nuK3q9i9L+q/Fv5OZnWhmZd+DhMYZVNTPkT8DVXni/Wsze6VVDDezN5jZSEnDVXni3SRVNkBIell3\nv3hKaZ2k/yfpM2Z2gpkdZ2anmdnlxZDvSPobq2wWHS3p5h4c60sZLGmIKsd1sHjF4qqq278j6d1m\nNtPMjlflLUlHjuGwKt+zzxXvoZeZTTaz1zZorqigdiuo3eZC3VZkV7cshrvvLklLVfk1xU8l3Xvk\nhuJ/dFerslltlSr/U/oXVd6IXm8flnSDpM7ia7zwKm9KaYOkP5P0WUlbJJ2mypvm9xe3f7e47btm\ntlPSE5L6dcFm5ieq/A/7yJ9bUkodqmyi+KKkbZL+qMp70JRSelqVt8wsUOXXYedIeriHc7helSfG\np4uv938lHdkc8hVVNmD8TtLjku7v4ddypZQ6VfnP5neKObxD0g+rbv83Sf8s6UFVvh+PFDftL/7+\nyJG8qPOfSzqzEXPFC6hdUbtNiLpVnnVrxZubkQGrtGd5TtJbU0rz+3o+QCMUv+l4StKQ4FeNQL9E\n7aIZtULd8spwizOz1xVvfxiiyq8ynpf02z6eFlBXZnatVXpbjpb0PyT9qFmflJEXahfNqNXqlsVw\n67tMlR6Hm1R5C8S1KaX9L/1PgKbzPlX6Yj4j6ZCk/9y30wG6jNpFM2qpuuVtEgAAAMgWrwwDAAAg\nWz1aDBfvR11mZn80s0a1+AAAAAAaottvkyg6E/xB0lxJayQ9Jum6otWIa9y4cWnatGnd+nrAEQsX\nLtycUmo79sj6oXbRU6tWrdLmzZvt2CPrh7pFPfCci2bV1drtyRXoLlTl+torJMnM7pX0ZlV647mm\nTZumjo6O6GagS8ys7CUoe4zaRU+1t7f3+tekblEPPOeiWXW1dnvyNonJevH1qNfoxZf6BQAAAPq1\nhm+gM7MbzazDzDo2bdrU6C8H1A21i2ZE3aJZUbvoKz1ZDK+VdErV5ycX2YuklG5PKbWnlNrb2nr1\nLUdAj1C7aEbULZoVtYu+0pPF8GOSZpjZqWY2WNKfq+ra1QAAAEB/1+0NdCmlg2b2XyT9TNIASXem\nlJbUbWYAAABAg/Wkm4RSSj+R9JM6zQUAAADoVVyBDgAAANliMQwAAIBssRgGAABAtnr0nmEAAPpC\nSqmh92/Wq1fORotqZJ0ePnzYzQ8ePOjmhw4dcnNvjlH9H3ec/xrqwIH+cjIaH91/Xz3ueGUYAAAA\n2WIxDAAAgGyxGAYAAEC2WAwDAAAgWyyGAQAAkC26SQAAAFQp2wWizPiy9x11gdi3b5+bb9++3c23\nbt3q5gcOHKjJBg8e7I4dNWqUm48ePdrNhw0b5uZluk/0RocJXhkGAABAtlgMAwAAIFsshgEAAJAt\nFsMAAADIFhvoALScel0ClUvy9g+NvvRyPTRyA1WkTH3mXsv12hBXj/uJLqMcbZTbvXu3m69evdrN\nFy1a5OYrVqxwc28j3oknnuiOPf3009181qxZbj5p0iQ3HzFihJsPGjTIzT31rGleGQYAAEC2WAwD\nAAAgWyyGAQAAkC0WwwAAAMgWi2EAAABkq0fdJMxslaROSYckHUwptddjUgBQrdHdBOpx/znv1u+L\nbg+N7shQtpuA1yEg6g5Q9r6jOXqXro3yAQMGlLrvXNSrjsqc06ibxP79+9183bp1bv7oo4+6+WOP\nPebmGzZscHPv0sjjx493x0bdHqLuE0OHDi11P16dRrVb9vHyUurRWu0/pJQ21+F+AAAAgF7F2yQA\nAACQrZ4uhpOkn5vZQjO7sR4TAgAAAHpLTxfDl6WUZkt6vaT3m9mrjx5gZjeaWYeZdWzatKmHXw7o\nPdQumhF1i2ZF7aKv9GgxnFJaW/y9UdL3JF3ojLk9pdSeUmpva2vryZcDehW1i2ZE3aJZUbvoK93e\nQGdmwyUdl1LqLD6+StIn6zYzIEP1uO59d+6nL+bSyLmX2WUvldt9XLb7QLPu1i9zHup1jqNd9l4e\njY06OETjo7mUvR9vfHQfkWjXfNkd+V53gLIdKZq1bqW+6W4S8erl+eefd8dGr4ZH3SGibhIrVqxw\n82HDhrn5mDFjujw2mvvGjRvdPOoycfzxx7v54MGDa7LeqNGedJOYIOl7xWQGSvpmSumndZkVAAAA\n0Au6vRhOKa2QdF4d5wIAAAD0KlqrAQAAIFsshgEAAJAtFsMAAADIVj0ux9zvPPLII27+hS98wc0n\nT57s5tFuyhtuuKEm83ZjvlSO1lNm53y0y/zAgQNuvmfPHjffvXt3qfH79+/vUvZSc9m7d6+bR7uM\no/vp7Ox08127dtVk0Ryj3fSnnXaam8+cOdPNx48f7+bezuZox3+z7r4vU7f16uAQ1Yp37iVp27Zt\nNdnmzZvdsTt27HDzgwcPunnZziNepwbJP9aoPqOfLdHO+9GjR7v5yJEj3TyaO7quXt10vMdGVKNL\nlixx8wULFrj50qVL3TyqizPOOMPNTz311Josqrl9+/a5efQzZ+vWrW4e1foJJ5xQk3nPw/XGIwYA\nAADZYjEMAACAbLEYBgAAQLZYDAMAACBbLIYBAACQrZbsJuF1e5Ck5cuX1+X+b7311pps1KhR7tiL\nLrqoLl+zL0ybNs3NP/rRj7r5lClTGjib/qPM7nvJ300c7chdv369m0e1G12Dfu3atW7uXT8+uqb8\n6tWr3TzaCR0df7QTOOrK4HUaiHYqDxkyxM1f/vKXu/mb3vQmN7/iiivcfOzYsTVZNO/+rmzdet0R\noq4RUaeGqM63bNni5suWLXPzxYsX12RR3UbnJ9odP2LECDePOj5EHTK8LivR8Uc76WfMmOHmUZ0f\nf/zxbt6snU2aQZnHi+R301mzZo07dv78+W6+aNGiLs6uIuoacfHFF3d5fFRDzz33nJtHx7R9+3Y3\njx4b3vcx+t7W87mYV4YBAACQLRbDAAAAyBaLYQAAAGSLxTAAAACyxWIYAAAA2WrJbhLf//733dzb\nkSxJs2bNcvPoOuGPPvpoTfaDH/zAHfuzn/3Mzb1rgUvSypUr3bysgQNrT+3EiRPdsVHXgEjUZeIj\nH/lIqftpNfXoJrFu3To3f/LJJ9086jIR1ZF3rqNd+bt373bzQYMGubl3TXkp3vE+cuRIN9+5c2dN\nFu1I9jpPSHEngKjLhLfjO3dePZepcSk+b1E9P/jgg27udU2JznH0/DRu3Dg3P+mkk9zcew6VpG3b\ntrm59ziKHs/RfYwZM8bNo049xx3nv57ldQLIqcOEd6xR7ZYV3U/UUWXz5s012cMPP+yO9dYWktTZ\n2enmF154oZtfcsklbt7e3u7mXneT6GfUrl273Dyqxeg5OuoQUeZ5p554ZRgAAADZYjEMAACAbLEY\nBgAAQLZYDAMAACBbx1wMm9mdZrbRzJ6qysaY2TwzW1787V/nEgAAAOjHutJN4i5JX5T0tarsZkkP\npJRuM7Obi8/7TSuBmTNnlsoj5557rptfd911Ndltt93mjl21apWbR90kvF3T3TF48OCaLOomEc1l\n06ZNbn7WWWd1f2KQFF9TfejQoW4edV4YNWqUm0+aNMnNvc4Op5xySqmvOX78eDefPn26m0+ePNnN\no93EHR0dNdm9997rjt26daubR4YPH+7mQ4YMcXNvh3Quu/LLHGfUTcLbSS9Jjz/+uJtHz5fe4+WM\nM85wx1566aVuHj3PjRgxws2j3fR/+tOf3Nx77o66Ruzfv9/No533UQeXqONFmbrNpZ7rJXreiros\nLFq0qCZbsGCBO3bHjh1ufs4557j5q1/9ajd/xSte4ebRc7f3+Ioe01Fnhz179rh5szjmK8MppV9L\nOvonzpsl3V18fLekt9R5XgAAAEDDdfc9wxNSSkcaKK6XNKFO8wEAAAB6TY830KXKa+ZhR2Qzu9HM\nOsysI/q1O9AfUbtoRtQtmhW1i77S3cXwBjObKEnF3/4lrCSllG5PKbWnlNrb2tq6+eWA3kftohlR\nt2hW1C76Sncvx/xDSTdIuq34278WcUaijU9lN5uV3eRXRnSpx2iTyytf+Uo3v+qqq+o2p1YSbUTx\nNid4GxwlacIE/x1Hs2fPdvOxY8e6ebQRw9tYE10uOdpwGeXRXKJNPlu2bHFzbxNVtJkj2ogYbdo7\n88wz3TzaWBfdv6dZNyKV2VgVjY0uRRtdjnj9+vVuHm1OOvvss2uyuXPnumNnzJjh5tFm0+iYog1R\n0WZLb3x0qfNog2t0ienoMVpmAx180fmPajHa5Bhtfp83b15NtmbNGndstF64/PLL3TzaLBptlIs2\nYnrPr7t373bHRptco8d6dBn06PveV7XbldZq35K0QNKZZrbGzP5SlUXwXDNbLunK4nMAAACgqRzz\nleGUUm0fsYor6jwXAAAAoFfxuxQAAABki8UwAAAAssViGAAAANnqbjcJ9HPeTtBrr73WHRvtmv38\n5z/v5tGO59zVo5tEtOM96qYQ7TKPduR65y7qhBJdpjYaH33NaO5RB4Knn366Juvs7HTHRpeMPv/8\n89389NNPd/McLscczTu6vGqZ44wuXbx69Wo33759u5tHXT0uuOCCmiw6l6NHj3bzqPNC9PwXfV+i\nTi1eh4CoU090aeioU0v0nBs95pq1RhupbNeI6Hlrw4YNbv7jH//YzX//+9/XZFHXoAsvvNDN58yZ\n4+YnnXSSm0e1Hh2T9/y6fPlyd+zChQvdPOqcUrYWvZ+XvVHnvDIMAACAbLEYBgAAQLZYDAMAACBb\nLIYBAACQLRbDAAAAyBbdJFrUXXfdVZOtX7/eHTt27Fg3nzp1aj2n1DLqsSs/2h0bXTs+6poQdXaI\n7sfrmuDt3pXiOZa9dvzzzz/v5g8//LCb/+IXv6jJos4TZ555ppu/5jWvcfPx48e7edTdg135LxbV\n+K5du9w86qawd+9eN4+6KZx88sk1WVT70RwPHDjg5lEnjC1btrj5U0895eZPPvlkTbZnzx53bNQ1\n5oQTTnDzqDtA2cdizqK6KNNhQZIeeOABN1+wYIGbe89/USeUyy67zM3b2trcPHp+imrd6zIlSStX\nrqzJvOdhSXr88cdL3Xe0voh+RpV5zq1HJ5wjeCQBAAAgWyyGAQAAkC0WwwAAAMgWi2EAAABki8Uw\nAAAAskU3iSb3zDPPuPmHPvShLt9HtAs2uu45Gifq7BDtnPe6Q0jldp9HO3IjZcevW7fOze+99143\n9zoQRLvvr776ajc/77zz3Dz6PtI1omsOHz7s5lE3iWinfrSTPLJx48aarGznhairSdQ1YMWKFW4e\n7aZfs2ZNTRbV7ejRo908qs9I2cdiLrzvS1SL+/fvd/MlS5a4+bx589z82WefdXOvc0TUTSKql7Ld\nIaJajzqkLFy4sCZbtmyZO3b16tVuHtVi1Amov3VI4ZVhAAAAZIvFMAAAALLFYhgAAADZYjEMAACA\nbB1zMWxmd5rZRjN7qiq7xczWmtni4o+/iwUAAADox7rSTeIuSV+U9LWj8s+llD5d9xmhlB/96Edu\n7u0mfdvb3uaOnT59el3nlKuoI4G3yzYaG+2kLbv7PlKma0I0Nsr37dvn5j/+8Y/dvKOjw829XcZz\n5sxxx771rW9182i3ftStI2dl6jbaMR7tDD/xxBPdfMyYMW4e7Tz3aiXqpBOd46gTRnRMW7dudfOn\nnnrKzb2OGuPGjXPHjh8/3s0HDx7s5n21w75Zeec66rCwfv16N49+tj7xxBNuHtWRd64nTJjgjo06\nW0RdI6JOKNGxep16JGnDhg012Z49e9yx0WM06mwUdciInjP6yjEfYSmlX0vynxUAAACAJtaT/25+\nwMyeKN5G4b8MAwAAAPRj3V0Mf0nSdEmzJa2T9JlooJndaGYdZtaxadOmbn45oPdRu2hG1C2aFbWL\nvtKtxXBKaUNK6VBK6bCkr0i68CXG3p5Sak8ptbe1tXV3nkCvo3bRjKhbNCtqF32lW4thM5tY9em1\nkvwdBQAAAEA/dsztfGb2LUlzJI0zszWSPi5pjpnNlpQkrZL0vgbOEYp3h37ve99zc29n56c+9Sl3\nLDvsG8vbrV+2m0SZHf9lld2pfujQITePdvffc889bh7tnJ45c2ZNdtNNN7ljp06d6ubRrvyyHTLw\nYtH3Keoacd5557l5dH62bNni5l7NRTvso84rQ4cOLZVH9blt2zY390yZMsXNX/ayl7l51AUlOqbo\nuTuXeo6e/7yOB9u3b3fH/uY3v3Hz3/72t26+d+9eN584caKbn3rqqTVZ1GGhbNeIqONJtF7YuHGj\nm+/cubMmi7pGRLV48sknu7l3/JI0bNgwN+8rx1wMp5Suc+I7GjAXAAAAoFfRvBAAAADZYjEMAACA\nbLEYBgAAQLZYDAMAACBb/evi0AjdcYe/Z3H+/Plu/o53vKMmmz59el3nhO4r200i2jVdj24S0VwO\nHz7s5tGO/y9/+ctuvmTJEjePdl/feOONNdkrXvEKd2zUCaBshwzU8upi4ED/R8a4cePc3OtqI0mn\nnHKKm0e75g8cOFCTRbVftmNItIN/5cqVbh51U/E6QVx00UXu2BkzZrj58OHD3Tz6vpfpGtGKHSai\nGti3b19NFp3PhQsXurnXYUGKa93rgiP53RSijgxRB5Ooa8SGDRvcfMeOHW6+evVqN1+zZk2X7zvq\neBJ1jjnnnHPcPKr1vnru5icGAAAAssViGAAAANliMQwAAIBssRgGAABAtthA188sXrzYzT/wgQ+4\neXQZ1E9+8pN1mxPqr+wmn3pslItE971nzx43/+lPf+rm999/v5tHl9597Wtf6+Zvf/vba7Lo0p1s\nlGucMhvoovMQbaCLNuFEm9O8Go3GRpeRjTYnrV27ttT4aOPPhAkTarK5c+e6Y6PjL7tRrhU3xZUR\nbfL1Nr/98Y9/dMdGm83Gjx/v5tHP3GhjnbeZ77nnnnPHRpeMXrFihZtHtbtq1apS470NqtElo9vb\n2938yiuvdPOpU6e6efRzoa9qmp8kAAAAyBaLYQAAAGSLxTAAAACyxWIYAAAA2WIxDAAAgGzRTaKP\n7N27182vu+46N492Tr/zne90cy693FrqtcPW25Uf7Zr/3e9+5+bf/OY33Ty6ZOj555/v5tdff72b\ne7u1c981319EXSOi8zNgwAA3j7omRJ1NvK4B0XNimY4Ukr/bX4ovxztmzBg3v+CCC2qy6HmYrhHl\nlKkLyX9Oi7rjRB1Pom4K0bmLulIsWrSoJos6nqxbt87No04YmzdvdvPoEuPR3L2OD1F3iGuuucbN\nzzzzTDePOqdEzw1erfdG/fPKMAAAALLFYhgAAADZYjEMAACAbLEYBgAAQLZYDAMAACBbx+wmYWan\nSPqapAmSkqTbU0pfMLMxkr4taZqkVZLenlLa1ripNqdot+sb3vAGN1+2bJmbz5w5080/8YlPdG9i\nyJJ3DfpnnnnGHXvfffe5+cKFC928ra3Nza+++mo3nzVrlpt7O55z303f35U9P9H46PmyzH1E+fPP\nP+/mq1evLjUXb+e9JJ177rk12ciRI92xZUXdFHhc+AYNGlSTRc9PUdeIzs5ON4+6UmzYsMHNt22r\nXRZFXSCijjxRd4hI1MEh6uzjdYiYM2eOO3bSpElufvzxx7u5dy6k8p1pGq0rrwwflPThlNLZki6S\n9H4zO1vSzZIeSCnNkPRA8TkAAADQNI65GE4prUspPV583ClpqaTJkt4s6e5i2N2S3tKoSQIAAACN\nUOo9w2Y2TdL5kh6VNCGldKRD9HpV3kbh/ZsbzazDzDo2bdrUg6kCvYvaRTOibtGsqF30lS4vhs1s\nhKT7JN2UUnrRpXlS5Q1N7puaUkq3p5TaU0rt0Xt2gP6I2kUzom7RrKhd9JUuLYbNbJAqC+FvpJTu\nL+INZjaxuH2ipI2NmSIAAADQGF3pJmGS7pC0NKX02aqbfijpBkm3FX//oCEzbHLR7tBf/vKXpe7n\nnnvucfMxY8aUnRIy4HWNkKS1a9fWZPPmzXPH/vznP3fzaBfwq171Kjd/05ve5ObDhw93c3bIt76o\nO0KUl3Ho0CE3j7oDbNmyxc2j+pw4caKbn3XWWTVZtJM+UrZrRC5dJqLjiZ6LvC4eUReQqFPD0KFD\n3Tx6+0bUTcGb4759+9yxkehV8unTp7v5RRdd5OZz58518xkzZtRkJ5xwgjvW6/YjSQMGDHDzsufO\nG98b9XzMxbCkSyX9haQnzWxxkX1MlUXwd8zsLyU9K+ntjZkiAAAA0BjHXAynlB6SFC3Lr6jvdAAA\nAIDewxXoAAAAkC0WwwAAAMgWi2EAAABkqysb6NAFO3bscPNoV2fk61//uptH1xRH3qKd8xs3+p0O\nH3rooZrsV7/6lTs26oTi7TyWpCuu8LcQRDueo93HaH316BoR3cfBgwfdfOfOnW4ejR87dqybT5o0\nyc29DgaRehw/4o4EXmeHKVOmuGOjriHTpk1z8127drl51CFi7969NVnUHWHIkCFuHnWNmjDBvdZZ\nWLvRsXodIsp2h4hE4+vRIaKeXSZ4ZRgAAADZYjEMAACAbLEYBgAAQLZYDAMAACBbLIYBAACQLbpJ\n1MlXv/pVN1+xYkWp+7nsssvcvNWuNY9yDh8+7OadnZ1u/oc//MHNFy5cWJOtWrXKHTt69Gg3b29v\nd/OLL77Yzb2d3VK8ExytI+qaUI88Grt//34393b1S9LgwYPdfNy4cW7e1tZW6n7KKNtlIvefC9Hx\ne+fixBNPdMeOGDHCzaOuIVH3keg52ptjVCuDBg1y86izQ/QcGuVl6qXRHU/KzKU36pyfRgAAAMgW\ni2EAAABki8UwAAAAssViGAAAANliMQwAAIBs0U2iG5YvX16T3XLLLb0/EbScsjvk169f7+ZLly51\n87Vr13Z5LtOmTXPzWbNmuXm0+z73He85KNsdoh6i3ftRPmTIEDc/+eST3TzqGjB27Fg39zoERLVP\n14j6KPMGNBzJAAAFd0lEQVR9iTo1DBzoL4OGDh3arTn1RL3Ocz3uJ7ea45VhAAAAZIvFMAAAALLF\nYhgAAADZYjEMAACAbB1zA52ZnSLpa5ImSEqSbk8pfcHMbpH0V5I2FUM/llL6SaMm2p/Mnz+/Jtu5\nc2ep+5g5c6abDxs2rFtzQnOJNtAcOnTIzaMNdNHlmKPxXn1NnTrVHTtlyhQ3P+2009w82ogSib4H\n3saNem3Eym1TSH9V9jx4l5eNLlEbXeo2urx4dD9RzUWXFx81alRNFm3aqsflcrszPhf97VK/6P+6\n8tProKQPp5QeN7ORkhaa2bzits+llD7duOkBAAAAjXPMxXBKaZ2kdcXHnWa2VNLkRk8MAAAAaLRS\n7xk2s2mSzpf0aBF9wMyeMLM7zcz9HZSZ3WhmHWbWsWnTJm8I0C9Ru2hG1C2aFbWLvtLlxbCZjZB0\nn6SbUko7JX1J0nRJs1V55fgz3r9LKd2eUmpPKbW3tbXVYcpA76B20YyoWzQrahd9pUuLYTMbpMpC\n+BsppfslKaW0IaV0KKV0WNJXJF3YuGkCAAAA9deVbhIm6Q5JS1NKn63KJxbvJ5akayU91ZgpNrdL\nLrnEzefNm+fmdJPIW7SzOerUMGbMGDefPXu2m3s77Xfv3u2OjbpJTJ7sbxmIds6XvVSvl7PLvn/r\ni8vIRmNHjhzp5tFz60knneTmUceHKPceo1Gniug+ItQz0Fhd6SZxqaS/kPSkmS0uso9Jus7MZqvS\nbm2VpPc1ZIYAAABAg3Slm8RDkrz/lmbRUxgAAACtiyvQAQAAIFsshgEAAJAtFsMAAADIVlc20OEo\n73nPe7qUAZFod3i0+3zEiBFuPnz4cDefPn26m19++eU1WdTVIVKvzg716ByA/q2R561spwavk0qj\nUbdAc+CVYQAAAGSLxTAAAACyxWIYAAAA2WIxDAAAgGyxGAYAAEC2rOxO8h59MbNNkp7ttS/YeOMk\nbe7rSfSS/nSsU1NKbb35BandptWfjpO6rY/+dE4bqT8dJ7Xbc/3pfDZafzrWLtVury6GW42ZdaSU\n2vt6Hr0hp2PNQS7nM5fjzEku5zSX48xFTuezGY+Vt0kAAAAgWyyGAQAAkC0Wwz1ze19PoBfldKw5\nyOV85nKcOcnlnOZynLnI6Xw23bHynmEAAABki1eGAQAAkC0Ww11kZnea2UYze6oqG2Nm88xsefH3\n6L6cYz2Y2Slm9qCZPW1mS8zsg0XecseaC2q39Y41B9Rt6x1rLqjd5jtWFsNdd5ek1x2V3SzpgZTS\nDEkPFJ83u4OSPpxSOlvSRZLeb2ZnqzWPNRd3idpttWPNwV2iblvtWHNxl6jdpjpWFsNdlFL6taSt\nR8VvlnR38fHdkt7Sq5NqgJTSupTS48XHnZKWSpqsFjzWXFC7rXesOaBuW+9Yc0HtNt+xshjumQkp\npXXFx+slTejLydSbmU2TdL6kR9Xix5qhlj6f1G7LaulzSd22tJY+n81euyyG6yRV2nK0TGsOMxsh\n6T5JN6WUdlbf1mrHmrtWO5/Ubh5a7VxSt/lotfPZCrXLYrhnNpjZREkq/t7Yx/OpCzMbpEphfyOl\ndH8Rt+SxZqwlzye12/Ja8lxSt1loyfPZKrXLYrhnfijphuLjGyT9oA/nUhdmZpLukLQ0pfTZqpta\n7lgz13Lnk9rNQsudS+o2Gy13PlupdrnoRheZ2bckzZE0TtIGSR+X9H1J35E0RdKzkt6eUjr6TfNN\nxcwukzRf0pOSDhfxx1R5H1BLHWsuqF1qtxlRt9Rts6J2m692WQwDAAAgW7xNAgAAANliMQwAAIBs\nsRgGAABAtlgMAwAAIFsshgEAAJAtFsMAAADIFothAAAAZIvFMAAAALL1/wEufHLDkoXM+QAAAABJ\nRU5ErkJggg==\n",
- "text/plain": [
- "<matplotlib.figure.Figure at 0x11e9ff7f0>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "f, ((ax1, ax2, ax3, ax4)) = plt.subplots(1,4, sharex='col', sharey='row',figsize=(12,3))\n",
- "ax1.imshow(np.reshape(image_test[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
- "ax1.set_title('True image')\n",
- "ax2.imshow(np.reshape(x_construction[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
- "ax2.set_title('Learned image')\n",
- "ax3.imshow(np.reshape(x_construction[999,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
- "ax3.set_title('Learned image')\n",
- "ax4.imshow(np.reshape(x_construction[9999,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
- "ax4.set_title('Learned image')\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 78,
- "metadata": {
- "collapsed": false
- },
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEICAYAAABLdt/UAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXuUZXV157/7vmjurQbjacQRqFuK4izGqIlE42saLZNg\ngxLHLJUU0s7MsuTmMcSBPLBWBieZMq9ZQSfIYI3iOH1vjMwYHzE4JATGrPggNAhMRHTQrioUH3QB\nAl1IN1V7/rj3FKdOncfvnHve9/tZa6/uuvc8fufcc75nn/3bv/0TVQUhhJDqUMu7AYQQQpKFwk4I\nIRWDwk4IIRWDwk4IIRWDwk4IIRWDwk4IIRWDwk5yRUSWReR1ebeDkCpBYSelQURURJ6b0LbOFpHv\nJLEtQooGhZ0QQioGhZ0UBhF5qYh8WUQeFpHvichVItIafff3o8XuFJHHROSto8/PE5E7Rut8SURe\n6NjesohcJiJ3iciPROQTIrJLRDoAPg/gWaNtPSYiz/Jozz4RuVtEHhWR74rIZaPPzxaR74jIe0Tk\n8Gg/c471zhWRr4rIIyJyn4i817XdV43a+vDo+3eMPj9ORP6ziKyKyA9E5BoROT7Rk0wmAgo7KRIb\nAN4NYA+AlwOYBfArAKCq/3K0zItUdUpVPyEiPwXgWgDvAmAB+BCAz4rIcY5tvgXAOQCeDeCFAN6h\nqkcAvB7A/aNtTanq/R7t+QiAd6nqbgAvAHCT47tnjtp5CoD9AJZE5Pmj744AuAjA0wCcC6AnIr8I\nACLSxfCh8mcATgLwYgB3jNb7QwBnjD577mjb/8Hw3BGyBYWdFAZVvU1Vv6KqT6rqMoZCvTdglXkA\nH1LVW1R1Q1U/BuAJAD/rWOa/qOr9qvoggL/CUDRNOQbgTBE5QVUfUtXbXd//rqo+oapfAPDXGD5E\noKr/R1X/r6puqupdAD7uOI5fBnCjqn5cVY+p6pqq3iEiMjqed6vqg6r6KID3AXhbhPYSAoDCTgqE\niJwhIp8Tke+LyCMYCtuegFW6AC4dhTQeFpGHAZwGwBlW+b7j/+sApiI06c0A9gFYEZEviMjLHd89\nNPL8bVbs/YrIy0TkZhF5QER+BOBix3GcBuBbHvs6CUAbwG2OY/nfo88JiQSFnRSJ/wrgHgDPU9UT\nALwHgAQsfx+ARVV9msPaqvpxg32FljVV1VtV9XwAzwDwaQDXOb7+iVGs3mYagB3O+XMAnwVwmqqe\nCOAax3HcB+B0j90dBvA4gH/hOJYTVTXKg4gQABR2Uix2A3gEwGMi8s8B9Fzf/wDAcxx//zcAF488\nZBGRzqjjcrfBvn4AwBKRE72+FJGWiMyJyImqemzUrk3XYv9xtNyrAZwH4H86juNBVf2xiLwUw/CL\nzQDA60TkLSLSEBFLRF6sqpuj47lSRJ4xasMpIvILBsdCyDYo7KRIXIahCD6Koch9wvX9ewF8bBSq\neIuqHgTwTgBXAXgIwL0A3mGyI1W9B8PY97dH29uRFQPg7QCWR2GhiwHMOb77/mif92Mo1hePtgkM\nO3x/T0QexbDzc8vTV9VVDMM7lwJ4EMOO0xeNvv7t0TF8ZbTPGwHYHbKEGCOcaIOQaIjI2QD6qnpq\n3m0hxAt67IQQUjEo7IQQUjEYiiGEkIpBj50QQipGI4+d7tmzR2dmZvLYNSGElJbbbrvtsKqGDlrL\nRdhnZmZw8ODBPHZNCCGlRURWTJZjKIYQQioGhZ0QQioGhZ0QQioGhZ0QQioGhZ0QQioGhZ0QF4PB\nADMzM6jVapiZmcFgMMi7SYREIpd0R0KKymAwwPz8PNbX1wEAKysrmJ+fBwDMzc0FrUpIYaDHToiD\nhYWFLVG3WV9fx8LCQk4tIiQ6FHZCHKyurkb6nJAiQmEnxMH09HSkz6sK+xnKDYWdEAeLi4tot9vb\nPmu321hcXMypRdlj9zOsrKxAVbf6GSju5YHCToiDubk5LC0todvtQkTQ7XaxtLQ0UR2n7GcoPxR2\nQlzMzc1heXkZm5ubWF5eLrSopxEyWVnxrjPl9zkpHkx3JKSkpJWaWa/XsbGx4fk5KQf02EnpKXtH\nX9z2RwmZRNmHl6gHfU4KiKpmbi95yUuUkCTo9/vabrcVwJa1223t9/uxt9ftdlVEtNvtxt6O6fZ7\nvV7s9ovItvVsE5Ed+3Tvo9VqqWVZnsfZ7XY9t9vtdhM9FyQ6AA6qgcZS2EmpSVKEkn5ImGzfT5xN\n2m967H7L+R1n2ueBxIfCTiYCU6/VhLQ9VROBjdJ+UwH2O0dBx5n2mwuJB4WdlB4TcUlSjJN8SETZ\n/jjtDzpH9ndJPkxIvlDYSakx9UaTDBuEPSScQlmv17e+M92X3/bdgp9E2MPrvCT1MCH5QWEnpcLt\neVqWFSpCtvj1er0toa3X69rr9Yz2EeUhESSUpkLst/1er7ejQ3XcMEiQp25ZljabTd/jZAimuFDY\nSWmI4106RTFJz77f7297qFiWZRTS6HQ6RsfZ6XS21qnVajseQmHtNBXesLCS13bYaVp8KOykNESJ\nA7vN9tS9vFKncDkF1b2cEz9xM2mL35uCvV23lwwM0w6dHrrf8djiayq8cfoekuqvoNefHhR2Uhqi\ndCqmYVNTU1seq5+w+n3uXsaPoIeX6fHXarVA4XUKqmVZ2mq1PB8CfsKbROcxvf50obCTwuEnKH6i\nNzU1ZSSmSYh7o9Hw9KjdAhW2HT8PNe2Hl5egNpvNHYOQvN4c6vV6YJ9GFI897ZTRSYfCTgqF3+Cc\nXq/nOzIyTGiBYWw7K48/rEPX2XZnTNx0vXEs7E3D/jfOuQoKMblJO2V00jEVdhYBI5ngVddEVXHN\nNdcAAI4//vit72u1Go4ePWq03SNHjiTb0ADW1taMljt69CguuugifPGLX8RHPvIR42MZh7D6Lva/\nQ22IxvXXX2+87PT0tGcVyEmbqCRvWASMZILf1HK2uDtFc3NzM6tmpcbm5iauueaaQFG3LAu9Xg+t\nVivDlkUnyrSAnKikGCQm7CJSF5GvisjnktomqQaDwQC1mv+lFseLLANBxyUiOHz4MK6++mrs3r07\nw1ZFJ4q3zYlKikGSoZhLAHwdwAkJbpOUHLtmOEu+bkdVISJ5NyOUVqsV2duem5ujkOdMIh67iJwK\n4FwAH05ie6Q6eMXW42BP8hDk+ZPk2b17N0W6hCR1l7wfwG8B8A2Oisi8iBwUkYMPPPBAQrslRSco\nPtvtdo238/znPx8iUon4e5l48MEH824CicHYwi4i5wH4oareFrScqi6p6lmqetZJJ5007m5JSQiL\nz1qWZbSdu+++u7Kx+LRIYio7ZrOUkyQ89lcCeKOILAP4CwCvFZF+AtslFcArS8JmZWUFjzzySOGz\nQsqIiBj3a9gPAHfMn9ks5WVsYVfVy1X1VFWdAfA2ADep6oVjt4yUnsFgsBVj9/Mejx07ht27d29l\nUZBkiPJ2Mz8/D1XFgQMHtmWz7N+/H5dccglEBCKCPXv2lG4+2YnFZBSTqQE4G8DnwpbjyNP8SLpA\nk9/2olZstOn1epmMIqU9ZV7D/YOKlrHuS36AJQWImzQmfvbbXtSZe5xtSKr+C83c3AT9fqz7kh8U\ndrIDv5olzhs1ikcfVPApak0Suw302LO3Wq224zcP+v1Y9yU/KOwThIkY9/v90Bs16iQPQduLWmPd\nbgO99fyt3W4nVu2RJAso7NXFXXfbb5ozJyav1kEeeL/f31Hf288sy/IUhiDBsCfGyFvUaE/9Hoyx\nFw9Q2MuN3xRtpp2Sbq8q6NXaLssaVHLVtPRss9n0fABYlqW9Xs9zO37r0PIzu9/D6xok+QEKe3np\n9/vaaDQ8BdBUYN1x0HE9dpN91ut138kxvGb0sduZt4hNkrVarW2Tb5j0u5DiAENhZ+GNAnLJJZfg\nySef3PH5sWPHjGuCP/3pT9/2d9BAk5WVFczMzGDfvn2eJVf37dtntM+NjQ089thjnt+tra15lrAd\nXqvED9Pcfnu5sNGm1157LQ4fPozNzU0sLy/jAx/4AMvsVhET9U/a6LEHgwQ8s2azuW1+S5N12u32\ntomVu92u9nq9SPnotOTNfqvxe7up1Wqh85ba5gUnny4PYCgmO5K+MZIUhKixa/crODs0i2F2fNuv\n/yJs/liv35aUDzAUkw12vfGVlRWoKlZWVjA/Pz/W0GvTwlhhqGrkadnc05p5TXOWBZ1OJ5f9FhU7\nBHfttdfuuD7W1ta2rrnFxUU0m03PbZiG1EgFMFH/pK1KHnsas7L7DefOyuwQjF9HaBb7z2IC6LKZ\nSSe3ff10Op0d348zypgUAzAUkw1RZ2V3xrztwTi2kLo/n0SLkvkzaWZfUybXnJ/41+t1xtJLDCjs\n2RDFY49aGGvSzC8lkja0MI/djrVH6SynuJcLUNizIUphLXZExhMs2lNZTvY15xWqazQakUN47FAt\nF6CwZ4c7K8adMmiailZVs9Mow4aoT+r5MbF6vb5tYFFS/R/ukCFTH4sNKOz5EOTBFzl2nFaIyCkO\nvV5vm3hPTU1tewhWsW/Br+ZKUczpsSdd1pkkDyjs+eAXTiiyN2qLb9LbdXrjXqLRarW0Vqvlfvxp\nn9u82+BnbtFOI8OLJAso7PlQZAEPu7nTEjbV8ePntlef9/lK27I6RhHZKv4Wdu2y/npxAAco5UOZ\nZnW3LAtLS0sAgJmZmVT2sbq6uu3fOIgIFhcXS3Vu49BsNnHxxRcbD84aZ45YVcX111+/7TO/85vm\neR8MBpiZmUGtVsPMzAznVE0KE/VP2qrssff7/dLEikVEZ2dnY8fXTbxLO2963HMSpWRx2c3rvLqr\nfdod0uOcD6+O0yxj7IzpRwcMxeRD1aZ2ExE988wzdwhz1gXC7Bu+6J3QaZk9EYnXpOHOz6OcG7+x\nFlllxTCmHx1Q2POhLN66qXU6nR2DhsYR2XE6S22hKXKHZFpmEuf2+z28JjJJ2jOO80CIEtNnGuYQ\nUNjzIW8ByMosy8olLFLmUIxX/RZTC/Ni/cJUzpm3kp4NyfmQdYu0+8HhJcymHjtDNk8BCns+VD19\njxbf4oaQTETMpDBYkuJo0t8Rtm+vUJ7XA8HvLdgvPFVlQGFPD7/XwjJ1nNLKYU7POigcERbWiOId\nm4ilSTjMZN9B+4vaWT4JXjwo7MkR9MrpvKCyjv3mldfdbrcr04FpdwLn3Q4/c86EFeTdhgl30LXi\nvM7d+7DXM32QRN130AMkzv1kWVZ6QlAAQGFPBlOvIesBNPV6XWdnZzMXmnq9rv1+v9BiaGp5PZCT\nNNNQS9CIaHuZsIe1U+SjLGvSt+DlbY9zP1U1NAMKezJEuemzEog8PXWTKdho2ZkzgySogzSsZMQ4\nHbtJmTssNO41Zl+vvV5vK0Rar9efGnF7++2qtZpqibx8ZCXsAE4DcDOAuwF8DcAlYeuUSdijiGin\n0yl0wacwC+ofiPMqTkvfnGIY5LWnUQsoDXM+iJII9/k9sHoXX6z6ileonnwyhd1zA8A/A/DTo//v\nBvBNAGcGrVMmYY/qNRSxsFWYCDvjnL7ezZjnhZaOtVqtbbNveZmdPZJ3W02s0Whk8vZwkYjq6aer\nXn45hd1og8BnAPxc0DJFFXavHvqqDGP388btmHmcc1WF80KrlnWBoax52N7RMlOAfhfQd550kt75\npjdR2EM3BswAWAVwgsd38wAOAjg4PT2d/hmIgN9rnzNGV4XQg58Qx00Tm9Th/WW0SUnDbQH6spHt\nPe44fbmIXgfoOqDPGS3zx4DeOPr/f2o09PGpqRRUJR2QtbADmAJwG4B/FbZskTz2MM+z2+2W5jU2\nyOw3EL8b3KQ+R1D+vv25ZVkU+4Jald6wOp1OqLPV6/X0g69/vW4AetHoszMAPQLoC0Z/XwHoWq2W\nssokB7IUdgBNADcA+PcmyxdJ2MNEW0Qq4a2bTD8XNDAlSo6zyXmlZWtFz9e37Qp4h1F+Ica2ngfo\nQ4B++pRTthyazwP6Qdf+HgAy05txQYadpwLgfwB4v+k6RRL2MNGumscelM/s/Nt0dh2/5csgIpNk\ns7OzRtdx3iGbKzAU45e57ISI25kC9GuA/gOgxzca2u/3df8znqGPAHo6oCeO7A8AfbBWU33oIdUf\n/zgPCYoEMhT2V41O5l0A7hjZvqB1iiTsQRe7M1WsKq+xXtUa/R5uzvCMyVuLPeqvSueLlq1dgaEH\nHXW9CwA9BOjG6N9bAL0f0GeOvu92u3rrhRd6vg1s2e//fk4qZA44QMmMsKp4zuW8vJmXAPpRQO8Z\nXVQfLcDNEWbNZnPbjPd+yzkHv5i+tbBDlRZkXfgL617EE/YLAH3Mta1NQN/rWk7vu0//5j3v0bee\nfLKeDegbTzhB//y44/RhQN968sn6qfe/Pw8JigQo7OaYFD7yE6x/B+j/A/QAhilUHy3AzWN0gzm8\n8bA6I6rmoRWKOi3InFkrtjmzVq4A9BiG4n4U0NsBfVPINg/B+0Fxv2Mfr6jVVH/0o233c7vd3vYg\nKUMRMVDYkyMotCCO/9+K+ML+CkC/AujjgH4b0F9P+QZzeuN+ou0cnFSFfgZa8excYFvWyhyg7wb0\nbEDfAOjnMBTpIHHfgP9bwDa7+eYd1/MV2P6GYJIdliegsCfDp6680vdC2eu6wOIK++mAPgroxwF9\nDaC/jaHX8m9TvKGcxaNMUiCrkBlEK5bZWStXhSz3JQw9d7/vD8H7/jzkWMZdesFvWyYzVeUJKOzj\n0+/39WnHHx/46ui8KOIK+zWAfgPQuuOzDwK6mtINFaVT2MbEY3d3ytJofubMWmmELHsZhl55zed7\nrxj7Y6PP7WXst8+wa96efL2o1SFBYY+OycTA7ldHp/kJe9d10TltL6ArGKZdOdd59ej7F3hsbxwT\nEe31ekaTgtTr9W3nxuuGsAeKmJRypdFs+yS2Z60E2aUIFnZgZ1bMBa7v45RoLmLMHRR2M6L82GGv\njn7CHtRhdCaGAr7ftc6e0ee/lMJNFWW+Uq9z5Rxhaos689ZpfuYW3esAfQLQVxqu/yVAb0ugHfa1\nG3WdIgEKezhR8q1NXh1NQzFOr/9ZGAr4+a5l6qPP35njDel3UVeldg4tffNLRbwJO52d3YDeDOiv\nADoL6C8C+tcY3itvSKg9UQdgFS3mDgp7OH6eutdrncmro4mwu73+Igu710z2/X6fok4ztkPwD0O6\nbS+gHwb0Wxi+zT4G6N8Dek5CbYlz3dJjL5GwB4VfvDyMJzDMUgl7dQwTdi+vvz3ahztmn1YoJup8\npfacmzZMe6RFMb9UxI0CtC3MosTYTScBHxdQ2L0JC78c8rkQH4H3q+MeQN88snsxfMV8M6BXwtzr\nXwH0fa7PXjXabxKdp/brZ9wa851OJ1Z8kkY7BO/76VAB2ua2Vqu1rd8oiqgHzTebJKCwexPmcRoP\ndsDw1XGv4bJBXv81gH4d23v9/wxDwU/qovUSd/tcULBpaZlJKmIW5jerWRLpjSYjt5MCFHZvgkSs\nXq8n4mH4bcPP67cHKA0wHHH3mxgOp05rgJK73G5Z5sOkldPCUhGzsE6n4/uW6tWXlISmpNHxCgq7\nN35PV3uKuMUzzxzbw4jq9QNDT/4WDEsKHEL6JQVss18ZmYNOm2Rz9yUloSn02DMkKFWv2Wxqr9cb\n28M4hPG9/iyNg4totKHFCckwxp6TsGcdTy5KXDGKMc5OK5o1Gg1tNpuZ79dLlMOyXpgVk7Gw5zXp\nQxHiilHMz2PftWtX7m2jldPGdRbsN+g8Umydnaq9Xi8zjzwMUNiHMO96PPPLJqDRgszus+p0OmNt\nx7KsWPdwkp6+yQxjWQFDYW+gwgwGA6ysrOTdjFKzubmZdxNICdnY2MCFF1449nbW1tawtrYWeb1j\nx46NvW+boZ7uZHV1NbF9JE0t7wakxWAwwPz8fN7NqCyWZaFer+fdDEJyY3p6Ou8m+FJZYV9YWMD6\n+nrezagk3W4Xhw8fxq5du/JuCiG50G63sbi4mHczfKmssBf5NaksNBrekbp9+/YBAI4cOZJlcwhJ\njShvn5ZlYWlpCXNzcym2aDxKI+yDwQAzMzOo1WqYmZnBYDAIXL7Ir0lZUavF/3kty8KJJ57o+d11\n110Xev4JKSoisuOzjY0Nz8+9ePzxx7f9HVWbMsGkhzVpi5oVE2cAQF5pjkWzqFktlmXp7Oxs5LrV\nNFqZbNxsL+ecwVmmQqJK6Y4mQ3a9Bgi4Z/zJ+2Iqg3HeUlrVLYnBeHYdmCzLCahWTNjDiuyEPTXp\nvdNotCTNFu4sC4CpVkzYw56KYYW9OEiJRptMc761x/XU3es5nUZ67GMIe5hHHvSD0VOn0SbT3OIa\ndxtBdWD8JnHv9XqRNM4UZCnsAM4B8A0A9wL4nbDl45QUCDq5YR45h8XTaNWzoP4grzK8Ubdv0gla\nWY8dQB3AtwA8B0ALwJ0AzgxaJ+laMf1+P5cqcDQaLT+zLCswKcI9gYZJAoUdsnE7j36OZWVj7ABe\nDuAGx9+XA7g8aJ0khZ0xdBptsi0sFOtMojDJ+vLSGL9QcJU99l8C8GHH328HcFXQOkkJO7NdaDSa\nqdkleIMcwVqttkNngsQ7Sh57EjXbUTRhBzAP4CCAg9PT05EPyAt66skac9hpVTdbdMOWixJuMRHs\npAYyYRJCMZz1J3njOaVV3UxHVScZbkkqZANDYU+iVsytAJ4nIs8WkRaAtwH4bALbDYX1YJJneO0Q\nUl02NjaMlltfX8fCwgL27du3o45M1OqOfkUJUytWaKL+YQZgH4BvYpgdsxC2fJIxdhTAA6DRJsXK\n0KeV9FunO+NORCLnqZfRY4eqXq+qZ6jq6apa3CLFhJCx2L9/f2AVxLwnXxERHDhwAO12O7Ftumdj\nUlVcf/31kbaxuLi4o02p1nQ3Uf+kLSmPnZ2nNFp21ul0Au85u+NwdnY213baBQDT3EecPPVSZcXE\nsSw7T8vw6kijld2cQuc3zD4rM818GcfymMhaNeNQTF6EdZ52u13s378/o9YQMrmo6tYkE0tLS7m2\nxe70tCwrcLl6vY7Z2Vl0u91I2y/6tHgAUGqPPWiAUliqEo1Gq66JiJHXbuLdt1ottSxrrBBKUmAS\nQjGq20sK2PmpJoMLaDRaeaxWq0W6ly3LUlWz+jDdbtd3uVqtlquQu8GkCHsY9NhptGpYlDdwu7qj\nSdkR27t3j7xutVqFEnVVCvsWXj9s3KHznF6PRsvP/GqzBC2vOuzMDRptam/XeX+7K0MWBVDYn8Jv\nPlT3D9npdHx/fLvXP++LO+qNoJp/lgKNloQ5a7OYOllhy7Xbbe31eplOSD0OoLCbY/KKFzYNn22m\nk3o4Lzg7dmhXnwu7GE3ije4LkyGp9OxdgP4doD8E9GFA/wHQnytAu6pm9j2YVFVX5+xIQfsrEqCw\nm2FykZhMjN3pdELnQbTNZDhy0Pr2G4dfSElEtt4+7FfQoLcR2ni2AuiHAD0f0NcB+t8B3QD0DQVo\nW1XM9qyTdFBsgpykonntoLCb4Xeh1Ot13/SmsBFkfuIfpcaEiRfBEEsxzPL47IuA3lSAtpXZnFlu\nXuGSsX+3kBmYgOKFZEBhNyOtqa3GHT7s9XBwPxgYXsnGusDwVvGwvT7rXAXoPQVoe9mt2WzmnrRQ\npJAMKOxmFDm+1uv1djx4nB4Ec/SzsRagL3PZdYCuA/ocn3VuA/SzBWh7auek1drRH+TMJPG6dr2s\nVqsVfoIXd7kE+02iXq9HrvI4LqCwm5HUzCZpEPbQoceej52LYQz9Ip/v/zWG3vzZBWhrUtZoNHxH\nX7rfTp2xcGc4xa+fJywjLW+zw7JTU1Oe32cp7qCwm5NE1bU0MJmSi0XOsrXnAfoQhqEWr+9/GtDH\nAL2yAG1N0vxGYJpcg/ZAnyAPvihvn3HaUa/XM9MEUNjLj59HblnW1oPI2QFke0dFuUmqZlOAfg3D\ndMaGx/fPBvR7gH4G0FoB2pu02cP0Ta5Rt3U6Hd9BQqZT1aVtduZNnJh+VmASqjtWHa/i/K1WC488\n8ghWVlagqlhbW8Pjjz+Ofr+PJ598EqqKAwcOoNFo5NTq6nABgEMANkb//h2An8Bw9vYnXcueBOAG\nACsYzg25mV0zM2NtbW3HZ6ZTux05csR3SjrTqeqSwLIsdLtdiAgsy4JlWRARdLtdLC0t4eqrr8bU\n1FSkbeY9uYgnJuqftNFjN8cdJvLzJpydvVFG5tG87QIMQyrqsE1A3+uxbAfQWwH9JqB7CtD2NM1N\nmfp5gvrOnPdZnG1nFcIFQzHVJOjCq9VqOjs7y7h7AnYI20XdtvuxPTtmN6A3APoEhg8Dd/ZM3seR\npDlDMXmWxHaP1Lbb4b436vX61kjwoAyWKH1VjUbD9x7MIukCFPbq4PQmihKPrLptwFvY3bY35Ht7\nez0MvfoHAT0C6F2jz/I+TlNzVjrMu9PeT0Cd94llWTsmofZbz/QBVavVth4OeaVJg8JeDfK+iSbV\nDsFbqA/F3N57RnYeoK8FdBHDh8elBTjWMJuamopUd8jZue83anTcDJkwAY0ivFHCL/bDIa2BjWGA\nwl58TEoT0EPPx7xi7I+NPk9qH31A7yzAsYaZPeLZNAbtJW5eue5ujxoYvhn45YuH7cNJ0LpuooaU\nut0uPXYvo7CHD4yip56/XYChh74x+jdI1LvwD8fs9VnnA4B+vQDHmbSZiptfDfRxPfagqe68cs6j\n3mv2xBx5DGwEhb3YBD3x6amXz0zLDtQxzKI5B8OBTr9WgLYnbe5JKqIOAAzzoG1hjbu+F+74fFBW\nmf1QyWNgIyjsxSbIK/F6RaWVy7zKDpyM7Z787xWgnWmaZVm+8XVnhopXmMbPgzapkBp0b0UJlRSx\n3Ago7MXGz6vgqNHym1/ZgTqgL8EwNHMFoI8D+psFaG+aFtTJ6Dcnqbv2utck9W5MMsfCPP2w7Rah\n3Ago7MXG74LO+0akjWdhZQec9rsYhmqOL0C78zA7e8bruzDP2h06CasQKSI6OztbKJGOAyjsxcOv\nCp7zQsv7ZqONZ5/EcBDTMw2WPRfDkMzpBWh30Swo68W0s9M5WY7fvKZe92CRQRbCDuBPANwD4C4A\nnwLwNJN/GYuVAAAQmklEQVT1JlHYTeN1LAVQHnNnzVyH4QjUVxqu/z4MwzGT6rEHWZDHbpqe6Hw4\nmIY+846hh4GMhP3nATRG//8jAH9kst4kCrvpKye99nKYXy2Zm7AzO2Y3oP8I6K9iOCfq6wH9U0CP\nAvoHBTiWollYB6lpP5Tz3oqy/7Rz0ccBWYdiALwJwMBk2UkU9rCeeuerIOcyLb4dwnZRD7K9gC4B\n+g0Mywk8AOiXAJ0rwHEU1bzEPUp9GveYkChJCWmPHh0H5CDsfwXgwoDv5wEcBHBweno69ROQBVF6\nzKNekEWeUYbmX0tmowBtq4o5s1jC4urOuVHdk2BHHRMyER47gBsB/JOHne9YZgHDGLuY7LQKHnvU\nHNd+v79Vac7kgs77pqIF2yF4C/uhArStSmZXlAxyjGynKs5o7aLlqYeBrDx2AO8A8GUAbdN1qiDs\nUdO0GDuvlmVRS4Y2tKCEApMOUj+zHwjMitkp6ucAuBvASVHWq4KwR63uFvWioxXfotSSmUQzfUMd\nx5yOVJwqjUEUUfSRkbDfC+A+AHeM7BqT9aog7FE9doZXaEW2tK7PNNN33eIcxXnq9XqBwl3EcgKq\nGYZi4lgVhN3kh+cUdbSimt3Z6BzAk8Z+ugHTObotyr3i5UFHyX5pNps7Rqs679+4I2LTBhT29Al7\n4rOYF62I5iWKaYYKe71e6JB/u00mnZ9B4jruA8redl4TaYQBCnu+MKZOK7o5y+uaerpx4+adTmfL\nIw8a7el0lrxSfk3CIb1eL3ZoyRZueuwUdk8YU6eVwZrNpvHAn3a7rbOzs7H3ZYuySaeklziblOy1\niTtPsC3cjLFT2D2hx04ri9nC5xbSVqu1FYe3LMtoyrowM/F4g2LlcTxmL5EOi7Hb601kVkxcmwRh\nZ4ydVkazBdUpZFEG/oSFakxi1EFOUdwYt5dIF1G4wwCFPX+YFUPL26KGIoCdXnGUa9iyrMCHgInH\nHRTGdJbiLYMQJw0o7MWCk1PT8jD72ov69ui8bqOsZ9d38XoY+HWSukU6TqGvSQEU9uIRpTodjZaE\nOa89U8+7Xq9vrRdnmL77eneLd1jHZBQnKO8slawBhb24MGOGlpXFzVePc62aetAmqYTuh4LfPvPO\nK88aUNiLC+PutKzMK9MjzBt2CqzpgyBKzDvO4J+i5pVnDSjsxYXCTsvS7NK3NmEhQefApbAYe5w4\ndxyRLmpeedaAwl5cyh6KyaJqHy1Z86qrEua52wLvN+mLcyKMKHjt274nLMvaVsMmSl55GdMXowIK\ne3EpcweqPfqPOfrlsnq9rr1eb0v4TB/O7Xbbd2CS+00gCs63hiBHx9QrnxSPHhT24tLv90OLIjlt\ndna2MA+DXq/HUBJNgWQ6Lk2ua5M4+qTE4GEo7DWQzJmbm8Pu3buNl//yl7+M5z73uSm2KJxOp4N+\nv4+rr74aa2trubaFFIPp6emxt7G6uprqMibrVhEKe8YMBgPMzMxEEsf19XXcdNNNKbYqnCNHjuDC\nCy9Eo9HItR2kGLTbbSwuLgJ46pqu1WqYmZnBYDAw3o7Jw2GcZZJ4+JQSE7c+aZvUUAxHn5bf7Jht\n0DKNRiP3dqZpUQYbjXtPMMa+HTDGXjzC4ontdjvx+PXs7OxWrRBms4xvNkH1VyzL8u1cnpqaKnwf\nRVD73DHrJGLbzmyWoKyYKNthVgyFPTOCev+7o+nJkr7p3Z20ZU+1zNtsxtnGrl27cj+OIPMTa6/0\nxqLONFRVkQeFvXgEeTdlDdOIyEQ9LMJ+yyKa14QVfsva3rLf926xLGI2SpXDMqCwF4+gCy5roWBY\nJro5xcpvbs1Go1HIUItXLXL321yr1Yp0Lbbbbe31eoUT0SI+bJICFPZi4veKOElebxnNLVZ+4mGP\n1owyTiFtswXNfe05ByyFdYgGbbtoYY+ihoeSABT2clGmV/tJM/vtxunt+i1rx6HzGJnbaDR27Nc5\nz2iYZ93r9bZ1tPuVEnAfb9Ggx66gsBeEuDH2Xbt2RZ4hp8yWtyfcarUCRbvb7eb2kG42mzu8cPvv\noPaq+oeWwqyIYskYu4LCXiA4EUe5zRaPPNvgFFoTZ8H2uOM4B0UWy6KFh5ICFPbywnj75JplWWO9\nldhC3e/3jcTafhDEaWdVxLJMgLViyovfMGgRybgl43Pccceh3W4nuk3LshLfZt602230+30cPnwY\n1157LSzLirWd6elpDAYDzM/PY2NjI3SfdlmAer1uvA/LsnD48GHMzc3FaiPJABP1DzMAl2L4JN9j\nsjw99mD8YoRx46B5W6/XM+qIM7FWqxUpnbBIw/ud4YFOp7PVKWuX1HUTNW3SvkZMPXV3x6nJ+Sty\n+GUSQFahGACnAbgBwAoo7InhFyMsYwy+2+0m1sEbNdtkdnY29+O3z4HztzXp3PNarlarbT0Q7MwV\nZ0dpWEw9SJidDwX7YVPVWHVZyVLY/xeAFwFYprCnTxlHqCbVZxDn4ZDkQKywUZl+1mw2t9U/8fPE\nvTJM3HVU3PF3p1CHPfTr9TqFueRkIuwAzgfwgdH/A4UdwDyAgwAOTk9PZ3AKqkvYzV40S8JjzzKk\n4jeaMs6I0k6nY/z7hOWEh+VnJzETESk2iQk7gBsB/JOHnQ/gFgAnqoGwO40eezBRX3/dQl+kaets\nQQnqH7AsS2dnZ32FaWpqKrNjsgcYef0GUb31qOmrYTnhYSMq/fZFT706JCbsvisCPwnghyNBXwbw\nJIBVAM8MW5fC7k8SgytMU93SMHsaP6+HklcM1/7cS7Q6nU5udXS84t1RzqktpqYPA5PfOMxjr/LA\nHDIkdWHfsSF67ImQ1HDoOB6m2+yJq3u9nnGsul6vh9YiidLOvPoTnIIYt1/DJHzjdW783thMhJud\nndWGwl5S4hQw8ruZ/TzhKGYLdBJxfDsdz9nWtCshhpUACDt21fEykYI6XP06S/1SXe122G8OFO7J\nI3Nhj2IUdn9MPPYomRLOUEbcDJEkR8KmPaq2Xq/r1NTUNmGNm0dvP0zHabP91uPehl+IxO/3N12f\nVBsKe0kJe902DQv4hW6KWCs8bWs2m7HeOJLw2O3QlD3lm/2ZvX3TGYmi/MakulDYS0xQnNRUZPxC\nN3mLbFnM/TBN4k3DKyxkWuc9ym9MqoupsLNWTAGZm5vD8vIyNjc3sby8vK0mx+rqqtE2/OrNhJFH\nPRrLsmLXRhmXer2OXq+Hfr+PbrcLEUG328XS0tK2855EbZqjR4/i2LFj2z5bX1/H/v37MRgMAACL\ni4s79uX3m8T9jckEYKL+SRs99viYeHRB8dewGei9BuckGRcP6w9Iaj9xzF2xMEo20LjmPg/O32lq\nampHO5rN5o5+FGbCVB8wFFNNvGLsziHrdizX7yYPmuvSuYxTKExqkEQRTz8RCppSrl6vez4Ukiou\n5hbMKAXXknrwBeWju83+zZi7PllQ2CvMOHnOQesHkUTqpC2CfoS9jXg9FNKondM1LIHgbEdSndIm\n58HZzqBl6b1XDwr7BJLU4KYo247rlXoR9uDweygk9dBx7ifqcSQ1gXWUYxGR0GXpvVcLCvsEkubs\n7EkIZ9jbQ9g+/HL5owz1N91P2Da96qcnVcrBmYcf1k6TBy7TIqsDhX0CKaLHHhTyiRpGsTs30yxd\nbNeJCYux+53TpNoW5v3bD8ko85qS8kNhn0DS7EiLI1hhD5Q4DwsRMfZo45hlWVvtCxJ30xIPcT14\nd39CUN2dsEJp9NirA4V9Qkkz9W3cTls3RZy02y3YYemhJiWV46SPxvGymSFTfSjsJFPiPFCS6pC1\nzV1Z0inKziH9Qeb2bsPW8RNOdz0frxTUoDeCuF42c9qrDYWdpIapeIQtFxbeierRuwcYebUnaH9e\nIh03Q8bLS/fqcI1SIIwQCjtJhSi58lFz6t2ebVDqn58HHiaKJp60E5O3CnfYxG8de3amoDbRyyZB\nUNhJKphm3iSVoRPk0aaZBWRj0mns3l+Ql8+OTDIOFHaSCqa58knm1Pt5tEnn7Qd1DtsPEZOwSZCX\nz9RDMg4UdpIKWXvsSbTFhCTLMQSV+aXHTsaBwk5SIekYexZtMSHpBxE7RUkaUNhJaiSVFZNlW8JI\noxwDO0VJ0pgKuwyXzZazzjpLDx48mPl+STUYDAZYWFjA6uoqpqensbi4uG1SjDjMzMxgZWVlx+fd\nbhfLy8tjbZuQpBCR21T1rLDlOIMSKRWDwQDz8/NYWVmBqmJlZQXz8/NbMxDFxWvmona7jcXFxbG2\nS0geUNhJqVhYWMD6+vq2z9bX17GwsABgKPwzMzOo1WqYmZkxFvy5uTksLS0FTo9HSFlgKIaUilqt\nBq9rVkRw4MABzM/PbxP+drtNgSaVgaEYUkn8JnCenp4O9eYJmRQo7KRUBMXCV1dXPdfx+5yQqkJh\nJ6UiKBYe5M0TMklQ2EnpmJubw/LyMjY3N7G8vLwVP2dmCyFDxhZ2Efl1EblHRL4mIn+cRKMIiQMz\nWwgZMlZWjIi8BsACgHNV9QkReYaq/jBsPWbFEEJIdLLKiukB+ENVfQIATESdEEJIuowr7GcAeLWI\n3CIiXxCRn/FbUETmReSgiBx84IEHxtwtIYQQPxphC4jIjQCe6fHVwmj9pwP4WQA/A+A6EXmOesR3\nVHUJwBIwDMWM02hCCCH+hAq7qr7O7zsR6QH4y5GQ/6OIbALYA4AuOSGE5MS4oZhPA3gNAIjIGQBa\nAA6P2yhCCCHxGTcrpgXgWgAvBnAUwGWqepPBeg8A2FkjNX32YHIfPDz2yYTHXi26qnpS2EK5FAHL\nCxE5aJIqVEV47Dz2SWOSj50jTwkhpGJQ2AkhpGJMmrAv5d2AHOGxTyY89glkomLshBAyCUyax04I\nIZWHwk4IIRVjYoVdRC4VERWRPXm3JStE5E9GJZbvEpFPicjT8m5T2ojIOSLyDRG5V0R+J+/2ZIWI\nnCYiN4vI3aOS2pfk3aasEZG6iHxVRD6Xd1uyZiKFXUROA/DzACZtzrS/BfACVX0hgG8CuDzn9qSK\niNQBfBDA6wGcCeACETkz31ZlxpMALlXVMzGs5fSrE3TsNpcA+HrejciDiRR2AFcC+C0AE9VzrKp/\no6pPjv78CoBT82xPBrwUwL2q+m1VPQrgLwCcn3ObMkFVv6eqt4/+/yiGAndKvq3KDhE5FcC5AD6c\nd1vyYOKEXUTOB/BdVb0z77bkzL8B8Pm8G5EypwC4z/H3dzBB4mYjIjMAfgrALfm2JFPej6Hztpl3\nQ/IgtLpjGQkpNfweDMMwlSTo2FX1M6NlFjB8VR9k2TaSPSIyBeCTAH5DVR/Juz1ZICLnAfihqt4m\nImfn3Z48qKSw+5UaFpGfBPBsAHeKCDAMRdwuIi9V1e9n2MTUCCqzDAAi8g4A5wGY9aqbXzG+C+A0\nx9+njj6bCESkiaGoD1T1L/NuT4a8EsAbRWQfgF0AThCRvqpemHO7MmOiByiJyDKAs1S1ahXgPBGR\ncwD8KYC9qlr5mvki0sCwk3gWQ0G/FcAvq+rXcm1YBsjQc/kYgAdV9Tfybk9ejDz2y1T1vLzbkiUT\nF2OfcK4CsBvA34rIHSJyTd4NSpNRR/GvAbgBw87D6yZB1Ee8EsDbAbx29FvfMfJgyQQw0R47IYRU\nEXrshBBSMSjshBBSMSjshBBSMSjshBBSMSjshBBSMSjshBBSMSjshBBSMf4/1HOo5p+uW1YAAAAA\nSUVORK5CYII=\n",
- "text/plain": [
- "<matplotlib.figure.Figure at 0x12fab6eb8>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAAC3CAYAAAD3oFO8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmQXeV55/HfA4jFyAIJLQgEEmKRWAIiFhgHs4Ultkmw\nQ7m8ToaUE5OZSjLOjCsVT6pmxjVVmXFSWWpqqiYTMvaAPTF2cELhcpwhBisDGErQCAm0IIRAWAK0\ngQFhK2x6549uOfee99fdr+7St889309Vl3Qenb73vec55/Sr2+9zn0gpCQAAAGiiwwY9AAAAAGBQ\nmAwDAACgsZgMAwAAoLGYDAMAAKCxmAwDAACgsZgMAwAAoLGYDAMAAKCxmAwXilF/GBEvjX39YUTE\noMeFQxcRV0XEqoh4NSK2DXo86FxE/G5ErI+IfRHxbET87qDHhM5ExL+NiGci4rWIeCEi/iwijhj0\nuNCZiDgyIjZFxI5BjwWdiYgvRsRbEfF6y9fSQY+rH5gMl7tZ0kckXSDpfEm/JOk3BjoidOrHkr4i\niYlT/YWkfylptqQPSPqtiPjEYIeEDn1b0kUppVmSztPovfbfDHZI6MLvStoz6EGga99MKc1s+Xpm\n0APqBybDFRHx8cr/gt6IiH+UdJOkP0kp7UgpPS/pjyX96iDHiomNl8uU0sMppa9JGsqLehhNkMs/\nSimtSSm9nVLaLOkuSZcOerwY3wS53JpSeungbpIOSDpjgEPFBCb4WamIOE3Sv5D0Xwc6SBSZKJdN\nwWS4IqX00/8FSTpJoxOm2yWdK2ldy67rxmKYpibIJWqmJJdjy5Yuk7RhAENEoYlyGRGfiojXJO3V\n6DvDfzG4kWIik1yT/13S70vaP6jxodwkufyliHg5IjZExL8e3Cj7i8nwOCLiMElfl/SPKaW/kDRT\n0qstu7wmaSbrhqc/k0vU1CS5/KJG72n/e6rHhUPncplS+vrYMomzJP1PSbsGOEQUqOYxIn5Z0uEp\npTsHPDQcInNN/rWksyXNk/RZSf8xIj45wCH2DZPh8f2BpHfrn9esvS5pVsu/Hyfp9ZRSmuqB4ZBV\nc4n6srmMiN/S6Nrh61NKbwxiYDhk416XKaUtGn2H/39M9aBwyH6ax4g4VtIfiXttXbVdkymljSml\nF1JK76SUHpT03yR9dJAD7BcqdY2xApxParSY462x8AaN/tru4bHtC8SvY6e9cXKJGhovlxHxGUlf\nkHR5SonK9RoovC6PkHT61I0Kh6qax4g4V9ISSfeP/dL0SEnHRcROSZeklLYNaqyYWOE1mTS6nn/o\n8M5wRURcqNH1Th9JKbVWwn5V0r+LiJMj4mRJn5d06wCGiELj5TIiDouIoyXNGN2MoyPiyEGNE5Ob\nIJeflvRfJF07rFXOw2aCXP56RMwf+/s5kv69pHsHM0pMZpw8rpd0iqQVY1+/rtGlLiskbR/EODG5\nCa7JD0fE7LGPlr1Y0uc0WqQ8dHhnOPdhjX5M0wMty4Hvl/QhSUslPTEW+1+iuGO6Gy+XfyhpVct+\n+yX9P0lXTuXgcEjGy+VySSdIeqQl/n9SSv9qykeIUuPlcqekP4iImRr9SK47JP2HgYwQJWweU0of\nPLgRES9LOpBS2jmA8aHceNfkKxr9GNKjJO2Q9KWU0m0DGWGfBUteAQAA0FQskwAAAEBjMRkGAABA\nYzEZBgAAQGMxGQYAAEBjdTUZjogPRMTmiHg6Ir7Qq0EBAAAAU6HjT5OIiMMlPSXpWo1+5MYjkj6Z\nUto43vfMnTs3LVmypKPnQ/ceffTRvSmleb14LHI5ONu2bdPevXt79sHn5HJweplL8jhY3F+HB7kc\nHqW57OZzhi+W9PTBD7qPiG9o9LPqxp0ML1myRCMjI108JboREc/16rHI5eCsXLmyp49HLgenl7kk\nj4PF/XV4kMvhUZrLbpZJnKz2jjI7xmLVgdwcESMRMbJnz57qP6NGyOXwIJfDgTwOD3I5PMhl/fS9\ngC6ldEtKaWVKaeW8eT35rQMGhFwOD3I5HMjj8CCXw4Nc1k83k+HnNdqD/KBFYzEAAACgFrqZDD8i\n6cyIOC0ijpT0CUnf7s2wAAAAgP7ruIAupfR2RPyWpLslHS7pKymlDT0bGQAAANBn3XyahFJK35X0\n3R6NBQAAAJhSdKADAABAYzEZBgAAQGN1tUwCAA5VaddLt5+LRZQ1cHP7lX4vyvJWmrNSpTkjj4em\nm5x0+vi9PDd6fV7U+fypUy6n4l7daS55ZxgAAACNxWQYAAAAjcVkGAAAAI3FZBgAAACNRQHdgPS6\nqASYrjotvDpw4EDRfu56OOww/p/fjV4Wy7k8Ok0oduq3bopTe/mcLuel13OJ0gIrdx+oy71hEIVx\nTje5rMa6KXx0eevlvaAeZwUAAADQB0yGAQAA0FhMhgEAANBYjVsz3M2aqrfffjuLvf7661lsx44d\nWWzfvn1t2yeddFK2z/z587PYUUcdlcX6vXamLqYil9u3b89ir776atv2okWLsn0WLFiQxYYtl51+\nGHuv1weXHq9OPyi+9DyrQzOIXjY8GVSDDad0XXKnBp3H6bIW2MXeeeedLPbmm29msVdeeSWL7dmz\nJ4u99NJLk47tXe96VxZz99y5c+cWfe8RR0zdVGi65NJdM2+99VYW+/GPf5zFXI5crDrvOfroo7N9\njjvuuCw2b968ov3c4x1++OFZrATvDAMAAKCxmAwDAACgsZgMAwAAoLG6WigTEdsk7ZP0jqS3U0or\nezEoAAAAYCr0YtX4VSmlvT14nIEpLRRwRVc//OEPs9gdd9yRxaqFAldeeWW2zzXXXJPFZsyYkcUG\nXcwxDFzxwAsvvJDFvvWtb2WxXbt2tW1fffXV2T4ul0ceeWTR2EoLx/qlm2YKLlYtsHHXkYu5whx3\nPbiYK0wsKf5x4y/9AHhXuFG6Xz90WuAoleWxmwYbpcfAHfuS886dO6XFqqWNGvp9TU5WQFV6n+hl\nAxV3nf7TP/1TFnNF5A8++GAWe/TRR7PYzp0727bfeOONbJ/jjz8+i5111llZ7Kqrrspi559/fhZz\nxVm91Ekue/lcpUWOe/fmU7knnngii/3gBz/IYlu3bs1i1aI6V6i4cOHCLHbBBRdkscsuuyyLuZzP\nnDkzi5VgmQQAAAAaq9vJcJJ0T0Q8GhE392JAAAAAwFTpdjL8/pTSCkkflPSbEXF5dYeIuDkiRiJi\nxH2mIOqDXA4PcjkcyOPwIJfDg1zWT1eT4ZTS82N/7pZ0p6SLzT63pJRWppRWug9SRn2Qy+FBLocD\neRwe5HJ4kMv66biALiKOlXRYSmnf2N+vk/SfezayHuhmUbr7XrfgfNOmTVlsw4YNkz6+6/TiijRw\naEoLQdzxf/rpp7PY+vXrJ3089/iuQGg6Fz62voaS4inJF9O4a6TasW/37t3ZPtWixPEe/9hjj81i\nxxxzTBZzRXUuJ9WCDvd97jldzHVDcgUjredBLwtnqo9XmkcXc9fHT37yk7Zt15mquk91TAe5Y+UK\nTCc7fhM9R5XrAOli7hwovcanUun9pOR4lf68c13GNm7cmMXuv//+LLZ69eos9swzz2Sx6nnlzkV3\nrmzevDmLufG6n7MXXXRRFuuXXhY+uv3c9bx///4s9uyzz2axe+65J4s9/PDDWcwV1b388stZrHo/\ncK/JFby5YjzXwfDGG2/MYsuWLctiJbr5NIkFku4cS+IRkr6eUvq/XTweAAAAMKU6ngynlJ6RlH/+\nBQAAAFAT/F4eAAAAjcVkGAAAAI3Viw50A9Hvji2u+MQtQnfFcq4r3amnnjrhtuSLOaZz0VWv9LqI\nqOTxXVHG448/nsVckcHixYvbthctWpTt4wo8uil46bfWY1RaLOc6Q/3oRz/KYtu2bWvbdsfZffyQ\nK55ynafe/e53F32vK4yq7ucqv+fPn5/F3LXqTHUuW+9b7hx3eXT3NZfH7du3t2274qfXXnsti7lj\nNWfOnCw2e/bsou8tKTR2RZXuOd35VNpJsN/3rV4p+fnmzgGX33/4h3/IYiMjI1nMFUC5c8rda6rF\nlS4f7t5TPT8l6aGHHspis2bNymLVe3o/dXNPKOku5wpb165dm8XuvPPOLOaK5aodASV/b3H312pn\nP/d9roPhli1bsti73vWuLLZgwYIs5jraleCdYQAAADQWk2EAAAA0FpNhAAAANBaTYQAAADRWLQro\n+l0s57gCOtdh5cknn8xirlPKe97znrZtV3TlFqC7wo06F9UNoujEPafL0WOPPZbF9u7dm8VWrFjR\ntn3yySdn+5R2zpouuWw9RqUFpa6IxR2vp556qm3bFZi689wVsy1dujSLuQI6V5jjCniqxSbuGncF\nN6Xn8VTmPKXUlid3DFyxiuvS5e5r69ata9t2XQNdkYu717kOdO7+516Dy1E1j66A1Z3Dbhwu5gz6\n2u3mZ1m1YM51kbv99tuz2AMPPJDFSq8ZVzReLbBy3+uKPqv3FMkX0D3//PNZzN3nXSfZfum0s5xU\nlss1a9Zk+9x2221ZzOXSFd+5ItPTTz89i7n7dbW7nOtQ6c49lzeX8+o9SZIuvfTSLFaCd4YBAADQ\nWEyGAQAA0FhMhgEAANBYTIYBAADQWLUooOs3t1DdLdp3HXncwnu3yP2ss85q2z7hhBOyfVynHRya\nkg49Ut4VTfKL8V3HnGoRl+ucVdIlazppLQYqLfpyx8YVY1WLIdy1ddppp2Wxiy++OIudeOKJWcwV\nXrlCDZeTasHfm2++me1z0kknZTGntNh1smPdjckKId3rc4Vw7lp4+umn27ZdkZorrKkWnErS3Llz\ns5jLoyvocbndvXt327Z7na64z3UX7KY4sl/cmEoLrFzRZLVzqiuwuvvuu7OYKzx2hVPnnXdeFnPX\nuLu2qo/nuhq682fVqlVZbMeOHVmseh5L/nyfSt3k8oknnmjb/upXv5rtc88992Sxffv2ZTHXve2S\nSy7JYmeffXYWW7JkSRarFki6TqPu5+d9992Xxdz3ug6xzz33XBYrUa+f2AAAAEAPMRkGAABAY006\nGY6Ir0TE7ohY3xKbExHfi4gtY3/m73MDAAAA01zJO8O3SvpAJfYFSfemlM6UdO/YNgAAAFArkxbQ\npZTui4gllfCHJV059vfbJP2jpN/r4bjalBYqdNrdrLTQZPXq1Vls586dWcwVBVSLgVyXo+ncoaxX\nOn093XSucwVb3//+97NYtQhH8oWO73//+9u2S3NZF6XFf66YwxWZvvjii23b1a5Eku8s564j17HK\nHWtXNOmKsV544YUsVuUKf5zS67ef58ZkxXnuvuYKTlw3r2rBpDsuF154YRZbvHhxFjv22GOzmFN6\nLlavXVfo5c6dZcuWZbFBdMmcTDddITdv3pzFvva1r7Vtu2K50vvhFVdckcV+4Rd+IYudccYZWcx1\nj6x273QFdO68cMVUroDu1VdfzWKdFl31isulu1a3bt2axb7+9a+3bd97773ZPq4zqLu/XnvttVns\n53/+57NY9QMBJN91sPqhAK4LqPs54rqUuvPRFf+67y3R6ZrhBSmlgz/hdkpa0OHjAAAAAAPTdQFd\nGv0v67j/lY6ImyNiJCJG3EdjoD7I5fAgl8OhNY/u3R/UB9fk8CCX9dPpZHhXRCyUpLE/8/evx6SU\nbkkprUwprXSfR4j6IJfDg1wOh9Y8us9eRX1wTQ4Pclk/nTbd+LakmyR9aezPu3o2ogFw63Vc84CR\nkZEs5ta7uLVR1Q+krq6LksrXEtZ5PWq/ubV1bs3Z/fffn8XcejuXy+p6KdcspddrRfud88ke310j\nL7/8chZza4ar67rch6y7tYjHHHNMFnPXTen64C1btmSx6gfwu8YM7rGc6bDmf7Ln279/fxZz66Zf\nf/31LFZdG++ujUWLFmUxt1bXNdhweXTXs1t3WD3H3Ll5yimnZDHXOGY6Km0M5dbIfvOb38xi1ZoJ\n91uF448/Pov94i/+Yhb7+Mc/nsWWL1+exdy15c7X6r3GNXdxuZwzZ04Wc2uL3X3erTHvl9LGUK4e\n6Y477shi1fXe7p1o9x/lG264IYt95CMfyWJufbBb6+1+DrrXVeXWLpfm0jUOcWvCS5R8tNrtkh6S\ntCwidkTEr2l0EnxtRGyRdM3YNgAAAFArJZ8m8clx/unqHo8FAAAAmFJ0oAMAAEBjMRkGAABAY3Va\nQFdbpYvXq4U1krRu3bos5hZ1X3DBBVmsuiC810VXFNWNcoVeroHAhg0bspgrzrrsssuyWLUAzJ0D\nw5Y3d424D0F3H3heLQQ555xzsn1ckYzjCp5cgaRrNLB+/fos9uSTT7Ztn3nmmdk+JUUg05G717nC\nOFds5ooGq+e9K2JyhTWuWM5xTQZcMYwrEqs2TXCFze7x3f1iOiot8r7zzjuz2KpVq7JY9Zp0jXCu\nv/76LPbpT386i5177rlZzBW/lhTLudhRRx2V7eOatrgCPXdPd/cQd270i3vNrhDMNUJxsWpTI3cN\nulx+9KMfzWJnn312FnPnhsulu99U93P3eff4bj/3c9Ydy07v17wzDAAAgMZiMgwAAIDGYjIMAACA\nxmIyDAAAgMZqXAGd4xbPP/jgg1nMdelZuHBhFrv88suzWLUIwC0Gd+pSYDUIpV2Z7rvvvizmOg4t\nWLAgi1177bVZrLq4f9hy6Y6rK0iqFm5IvptZtcDNFWe5a8sVUbhrdevWrVnsoYceymJr1qzJYtWC\nP3c9u4Ibd565wg1XwDNVXHGJKzZ0RWqu0K76eO71uvy4fLv9XOesTZs2ZTGX72rMFVOV5tHF3LF0\n10kvtT7+T37yk+zfq13kJOnv/u7vsti2bduyWPW8dIXCn/jEJ7KYK351x9rdE93xKrl3lhYou3PK\nnaOuoNO9hl5qfe1unO48//u///ss5s79ai6vuOKKbB/XJdAVPrrCRFfs75RcI6U/A93PG3ddukJN\n9xpK8M4wAAAAGovJMAAAABqLyTAAAAAai8kwAAAAGqtxBXRuEb/rwPTAAw9kMbeA23Wbc12sSgpp\n6lJgNZ25Tj733ntvFnOFFZdcckkWO+OMM7JYSUFBnXPprhFX9OGKetx+1WKsp556KtvHdZlyRS0u\nv65ob8uWLVnMdZXcv39/27YrMPvRj36UxVxRWGnxSeu50c8iLHeOu3G7/VwBS7XjoCtIdI/vjoE7\npi6PrqPhE088kcWqxXcnnHBCto87X9143X6lHbF6JaXU9vPGnc/f+c53spjrrOkKB6udxq677rpJ\n95Gko48+Oot1022zpGuZK8xy548r6HWP7zq0nXrqqVmsV1JKba/BdQ50hY+PPPJIFnPX5fLly9u2\nP/jBD2b7uGI5d38t7Yxbkjcpv0ZcLnft2pXF3L3A3afcdb5o0aIsVoJ3hgEAANBYTIYBAADQWJNO\nhiPiKxGxOyLWt8S+GBHPR8Tasa8P9XeYAAAAQO+VvDN8q6QPmPifpZRWjH19t7fDAgAAAPpv0qqu\nlNJ9EbGk/0OZGm4Rtis6WL9+fRZzRXCu6GrOnDlZrLq4vM4FVtOFy+XmzZuz2GOPPZbFXKGA68I0\na9asLDbsuXTFES7mCsbmzp2bxaqFLS5HrvjFXW+uuKm0s5jbr1rQ4TqvuQ5trtDOnSuu4Kjfncsm\neh6Xs5kzZ2YxVzC2ffv2tu033ngj2+fxxx/PYu5ac4VArvjS5cwVT1W/1+XRFcu53LrXNdUd6A4c\nONB2jFatWpXt47qkutfjfh69973vbdt+3/vel+3jCs26udeVHq/qsXavaWRkJIs999xzWcydx2ed\ndVYWu+iii4rG1omUUtt57Ap5f/CDH2Qx15XzuOOOy2I/93M/N+G25O9N/SwAPajk/rpx48YstmPH\njizmiqyXLVuWxVx+S3RzNH47Ih4fW0Yxu4vHAQAAAAai08nwn0taKmmFpBcl/cl4O0bEzRExEhEj\nrvc86oNcDg9yORxa8+jeSUJ9tObSffwW6oPrsn46mgynlHallN5JKR2Q9JeSLp5g31tSSitTSivn\nzZvX6TgxDZDL4UEuh0NrHt0SFdRHay7d56eiPrgu66ejyXBELGzZ/GVJ+QJbAAAAYJqbtIAuIm6X\ndKWkuRGxQ9J/knRlRKyQlCRtk/QbfRxjx9yCfVe48f3vfz+LuWIL9w7a5ZdfnsVmzJhROkQUKu2K\n5jr5vPLKK1mMXB4aV7zgOi2uWLEii1VzVy3EknwuHVfU44pDSlW7T7rXWVpo4gr+3Pf2q+AyItoe\n2xWuuXepzjnnnCzmChW3bdvWtu0K2Vxhoet0dcwxx2Sx0kI+t6ynWjxVWqzl9ivNYz8LkN555522\nc9MVjLnj4F6P68i1cuXKtm13XnTTjcxxRYiuCLp6v3YdRN3PbFdod+KJJ2YxV/R+/vnnZ7FeOXDg\nQFunS1f46Irq3LFxnfLe8573tG27n22luSzlculi1YI517Vy9erVWcx1Gl28eHEWqxaCStJJJ52U\nxUqUfJrEJ034yx09GwAAADCN0IEOAAAAjcVkGAAAAI3FZBgAAACNNema4emqZNG+W4DuOl09/PDD\nWcwtLq8uVJekpUuXZrGp6OwyTEpy6Rbnu89vfOCBB4qes1pAIvnihGHrLlfCnb+uc5k792+44YYs\nVi18ePbZZ7N9XMGqK2RyBXSu8Mp1OnKFXM8880zbtis2WrBgQRZzH33ljtFkxVi9Pr9aH88Vn5UW\nFLn9nnzyybbt3bt3Z/u4e6477q6T1vHHH5/FXAc6VwRdzaMrhHTP6XLrxuGOZT/v82+99ZZ27dr1\n0+0tW7Zk+7iiU3e+LV++PItVr133+hx3Hy49h13e3Dl01113tW1/61vfyvbZunVrFnP3BnduX3/9\n9Vls/vz5WaxXDhw40FZY6orlXOGpy4nLZbWQ2Z37pTlyP4vdNe3OPVeo/tBDD7Vtf+Mb38j2Wbdu\nXRZz9wzXWe+KK67IYp0WVDNrAwAAQGMxGQYAAEBjMRkGAABAY027NcOlH+DtVNczufVma9euzWI7\nd+7MYm4dolt/5NYpYXydfhj+22+/ne2zcePGLPb8889nMZfLK6+8Mou5dUpN5NZCHn300VnMfbi7\nW6+1cOHCtu3WtZAHuTVzbv2jWw/nuPOg2mBDyu8Z7gPblyxZksVmz56dxVyDlqmuH2hdG+jG49bN\nuvPeraWtrld0TTfeeOONLObOHRdz6xrduVJduyzl+XDXvKsJqJ6b442t100LJlNt1OCaoDju+nBr\n2avnvVtj77jXXNJsQZI2bNiQxb7zne9ksWrjJLeu2J2zF198cRb72Mc+lsWWLVuWxUrXTHfiwIED\nbflzzUHcz0XXmMbNN6o5cddg6X3IrQ929Rw//OEPs9iqVauyWDW/mzZtyvZx56xbH3zjjTdmMXdN\nd9ooi3eGAQAA0FhMhgEAANBYTIYBAADQWEyGAQAA0FjTroCulFtwXo3t27cv2+exxx7LYq7QzhVW\n/OzP/mwWc0U+TWzU4HRTDFn9XlcUsHr16izmcumKot73vvdlsakukplOWl+nK7YojbnroVqc4opf\nWouFDnLFHK5Yx32vG5srkqk2WHDNNFzRmSvSmA7nSusY3Pnsjot7La6IrFq84woo3fVXcq+WfJGY\n28+dA9Vz6uSTT872qTZ/kXxxmTtGLrf9zHdEtOXFFaa6a81dMy+88EIWW7NmTdu2a4jhntPtV1rk\n6O7Xjz/+eBarFma6QjLXbOGzn/1sFrvwwguzmLv/9LvQdbJmOC6X7loqOdbuena5dD9TX3rppSzm\nPnTg0UcfzWKueVl1vK5YzuXyM5/5TBY799xzs5i7T3WaS94ZBgAAQGMxGQYAAEBjMRkGAABAY006\nGY6IUyJiVURsjIgNEfG5sficiPheRGwZ+zP/FHoAAABgGispoHtb0udTSmsi4t2SHo2I70n6VUn3\nppS+FBFfkPQFSb93KE/eTYGVHWilS9mLL76Y7eM6p7gF16effnoWc0UZU91hqimqRTJ79uzJ9nn2\n2WezmCtO+Jmf+ZksRi67V1poVL3OXQGLK/pwBR6ugMft54qx3HirRWGugK60W9p0U1r05e7DJbkt\nLVJ0RV0uZ65gyHUmdN0oq3k75ZRTsn1cwZ8rMpwOZsyYofnz5/90++yzz872efrpp7OYO14jIyNZ\nrHr8169fn+3jzhVXlO46EbpCr1deeaXoOZYuXdq27QqsbrrppixW7ZAo+eK7qS6UPuyww9o6IrpC\nTtdZznXGffDBB7NY9fpynf5c4ZrrhOfmTNu2bctiritgyQcRXH311dk+LpfufO93Lif96Z9SejGl\ntGbs7/skbZJ0sqQPS7ptbLfbJH2koxEAAAAAA3JIb4VFxBJJF0paLWlBSungfyN2Ssqb2Y9+z80R\nMRIRI+7dPdQHuRwe5HI4kMfh0ZrLl19+edDDQRfIZf0UT4YjYqakv5H0Oymltt/FpNHfs9k1Dyml\nW1JKK1NKK92vqVAf5HJ4kMvhQB6HR2su58yZM+jhoAvksn6KJsMRMUOjE+G/Sin97Vh4V0QsHPv3\nhZLyRSQAAADANDZpAV2Mrkb+sqRNKaU/bfmnb0u6SdKXxv68qy8jHIcr+qgu4HYLv91if7d4vbqI\nX8q7VUkUXR3Uy25zUp7L7du3Z/u4gozZs/MPNXGda1znqToURU2F0uNQmvPqNVLavc3F3nzzzaLn\ndB2dWotYDqoWYLjzwunmfJ9u51mnRXXuGJd2KnRFcO453XO4+/Cpp57atu26TrriPqe0Y14/83jE\nEUdo7ty5P92+/PLLs32eeeaZLLZu3bos5u6TjzzySNv2U089le3jiq5Kc+SutQsuuCCLnXnmmVms\n+lpdF7nWY3OQy29pkW8/HX744W1dLC+99NJsn82bN2exVatWZbG9e/dmsbvvvrttu5pbyRcBu8JW\nd126/Lqfs+edd14Wu+qqq9q2XQGd6/brzr1+57Lk0yQulfQrkp6IiIN9+X5fo5Pgv46IX5P0nKSP\n9WxUAAAAwBSYdDKcUnpA0njT73yaDwAAANQEv+MHAABAYzEZBgAAQGOVrBkeOLdo3y3+3r9/f9v2\njh07sn19fRF3AAAH/klEQVSqnc0ktXX6Oei0007LYt10oqq+hulWRDNILifVQqnnnnuu6PtOPPHE\nLOZy6YoCSpHLQ1PSuaw0H+4adEVvrQUrE31v9TxzHY163SmzrkryWHotuGInV8jsrmfXPbJ6L5g1\na1bR2NzPkdJ897trWWunxmuuuSbbx3VZdMWFTz75ZBYrOe9dPlyHxkWLFmUxVyznCseqhY/ueUsL\n46ZrMfthhx3Wdu9ZuXJlto8rcnSv2xXHVb/XnReuMM7dN9315grQXS4vu+yyLFbtBFnaRa6bXPat\nAx0AAAAwrJgMAwAAoLGYDAMAAKCxmAwDAACgsaa8gG6y4oTS7j8uVi2iaC1AOMgVX7hCAddNxT1e\naREJRVajOs2lK4hxhXFuv+XLl2ex6dqtaLoqzVuJXl8frgiq2sFwvOeoFnS4IrvSTmt1ve57WSBY\negzc8XOdCV0nKtfdrHrdu+u7m1xMdR4jou14uI56n/rUp7KYK85au3ZtFtu1a1fbtjsHXBHq4sWL\ns5i7v7qiutJrq3qsS499aSfFQeSy9XXOmzcv2+eGG27IYu64rl69Oott2bKlbfu1117L9nHH2XV+\nc4WPK1asyGILFizIYu5arRbHlRbGDeK+yTvDAAAAaCwmwwAAAGgsJsMAAABoLCbDAAAAaKwpL6Br\nXRjdTeGG61pS7b5z4403Zvt86EMfymKucMMVy1F01R/uGFaLZK677rpsn6uuuqrosdzCfpfzuhZA\nDUovC1a6KbxyhTmuU9YZZ5yRxaqduJYsWZLt47p6ufPH3ZOm2znV72K5bh6/NLeumKz6vK5IyeXR\n3dNL89hvrc/pzrc5c+Zksfe+971ZzBXVuaLTQx3TQS5vve4q1qluiu/6NQ6XS1es6PLmitmq3eVc\nbrvJWzcFxINABzoAAADgEDEZBgAAQGNNOhmOiFMiYlVEbIyIDRHxubH4FyPi+YhYO/aVrz8AAAAA\nprGSNcNvS/p8SmlNRLxb0qMR8b2xf/uzlNIf93JApWvs3Nq06vovt1bUrc3pxnRZJ1MXpceruq7K\nrbOainFgVL8/0N6tQXNr1Y499tgs5q5ztz7UrQeurpFzDR3c45c0CxgvNki9Xufb6XOWru1362Ld\nusZTTz21bds1Upo9e3bRc06Xtd+TPX7p87vXg6nVz1y6cxiHbtLJcErpRUkvjv19X0RskpS3cQMA\nAABq5pDWDEfEEkkXSjrYE/C3I+LxiPhKROT/7QYAAACmseLJcETMlPQ3kn4npfSapD+XtFTSCo2+\nc/wn43zfzRExEhEje/bs6cGQMSjkcniQy+FAHocHuRwe5LJ+iibDETFDoxPhv0op/a0kpZR2pZTe\nSSkdkPSXki5235tSuiWltDKltNJ99iPqg1wOD3I5HMjj8CCXw4Nc1s+ka4ZjdGX3lyVtSin9aUt8\n4dh6Ykn6ZUnrD/XJuylAmG6FKU1Xmo9eFl11Mw50r9NjXfp9pUV1rnGCa5pTcu51UyhV13Ov39df\nacwVB7nczpo1a9LHc+dOaayueQTQuZJPk7hU0q9IeiIi1o7Ffl/SJyNihaQkaZuk3+jLCAEAAIA+\nKfk0iQckuf8qf7f3wwEAAACmDh3oAAAA0FhMhgEAANBYJWuGgZ6hOAWl6tDRDZMrLZZzxWy9fE4A\nGA/vDAMAAKCxmAwDAACgsZgMAwAAoLGYDAMAAKCxwnVl6tuTReyR9NyUPWG7uZL2Dui5e6EX41+c\nUupJb0hy2ZVux9+zPEoDzWXd8yhNo1xyTXaNXI6qey75WfnPyGVhLqd0MjxIETGSUlo56HF0qu7j\n76W6H4u6j79XhuE4DMNr6IVhOA7D8Bp6oe7Hoe7j76W6H4upHD/LJAAAANBYTIYBAADQWE2aDN8y\n6AF0qe7j76W6H4u6j79XhuE4DMNr6IVhOA7D8Bp6oe7Hoe7j76W6H4spG39j1gwDAAAAVU16ZxgA\nAABoM5ST4Yj4SkTsjoj1LbE5EfG9iNgy9ufsQY5xIhFxSkSsioiNEbEhIj43Fq/Na+gF8jg8yOXw\nIJfDg1wOh7rnURp8LodyMizpVkkfqMS+IOnelNKZku4d256u3pb0+ZTSOZIukfSbEXGO6vUaeuFW\nkcdhcavI5bC4VeRyWNwqcjkMblW98ygNOpcppaH8krRE0vqW7c2SFo79faGkzYMe4yG8lrskXVvn\n10Aem51HcjlcX+RyeL7I5XB8DVMeB5HLYX1n2FmQUnpx7O87JS0Y5GBKRcQSSRdKWq2avoYeq+Ux\nII9WLY8DubRqeRzIpVXL40AuM7U9BoPIZZMmwz+VRv+LMe0/RiMiZkr6G0m/k1J6rfXf6vIa+qku\nx4A8Tq4ux4FcTq4ux4FcTq4ux4FcTqxOx2BQuWzSZHhXRCyUpLE/dw94PBOKiBkaPSH+KqX0t2Ph\nWr2GPqnVMSCPE6rVcSCXE6rVcSCXE6rVcSCX46rdMRhkLps0Gf62pJvG/n6TRtejTEsREZK+LGlT\nSulPW/6pNq+hj2pzDMjjpGpzHMjlpGpzHMjlpGpzHMjlhGp1DAaey0Evku7Hl6TbJb0o6S1JOyT9\nmqQTNFqJuEXSPZLmDHqcE4z//Rr9VcDjktaOfX2oTq+BPJJHcjmcX+RyeL7I5XB81T2P0yGXdKAD\nAABAYzVpmQQAAADQhskwAAAAGovJMAAAABqLyTAAAAAai8kwAAAAGovJMAAAABqLyTAAAAAai8kw\nAAAAGuv/A7waJ9gxyNuaAAAAAElFTkSuQmCC\n",
- "text/plain": [
- "<matplotlib.figure.Figure at 0x11f9cd438>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "z1 = z[:,0]\n",
- "z2 = z[:,1]\n",
- "\n",
- "fig = plt.figure()\n",
- "ax = fig.add_subplot(111)\n",
- "ax.plot(z1,z2,'ko')\n",
- "plt.title(\"latent space\")\n",
- "\n",
- "#np.where((z1>3) & (z2<2) & (z2>0))\n",
- "#select the points from the latent space\n",
- "a_vec = [2,5,7,789,25,9993]\n",
- "for i in range(len(a_vec)):\n",
- " ax.plot(z1[a_vec[i]],z2[a_vec[i]],'ro') \n",
- " ax.annotate('z%d' %i, xy=(z1[a_vec[i]],z2[a_vec[i]]), \n",
- " xytext=(z1[a_vec[i]],z2[a_vec[i]]),color = 'r',fontsize=15)\n",
- "\n",
- "\n",
- "f, ((ax0, ax1, ax2, ax3, ax4,ax5)) = plt.subplots(1,6, sharex='col', sharey='row',figsize=(12,2.5))\n",
- "for i in range(len(a_vec)):\n",
- " eval('ax%d' %(i)).imshow(np.reshape(x_construction[a_vec[i],:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
- " eval('ax%d' %(i)).set_title('z%d'%i)\n",
- "\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Above is a plot of points in the 2D latent space and their corresponding decoded images, it can be seen that points that are close in the latent space get mapped to the same digit from the decoder, and we can see how it evolves from left to right."
- ]
- }
- ],
- "metadata": {
- "anaconda-cloud": {},
- "kernelspec": {
- "display_name": "Python [Root]",
- "language": "python",
- "name": "Python [Root]"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 2
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython2",
- "version": "2.7.12"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services