[GitHub] ThomasDelteil commented on a change in pull request #12542: [MXNET-949] Module API to Gluon API tutorial

[GitBox <gi...@apache.org>]

ThomasDelteil commented on a change in pull request #12542: [MXNET-949] Module API to Gluon API tutorial
URL: https://github.com/apache/incubator-mxnet/pull/12542#discussion_r217848399
 
 

 ##########
 File path: docs/tutorials/python/module_to_gluon.md
 ##########
 @@ -0,0 +1,297 @@
+
+# Converting Module API code to the Gluon API
+
+Sometimes, you find yourself in the situation where the model you want to use has been written using the symbolic Module API rather than the simpler, easier-to-debug, more flexible, imperative Gluon API. In this tutorial, we will give you a comprehensive guide you can use in order to see how you can transform your Module code, to work with the Gluon API.
+
+The different steps to take into consideration are:
+
+I) Data loading
+
+II) Model definition
+
+III) Loss
+
+IV) Training Loop
+
+V) Exporting Models
+
+In the following section we will look at 1:1 mappings between the Module and the Gluon ways of training a neural networks.
+
+## I - Data Loading
+
+
+```python
+import logging
+logging.basicConfig(level=logging.INFO)
+
+import numpy as np
+import mxnet as mx
+from mxnet.gluon.data import ArrayDataset, DataLoader
+from mxnet.gluon import nn
+from mxnet import gluon
+
+batch_size = 5
+dataset_length = 200
+```
+
+#### Module
+
+When using the Module API we use a [`DataIter`](https://mxnet.incubator.apache.org/api/python/io/io.html?highlight=dataiter#mxnet.io.DataIter), in addition to the data itself, the [`DataIter`](https://mxnet.incubator.apache.org/api/python/io/io.html?highlight=dataiter#mxnet.io.DataIter) contains information about the name of the input symbols.
+
+Let's create some random data, following the same format as grayscale 28x28 images.
+
+
+```python
+train_data = np.random.rand(dataset_length, 28,28).astype('float32')
+train_label = np.random.randint(0, 10, (dataset_length,)).astype('float32')
+```
+
+
+```python
+data_iter = mx.io.NDArrayIter(data=train_data, label=train_label, batch_size=batch_size, shuffle=False, data_name='data', label_name='softmax_label')
+for batch in data_iter:
+    print(batch.data[0].shape, batch.label[0])
+    break;
+```
+
+    (5, 28, 28) 
+    [5. 0. 3. 4. 9.]
+    <NDArray 5 @cpu(0)>
+
+
+#### Gluon
+
+With Gluon, the preferred method is to use a [`DataLoader`](https://mxnet.incubator.apache.org/api/python/gluon/data.html?highlight=dataloader#mxnet.gluon.data.DataLoader) that make use of a [`Dataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html?highlight=dataset#mxnet.gluon.data.Dataset) to prefetch asynchronously the data.
+
+
+```python
+dataset = ArrayDataset(train_data, train_label)
+dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=False, num_workers=0)
+for data, label in dataloader:
+    print(data.shape, label)
+    break
+```
+
+    (5, 28, 28) 
+    [5. 0. 3. 4. 9.]
+    <NDArray 5 @cpu(0)>
+
+
+#### Notable differences
+
+- Gluon keeps a strict separation between data holding, and data loading / fetching. The `Dataset` role is to hold onto some data, in or out of memory, and the `DataLoader` role is to request certain indices of the dataset, in the main thread or through multi-processing workers. This flexible API allows to efficiently pre-fetch data and separate the concerns. 
+- In the module API, `DataIter`s are responsible for both holding the data and iterating through it. Some `DataIter` support multi-threading like the [`ImageRecordIter`](https://mxnet.incubator.apache.org/api/python/io/io.html#mxnet.io.ImageRecordIter), while other don't like the `NDArrayIter`.
+
+You can checkout the [`Dataset` and `DataLoader` tutorial](https://mxnet.incubator.apache.org/tutorials/gluon/datasets.html). You can either rewrite your code in order to use one of the provided [`Dataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html?highlight=dataset#mxnet.gluon.data.Dataset) class, like the [`ArrayDataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html?highlight=arraydataset#mxnet.gluon.data.ArrayDataset) or the [`ImageFolderDataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html?highlight=imagefolderdataset#mxnet.gluon.data.vision.datasets.ImageFolderDataset), or you can simply wrap your existing [`DataIter`](https://mxnet.incubator.apache.org/api/python/io/io.html?highlight=dataiter#mxnet.io.DataIter) to have a similar usage pattern as a `DataLoader`:
+
+
+```python
+class DataIterLoader():
+    def __init__(self, data_iter):
+        self.data_iter = data_iter
+
+    def __iter__(self):
+        self.data_iter.reset()
+        return self
+
+    def __next__(self):
+        batch = self.data_iter.__next__()
+        assert len(batch.data) == len(batch.label) == 1
+        data = batch.data[0]
+        label = batch.label[0]
+        return data, label
+
+    def next(self):
+        return self.__next__() # for Python 2
+```
+
+
+```python
+data_iter = mx.io.NDArrayIter(data=train_data, label=train_label, batch_size=batch_size)
+data_iter_loader = DataIterLoader(data_iter)
+for data, label in data_iter_loader:
+    print(data.shape, label)
+    break
+```
+
+    (5, 28, 28) 
+    [5. 0. 3. 4. 9.]
+    <NDArray 5 @cpu(0)>
+
+
+## II - Model definition
+
+Let's look at the model definition from the [MNIST Module Tutorial](https://mxnet.incubator.apache.org/tutorials/python/mnist.html):
+
+
+```python
+ctx = mx.gpu()
+```
+
+#### Module
+
+For the Module API, you define the data flow by setting `data` keyword argument of one layer to the next.
+You then bind the symbolic model to a specific compute context and specify the symbol names for the data and the label.
+
+```python
+data = mx.sym.var('data')
+data = mx.sym.flatten(data=data)
+fc1  = mx.sym.FullyConnected(data=data, num_hidden=128)
+act1 = mx.sym.Activation(data=fc1, act_type="relu")
+fc2  = mx.sym.FullyConnected(data=act1, num_hidden = 64)
+act2 = mx.sym.Activation(data=fc2, act_type="relu")
+fc3  = mx.sym.FullyConnected(data=act2, num_hidden=10)
+mlp  = mx.sym.SoftmaxOutput(data=fc3, name='softmax')
+
+# Bind model to Module
+mlp_model = mx.mod.Module(symbol=mlp, context=ctx, data_names=['data'], label_names=['softmax_label'])
+```
+
+#### Gluon
+
+In Gluon, for a sequential model like that, you would create a `Sequential` block, in that case a `HybridSequential` block to allow for future hybridization since we are only using hybridizable blocks. Learn more [about hybridization](https://mxnet.incubator.apache.org/tutorials/gluon/hybrid.html). The flow of the data will be automatically set from one layer to the next, since they are held in a `Sequential` block.
+
+
+```python
+net = nn.HybridSequential()
+with net.name_scope():
+    net.add(
+        nn.Flatten(),
+        nn.Dense(units=128, activation="relu"),
+        nn.Dense(units=64, activation="relu"),
+        nn.Dense(units=10)
+    )
+```
+
+## III - Loss
+
+The loss, that you are trying to minimize using an optimization algorithm like SGD, is defined differently in the Module API than in Gluon.
+
+In the module API, the loss is part of the network. It has usually a forward pass result, that is the inference value, and a backward pass that is the gradient of the output with respect to that particular loss.
+
+For example the [sym.SoftmaxOutput](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html?highlight=softmaxout#mxnet.symbol.SoftmaxOutput) is a softmax output in the forward pass and the gradient with respect to the cross-entropy loss in the backward pass.
 
 Review comment:
   Sure that's a good idea

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services