You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/04/27 00:43:49 UTC
[GitHub] solin319 closed pull request #8423: Re-implement segnet in MXnet
solin319 closed pull request #8423: Re-implement segnet in MXnet
URL: https://github.com/apache/incubator-mxnet/pull/8423
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/example/segnet/README.md b/example/segnet/README.md
new file mode 100644
index 00000000000..efd4d9bbf17
--- /dev/null
+++ b/example/segnet/README.md
@@ -0,0 +1,108 @@
+# SegNet
+
+SegNet is a deep encoder-decoder architecture for multi-class pixelwise segmentation researched and developed by members of the [Computer Vision and Robotics Group](http://mi.eng.cam.ac.uk/Main/CVR) at the University of Cambridge, UK.
+
+This mxnet version reference a caffe version https://github.com/alexgkendall/SegNet-Tutorial .
+
+The segnet_basic and segnet networks were included in this mxnet version.
+
+## Requirement
+
+Need python package Pillow.
+```
+pip install Pillow
+```
+Build MXNet with new pooling and upsampling operators.
+```
+# copy new operators to src/operator/
+cp segnet/op/* incubator-mxnet/src/operator/
+# rebuild MXNet from source
+cd incubator-mxnet/
+make
+cd python/
+python setup.py install
+```
+
+## Dataset
+
+The model can be trained for road scene understanding using the [CamVid dataset](http://mi.eng.cam.ac.uk/research/projects/VideoRec/CamVid/). The Cambridge-driving Labeled Video Database (CamVid) is the first collection of videos with object class semantic labels, complete with metadata. The database provides ground truth labels that associate each pixel with one of [32 semantic classes](http://mi.eng.cam.ac.uk/research/projects/VideoRec/CamVid/#ClassLabels).
+
+This dataset is small, consisting of 367 training and 233 testing RGB images (day and dusk scenes) at
+360*480 resolution. The challenge is to segment 11 classes such
+as road, building, cars, pedestrians, signs, poles, side-walk etc.
+
+## Train
+
+- Getting Started
+
+ Install python package `Pillow` (required by `image_segment.py`).
+
+```
+[sudo] pip install Pillow
+```
+
+- train model
+```
+python train_segnet.py \
+--gpus=0,1,2,3 \
+--lr=0.005 \
+--wd=0.0005 \
+--network=segnet_basic \
+--batch-size=12
+```
+
+ The output log may look like this
+```
+Epoch[132] Batch [5] Speed: 14.15 samples/sec accuracy=0.903786
+Epoch[132] Batch [10] Speed: 14.80 samples/sec accuracy=0.913140
+Epoch[132] Batch [15] Speed: 13.62 samples/sec accuracy=0.909260
+Epoch[132] Batch [20] Speed: 14.09 samples/sec accuracy=0.893901
+Epoch[132] Batch [25] Speed: 14.74 samples/sec accuracy=0.911765
+Epoch[132] Train-accuracy=0.906672
+Epoch[132] Time cost=25.005
+Epoch[132] Validation-accuracy=0.819186
+
+```
+- Using the pre-trained model for image segmentation
+
+ It load parameters from model_prefix and load_epoch as normal.
+
+ We can also load convolution parameters from vgg16 model by use model_prefix_for_vgg16 and load_epoch_for_vgg16.
+- The accuracy can reach 0.84 with segnet_basic. When train segnet with vgg pre-trained parameters, the accuracy will reach about 0.89 .
+- Use bayesian+segnet+vgg_pretrained to train the model in the first 290 epoch and use segnet to train the last
+10 epoch, the accuracy can reach 0.91 .
+## Score
+
+- score the model
+
+```
+python score.py --batch-size=1 --load_epoch=270 --score_file=test.txt
+```
+
+The output log may look like this. Each line shows the global segment accuracy in one picture. The segment result can be saved in a package named "res_pic".
+
+```
+INFO:root:('accuracy', 0.80057751076185835)
+INFO:root:('accuracy', 0.80030805232087987)
+INFO:root:('accuracy', 0.80030805232087987)
+INFO:root:('accuracy', 0.80014792169887283)
+INFO:root:('accuracy', 0.80014792169887283)
+INFO:root:('accuracy', 0.80037548100048095)
+INFO:root:('accuracy', 0.80037548100048095)
+
+```
+
+## Segment one pic
+
+```
+python segment_one_pic.py \
+--image_name=0016E5_04530.png \
+--label_name=0016E5_04530.png \
+--load_epoch=270 \
+```
+
+The program will output segment result and label in a package named "res_pic".
+
+The left one is a label picture with 11 classes. The right one below is a segment result.
+
+ 
diff --git a/example/segnet/__init__.py b/example/segnet/__init__.py
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/example/segnet/common/__init__.py b/example/segnet/common/__init__.py
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/example/segnet/common/contrib_metrics.py b/example/segnet/common/contrib_metrics.py
new file mode 100644
index 00000000000..5bbf182ec28
--- /dev/null
+++ b/example/segnet/common/contrib_metrics.py
@@ -0,0 +1,157 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import numpy
+import mxnet as mx
+
+class Accuracy(mx.metric.EvalMetric):
+ """Computes accuracy classification score.
+
+ The accuracy score is defined as
+
+ .. math::
+ \\text{accuracy}(y, \\hat{y}) = \\frac{1}{n} \\sum_{i=0}^{n-1}
+ \\text{1}(\\hat{y_i} == y_i)
+
+ Parameters
+ ----------
+ axis : int, default=1
+ The axis that represents classes
+ name : str
+ Name of this metric instance for display.
+ output_names : list of str, or None
+ Name of predictions that should be used when updating with update_dict.
+ By default include all predictions.
+ label_names : list of str, or None
+ Name of labels that should be used when updating with update_dict.
+ By default include all labels.
+ ignore_label : int
+ Number of label that should not be computed.
+
+ Examples
+ --------
+ >>> predicts = [mx.nd.array([[0.3, 0.7], [0, 1.], [0.4, 0.6]])]
+ >>> labels = [mx.nd.array([0, 1, 1])]
+ >>> acc = mx.metric.Accuracy()
+ >>> acc.update(preds = predicts, labels = labels)
+ >>> print acc.get()
+ ('accuracy', 0.6666666666666666)
+ """
+ def __init__(self, axis=1, name='accuracy',
+ output_names=None, label_names=None, ignore_label=-1):
+ super(Accuracy, self).__init__(
+ name, axis=axis,
+ output_names=output_names, label_names=label_names)
+ self.axis = axis
+ self.ignore_label = ignore_label
+
+ def update(self, labels, preds):
+ """Updates the internal evaluation result.
+
+ Parameters
+ ----------
+ labels : list of `NDArray`
+ The labels of the data.
+
+ preds : list of `NDArray`
+ Predicted values.
+ """
+ mx.metric.check_label_shapes(labels, preds)
+
+ for label, pred_label in zip(labels, preds):
+ if pred_label.shape != label.shape:
+ pred_label = mx.ndarray.argmax(pred_label, axis=self.axis)
+ pred_label = pred_label.asnumpy().astype('int32')
+ label = label.asnumpy().astype('int32')
+
+ mx.metric.check_label_shapes(label, pred_label)
+
+ self.sum_metric += (pred_label.flat == label.flat).sum()
+ self.num_inst += len(pred_label.flat)
+ self.sum_metric -= len(pred_label[pred_label == self.ignore_label])
+ self.num_inst -= len(label[label == self.ignore_label])
+
+class CrossEntropy(mx.metric.EvalMetric):
+ """Computes Cross Entropy loss.
+
+ The cross entropy over a batch of sample size :math:`N` is given by
+
+ .. math::
+ -\\sum_{n=1}^{N}\\sum_{k=1}^{K}t_{nk}\\log (y_{nk}),
+
+ where :math:`t_{nk}=1` if and only if sample :math:`n` belongs to class :math:`k`.
+ :math:`y_{nk}` denotes the probability of sample :math:`n` belonging to
+ class :math:`k`.
+
+ Parameters
+ ----------
+ eps : float
+ Cross Entropy loss is undefined for predicted value is 0 or 1,
+ so predicted values are added with the small constant.
+ name : str
+ Name of this metric instance for display.
+ output_names : list of str, or None
+ Name of predictions that should be used when updating with update_dict.
+ By default include all predictions.
+ label_names : list of str, or None
+ Name of labels that should be used when updating with update_dict.
+ By default include all labels.
+
+ Examples
+ --------
+ >>> predicts = [mx.nd.array([[0.3, 0.7], [0, 1.], [0.4, 0.6]])]
+ >>> labels = [mx.nd.array([0, 1, 1])]
+ >>> ce = mx.metric.CrossEntropy()
+ >>> ce.update(labels, predicts)
+ >>> print ce.get()
+ ('cross-entropy', 0.57159948348999023)
+ """
+ def __init__(self, eps=1e-12, name='cross-entropy',
+ output_names=None, label_names=None):
+ super(CrossEntropy, self).__init__(
+ name, eps=eps,
+ output_names=output_names, label_names=label_names)
+ self.eps = eps
+
+ def update(self, labels, preds):
+ """Updates the internal evaluation result.
+
+ Parameters
+ ----------
+ labels : list of `NDArray`
+ The labels of the data.
+
+ preds : list of `NDArray`
+ Predicted values.
+ """
+ mx.metric.check_label_shapes(labels, preds)
+ for label, pred in zip(labels, preds):
+ if len(pred.shape) > 2:
+ for i in range(len(pred.shape) - 1, 1, -1):
+ pred = pred.swapaxes(1, i)
+ pred = pred.reshape((-1, pred.shape[-1]))
+ label = label.asnumpy()
+ pred = pred.asnumpy()
+
+ label = label.ravel()
+ assert label.shape[0] == pred.size / pred.shape[-1]
+ while label.max() >= pred.shape[1]:
+ pred = numpy.append(pred, numpy.ones((pred.shape[0], 1)), 1)
+ prob = pred[numpy.arange(label.shape[0]), numpy.int64(label)]
+ self.sum_metric += (-numpy.log(prob + self.eps)).sum()
+ self.num_inst += label.shape[0]
+
\ No newline at end of file
diff --git a/example/segnet/data_iter.py b/example/segnet/data_iter.py
new file mode 100644
index 00000000000..066f46a0e7e
--- /dev/null
+++ b/example/segnet/data_iter.py
@@ -0,0 +1,168 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: skip-file
+""" file iterator for pasval voc 2012"""
+
+import mxnet as mx
+import numpy as np
+import random
+import sys, os
+from mxnet.io import DataIter
+
+class FileIter(DataIter):
+ """FileIter object in fcn-xs example. Taking a file list file to get dataiter.
+ in this example, we use the whole image training for fcn-xs, that is to say
+ we do not need resize/crop the image to the same size, so the batch_size is
+ set to 1 here
+ Parameters
+ ----------
+ batch_size : int
+ Number of examples per batch.
+ root_dir : string
+ the root dir of image/label lie in
+ flist_name : string
+ the list file of iamge and label, every line owns the form:
+ index \t image_data_path \t image_label_path
+ cut_off_size : int
+ if the maximal size of one image is larger than cut_off_size, then it will
+ crop the image with the minimal size of that image
+ data_name : string
+ the data name used in symbol data(default data name)
+ label_name : string
+ the label name used in symbol softmax_label(default label name)
+ data_shape : tuple
+ Data shape in (channels, height, width) format.
+ For now, only RGB image with 3 channels is supported.
+ part_index : int
+ Partition index.
+ num_parts : int
+ Total number of partitions.
+ shuffle : bool
+ Shuffle the data rank.
+ """
+ def __init__(self, batch_size, root_dir, flist_name,
+ rgb_mean = (117, 117, 117),
+ cut_off_size = None,
+ data_name = "data",
+ label_name = "softmax_label",
+ data_shape = (3, 360, 480),
+ part_index = 0,
+ num_parts = 1,
+ shuffle = True):
+ super(FileIter, self).__init__()
+ self.root_dir = root_dir
+ self.flist_name = os.path.join(self.root_dir, flist_name)
+ self.mean = np.array(rgb_mean) # (R, G, B)
+ self.cut_off_size = cut_off_size
+ self.data_name = data_name
+ self.label_name = label_name
+
+ self.num_data = len(open(self.flist_name, 'r').readlines())
+ self.f = open(self.flist_name, 'r')
+ self.names = self.f.read().split("\n")[0:-1]
+ self.cursor = part_index
+ self.num_parts = num_parts
+ self.part_index = part_index
+ self.batch_size = batch_size
+ self.data_shape = data_shape
+ self.shuffle = shuffle
+ self.data, self.label = self._read()
+
+ def _read(self):
+ """get two list, each list contains two elements: name and nd.array value"""
+ #img_name= self.f.readline().strip('\n').split("\t")[0].split(' ')
+ img_name = self.names[self.cursor + self.num_parts].split(' ')
+ data = {}
+ label = {}
+ data[self.data_name], label[self.label_name] = self._read_img(img_name[0], img_name[1])
+ return list(data.items()), list(label.items())
+
+ def _read_img(self, img_name, label_name):
+ img = mx.image.imread(os.path.join(self.root_dir, img_name))
+ label = mx.image.imread(os.path.join(self.root_dir, label_name), flag=0)
+ img = img.astype('float32')
+ label = label.astype('float32')
+ if self.cut_off_size is not None:
+ max_hw = max(img.shape[0], img.shape[1])
+ min_hw = min(img.shape[0], img.shape[1])
+ if min_hw > self.cut_off_size:
+ rand_start_max = int(np.random.uniform(0, max_hw - self.cut_off_size - 1))
+ rand_start_min = int(np.random.uniform(0, min_hw - self.cut_off_size - 1))
+ if img.shape[0] == max_hw :
+ img = img[rand_start_max : rand_start_max + self.cut_off_size, rand_start_min : rand_start_min + self.cut_off_size]
+ label = label[rand_start_max : rand_start_max + self.cut_off_size, rand_start_min : rand_start_min + self.cut_off_size]
+ else :
+ img = img[rand_start_min : rand_start_min + self.cut_off_size, rand_start_max : rand_start_max + self.cut_off_size]
+ label = label[rand_start_min : rand_start_min + self.cut_off_size, rand_start_max : rand_start_max + self.cut_off_size]
+ elif max_hw > self.cut_off_size:
+ rand_start = int(np.random.uniform(0, max_hw - min_hw - 1))
+ if img.shape[0] == max_hw :
+ img = img[rand_start : rand_start + min_hw, :]
+ label = label[rand_start : rand_start + min_hw, :]
+ else :
+ img = img[:, rand_start : rand_start + min_hw]
+ label = label[:, rand_start : rand_start + min_hw]
+ reshaped_mean = self.mean.reshape((1, 1, 3))
+ img = img - reshaped_mean
+ img = mx.nd.moveaxis(img, 2, 0)
+ img = mx.nd.expand_dims(img, axis=0)
+ label = mx.nd.moveaxis(label, 2, 0)
+ return (img, label)
+
+ @property
+ def provide_data(self):
+ """The name and shape of data provided by this iterator"""
+ return [(k, tuple([self.batch_size] + list(v.shape[1:]))) for k, v in self.data]
+
+ @property
+ def provide_label(self):
+ """The name and shape of label provided by this iterator"""
+ return [(k, tuple([self.batch_size] + list(v.shape[1:]))) for k, v in self.label]
+
+ def get_batch_size(self):
+ return self.batch_size
+
+ def reset(self):
+ self.cursor = self.part_index
+ #self.f.close()
+ #self.f = open(self.flist_name, 'r')
+ if self.shuffle:
+ random.shuffle(self.names)
+
+ def iter_next(self):
+ self.cursor += self.num_parts
+ if(self.cursor < self.num_data-1):
+ return True
+ else:
+ return False
+
+ def next(self):
+ """return one dict which contains "data" and "label" """
+ c, h, w = self.data_shape
+ batch_data = mx.ndarray.empty((self.batch_size, c, h, w))
+ batch_label = mx.ndarray.empty((self.batch_size, h, w))
+ i = 0
+ while i < self.batch_size:
+ if self.iter_next():
+ self.data, self.label = self._read()
+ batch_data[i][:] = self.data[0][1][0]
+ batch_label[i][:] = self.label[0][1][0]
+ i += 1
+ else:
+ raise StopIteration
+ return mx.io.DataBatch([batch_data], [batch_label], pad = self.batch_size - i)
diff --git a/example/segnet/op/pooling_mask_max_2d-inl.h b/example/segnet/op/pooling_mask_max_2d-inl.h
new file mode 100644
index 00000000000..18ac9587574
--- /dev/null
+++ b/example/segnet/op/pooling_mask_max_2d-inl.h
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file pooling_mask_max_2d-inl.h
+ * \brief
+ * \author Pengfei Li
+*/
+
+#ifndef MXNET_OPERATOR_POOLING_MASK_MAX_2D_INL_H_
+#define MXNET_OPERATOR_POOLING_MASK_MAX_2D_INL_H_
+
+#include <dmlc/logging.h>
+#include <dmlc/parameter.h>
+#include <mxnet/operator.h>
+#include <algorithm>
+#include <map>
+#include <vector>
+#include <string>
+#include <utility>
+#include "./operator_common.h"
+
+namespace mxnet {
+namespace op {
+
+namespace pooling_mask_enum {
+enum PoolingMaskOpInputs {kData};
+enum PoolingMaskOpOutputs {kOut, kMask};
+enum PoolingOpPadConventionType {kValid, kFull};
+} // namespace pooling_mask_enum
+
+struct PoolingMaskParam : public dmlc::Parameter<PoolingMaskParam> {
+ TShape kernel;
+ TShape stride;
+ TShape pad;
+ int pooling_convention;
+ DMLC_DECLARE_PARAMETER(PoolingMaskParam) {
+ DMLC_DECLARE_FIELD(kernel)
+ .enforce_nonzero()
+ .describe("pooling kernel size: (y, x) or (d, y, x)");
+
+ DMLC_DECLARE_FIELD(stride).set_default(TShape())
+ .enforce_nonzero()
+ .describe("stride: for pooling (y, x) or (d, y, x)");
+
+ DMLC_DECLARE_FIELD(pad).set_default(TShape())
+ .describe("pad for pooling: (y, x) or (d, y, x)");
+
+ DMLC_DECLARE_FIELD(pooling_convention).set_default(pooling_mask_enum::kValid)
+ .add_enum("full", pooling_mask_enum::kFull)
+ .add_enum("valid", pooling_mask_enum::kValid)
+ .describe("Pooling convention to be applied.");
+ }
+};
+
+template<typename xpu, typename DType>
+class PoolingMaskOp : public Operator {
+ public:
+ explicit PoolingMaskOp(PoolingMaskParam p) {
+ this->param_ = p;
+ }
+
+ virtual void Forward(const OpContext& ctx,
+ const std::vector<TBlob>& in_data,
+ const std::vector<OpReqType>& req,
+ const std::vector<TBlob>& out_data,
+ const std::vector<TBlob>& aux_args) {
+ using namespace mshadow;
+ CHECK_EQ(in_data.size(), 1U);
+ CHECK_EQ(out_data.size(), 2U);
+ Stream<xpu> *s = ctx.get_stream<xpu>();
+ pool_mask_forward(s, in_data[pooling_mask_enum::kData].dptr<DType>(),
+ out_data[pooling_mask_enum::kOut].dptr<DType>(),
+ out_data[pooling_mask_enum::kMask].dptr<int>(),
+ in_data[pooling_mask_enum::kData].shape_,
+ out_data[pooling_mask_enum::kOut].shape_,
+ param_.kernel, param_.stride, param_.pad);
+ }
+
+ virtual void Backward(const OpContext& ctx,
+ const std::vector<TBlob>& out_grad,
+ const std::vector<TBlob>& in_data,
+ const std::vector<TBlob>& out_data,
+ const std::vector<OpReqType>& req,
+ const std::vector<TBlob>& in_grad,
+ const std::vector<TBlob>& aux_args) {
+ using namespace mshadow;
+ CHECK_EQ(out_grad.size(), 2U);
+ CHECK_EQ(in_data.size(), 1U);
+ CHECK_EQ(out_data.size(), 2U);
+ CHECK_EQ(req.size(), 1U);
+ CHECK_EQ(in_grad.size(), 1U);
+ Stream<xpu> *s = ctx.get_stream<xpu>();
+ Tensor<xpu, 4, DType> grad_in = in_grad[pooling_mask_enum::kData].get<xpu, 4, DType>(s);
+ if (kWriteTo == req[pooling_mask_enum::kData]) {
+ grad_in = 0.0f;
+ }
+ pool_mask_backward(s, in_grad[pooling_mask_enum::kData].dptr<DType>(),
+ out_grad[pooling_mask_enum::kData].dptr<DType>(),
+ out_data[pooling_mask_enum::kMask].dptr<int>(),
+ in_grad[pooling_mask_enum::kData].shape_,
+ out_grad[pooling_mask_enum::kOut].shape_,
+ param_.kernel, param_.stride, param_.pad);
+ }
+ private:
+ PoolingMaskParam param_;
+ void pool_mask_forward(mshadow::Stream<cpu>* s,
+ const DType* in_data, DType* out_data, int* mask,
+ const TShape& ishape, const TShape& oshape,
+ const TShape& kernel, const TShape& stride, const TShape& pad);
+
+ void pool_mask_backward(mshadow::Stream<cpu>* s,
+ DType* in_grad, const DType* out_grad, const int* mask,
+ const TShape& ishape, const TShape& oshape,
+ const TShape& kernel, const TShape& stride, const TShape& pad);
+
+#if MXNET_USE_CUDA
+ void pool_mask_forward(mshadow::Stream<gpu>* s,
+ const DType* in_data, DType* out_data, int* mask,
+ const TShape& ishape, const TShape& oshape,
+ const TShape& kernel, const TShape& stride, const TShape& pad);
+ void pool_mask_backward(mshadow::Stream<gpu>* s,
+ DType* in_grad, const DType* out_grad, const int* mask,
+ const TShape& ishape, const TShape& oshape,
+ const TShape& kernel, const TShape& stride, const TShape& pad);
+#endif // MXNET_USE_CUDA
+}; // class PoolingMaskOp
+
+// Decalre Factory function, used for dispatch specialization
+template<typename xpu>
+Operator* CreateOp(PoolingMaskParam param, int dtype);
+
+#if DMLC_USE_CXX11
+class PoolingMaskProp : public OperatorProperty {
+ public:
+
+ void Init(const std::vector<std::pair<std::string, std::string> >& kwargs) override {
+ using namespace mshadow;
+ param_.Init(kwargs);
+ if (param_.kernel.ndim() == 1) {
+ if (param_.stride.ndim() == 0) param_.stride = Shape1(1);
+ if (param_.pad.ndim() == 0) param_.pad = Shape1(0);
+ } else if (param_.kernel.ndim() == 2) {
+ if (param_.stride.ndim() == 0) param_.stride = Shape2(1, 1);
+ if (param_.pad.ndim() == 0) param_.pad = Shape2(0, 0);
+ } else {
+ CHECK_EQ(param_.kernel.ndim(), 3U) << param_.kernel.ndim() << "D pooling not supported";
+ if (param_.stride.ndim() == 0) param_.stride = Shape3(1, 1, 1);
+ if (param_.pad.ndim() == 0) param_.pad = Shape3(0, 0, 0);
+ }
+ CHECK_EQ(param_.stride.ndim(), param_.kernel.ndim())
+ << "stride and kernel should have the same length";
+ CHECK_EQ(param_.pad.ndim(), param_.kernel.ndim())
+ << "pad and kernel should have the same length";
+ }
+
+ std::vector<std::string> ListOutputs() const override {
+ return {"output", "mask"};
+ }
+
+ int NumOutputs() const override {
+ return 2;
+ }
+
+ int NumVisibleOutputs() const override {
+ return 2;
+ }
+
+ std::map<std::string, std::string> GetParams() const override {
+ return param_.__DICT__();
+ }
+
+ bool InferShape(std::vector<TShape> *in_shape,
+ std::vector<TShape> *out_shape,
+ std::vector<TShape> *aux_shape) const override {
+ CHECK_EQ(in_shape->size(), 1U);
+ const TShape &dshape = (*in_shape)[0];
+ CHECK_GE(dshape.ndim(), 3U) << "Pooling: Input data should be 3D in (batch, channel, x)"
+ << " Or 4D in (batch, channel, y, x) "
+ << " Or 5D in (batch, channel, d, y, x)";
+ TShape oshape = dshape;
+ if (dshape.ndim() == 0) return false;
+ if (param_.kernel.ndim() == 2) {
+ CHECK_EQ(dshape.ndim(), 4U) << "Pooling: Input data should be 4D in (batch, channel, y, x)";
+
+ CHECK(param_.kernel[0] <= dshape[2] + 2 * param_.pad[0])
+ << "kernel size (" << param_.kernel[0] << ") exceeds input (" << dshape[2]
+ << " padded to " << (dshape[2] + 2*param_.pad[0]) << ")";
+ CHECK(param_.kernel[1] <= dshape[3] + 2 * param_.pad[1])
+ << "kernel size (" << param_.kernel[1] << ") exceeds input (" << dshape[3]
+ << " padded to " << (dshape[3] + 2*param_.pad[1]) << ")";
+ if (param_.pooling_convention == pooling_mask_enum::kValid) {
+ oshape[2] = 1 + (dshape[2] + 2 * param_.pad[0] - param_.kernel[0]) /
+ param_.stride[0];
+ oshape[3] = 1 + (dshape[3] + 2 * param_.pad[1] - param_.kernel[1]) /
+ param_.stride[1];
+ } else {
+ oshape[2] = 1 + static_cast<int>(ceil(static_cast<float>(
+ dshape[2] + 2 * param_.pad[0] -
+ param_.kernel[0]) / param_.stride[0]));
+ oshape[3] = 1 + static_cast<int>(ceil(static_cast<float>(
+ dshape[3] + 2 * param_.pad[1] -
+ param_.kernel[1]) / param_.stride[1]));
+ }
+
+ out_shape->clear();
+ out_shape->push_back(oshape); // save output shape
+ out_shape->push_back(oshape); // save output shape
+ } else {
+ LOG(FATAL) << "Not Implemented.";
+ }
+ return true;
+ }
+
+ bool InferType(std::vector<int> *in_type,
+ std::vector<int> *out_type,
+ std::vector<int> *aux_type) const override {
+ CHECK_EQ(in_type->size(), 1U);
+ int dtype = (*in_type)[0];
+
+ if (dtype == -1) {
+ LOG(FATAL) << "Input type to pooling is not specified.";
+ return false;
+ }
+
+ out_type->clear();
+ out_type->push_back(dtype);
+ out_type->push_back(mshadow::TypeFlag::kInt32);
+ return true;
+ }
+
+ OperatorProperty* Copy() const override {
+ PoolingMaskProp *prop_sym = new PoolingMaskProp();
+ prop_sym->param_ = this->param_;
+ return prop_sym;
+ }
+
+ std::string TypeString() const override {
+ return "PoolingMask";
+ }
+
+ std::vector<int> DeclareBackwardDependency(
+ const std::vector<int> &out_grad,
+ const std::vector<int> &in_data,
+ const std::vector<int> &out_data) const override {
+ return {out_grad[pooling_mask_enum::kOut], in_data[pooling_mask_enum::kData],
+ out_data[pooling_mask_enum::kOut], out_data[pooling_mask_enum::kMask]};
+ }
+ /*
+ std::vector<std::pair<int, void*> > BackwardInplaceOption(
+ const std::vector<int> &out_grad,
+ const std::vector<int> &in_data,
+ const std::vector<int> &out_data,
+ const std::vector<void*> &in_grad) const override {
+ return {{in_data[pooling_mask_enum::kData], in_grad[pooling_mask_enum::kData]}};
+ }
+ */
+ std::vector<ResourceRequest> ForwardResource(
+ const std::vector<TShape> &in_shape) const override {
+ return {ResourceRequest::kTempSpace};
+ }
+
+ std::vector<ResourceRequest> BackwardResource(
+ const std::vector<TShape> &in_shape) const override {
+ return {ResourceRequest::kTempSpace};
+ }
+
+ Operator* CreateOperator(Context ctx) const override {
+ LOG(FATAL) << "Not Implemented.";
+ return NULL;
+ }
+
+ Operator* CreateOperatorEx(Context ctx, std::vector<TShape> *in_shape,
+ std::vector<int> *in_type) const override;
+ private:
+ PoolingMaskParam param_;
+}; // class PoolingMaskProp
+#endif
+} // namespace op
+} // namespace mxnet
+
+#endif // MXNET_OPERATOR_POOLING_INL_H_
\ No newline at end of file
diff --git a/example/segnet/op/pooling_mask_max_2d.cc b/example/segnet/op/pooling_mask_max_2d.cc
new file mode 100644
index 00000000000..85fa24b8b04
--- /dev/null
+++ b/example/segnet/op/pooling_mask_max_2d.cc
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file pooling_mask_max_2d.cc
+ * \brief
+ * \author Pengfei Li
+*/
+
+#include <mxnet/base.h>
+#include <mxnet/operator.h>
+#include <algorithm>
+#include "./mxnet_op.h"
+#include "./pooling_mask_max_2d-inl.h"
+
+namespace mxnet {
+namespace op {
+
+template<typename xpu, typename DType>
+void PoolingMaskOp<xpu, DType>::pool_mask_forward(mshadow::Stream<cpu>* s,
+ const DType* in_data, DType* out_data, int* mask,
+ const TShape& ishape, const TShape& oshape,
+ const TShape& kernel, const TShape& stride, const TShape& pad) {
+ using mshadow::red::limits::MinValue;
+ const int height = ishape[2], width = ishape[3];
+ const int pooled_height = oshape[2], pooled_width = oshape[3];
+ const int kernel_h = kernel[0], kernel_w = kernel[1];
+ const int pad_h = pad[0], pad_w = pad[1];
+ const int stride_h = stride[0], stride_w = stride[1];
+ const index_t in_data_offset = ishape[2] * ishape[3];
+ const index_t out_data_offset = oshape[2] * oshape[3];
+ for (index_t n = 0; n < oshape[0]; ++n) {
+ for (index_t c = 0; c < oshape[1]; ++c) {
+ for (int ph = 0; ph < pooled_height; ++ph) {
+ for (int pw = 0; pw < pooled_width; ++pw) {
+ int hstart = ph * stride_h - pad_h;
+ int wstart = pw * stride_w - pad_w;
+ int hend = std::min(hstart + kernel_h, height);
+ int wend = std::min(wstart + kernel_w, width);
+ hstart = std::max(hstart, 0);
+ wstart = std::max(wstart, 0);
+ const int pool_index = ph * pooled_width + pw;
+ DType max_val = MinValue<DType>();
+ for (int h = hstart; h < hend; ++h) {
+ for (int w = wstart; w < wend; ++w) {
+ const int in_index = h * width + w;
+ if (in_data[in_index] > max_val) {
+ max_val = in_data[in_index];
+ mask[pool_index] = in_index;
+ }
+ }
+ }
+ out_data[pool_index] = max_val;
+ }
+ }
+ in_data += in_data_offset;
+ out_data += out_data_offset;
+ mask += out_data_offset;
+ }
+ }
+}
+
+template<typename xpu, typename DType>
+void PoolingMaskOp<xpu, DType>::pool_mask_backward(mshadow::Stream<cpu>* s,
+ DType* in_grad, const DType* out_grad, const int* mask,
+ const TShape& ishape, const TShape& oshape,
+ const TShape& kernel, const TShape& stride, const TShape& pad) {
+ const index_t in_offset = ishape[2] * ishape[3];
+ const index_t out_offset = oshape[2] * oshape[3];
+ for (index_t n = 0; n < oshape[0]; ++n) {
+ for (index_t c = 0; c < oshape[1]; ++c) {
+ for (int ph = 0; ph < oshape[2]; ++ph) {
+ for (int pw = 0; pw < oshape[3]; ++pw) {
+ const int pool_index = ph * oshape[3] + pw;
+ const int max_idx = mask[pool_index];
+ in_grad[max_idx] += out_grad[pool_index];
+ }
+ }
+ in_grad += in_offset;
+ out_grad += out_offset;
+ mask += out_offset;
+ }
+ }
+}
+
+template<>
+Operator *CreateOp<cpu>(PoolingMaskParam param, int dtype) {
+ Operator *op = NULL;
+ MSHADOW_REAL_TYPE_SWITCH(dtype, DType, {
+ op = new PoolingMaskOp<cpu, DType>(param);
+ });
+ return op;
+}
+
+// DO_BIND_DISPATCH comes from operator_common.h
+Operator* PoolingMaskProp::CreateOperatorEx(Context ctx, std::vector<TShape> *in_shape,
+ std::vector<int> *in_type) const {
+ std::vector<TShape> out_shape, aux_shape;
+ std::vector<int> out_type, aux_type;
+ CHECK(InferType(in_type, &out_type, &aux_type));
+ CHECK(InferShape(in_shape, &out_shape, &aux_shape));
+ DO_BIND_DISPATCH(CreateOp, param_, (*in_type)[0]);
+}
+
+DMLC_REGISTER_PARAMETER(PoolingMaskParam);
+
+MXNET_REGISTER_OP_PROPERTY(PoolingMask, PoolingMaskProp)
+.describe(R"code(Performs max 2d pooling on the input.
+
+The shapes for 2-D pooling are
+
+- **data**: *(batch_size, channel, height, width)*
+- **mask**: *(batch_size, num_filter, out_height, out_width)*
+- **out**: *(batch_size, num_filter, out_height, out_width)*, with::
+
+ out_height = f(height, kernel[0], pad[0], stride[0])
+ out_width = f(width, kernel[1], pad[1], stride[1])
+
+The definition of *f* depends on ``pooling_convention``, which has two options:
+
+- **valid** (default)::
+
+ f(x, k, p, s) = floor((x+2*p-k)/s)+1
+
+- **full**, which is compatible with Caffe::
+
+ f(x, k, p, s) = ceil((x+2*p-k)/s)+1
+
+Only max pooling is supported.
+
+)code" ADD_FILELINE)
+.add_argument("data", "NDArray-or-Symbol", "Input data to the pooling operator.")
+.add_arguments(PoolingMaskParam::__FIELDS__());
+} // namespace op
+} // namespace mxnet
\ No newline at end of file
diff --git a/example/segnet/op/pooling_mask_max_2d.cu b/example/segnet/op/pooling_mask_max_2d.cu
new file mode 100644
index 00000000000..42115d628a1
--- /dev/null
+++ b/example/segnet/op/pooling_mask_max_2d.cu
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file pooling_mask_max_2d.cu
+ * \brief
+ * \author Pengfei Li
+*/
+
+#include <vector>
+#include <mxnet/base.h>
+#include <mxnet/operator.h>
+#include "mxnet_op.h"
+#include "../common/cuda_utils.h"
+#include "./pooling_mask_max_2d-inl.h"
+
+namespace mxnet {
+namespace op {
+
+template <typename DType>
+__global__ void pool_max_2d_gpu_kernel(const int nthreads, const DType* in_data,
+ const int channels, const int height, const int width,
+ const int pooled_height, const int pooled_width,
+ const int kernel_h, const int kernel_w, const int stride_h,
+ const int stride_w, const int pad_h, const int pad_w,
+ DType* out_data, int* mask) {
+ using mshadow::red::limits::MinValue;
+ // index is the output image's pixel index in NCHW
+ CUDA_KERNEL_LOOP(index, nthreads) {
+ const int pw = index % pooled_width;
+ const int ph = (index / pooled_width) % pooled_height;
+ const int c = (index / pooled_width / pooled_height) % channels;
+ const int n = index / pooled_width / pooled_height / channels;
+ int hstart = ph * stride_h - pad_h;
+ int wstart = pw * stride_w - pad_w;
+ const int hend = min(hstart + kernel_h, height);
+ const int wend = min(wstart + kernel_w, width);
+ hstart = max(hstart, 0);
+ wstart = max(wstart, 0);
+ const DType* in_slice =
+ in_data + (n * channels + c) * height * width;
+ DType max_val = MinValue<DType>();
+ int max_index = 0;
+ for (int h = hstart; h < hend; ++h) {
+ for (int w = wstart; w < wend; ++w) {
+ const int cur_index = h * width + w;
+ const DType in_val = in_slice[cur_index];
+ if (in_val > max_val) {
+ max_val = in_val;
+ max_index = cur_index;
+ }
+ }
+ }
+ out_data[index] = max_val;
+ mask[index] = max_index;
+ }
+}
+
+template <typename DType>
+__global__ void unpool_max_2d_gpu_kernel(const int nthreads, const DType* out_grad,
+ const int channels, const int height, const int width,
+ const int pooled_height, const int pooled_width,
+ const int kernel_h, const int kernel_w,
+ const int stride_h, const int stride_w,
+ const int pad_h, const int pad_w,
+ DType* in_grad, const int* mask) {
+ // index is the output image's pixel index in NCHW
+ // the order has to be consistent with pooling max
+ // to avoid adding out_grad to the wrong in_grad
+ // in the case where there are multiple max pixels
+ // covered by a kernel window
+ CUDA_KERNEL_LOOP(index, nthreads) {
+ const int c = (index / pooled_width / pooled_height) % channels;
+ const int n = index / pooled_width / pooled_height / channels;
+ // in data/grad offset batch and channel dims
+ int in_offset = (n * channels + c) * height * width;
+ int max_idx = mask[index];
+
+ // In the case where pad > 0 and kernel = 1, for example,
+ // max_idx can be -1 reaching this step.
+ if (max_idx >= 0) {
+ atomicAdd(&in_grad[in_offset+max_idx], out_grad[index]);
+ }
+ }
+}
+
+template<typename xpu, typename DType>
+void PoolingMaskOp<xpu, DType>::pool_mask_forward(mshadow::Stream<gpu>* s,
+ const DType* in_data, DType* out_data, int* mask,
+ const TShape& ishape, const TShape& oshape,
+ const TShape& kernel, const TShape& stride, const TShape& pad) {
+ using namespace mxnet_op;
+ pool_max_2d_gpu_kernel<<<cuda_get_num_blocks(oshape.Size()), mshadow::cuda::kBaseThreadNum,
+ 0, mshadow::Stream<gpu>::GetStream(s)>>>(
+ oshape.Size(), in_data, ishape[1], ishape[2], ishape[3],
+ oshape[2], oshape[3], kernel[0], kernel[1],
+ stride[0], stride[1], pad[0], pad[1], out_data, mask);
+ MSHADOW_CUDA_POST_KERNEL_CHECK(pool_max_2d_gpu_kernel);
+}
+
+template<typename xpu, typename DType>
+void PoolingMaskOp<xpu, DType>::pool_mask_backward(mshadow::Stream<gpu>* s,
+ DType* in_grad, const DType* out_grad, const int* mask,
+ const TShape& ishape, const TShape& oshape,
+ const TShape& kernel, const TShape& stride, const TShape& pad) {
+ using namespace mxnet_op;
+ unpool_max_2d_gpu_kernel<<<cuda_get_num_blocks(oshape.Size()), mshadow::cuda::kBaseThreadNum,
+ 0, mshadow::Stream<gpu>::GetStream(s)>>>(
+ oshape.Size(), out_grad,
+ ishape[1], ishape[2], ishape[3],
+ oshape[2], oshape[3], kernel[0], kernel[1],
+ stride[0], stride[1], pad[0], pad[1], in_grad, mask);
+ MSHADOW_CUDA_POST_KERNEL_CHECK(unpool_max_2d_gpu_kernel);
+}
+
+template<>
+Operator *CreateOp<gpu>(PoolingMaskParam param, int dtype) {
+ Operator *op = NULL;
+ MSHADOW_REAL_TYPE_SWITCH(dtype, DType, {
+ op = new PoolingMaskOp<gpu, DType>(param);
+ });
+ return op;
+}
+
+} // namespace op
+} // namespace mxnet
diff --git a/example/segnet/op/upsampling_mask_max_2d-inl.h b/example/segnet/op/upsampling_mask_max_2d-inl.h
new file mode 100644
index 00000000000..6ba3cb8907c
--- /dev/null
+++ b/example/segnet/op/upsampling_mask_max_2d-inl.h
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file upsampling_mask_max_2d-inl.h
+ * \brief
+ * \author Pengfei Li
+*/
+
+#ifndef MXNET_OPERATOR_UPSAMPLING_MASK_MAX_2D_INL_H_
+#define MXNET_OPERATOR_UPSAMPLING_MASK_MAX_2D_INL_H_
+
+#include <dmlc/logging.h>
+#include <dmlc/parameter.h>
+#include <mxnet/operator.h>
+#include <algorithm>
+#include <map>
+#include <vector>
+#include <string>
+#include <utility>
+#include "./operator_common.h"
+
+namespace mxnet {
+namespace op {
+
+namespace upsampling_mask_enum {
+enum UpSamplingMaskOpInputs {kData, kMask};
+enum UpSamplingMaskOpOutputs {kOut};
+} // namespace upsampling_mask_enum
+
+struct UpSamplingMaskParam : public dmlc::Parameter<UpSamplingMaskParam> {
+ TShape out_shape;
+ DMLC_DECLARE_PARAMETER(UpSamplingMaskParam) {
+ DMLC_DECLARE_FIELD(out_shape)
+ .enforce_nonzero()
+ .describe("upsampling output size: (y, x)");
+ }
+}; // struct UpSamplingMaskParam
+
+template<typename xpu, typename DType>
+class UpSamplingMaskOp : public Operator {
+ public:
+ explicit UpSamplingMaskOp(UpSamplingMaskParam p) {
+ this->param_ = p;
+ }
+
+ virtual void Forward(const OpContext &ctx,
+ const std::vector<TBlob> &in_data,
+ const std::vector<OpReqType> &req,
+ const std::vector<TBlob> &out_data,
+ const std::vector<TBlob> &aux_args) {
+ using namespace mshadow;
+ CHECK_EQ(in_data.size(), 2U);
+ CHECK_EQ(out_data.size(), 1U);
+ Stream<xpu> *s = ctx.get_stream<xpu>();
+ Tensor<xpu, 4, DType> data_out = out_data[upsampling_mask_enum::kData].get<xpu, 4, DType>(s);
+ data_out = 0.0f;
+ upsample_mask_forward(s, in_data[upsampling_mask_enum::kData].dptr<DType>(),
+ out_data[upsampling_mask_enum::kOut].dptr<DType>(),
+ in_data[upsampling_mask_enum::kMask].dptr<int>(),
+ in_data[upsampling_mask_enum::kData].shape_,
+ out_data[upsampling_mask_enum::kOut].shape_);
+ }
+
+ virtual void Backward(const OpContext &ctx,
+ const std::vector<TBlob> &out_grad,
+ const std::vector<TBlob> &in_data,
+ const std::vector<TBlob> &out_data,
+ const std::vector<OpReqType> &req,
+ const std::vector<TBlob> &in_grad,
+ const std::vector<TBlob> &aux_args) {
+ using namespace mshadow;
+ CHECK_EQ(out_grad.size(), 1U);
+ CHECK_EQ(in_data.size(), 2U);
+ CHECK_EQ(out_data.size(), 1U);
+ CHECK_EQ(req.size(), 2U);
+ CHECK_EQ(in_grad.size(), 2U);
+ Stream<xpu> *s = ctx.get_stream<xpu>();
+ Tensor<xpu, 4, DType> grad_in = in_grad[upsampling_mask_enum::kData].get<xpu, 4, DType>(s);
+ //if (kWriteTo == req[upsampling_mask_enum::kData]) {
+ grad_in = 0.0f;
+ //}
+ upsample_mask_backward(s, in_grad[upsampling_mask_enum::kData].dptr<DType>(),
+ out_grad[upsampling_mask_enum::kData].dptr<DType>(),
+ in_data[upsampling_mask_enum::kMask].dptr<int>(),
+ in_grad[upsampling_mask_enum::kData].shape_,
+ out_grad[upsampling_mask_enum::kOut].shape_);
+
+ }
+
+private:
+ UpSamplingMaskParam param_;
+ void upsample_mask_forward(mshadow::Stream<cpu>* s,
+ const DType* in_data, DType* out_data, int* mask,
+ const TShape& ishape, const TShape& oshape);
+
+ void upsample_mask_backward(mshadow::Stream<cpu>* s,
+ DType* in_grad, const DType* out_grad, const int* mask,
+ const TShape& ishape, const TShape& oshape);
+
+#if MXNET_USE_CUDA
+ void upsample_mask_forward(mshadow::Stream<gpu>* s,
+ const DType* in_data, DType* out_data, int* mask,
+ const TShape& ishape, const TShape& oshape);
+ void upsample_mask_backward(mshadow::Stream<gpu>* s,
+ DType* in_grad, const DType* out_grad, const int* mask,
+ const TShape& ishape, const TShape& oshape);
+#endif // MXNET_USE_CUDA
+}; // class UnSamplingMaskOp
+
+template<typename xpu>
+Operator *CreateOp(UpSamplingMaskParam param, int dtype);
+
+#if DMLC_USE_CXX11
+class UpSamplingMaskProp : public OperatorProperty {
+ public:
+ void Init(const std::vector<std::pair<std::string, std::string> >& kwargs) override {
+ param_.Init(kwargs);
+ }
+
+ std::vector<std::string> ListArguments() const override {
+ return {"data", "mask"};
+ }
+
+ std::map<std::string, std::string> GetParams() const override {
+ return param_.__DICT__();
+ }
+
+ bool InferShape(std::vector<TShape> *in_shape,
+ std::vector<TShape> *out_shape,
+ std::vector<TShape> *aux_shape) const override {
+ CHECK_EQ(in_shape->size(), 2U);
+ CHECK_EQ(param_.out_shape.ndim(), 2U);
+ const TShape &dshape = (*in_shape)[0];
+ const TShape &dshape1 = (*in_shape)[1];
+ CHECK_EQ(dshape, dshape1);
+ TShape oshape = dshape;
+ oshape[2] = param_.out_shape[0];
+ oshape[3] = param_.out_shape[1];
+ out_shape->clear();
+ out_shape->push_back(oshape);
+ return true;
+ }
+
+ bool InferType(std::vector<int> *in_type,
+ std::vector<int> *out_type,
+ std::vector<int> *aux_type) const override {
+ CHECK_GE(in_type->size(), 2U);
+ int dtype = (*in_type)[0];
+ CHECK_NE(dtype, -1) << "First input must have specified type";
+ // for (index_t i = 0; i < in_type->size(); ++i) {
+ // if ((*in_type)[i] == -1) {
+ // (*in_type)[i] = dtype;
+ // } else {
+ // UNIFORM_TYPE_CHECK((*in_type)[i], dtype, ListArguments()[i]);
+ // }
+ // }
+ out_type->clear();
+ out_type->push_back(dtype);
+ return true;
+ }
+
+ OperatorProperty* Copy() const override {
+ auto ptr = new UpSamplingMaskProp();
+ ptr->param_ = this->param_;
+ return ptr;
+ }
+
+ std::string TypeString() const override {
+ return "UpSamplingMask";
+ }
+
+ std::vector<int> DeclareBackwardDependency(
+ const std::vector<int> &out_grad,
+ const std::vector<int> &in_data,
+ const std::vector<int> &out_data) const override {
+ return {out_grad[upsampling_mask_enum::kOut], in_data[upsampling_mask_enum::kMask]};
+ }
+
+ std::vector<std::pair<int, void*> > BackwardInplaceOption(
+ const std::vector<int> &out_grad,
+ const std::vector<int> &in_data,
+ const std::vector<int> &out_data,
+ const std::vector<void*> &in_grad) const override {
+ return {};
+ }
+
+ std::vector<ResourceRequest> ForwardResource(
+ const std::vector<TShape> &in_shape) const override {
+ return {ResourceRequest::kTempSpace};
+ }
+
+ std::vector<ResourceRequest> BackwardResource(
+ const std::vector<TShape> &in_shape) const override {
+ return {ResourceRequest::kTempSpace};
+ }
+
+ Operator* CreateOperator(Context ctx) const override {
+ LOG(FATAL) << "Not Implemented";
+ return NULL;
+ }
+
+ Operator* CreateOperatorEx(Context ctx, std::vector<TShape> *in_shape,
+ std::vector<int> *in_type) const override;
+ private:
+ UpSamplingMaskParam param_;
+}; // class UpSamplingMaskProp
+#endif // DMLC_USE_CXX11
+
+} // namespace op
+} // namespace mxnet
+
+#endif // MXNET_OPERATOR_UPSAMPLING_INL_H_
\ No newline at end of file
diff --git a/example/segnet/op/upsampling_mask_max_2d.cc b/example/segnet/op/upsampling_mask_max_2d.cc
new file mode 100644
index 00000000000..e32b083b4d3
--- /dev/null
+++ b/example/segnet/op/upsampling_mask_max_2d.cc
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file upsampling_mask_max_2d.cc
+ * \brief
+ * \author Pengfei Li
+*/
+
+#include <mxnet/base.h>
+#include <mxnet/operator.h>
+#include <algorithm>
+#include "./mxnet_op.h"
+#include "./upsampling_mask_max_2d-inl.h"
+
+namespace mxnet {
+namespace op {
+
+template<typename xpu, typename DType>
+void UpSamplingMaskOp<xpu, DType>::upsample_mask_forward(mshadow::Stream<cpu>* s,
+ const DType* in_data, DType* out_data, int* mask,
+ const TShape& ishape, const TShape& oshape) {
+ const index_t in_offset = ishape[2] * ishape[3];
+ const index_t out_offset = oshape[2] * oshape[3];
+ for (index_t n = 0; n < ishape[0]; ++n) {
+ for (index_t c = 0; c < ishape[1]; ++c) {
+ for (int ph = 0; ph < ishape[2]; ++ph) {
+ for (int pw = 0; pw < ishape[3]; ++pw) {
+ const int upsample_index = ph * ishape[3] + pw;
+ const int max_idx = mask[upsample_index];
+ if (max_idx < out_offset) {
+ out_data[max_idx] = in_data[upsample_index];
+ }
+ }
+ }
+ mask += in_offset;
+ in_data += in_offset;
+ out_data += out_offset;
+ }
+ }
+}
+
+template<typename xpu, typename DType>
+void UpSamplingMaskOp<xpu, DType>::upsample_mask_backward(mshadow::Stream<cpu>* s,
+ DType* in_grad, const DType* out_grad, const int* mask,
+ const TShape& ishape, const TShape& oshape) {
+ const index_t in_offset = ishape[2] * ishape[3];
+ const index_t out_offset = oshape[2] * oshape[3];
+ for (index_t n = 0; n < ishape[0]; ++n) {
+ for (index_t c = 0; c < ishape[1]; ++c) {
+ for (int ph = 0; ph < ishape[2]; ++ph) {
+ for (int pw = 0; pw < ishape[3]; ++pw) {
+ const int upsample_index = ph * ishape[3] + pw;
+ const int max_idx = mask[upsample_index];
+ if (max_idx < out_offset) {
+ in_grad[upsample_index] += out_grad[max_idx];
+ }
+ }
+ }
+ mask += in_offset;
+ in_grad += in_offset;
+ out_grad += out_offset;
+ }
+ }
+}
+
+template<>
+Operator *CreateOp<cpu>(UpSamplingMaskParam param, int dtype) {
+ Operator *op = NULL;
+ MSHADOW_REAL_TYPE_SWITCH(dtype, DType, {
+ op = new UpSamplingMaskOp<cpu, DType>(param);
+ });
+ return op;
+}
+
+// DO_BIND_DISPATCH comes from operator_common.h
+Operator* UpSamplingMaskProp::CreateOperatorEx(Context ctx, std::vector<TShape> *in_shape,
+ std::vector<int> *in_type) const {
+ std::vector<TShape> out_shape, aux_shape;
+ std::vector<int> out_type, aux_type;
+ CHECK(InferType(in_type, &out_type, &aux_type));
+ CHECK(InferShape(in_shape, &out_shape, &aux_shape));
+ DO_BIND_DISPATCH(CreateOp, param_, (*in_type)[0]);
+}
+
+DMLC_REGISTER_PARAMETER(UpSamplingMaskParam);
+
+MXNET_REGISTER_OP_PROPERTY(UpSamplingMask, UpSamplingMaskProp)
+.describe(R"code(Performs max 2d UpSampling on the input.
+
+The shapes for 2-D UpSampling are
+
+- **data**: *(batch_size, channel, height, width)*
+- **mask**: *(batch_size, num_filter, out_height, out_width)*
+- **out**: *(batch_size, num_filter, out_height, out_width)*, with::
+
+ out_height = f(height, kernel[0], pad[0], stride[0])
+ out_width = f(width, kernel[1], pad[1], stride[1])
+
+Only mask UpSampling is supported.
+
+)code" ADD_FILELINE)
+.add_argument("data", "NDArray-or-Symbol", "Input data to the UpSamplingMask operator.")
+.add_argument("mask", "NDArray-or-Symbol", "Input mask to the UpSamplingMask operator.")
+.add_arguments(UpSamplingMaskParam::__FIELDS__());
+} // namespace op
+} // namespace mxnet
\ No newline at end of file
diff --git a/example/segnet/op/upsampling_mask_max_2d.cu b/example/segnet/op/upsampling_mask_max_2d.cu
new file mode 100644
index 00000000000..7f778eb6490
--- /dev/null
+++ b/example/segnet/op/upsampling_mask_max_2d.cu
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file upsampling_mask_max_2d.cu
+ * \brief
+ * \author Pengfei Li
+*/
+#include <vector>
+#include <mxnet/base.h>
+#include <mxnet/operator.h>
+#include "mxnet_op.h"
+#include "../common/cuda_utils.h"
+#include "./upsampling_mask_max_2d-inl.h"
+
+namespace mxnet {
+namespace op {
+
+template <typename DType>
+__global__ void upsample_max_2d_gpu_kernel(const int nthreads, const DType* in_data,
+ const int channels, const int height, const int width,
+ const int upsampleed_height, const int upsampleed_width,
+ DType* out_data, int* mask) {
+ CUDA_KERNEL_LOOP(index, nthreads) {
+ const int c = (index / width / height) % channels;
+ const int n = index / width / height / channels;
+ int out_offset = (n * channels + c) * upsampleed_height * upsampleed_width;
+ int max_idx = mask[index];
+ if (max_idx < out_offset) {
+ out_data[out_offset+max_idx] = in_data[index];
+ }
+ }
+}
+
+template <typename DType>
+__global__ void unupsample_max_2d_gpu_kernel(const int nthreads, const DType* out_grad,
+ const int channels, const int height, const int width,
+ const int upsampleed_height, const int upsampleed_width,
+ DType* in_grad, const int* mask) {
+ CUDA_KERNEL_LOOP(index, nthreads) {
+ const int c = (index / width / height) % channels;
+ const int n = index / width / height / channels;
+ int out_offset = (n * channels + c) * upsampleed_height * upsampleed_width;
+ int max_idx = mask[index];
+ if (max_idx < out_offset) {
+ atomicAdd(&in_grad[index], out_grad[out_offset+max_idx]);
+ }
+ }
+}
+
+template<typename xpu, typename DType>
+void UpSamplingMaskOp<xpu, DType>::upsample_mask_forward(mshadow::Stream<gpu>* s,
+ const DType* in_data, DType* out_data, int* mask,
+ const TShape& ishape, const TShape& oshape) {
+ using namespace mxnet_op;
+ upsample_max_2d_gpu_kernel<<<cuda_get_num_blocks(ishape.Size()), mshadow::cuda::kBaseThreadNum,
+ 0, mshadow::Stream<gpu>::GetStream(s)>>>(
+ ishape.Size(), in_data, ishape[1], ishape[2], ishape[3],
+ oshape[2], oshape[3], out_data, mask);
+ MSHADOW_CUDA_POST_KERNEL_CHECK(upsample_max_2d_gpu_kernel);
+}
+
+template<typename xpu, typename DType>
+void UpSamplingMaskOp<xpu, DType>::upsample_mask_backward(mshadow::Stream<gpu>* s,
+ DType* in_grad, const DType* out_grad, const int* mask,
+ const TShape& ishape, const TShape& oshape) {
+ using namespace mxnet_op;
+ unupsample_max_2d_gpu_kernel<<<cuda_get_num_blocks(ishape.Size()), mshadow::cuda::kBaseThreadNum,
+ 0, mshadow::Stream<gpu>::GetStream(s)>>>(
+ ishape.Size(), out_grad,
+ ishape[1], ishape[2], ishape[3],
+ oshape[2], oshape[3], in_grad, mask);
+ MSHADOW_CUDA_POST_KERNEL_CHECK(unupsample_max_2d_gpu_kernel);
+}
+
+template<>
+Operator *CreateOp<gpu>(UpSamplingMaskParam param, int dtype) {
+ Operator *op = NULL;
+ MSHADOW_REAL_TYPE_SWITCH(dtype, DType, {
+ op = new UpSamplingMaskOp<gpu, DType>(param);
+ });
+ return op;
+}
+
+} // namespace op
+} // namespace mxnet
diff --git a/example/segnet/score.py b/example/segnet/score.py
new file mode 100644
index 00000000000..bc50843c213
--- /dev/null
+++ b/example/segnet/score.py
@@ -0,0 +1,132 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import argparse
+import mxnet as mx
+import time
+import os
+import logging
+import pdb
+import numpy as np
+from PIL import Image
+from data_iter import FileIter
+from common import contrib_metrics
+
+def getpallete(num_cls):
+ """
+ this function is to get the colormap for visualizing the segmentation mask
+ """
+ pallete = [0] * (num_cls * 3)
+ for j in xrange(0, num_cls):
+ lab = j
+ pallete[j*3+0] = 0
+ pallete[j*3+1] = 0
+ pallete[j*3+2] = 0
+ i = 0
+ while lab > 0:
+ pallete[j*3+0] |= (((lab >> 0) & 1) << (7-i))
+ pallete[j*3+1] |= (((lab >> 1) & 1) << (7-i))
+ pallete[j*3+2] |= (((lab >> 2) & 1) << (7-i))
+ i = i + 1
+ lab >>= 3
+ return pallete
+
+def score(args):
+ """
+ score segnet
+ """
+ pallete = getpallete(256)
+ # set up logger
+ logging.basicConfig()
+ logger = logging.getLogger()
+ logger.setLevel(logging.INFO)
+ if args.log_file:
+ file_path = logging.FileHandler(args.log_file)
+ logger.addHandler(file_path)
+
+ data = FileIter(
+ batch_size=args.batch_size,
+ root_dir=args.data_path,
+ flist_name=args.score_file,
+ rgb_mean=[float(i) for i in args.rgb_mean.split(',')],
+ shuffle=True)
+ # load model
+ from importlib import import_module
+ network = import_module('symbols.'+args.network).get_symbol(args.num_classes)
+
+ try:
+ _, arg_params, aux_params = mx.model.load_checkpoint(args.model_prefix, args.load_epoch)
+ except ():
+ logging.info('model load error!')
+
+ # create module
+ if args.gpus == '':
+ devs = mx.cpu()
+ else:
+ devs = [mx.gpu(int(i)) for i in args.gpus.split(',')]
+
+ mod = mx.mod.Module(symbol=network, context=devs, label_names=['softmax_label',])
+ mod.bind(for_training=False,
+ data_shapes=data.provide_data,
+ label_shapes=data.provide_label)
+ mod.set_params(arg_params, aux_params)
+ # evaluation metrices
+ metric = contrib_metrics.Accuracy(ignore_label=11)
+ tic = time.time()
+ num = 0
+ data.reset()
+ metric.reset()
+ for batch in data:
+ mod.forward(batch, is_train=False)
+ out_img = np.uint8(np.squeeze(mod.get_outputs()[0].asnumpy().argmax(axis=1)))
+ out_img = Image.fromarray(out_img)
+ out_img.putpalette(pallete)
+ out_img.save('res_pic/' + str(num) + 'res.png')
+ mod.update_metric(metric, batch.label)
+ logging.info(metric.get())
+ num += args.batch_size
+ logging.info(metric.get())
+ return (num / (time.time() - tic), )
+
+
+if __name__ == '__main__':
+ parser = argparse.ArgumentParser(description='score a model on a dataset')
+ parser.add_argument('--gpus', type=str, default='0')
+ parser.add_argument('--batch_size', type=int, default=1)
+ parser.add_argument('--rgb_mean', type=str, default='123.68, 116.779, 103.939')
+ parser.add_argument('--image_shape', type=str, default='3,360,480')
+ parser.add_argument('--data_nthreads', type=int, default=4,
+ help='number of threads for data decoding')
+ parser.add_argument('--data_path', type=str, default="/data/CamVid",
+ help='data path')
+ parser.add_argument('--score_file', type=str, default="test.txt",
+ help='test data list file name in data path')
+ parser.add_argument('--model_prefix', type=str, default='models/segnet',
+ help='model prefix')
+ parser.add_argument('--load_epoch', type=int, default=200,
+ help='load the model on an epoch using the model-load-prefix')
+ parser.add_argument('--log_file', type=str, default="log.txt",
+ help='the name of log file')
+ parser.add_argument('--network', type=str, required=True,
+ help='the neural network to use')
+ parser.add_argument('--num_classes', type=int, default=11,
+ help='num classes to clissufy')
+
+ args = parser.parse_args()
+
+ (speed,) = score(args)
+ logging.info('Finished with %f images per second', speed)
diff --git a/example/segnet/segment_one_pic.py b/example/segnet/segment_one_pic.py
new file mode 100644
index 00000000000..0e67e565d25
--- /dev/null
+++ b/example/segnet/segment_one_pic.py
@@ -0,0 +1,137 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import argparse
+import mxnet as mx
+import time
+import os
+import logging
+import numpy as np
+from PIL import Image
+from data_iter import FileIter
+from common import contrib_metrics
+
+def getpallete(num_cls):
+ """
+ this function is to get the colormap for visualizing the segmentation mask
+ """
+ pallete = [0] * (num_cls * 3)
+ for j in xrange(0, num_cls):
+ lab = j
+ pallete[j*3+0] = 0
+ pallete[j*3+1] = 0
+ pallete[j*3+2] = 0
+ i = 0
+ while lab > 0:
+ pallete[j*3+0] |= (((lab >> 0) & 1) << (7-i))
+ pallete[j*3+1] |= (((lab >> 1) & 1) << (7-i))
+ pallete[j*3+2] |= (((lab >> 2) & 1) << (7-i))
+ i = i + 1
+ lab >>= 3
+ return pallete
+
+def read_img(args):
+ """
+ read picture to numpy.array
+ """
+ img = mx.image.imread(os.path.join(args.image_path, args.image_name))
+ img = img.astype('float32')
+ mean = mx.nd.array([float(i) for i in args.rgb_mean.split(',')])
+ reshaped_mean = mean.reshape((1, 1, 3))
+ img = img - reshaped_mean
+ img = mx.nd.moveaxis(img, 2, 0)
+ img = mx.nd.expand_dims(img, axis=0)
+ return img
+
+def read_label(args):
+ """
+ read label to numpy.array
+ """
+ label = mx.image.imread(os.path.join(args.label_path, args.label_name), flag=0)
+ label = label.astype('float32')
+ label = mx.nd.moveaxis(label, 2, 0)
+ return label
+
+def score(args):
+ """
+ test one pic and save the result
+ """
+ pallete = getpallete(256)
+ # create module
+ if args.gpus == '':
+ devs = mx.cpu()
+ else:
+ devs = [mx.gpu(int(i)) for i in args.gpus.split(',')]
+ # create data iterator
+ img = read_img(args)
+ label = read_label(args)
+ data_shape = tuple([int(i) for i in args.image_shape.split(',')])
+ batch_data = mx.ndarray.array(img[0])
+ batch_data = batch_data.reshape((tuple([1]) + data_shape))
+ data = mx.io.DataBatch([batch_data], [])
+ try:
+ sym, arg_params, aux_params = mx.model.load_checkpoint(args.model_prefix, args.load_epoch)
+ except ():
+ logging.info('model load error!')
+
+
+ mod = mx.mod.Module(symbol=sym, context=devs, label_names=['softmax_label',])
+ mod.bind(for_training=False,
+ data_shapes=[('data', (tuple([1]) + data_shape))],
+ label_shapes=[('softmax_label', (tuple([1] + [data_shape[1]*data_shape[2]])))])
+ mod.set_params(arg_params, aux_params)
+ metric = contrib_metrics.Accuracy(ignore_label=11)
+ tic = time.time()
+ mod.forward(data, is_train=False)
+ out_img = np.uint8(np.squeeze(mod.get_outputs()[0].asnumpy().argmax(axis=1)))
+ metric.reset()
+ metric.update([label], [mod.get_outputs()[0]])
+ logging.info(metric.get())
+ out_img = Image.fromarray(np.uint8(label[0].asnumpy()))
+ out_img.putpalette(pallete)
+ out_img.save('res_pic/' + args.image_name)
+ out_label = Image.fromarray(label)
+ out_label.putpalette(pallete)
+ out_label.save('res_pic/label' + args.image_name)
+ return (1 / (time.time() - tic), )
+
+
+if __name__ == '__main__':
+ parser = argparse.ArgumentParser(description='score a model on a dataset')
+ parser.add_argument('--gpus', type=str, default='0')
+ parser.add_argument('--rgb_mean', type=str, default='123.68, 116.779, 103.939')
+ parser.add_argument('--image_shape', type=str, default='3,360,480')
+ parser.add_argument('--data_nthreads', type=int, default=4,
+ help='number of threads for data decoding')
+ parser.add_argument('--image_path', type=str, default="/data/CamVid/Image/",
+ help='data path')
+ parser.add_argument('--image_name', type=str, default="0016E5_04530.png",
+ help='test data list file name in data path')
+ parser.add_argument('--label_path', type=str, default="/data/CamVid/Label/",
+ help='data path')
+ parser.add_argument('--label_name', type=str, default="0016E5_04530.png",
+ help='test data list file name in data path')
+ parser.add_argument('--model_prefix', type=str, default='models/segnet',
+ help='model prefix')
+ parser.add_argument('--load_epoch', type=int, default=70,
+ help='load the model on an epoch using the model-load-prefix')
+ parser.add_argument('--log_file', type=str, default="log.txt",
+ help='the name of log file')
+ args = parser.parse_args()
+
+ (speed,) = score(args)
+ logging.info('Finished with %f images per second', speed)
diff --git a/example/segnet/symbols/__init__.py b/example/segnet/symbols/__init__.py
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/example/segnet/symbols/segnet.py b/example/segnet/symbols/segnet.py
new file mode 100644
index 00000000000..b80316399ca
--- /dev/null
+++ b/example/segnet/symbols/segnet.py
@@ -0,0 +1,199 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import mxnet as mx
+
+def vgg_conv_deconv(input_data, num_classes, fix_gamma=False, eps=0.00002,
+ bn_mom=0.9, input_height=360, input_width=480):
+ """
+ get vgg encode and decode layers
+ """
+ # group 1
+ conv1_1 = mx.symbol.Convolution(data=input_data, kernel=(3, 3), pad=(1, 1), num_filter=64,
+ name="conv1_1")
+ bn_conv1_1 = mx.symbol.BatchNorm(data=conv1_1, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv1_1')
+ relu1_1 = mx.symbol.Activation(data=bn_conv1_1, act_type="relu", name="relu1_1")
+ conv1_2 = mx.symbol.Convolution(data=relu1_1, kernel=(3, 3), pad=(1, 1), num_filter=64,
+ name="conv1_2")
+ bn_conv1_2 = mx.symbol.BatchNorm(data=conv1_2, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv1_2')
+ relu1_2 = mx.symbol.Activation(data=bn_conv1_2, act_type="relu", name="relu1_2")
+ pool1 = mx.symbol.PoolingMask(data=relu1_2, kernel=(2, 2), stride=(2, 2), name="pool1")
+ # group 2
+ conv2_1 = mx.symbol.Convolution(data=pool1[0], kernel=(3, 3), pad=(1, 1), num_filter=128,
+ name="conv2_1")
+ bn_conv2_1 = mx.symbol.BatchNorm(data=conv2_1, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv2_1')
+ relu2_1 = mx.symbol.Activation(data=bn_conv2_1, act_type="relu", name="relu2_1")
+ conv2_2 = mx.symbol.Convolution(data=relu2_1, kernel=(3, 3), pad=(1, 1), num_filter=128,
+ name="conv2_2")
+ bn_conv2_2 = mx.symbol.BatchNorm(data=conv2_2, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv2_2')
+ relu2_2 = mx.symbol.Activation(data=bn_conv2_2, act_type="relu", name="relu2_2")
+ pool2 = mx.symbol.PoolingMask(data=relu2_2, kernel=(2, 2), stride=(2, 2), name="pool2")
+ # group 3
+ conv3_1 = mx.symbol.Convolution(data=pool2[0], kernel=(3, 3), pad=(1, 1), num_filter=256,
+ name="conv3_1")
+ bn_conv3_1 = mx.symbol.BatchNorm(data=conv3_1, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv3_1')
+ relu3_1 = mx.symbol.Activation(data=bn_conv3_1, act_type="relu", name="relu3_1")
+ conv3_2 = mx.symbol.Convolution(data=relu3_1, kernel=(3, 3), pad=(1, 1), num_filter=256,
+ name="conv3_2")
+ bn_conv3_2 = mx.symbol.BatchNorm(data=conv3_2, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv3_2')
+ relu3_2 = mx.symbol.Activation(data=bn_conv3_2, act_type="relu", name="relu3_2")
+ conv3_3 = mx.symbol.Convolution(data=relu3_2, kernel=(3, 3), pad=(1, 1), num_filter=256,
+ name="conv3_3")
+ bn_conv3_3 = mx.symbol.BatchNorm(data=conv3_3, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv3_3')
+ relu3_3 = mx.symbol.Activation(data=bn_conv3_3, act_type="relu", name="relu3_3")
+ pool3 = mx.symbol.PoolingMask(data=relu3_3, kernel=(2, 2), stride=(2, 2), name="pool3")
+ # group 4
+ conv4_1 = mx.symbol.Convolution(data=pool3[0], kernel=(3, 3), pad=(1, 1), num_filter=512,
+ name="conv4_1")
+ bn_conv4_1 = mx.symbol.BatchNorm(data=conv4_1, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv4_1')
+ relu4_1 = mx.symbol.Activation(data=bn_conv4_1, act_type="relu", name="relu4_1")
+ conv4_2 = mx.symbol.Convolution(data=relu4_1, kernel=(3, 3), pad=(1, 1), num_filter=512,
+ name="conv4_2")
+ bn_conv4_2 = mx.symbol.BatchNorm(data=conv4_2, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv4_2')
+ relu4_2 = mx.symbol.Activation(data=bn_conv4_2, act_type="relu", name="relu4_2")
+ conv4_3 = mx.symbol.Convolution(data=relu4_2, kernel=(3, 3), pad=(1, 1), num_filter=512,
+ name="conv4_3")
+ bn_conv4_3 = mx.symbol.BatchNorm(data=conv4_3, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv4_3')
+ relu4_3 = mx.symbol.Activation(data=bn_conv4_3, act_type="relu", name="relu4_3")
+ pool4 = mx.symbol.PoolingMask(data=relu4_3, kernel=(2, 2), stride=(2, 2), name="pool4")
+ # group 5
+ conv5_1 = mx.symbol.Convolution(data=pool4[0], kernel=(3, 3), pad=(1, 1), num_filter=512,
+ name="conv5_1")
+ bn_conv5_1 = mx.symbol.BatchNorm(data=conv5_1, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv5_1')
+ relu5_1 = mx.symbol.Activation(data=bn_conv5_1, act_type="relu", name="relu5_1")
+ conv5_2 = mx.symbol.Convolution(data=relu5_1, kernel=(3, 3), pad=(1, 1), num_filter=512,
+ name="conv5_2")
+ bn_conv5_2 = mx.symbol.BatchNorm(data=conv5_2, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv5_2')
+ relu5_2 = mx.symbol.Activation(data=bn_conv5_2, act_type="relu", name="relu5_2")
+ conv5_3 = mx.symbol.Convolution(data=relu5_2, kernel=(3, 3), pad=(1, 1), num_filter=512,
+ name="conv5_3")
+ bn_conv5_3 = mx.symbol.BatchNorm(data=conv5_3, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv5_3')
+ relu5_3 = mx.symbol.Activation(data=bn_conv5_3, act_type="relu", name="relu5_3")
+ pool5 = mx.symbol.PoolingMask(data=relu5_3, kernel=(2, 2), stride=(2, 2), name="pool5")
+ # deconve
+ # group 5
+ pool5_d = mx.symbol.UpSamplingMask(data=pool5[0], mask=pool5[1],
+ out_shape=(input_height/16, input_width/16), name="pool5_d")
+ conv5_3_d = mx.symbol.Convolution(data=pool5_d, kernel=(3, 3), pad=(1, 1), num_filter=512,
+ name="conv5_3_d")
+ bn_conv5_3_d = mx.symbol.BatchNorm(data=conv5_3_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv5_3_d')
+ relu5_3_d = mx.symbol.Activation(data=bn_conv5_3_d, act_type="relu", name="relu5_3_d")
+ conv5_2_d = mx.symbol.Convolution(data=relu5_3_d, kernel=(3, 3), pad=(1, 1), num_filter=512,
+ name="conv5_2_d")
+ bn_conv5_2_d = mx.symbol.BatchNorm(data=conv5_2_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv5_2_d')
+ relu5_2_d = mx.symbol.Activation(data=bn_conv5_2_d, act_type="relu", name="relu5_2_d")
+ conv5_1_d = mx.symbol.Convolution(data=relu5_2_d, kernel=(3, 3), pad=(1, 1), num_filter=512,
+ name="conv5_1_d")
+ bn_conv5_1_d = mx.symbol.BatchNorm(data=conv5_1_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv5_1_d')
+ relu5_1_d = mx.symbol.Activation(data=bn_conv5_1_d, act_type="relu", name="relu5_1_d")
+ # group 4
+ pool4_d = mx.symbol.UpSamplingMask(data=relu5_1_d, mask=pool4[1],
+ out_shape=(input_height/8, input_width/8), name="pool4_d")
+ conv4_3_d = mx.symbol.Convolution(data=pool4_d, kernel=(3, 3), pad=(1, 1), num_filter=512,
+ name="conv4_3_d")
+ bn_conv4_3_d = mx.symbol.BatchNorm(data=conv4_3_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv4_3_d')
+ relu4_3_d = mx.symbol.Activation(data=bn_conv4_3_d, act_type="relu", name="relu4_3_d")
+ conv4_2_d = mx.symbol.Convolution(data=relu4_3_d, kernel=(3, 3), pad=(1, 1), num_filter=512,
+ name="conv4_2_d")
+ bn_conv4_2_d = mx.symbol.BatchNorm(data=conv4_2_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv4_2_d')
+ relu4_2_d = mx.symbol.Activation(data=bn_conv4_2_d, act_type="relu", name="relu4_2_d")
+ conv4_1_d = mx.symbol.Convolution(data=relu4_2_d, kernel=(3, 3), pad=(1, 1), num_filter=256,
+ name="conv4_1_d")
+ bn_conv4_1_d = mx.symbol.BatchNorm(data=conv4_1_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv4_1_d')
+ relu4_1_d = mx.symbol.Activation(data=bn_conv4_1_d, act_type="relu", name="relu4_1_d")
+ # group 3
+ pool3_d = mx.symbol.UpSamplingMask(data=relu4_1_d, mask=pool3[1],
+ out_shape=(input_height/4, input_width/4), name="pool3_d")
+ conv3_3_d = mx.symbol.Convolution(data=pool3_d, kernel=(3, 3), pad=(1, 1), num_filter=256,
+ name="conv3_3_d")
+ bn_conv3_3_d = mx.symbol.BatchNorm(data=conv3_3_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv3_3_d')
+ relu3_3_d = mx.symbol.Activation(data=bn_conv3_3_d, act_type="relu", name="relu3_3_d")
+ conv3_2_d = mx.symbol.Convolution(data=relu3_3_d, kernel=(3, 3), pad=(1, 1), num_filter=256,
+ name="conv3_2_d")
+ bn_conv3_2_d = mx.symbol.BatchNorm(data=conv3_2_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv3_2_d')
+ relu3_2_d = mx.symbol.Activation(data=bn_conv3_2_d, act_type="relu", name="relu3_2_d")
+ conv3_1_d = mx.symbol.Convolution(data=relu3_2_d, kernel=(3, 3), pad=(1, 1), num_filter=128,
+ name="conv3_1_d")
+ bn_conv3_1_d = mx.symbol.BatchNorm(data=conv3_1_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv3_1_d')
+ relu3_1_d = mx.symbol.Activation(data=bn_conv3_1_d, act_type="relu", name="relu3_1_d")
+ # group 2
+ pool2_d = mx.symbol.UpSamplingMask(data=relu3_1_d, mask=pool2[1],
+ out_shape=(input_height/2, input_width/2), name="pool2_d")
+ conv2_2_d = mx.symbol.Convolution(data=pool2_d, kernel=(3, 3), pad=(1, 1), num_filter=128,
+ name="conv2_2_d")
+ bn_conv2_2_d = mx.symbol.BatchNorm(data=conv2_2_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv2_2_d')
+ relu2_2_d = mx.symbol.Activation(data=bn_conv2_2_d, act_type="relu", name="relu2_2_d")
+ conv2_1_d = mx.symbol.Convolution(data=relu2_2_d, kernel=(3, 3), pad=(1, 1), num_filter=64,
+ name="conv2_1_d")
+ bn_conv2_1_d = mx.symbol.BatchNorm(data=conv2_1_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv2_1_d')
+ relu2_1_d = mx.symbol.Activation(data=bn_conv2_1_d, act_type="relu", name="relu2_1_d")
+ # group 1
+ pool1_d = mx.symbol.UpSamplingMask(data=relu2_1_d, mask=pool1[1],
+ out_shape=(input_height, input_width), name="pool1_d")
+ conv1_2_d = mx.symbol.Convolution(data=pool1_d, kernel=(3, 3), pad=(1, 1), num_filter=64,
+ name="conv1_2_d")
+ bn_conv1_2_d = mx.symbol.BatchNorm(data=conv1_2_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv1_2_d')
+ relu1_2_d = mx.symbol.Activation(data=bn_conv1_2_d, act_type="relu", name="relu1_2_d")
+ conv1_1_d = mx.symbol.Convolution(data=relu1_2_d, kernel=(3, 3), pad=(1, 1), num_filter=num_classes,
+ name="conv1_1_d")
+ return conv1_1_d
+
+def get_symbol(num_classes, natural_balance=False):
+ """
+ get segnet network
+ """
+ data = mx.symbol.Variable(name='data')
+ network = vgg_conv_deconv(data, num_classes)
+ if natural_balance:
+ weight = mx.symbol.Variable(name="softmax_weight", lr_mult=0)
+ label = mx.symbol.Variable(name='softmax_label', lr_mult=0)
+ softmax = mx.symbol.softmax(data=network, axis=1, name='softmax')
+ label = mx.symbol.one_hot(indices=label, depth=num_classes)
+ label = mx.symbol.swapaxes(label, 1, 3)
+ label = mx.symbol.swapaxes(label, 2, 3)
+ cross_entropy = mx.symbol.broadcast_mul(mx.symbol.log(softmax), label)
+ loss = mx.sym.MakeLoss(-cross_entropy*weight, grad_scale=1.0/(360*480))
+ return loss
+ else:
+ softmax = mx.symbol.SoftmaxOutput(data=network, multi_output=True, use_ignore=True,
+ ignore_label=11, name='softmax')
+ return softmax
diff --git a/example/segnet/symbols/segnet_basic.py b/example/segnet/symbols/segnet_basic.py
new file mode 100644
index 00000000000..57aefb02b6c
--- /dev/null
+++ b/example/segnet/symbols/segnet_basic.py
@@ -0,0 +1,99 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import mxnet as mx
+
+def conv_deconv(input_data, num_classes, input_height=360, input_width=480):
+ """
+ get encode and decode layers
+ """
+ # norm
+ norm_data = mx.symbol.LRN(data=input_data, alpha=0.0001, beta=0.75, nsize=5, name='lrn_norm')
+ # group 1
+ conv1 = mx.symbol.Convolution(data=norm_data, kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv1')
+ bn_conv1 = mx.symbol.BatchNorm(data=conv1, name='bn_conv1')
+ relu1 = mx.symbol.Activation(data=bn_conv1, act_type='relu', name='relu1')
+ pool1 = mx.symbol.PoolingMask(data=relu1, kernel=(2, 2), stride=(2, 2), name='pool1')
+ # group 2
+ conv2 = mx.symbol.Convolution(data=pool1[0], kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv2')
+ bn_conv2 = mx.symbol.BatchNorm(data=conv2, name='bn_conv2')
+ relu2 = mx.symbol.Activation(data=bn_conv2, act_type='relu', name='relu2')
+ pool2 = mx.symbol.PoolingMask(data=relu2, kernel=(2, 2), stride=(2, 2), name='pool2')
+ # group 3
+ conv3 = mx.symbol.Convolution(data=pool2[0], kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv3')
+ bn_conv3 = mx.symbol.BatchNorm(data=conv3, name='bn_conv3')
+ relu3 = mx.symbol.Activation(data=bn_conv3, act_type='relu', name='relu3')
+ pool3 = mx.symbol.PoolingMask(data=relu3, kernel=(2, 2), stride=(2, 2), name='pool3')
+ # group 4
+ conv4 = mx.symbol.Convolution(data=pool3[0], kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv4')
+ bn_conv4 = mx.symbol.BatchNorm(data=conv4, name='bn_conv4')
+ relu4 = mx.symbol.Activation(data=bn_conv4, act_type='relu', name='relu4')
+ pool4 = mx.symbol.PoolingMask(data=relu4, kernel=(2, 2), stride=(2, 2), name='pool4')
+ # deconve
+ # group 4
+ pool4_d = mx.symbol.UpSamplingMask(data=pool4[0], mask=pool4[1],
+ out_shape=(input_height/8, input_width/8), name="pool4_d")
+ conv4_d = mx.symbol.Convolution(data=pool4_d, kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv4_d')
+ bn_conv4_d = mx.symbol.BatchNorm(data=conv4_d, name='bn_conv4_d')
+ # group 3
+ pool3_d = mx.symbol.UpSamplingMask(data=bn_conv4_d, mask=pool3[1],
+ out_shape=(input_height/4, input_width/4), name="pool3_d")
+ conv3_d = mx.symbol.Convolution(data=pool3_d, kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv3_d')
+ bn_conv3_d = mx.symbol.BatchNorm(data=conv3_d, name='bn_conv3_d')
+ # group 2
+ pool2_d = mx.symbol.UpSamplingMask(data=bn_conv3_d, mask=pool2[1],
+ out_shape=(input_height/2, input_width/2), name="pool2_d")
+ conv2_d = mx.symbol.Convolution(data=pool2_d, kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv2_d')
+ bn_conv2_d = mx.symbol.BatchNorm(data=conv2_d, name='bn_conv2_d')
+ # group 1
+ pool1_d = mx.symbol.UpSamplingMask(data=bn_conv2_d, mask=pool1[1],
+ out_shape=(input_height, input_width), name="pool1_d")
+ conv1_d = mx.symbol.Convolution(data=pool1_d, kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv1_d')
+ bn_conv1_d = mx.symbol.BatchNorm(data=conv1_d, name='bn_conv1_d')
+ # classfiier
+ conv_classfiier = mx.symbol.Convolution(data=bn_conv1_d, kernel=(1, 1),
+ num_filter=num_classes, name='conv_classfiier')
+ return conv_classfiier
+
+def get_symbol(num_classes, natural_balance=False):
+ """
+ get segnet_basic network
+ """
+ data = mx.symbol.Variable(name='data')
+ network = conv_deconv(data, num_classes)
+ if natural_balance:
+ weight = mx.symbol.Variable(name="softmax_weight", lr_mult=0)
+ label = mx.symbol.Variable(name='softmax_label', lr_mult=0)
+ softmax = mx.symbol.softmax(data=network, axis=1, name='softmax')
+ label = mx.symbol.one_hot(indices=label, depth=num_classes)
+ label = mx.symbol.swapaxes(label, 1, 3)
+ label = mx.symbol.swapaxes(label, 2, 3)
+ cross_entropy = mx.symbol.broadcast_mul(mx.symbol.log(softmax), label)
+ loss = mx.sym.MakeLoss(-cross_entropy*weight, grad_scale=1.0/(360*480))
+ return loss
+ else:
+ softmax = mx.symbol.SoftmaxOutput(data=network, multi_output=True, use_ignore=True,
+ ignore_label=11, name='softmax')
+ return softmax
diff --git a/example/segnet/symbols/segnet_basic_bayesian.py b/example/segnet/symbols/segnet_basic_bayesian.py
new file mode 100644
index 00000000000..41fbb81efe5
--- /dev/null
+++ b/example/segnet/symbols/segnet_basic_bayesian.py
@@ -0,0 +1,79 @@
+import mxnet as mx
+def vgg_conv_deconv(input_data, num_classes, input_height=360, input_width=480):
+ # norm
+ norm_data = mx.symbol.LRN(data=input_data, alpha=0.0001, beta=0.75, nsize=5, name='lrn_norm')
+ # group 1
+ conv1 = mx.symbol.Convolution(data=norm_data, kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv1')
+ bn_conv1 = mx.symbol.BatchNorm(data=conv1, name='bn_conv1')
+ relu1 = mx.symbol.Activation(data=bn_conv1, act_type='relu', name='relu1')
+ pool1 = mx.symbol.PoolingMask(data=relu1, kernel=(2, 2), stride=(2, 2), name='pool1')
+ # group 2
+ conv2 = mx.symbol.Convolution(data=pool1[0], kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv2')
+ bn_conv2 = mx.symbol.BatchNorm(data=conv2, name='bn_conv2')
+ relu2 = mx.symbol.Activation(data=bn_conv2, act_type='relu', name='relu2')
+ pool2 = mx.symbol.PoolingMask(data=relu2, kernel=(2, 2), stride=(2, 2), name='pool2')
+ # group 3
+ conv3 = mx.symbol.Convolution(data=pool2[0], kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv3')
+ bn_conv3 = mx.symbol.BatchNorm(data=conv3, name='bn_conv3')
+ relu3 = mx.symbol.Activation(data=bn_conv3, act_type='relu', name='relu3')
+ pool3 = mx.symbol.PoolingMask(data=relu3, kernel=(2, 2), stride=(2, 2), name='pool3')
+ encdrop3 = mx.sym.Dropout(data=pool3[0], p=0.5, name="encdrop3")
+ # group 4
+ conv4 = mx.symbol.Convolution(data=encdrop3, kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv4')
+ bn_conv4 = mx.symbol.BatchNorm(data=conv4, name='bn_conv4')
+ relu4 = mx.symbol.Activation(data=bn_conv4, act_type='relu', name='relu4')
+ pool4 = mx.symbol.PoolingMask(data=relu4, kernel=(2, 2), stride=(2, 2), name='pool4')
+ encdrop4 = mx.sym.Dropout(data=pool4[0], p=0.5, name="encdrop4")
+
+ # group 4
+ pool4_d = mx.symbol.UpSamplingMask(data=encdrop4, mask=pool4[1],
+ out_shape=(input_height/8, input_width/8), name="pool4_d")
+ conv4_d = mx.symbol.Convolution(data=pool4_d, kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv4_d')
+ bn_conv4_d = mx.symbol.BatchNorm(data=conv4_d, name='bn_conv4_d')
+ decdrop4 = mx.sym.Dropout(data=bn_conv4_d, p=0.5, name="decdrop4")
+ # group 3
+ pool3_d = mx.symbol.UpSamplingMask(data=decdrop4, mask=pool3[1],
+ out_shape=(input_height/4, input_width/4), name="pool3_d")
+ conv3_d = mx.symbol.Convolution(data=pool3_d, kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv3_d')
+ bn_conv3_d = mx.symbol.BatchNorm(data=conv3_d, name='bn_conv3_d')
+ decdrop3 = mx.sym.Dropout(data=bn_conv3_d, p=0.5, name="decdrop3")
+ # group 2
+ pool2_d = mx.symbol.UpSamplingMask(data=decdrop3, mask=pool2[1],
+ out_shape=(input_height/2, input_width/2), name="pool2_d")
+ conv2_d = mx.symbol.Convolution(data=pool2_d, kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv2_d')
+ bn_conv2_d = mx.symbol.BatchNorm(data=conv2_d, name='bn_conv2_d')
+ # group 1
+ pool1_d = mx.symbol.UpSamplingMask(data=bn_conv2_d, mask=pool1[1],
+ out_shape=(input_height, input_width), name="pool1_d")
+ conv1_d = mx.symbol.Convolution(data=pool1_d, kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv1_d')
+ bn_conv1_d = mx.symbol.BatchNorm(data=conv1_d, name='bn_conv1_d')
+ # classfiier
+ conv_classfiier = mx.symbol.Convolution(data=bn_conv1_d, kernel=(1, 1),
+ num_filter=num_classes, name='conv_classfiier')
+ return conv_classfiier
+
+def get_symbol(num_classes, natural_balance=False, **kwargs):
+ data = mx.symbol.Variable(name='data')
+ network = vgg_conv_deconv(data, num_classes)
+ if natural_balance:
+ weight = mx.symbol.Variable(name="softmax_weight", lr_mult=0)
+ label = mx.symbol.Variable(name='softmax_label', lr_mult=0)
+ softmax = mx.symbol.softmax(data=network, axis=1, name='softmax')
+ label = mx.symbol.one_hot(indices=label, depth=num_classes)
+ label = mx.symbol.swapaxes(label, 1, 3)
+ label = mx.symbol.swapaxes(label, 2, 3)
+ cross_entropy = mx.symbol.broadcast_mul(mx.symbol.log(softmax), label)
+ loss = mx.sym.MakeLoss(-cross_entropy*weight, grad_scale=1.0/(360*480))
+ return loss
+ else:
+ softmax = mx.symbol.SoftmaxOutput(data=network, multi_output=True, use_ignore=True,
+ ignore_label=11, name='softmax')
+ return softmax
diff --git a/example/segnet/symbols/segnet_basic_with_dropout.py b/example/segnet/symbols/segnet_basic_with_dropout.py
new file mode 100644
index 00000000000..5056db3c9f1
--- /dev/null
+++ b/example/segnet/symbols/segnet_basic_with_dropout.py
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import mxnet as mx
+
+def conv_deconv(input_data, num_classes, input_height=360, input_width=480):
+ """
+ get encode and decode layers
+ """
+ # norm
+ norm_data = mx.symbol.LRN(data=input_data, alpha=0.0001, beta=0.75, nsize=5, name='lrn_norm')
+ # group 1
+ conv1 = mx.symbol.Convolution(data=norm_data, kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv1')
+ bn_conv1 = mx.symbol.BatchNorm(data=conv1, name='bn_conv1')
+ relu1 = mx.symbol.Activation(data=bn_conv1, act_type='relu', name='relu1')
+ pool1 = mx.symbol.PoolingMask(data=relu1, kernel=(2, 2), stride=(2, 2), name='pool1')
+ # group 2
+ conv2 = mx.symbol.Convolution(data=pool1[0], kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv2')
+ bn_conv2 = mx.symbol.BatchNorm(data=conv2, name='bn_conv2')
+ relu2 = mx.symbol.Activation(data=bn_conv2, act_type='relu', name='relu2')
+ pool2 = mx.symbol.PoolingMask(data=relu2, kernel=(2, 2), stride=(2, 2), name='pool2')
+ # group 3
+ conv3 = mx.symbol.Convolution(data=pool2[0], kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv3')
+ bn_conv3 = mx.symbol.BatchNorm(data=conv3, name='bn_conv3')
+ relu3 = mx.symbol.Activation(data=bn_conv3, act_type='relu', name='relu3')
+ pool3 = mx.symbol.PoolingMask(data=relu3, kernel=(2, 2), stride=(2, 2), name='pool3')
+ # group 4
+ conv4 = mx.symbol.Convolution(data=pool3[0], kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv4')
+ bn_conv4 = mx.symbol.BatchNorm(data=conv4, name='bn_conv4')
+ relu4 = mx.symbol.Activation(data=bn_conv4, act_type='relu', name='relu4')
+ pool4 = mx.symbol.PoolingMask(data=relu4, kernel=(2, 2), stride=(2, 2), name='pool4')
+ # deconve
+ # group 4
+ pool4_d = mx.symbol.UpSamplingMask(data=pool4[0], mask=pool4[1],
+ out_shape=(input_height/8, input_width/8), name="pool4_d")
+ conv4_d = mx.symbol.Convolution(data=pool4_d, kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv4_d')
+ bn_conv4_d = mx.symbol.BatchNorm(data=conv4_d, name='bn_conv4_d')
+ drop1 = mx.symbol.Dropout(data=bn_conv4_d, p=0.5, name='drop1')
+ # group 3
+ pool3_d = mx.symbol.UpSamplingMask(data=drop1, mask=pool3[1],
+ out_shape=(input_height/4, input_width/4), name="pool3_d")
+ conv3_d = mx.symbol.Convolution(data=pool3_d, kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv3_d')
+ bn_conv3_d = mx.symbol.BatchNorm(data=conv3_d, name='bn_conv3_d')
+ drop2 = mx.symbol.Dropout(data=bn_conv3_d, p=0.5, name='drop2')
+ # group 2
+ pool2_d = mx.symbol.UpSamplingMask(data=drop2, mask=pool2[1],
+ out_shape=(input_height/2, input_width/2), name="pool2_d")
+ conv2_d = mx.symbol.Convolution(data=pool2_d, kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv2_d')
+ bn_conv2_d = mx.symbol.BatchNorm(data=conv2_d, name='bn_conv2_d')
+ # group 1
+ pool1_d = mx.symbol.UpSamplingMask(data=bn_conv2_d, mask=pool1[1],
+ out_shape=(input_height, input_width), name="pool1_d")
+ conv1_d = mx.symbol.Convolution(data=pool1_d, kernel=(7, 7), pad=(3, 3), num_filter=64,
+ no_bias=True, name='conv1_d')
+ bn_conv1_d = mx.symbol.BatchNorm(data=conv1_d, name='bn_conv1_d')
+ # classfiier
+ conv_classfiier = mx.symbol.Convolution(data=bn_conv1_d, kernel=(1, 1),
+ num_filter=num_classes, name='conv_classfiier')
+ return conv_classfiier
+
+def get_symbol(num_classes, natural_balance=False):
+ """
+ get segnet_basic network
+ """
+ data = mx.symbol.Variable(name='data')
+ network = conv_deconv(data, num_classes)
+ if natural_balance:
+ weight = mx.symbol.Variable(name="softmax_weight", lr_mult=0)
+ label = mx.symbol.Variable(name='softmax_label', lr_mult=0)
+ softmax = mx.symbol.softmax(data=network, axis=1, name='softmax')
+ label = mx.symbol.one_hot(indices=label, depth=num_classes)
+ label = mx.symbol.swapaxes(label, 1, 3)
+ label = mx.symbol.swapaxes(label, 2, 3)
+ cross_entropy = mx.symbol.broadcast_mul(mx.symbol.log(softmax), label)
+ loss = mx.sym.MakeLoss(-cross_entropy*weight, grad_scale=1.0/(360*480))
+ return loss
+ else:
+ softmax = mx.symbol.SoftmaxOutput(data=network, multi_output=True, use_ignore=True,
+ ignore_label=11, name='softmax')
+ return softmax
diff --git a/example/segnet/symbols/segnet_bayesian.py b/example/segnet/symbols/segnet_bayesian.py
new file mode 100644
index 00000000000..5eadae30294
--- /dev/null
+++ b/example/segnet/symbols/segnet_bayesian.py
@@ -0,0 +1,187 @@
+import mxnet as mx
+def vgg_conv_deconv(input_data, num_classes, fix_gamma=False, eps=0.00002,
+ bn_mom=0.9, input_height=360, input_width=480):
+ """
+ get vgg encode and decode layers
+ """
+ # group 1
+ conv1_1 = mx.symbol.Convolution(data=input_data, kernel=(3, 3), pad=(1, 1), num_filter=64,
+ name="conv1_1")
+ bn_conv1_1 = mx.symbol.BatchNorm(data=conv1_1, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv1_1')
+ relu1_1 = mx.symbol.Activation(data=bn_conv1_1, act_type="relu", name="relu1_1")
+ conv1_2 = mx.symbol.Convolution(data=relu1_1, kernel=(3, 3), pad=(1, 1), num_filter=64,
+ name="conv1_2")
+ bn_conv1_2 = mx.symbol.BatchNorm(data=conv1_2, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv1_2')
+ relu1_2 = mx.symbol.Activation(data=bn_conv1_2, act_type="relu", name="relu1_2")
+ pool1 = mx.symbol.PoolingMask(data=relu1_2, kernel=(2, 2), stride=(2, 2), name="pool1")
+ # group 2
+ conv2_1 = mx.symbol.Convolution(data=pool1[0], kernel=(3, 3), pad=(1, 1), num_filter=128,
+ name="conv2_1")
+ bn_conv2_1 = mx.symbol.BatchNorm(data=conv2_1, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv2_1')
+ relu2_1 = mx.symbol.Activation(data=bn_conv2_1, act_type="relu", name="relu2_1")
+ conv2_2 = mx.symbol.Convolution(data=relu2_1, kernel=(3, 3), pad=(1, 1), num_filter=128,
+ name="conv2_2")
+ bn_conv2_2 = mx.symbol.BatchNorm(data=conv2_2, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv2_2')
+ relu2_2 = mx.symbol.Activation(data=bn_conv2_2, act_type="relu", name="relu2_2")
+ pool2 = mx.symbol.PoolingMask(data=relu2_2, kernel=(2, 2), stride=(2, 2), name="pool2")
+ # group 3
+ conv3_1 = mx.symbol.Convolution(data=pool2[0], kernel=(3, 3), pad=(1, 1), num_filter=256,
+ name="conv3_1")
+ bn_conv3_1 = mx.symbol.BatchNorm(data=conv3_1, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv3_1')
+ relu3_1 = mx.symbol.Activation(data=bn_conv3_1, act_type="relu", name="relu3_1")
+ conv3_2 = mx.symbol.Convolution(data=relu3_1, kernel=(3, 3), pad=(1, 1), num_filter=256,
+ name="conv3_2")
+ bn_conv3_2 = mx.symbol.BatchNorm(data=conv3_2, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv3_2')
+ relu3_2 = mx.symbol.Activation(data=bn_conv3_2, act_type="relu", name="relu3_2")
+ conv3_3 = mx.symbol.Convolution(data=relu3_2, kernel=(3, 3), pad=(1, 1), num_filter=256,
+ name="conv3_3")
+ bn_conv3_3 = mx.symbol.BatchNorm(data=conv3_3, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv3_3')
+ relu3_3 = mx.symbol.Activation(data=bn_conv3_3, act_type="relu", name="relu3_3")
+ pool3 = mx.symbol.PoolingMask(data=relu3_3, kernel=(2, 2), stride=(2, 2), name="pool3")
+ encdrop3 = mx.sym.Dropout(data=pool3[0], p=0.5, name="encdrop3")
+ # group 4
+ conv4_1 = mx.symbol.Convolution(data=encdrop3, kernel=(3, 3), pad=(1, 1), num_filter=512,
+ name="conv4_1")
+ bn_conv4_1 = mx.symbol.BatchNorm(data=conv4_1, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv4_1')
+ relu4_1 = mx.symbol.Activation(data=bn_conv4_1, act_type="relu", name="relu4_1")
+ conv4_2 = mx.symbol.Convolution(data=relu4_1, kernel=(3, 3), pad=(1, 1), num_filter=512,
+ name="conv4_2")
+ bn_conv4_2 = mx.symbol.BatchNorm(data=conv4_2, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv4_2')
+ relu4_2 = mx.symbol.Activation(data=bn_conv4_2, act_type="relu", name="relu4_2")
+ conv4_3 = mx.symbol.Convolution(data=relu4_2, kernel=(3, 3), pad=(1, 1), num_filter=512,
+ name="conv4_3")
+ bn_conv4_3 = mx.symbol.BatchNorm(data=conv4_3, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv4_3')
+ relu4_3 = mx.symbol.Activation(data=bn_conv4_3, act_type="relu", name="relu4_3")
+ pool4 = mx.symbol.PoolingMask(data=relu4_3, kernel=(2, 2), stride=(2, 2), name="pool4")
+ encdrop4 = mx.sym.Dropout(data=pool4[0], p=0.5, name="encdrop4")
+ # group 5
+ conv5_1 = mx.symbol.Convolution(data=encdrop4, kernel=(3, 3), pad=(1, 1), num_filter=512,
+ name="conv5_1")
+ bn_conv5_1 = mx.symbol.BatchNorm(data=conv5_1, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv5_1')
+ relu5_1 = mx.symbol.Activation(data=bn_conv5_1, act_type="relu", name="relu5_1")
+ conv5_2 = mx.symbol.Convolution(data=relu5_1, kernel=(3, 3), pad=(1, 1), num_filter=512,
+ name="conv5_2")
+ bn_conv5_2 = mx.symbol.BatchNorm(data=conv5_2, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv5_2')
+ relu5_2 = mx.symbol.Activation(data=bn_conv5_2, act_type="relu", name="relu5_2")
+ conv5_3 = mx.symbol.Convolution(data=relu5_2, kernel=(3, 3), pad=(1, 1), num_filter=512,
+ name="conv5_3")
+ bn_conv5_3 = mx.symbol.BatchNorm(data=conv5_3, fix_gamma=fix_gamma, eps=eps, momentum=bn_mom,
+ name='bn_conv5_3')
+ relu5_3 = mx.symbol.Activation(data=bn_conv5_3, act_type="relu", name="relu5_3")
+ pool5 = mx.symbol.PoolingMask(data=relu5_3, kernel=(2, 2), stride=(2, 2), name="pool5")
+ encdrop5 = mx.sym.Dropout(data=pool5[0], p=0.5, name="encdrop5")
+ # deconve
+ # group 5
+ pool5_d = mx.symbol.UpSamplingMask(data=encdrop5, mask=pool5[1],
+ out_shape=(input_height/16, input_width/16), name="pool5_d")
+ conv5_3_d = mx.symbol.Convolution(data=pool5_d, kernel=(3, 3), pad=(1, 1), num_filter=512,
+ name="conv5_3_d")
+ bn_conv5_3_d = mx.symbol.BatchNorm(data=conv5_3_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv5_3_d')
+ relu5_3_d = mx.symbol.Activation(data=bn_conv5_3_d, act_type="relu", name="relu5_3_d")
+ conv5_2_d = mx.symbol.Convolution(data=relu5_3_d, kernel=(3, 3), pad=(1, 1), num_filter=512,
+ name="conv5_2_d")
+ bn_conv5_2_d = mx.symbol.BatchNorm(data=conv5_2_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv5_2_d')
+ relu5_2_d = mx.symbol.Activation(data=bn_conv5_2_d, act_type="relu", name="relu5_2_d")
+ conv5_1_d = mx.symbol.Convolution(data=relu5_2_d, kernel=(3, 3), pad=(1, 1), num_filter=512,
+ name="conv5_1_d")
+ bn_conv5_1_d = mx.symbol.BatchNorm(data=conv5_1_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv5_1_d')
+ relu5_1_d = mx.symbol.Activation(data=bn_conv5_1_d, act_type="relu", name="relu5_1_d")
+ decdrop5 = mx.sym.Dropout(data=relu5_1_d, p=0.5, name="decdrop5")
+ # group 4
+ pool4_d = mx.symbol.UpSamplingMask(data=decdrop5, mask=pool4[1],
+ out_shape=(input_height/8, input_width/8), name="pool4_d")
+ conv4_3_d = mx.symbol.Convolution(data=pool4_d, kernel=(3, 3), pad=(1, 1), num_filter=512,
+ name="conv4_3_d")
+ bn_conv4_3_d = mx.symbol.BatchNorm(data=conv4_3_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv4_3_d')
+ relu4_3_d = mx.symbol.Activation(data=bn_conv4_3_d, act_type="relu", name="relu4_3_d")
+ conv4_2_d = mx.symbol.Convolution(data=relu4_3_d, kernel=(3, 3), pad=(1, 1), num_filter=512,
+ name="conv4_2_d")
+ bn_conv4_2_d = mx.symbol.BatchNorm(data=conv4_2_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv4_2_d')
+ relu4_2_d = mx.symbol.Activation(data=bn_conv4_2_d, act_type="relu", name="relu4_2_d")
+ conv4_1_d = mx.symbol.Convolution(data=relu4_2_d, kernel=(3, 3), pad=(1, 1), num_filter=256,
+ name="conv4_1_d")
+ bn_conv4_1_d = mx.symbol.BatchNorm(data=conv4_1_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv4_1_d')
+ relu4_1_d = mx.symbol.Activation(data=bn_conv4_1_d, act_type="relu", name="relu4_1_d")
+ decdrop4 = mx.sym.Dropout(data=relu4_1_d, p=0.5, name="decdrop4")
+ # group 3
+ pool3_d = mx.symbol.UpSamplingMask(data=decdrop4, mask=pool3[1],
+ out_shape=(input_height/4, input_width/4), name="pool3_d")
+ conv3_3_d = mx.symbol.Convolution(data=pool3_d, kernel=(3, 3), pad=(1, 1), num_filter=256,
+ name="conv3_3_d")
+ bn_conv3_3_d = mx.symbol.BatchNorm(data=conv3_3_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv3_3_d')
+ relu3_3_d = mx.symbol.Activation(data=bn_conv3_3_d, act_type="relu", name="relu3_3_d")
+ conv3_2_d = mx.symbol.Convolution(data=relu3_3_d, kernel=(3, 3), pad=(1, 1), num_filter=256,
+ name="conv3_2_d")
+ bn_conv3_2_d = mx.symbol.BatchNorm(data=conv3_2_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv3_2_d')
+ relu3_2_d = mx.symbol.Activation(data=bn_conv3_2_d, act_type="relu", name="relu3_2_d")
+ conv3_1_d = mx.symbol.Convolution(data=relu3_2_d, kernel=(3, 3), pad=(1, 1), num_filter=128,
+ name="conv3_1_d")
+ bn_conv3_1_d = mx.symbol.BatchNorm(data=conv3_1_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv3_1_d')
+ relu3_1_d = mx.symbol.Activation(data=bn_conv3_1_d, act_type="relu", name="relu3_1_d")
+ decdrop3 = mx.sym.Dropout(data=relu3_1_d, p=0.5, name="decdrop3")
+ # group 2
+ pool2_d = mx.symbol.UpSamplingMask(data=decdrop3, mask=pool2[1],
+ out_shape=(input_height/2, input_width/2), name="pool2_d")
+ conv2_2_d = mx.symbol.Convolution(data=pool2_d, kernel=(3, 3), pad=(1, 1), num_filter=128,
+ name="conv2_2_d")
+ bn_conv2_2_d = mx.symbol.BatchNorm(data=conv2_2_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv2_2_d')
+ relu2_2_d = mx.symbol.Activation(data=bn_conv2_2_d, act_type="relu", name="relu2_2_d")
+ conv2_1_d = mx.symbol.Convolution(data=relu2_2_d, kernel=(3, 3), pad=(1, 1), num_filter=64,
+ name="conv2_1_d")
+ bn_conv2_1_d = mx.symbol.BatchNorm(data=conv2_1_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv2_1_d')
+ relu2_1_d = mx.symbol.Activation(data=bn_conv2_1_d, act_type="relu", name="relu2_1_d")
+ # group 1
+ pool1_d = mx.symbol.UpSamplingMask(data=relu2_1_d, mask=pool1[1],
+ out_shape=(input_height, input_width), name="pool1_d")
+ conv1_2_d = mx.symbol.Convolution(data=pool1_d, kernel=(3, 3), pad=(1, 1), num_filter=64,
+ name="conv1_2_d")
+ bn_conv1_2_d = mx.symbol.BatchNorm(data=conv1_2_d, fix_gamma=fix_gamma, eps=eps,
+ momentum=bn_mom, name='bn_conv1_2_d')
+ relu1_2_d = mx.symbol.Activation(data=bn_conv1_2_d, act_type="relu", name="relu1_2_d")
+ conv1_1_d = mx.symbol.Convolution(data=relu1_2_d, kernel=(3, 3), pad=(1, 1), num_filter=num_classes,
+ name="conv1_1_d")
+ return conv1_1_d
+
+def get_symbol(num_classes, natural_balance=False, **kwargs):
+ """
+ get segnet network
+ """
+ data = mx.symbol.Variable(name='data')
+ network = vgg_conv_deconv(data, num_classes)
+ if natural_balance:
+ weight = mx.symbol.Variable(name="softmax_weight", lr_mult=0)
+ label = mx.symbol.Variable(name='softmax_label', lr_mult=0)
+ softmax = mx.symbol.softmax(data=network, axis=1, name='softmax')
+ label = mx.symbol.one_hot(indices=label, depth=num_classes)
+ label = mx.symbol.swapaxes(label, 1, 3)
+ label = mx.symbol.swapaxes(label, 2, 3)
+ cross_entropy = mx.symbol.broadcast_mul(mx.symbol.log(softmax), label)
+ loss = mx.sym.MakeLoss(-cross_entropy*weight, grad_scale=1.0/(360*480))
+ return loss
+ else:
+ softmax = mx.symbol.SoftmaxOutput(data=network, multi_output=True, use_ignore=True,
+ ignore_label=11, name='softmax')
+ return softmax
diff --git a/example/segnet/train_segnet.py b/example/segnet/train_segnet.py
new file mode 100644
index 00000000000..6609db42383
--- /dev/null
+++ b/example/segnet/train_segnet.py
@@ -0,0 +1,206 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import argparse
+import mxnet as mx
+import numpy as np
+import logging
+from data_iter import FileIter
+from common import contrib_metrics
+
+def _load_model(args, rank=0):
+ if 'load_epoch' not in args or args.load_epoch is None:
+ return (None, None, None)
+ assert args.model_prefix is not None
+ model_prefix = args.model_prefix
+ if rank > 0 and os.path.exists("%s-%d-symbol.json" % (model_prefix, rank)):
+ model_prefix += "-%d" % (rank)
+ sym, arg_params, aux_params = mx.model.load_checkpoint(
+ model_prefix, args.load_epoch)
+ logging.info('Loaded model %s_%04d.params', model_prefix, args.load_epoch)
+ return (sym, arg_params, aux_params)
+
+def _load_model_for_vgg16(args, rank=0):
+ if 'load_epoch' not in args or args.load_epoch_for_vgg16 is None:
+ return (None, None, None)
+ assert args.model_prefix_for_vgg16 is not None
+ model_prefix = args.model_prefix_for_vgg16
+ if rank > 0 and os.path.exists("%s-%d-symbol.json" % (model_prefix, rank)):
+ model_prefix += "-%d" % (rank)
+ sym, arg_params, aux_params = mx.model.load_checkpoint(
+ model_prefix, args.load_epoch_for_vgg16)
+ logging.info('Loaded model %s_%04d.params', model_prefix, args.load_epoch_for_vgg16)
+ return (sym, arg_params, aux_params)
+
+def _load_vgg_model(args, rank=0):
+ sym, arg_params, aux_params = _load_model(args, rank)
+ _, arg_params_vgg, aux_params_vgg = _load_model_for_vgg16(args, rank)
+ for name, val in arg_params_vgg.items():
+ if arg_params.get(name) is not None:
+ arg_params[name] = val
+ for name, val in aux_params_vgg.items():
+ if aux_params.get(name) is not None:
+ aux_params[name] = val
+ return (sym, arg_params, aux_params)
+
+def train(args):
+ """
+ train segnet
+ """
+ # set up logger
+ logging.basicConfig()
+ logger = logging.getLogger()
+ logger.setLevel(logging.INFO)
+ if args.log_file:
+ file_path = logging.FileHandler(args.log_file)
+ logger.addHandler(file_path)
+
+ if args.gpus is None or args.gpus is '':
+ ctx = mx.cpu()
+ else:
+ ctx = [mx.gpu(int(i)) for i in args.gpus.split(',')]
+
+ train_dataiter = FileIter(
+ batch_size=args.batch_size,
+ root_dir=args.data_path,
+ flist_name=args.train_file,
+ rgb_mean=[float(i) for i in args.rgb_mean.split(',')],
+ shuffle=True
+ )
+ val_dataiter = FileIter(
+ batch_size=args.batch_size,
+ root_dir=args.data_path,
+ flist_name=args.val_file,
+ rgb_mean=[float(i) for i in args.rgb_mean.split(',')],
+ shuffle=False)
+ # load model
+ from importlib import import_module
+ network = import_module('symbols.'+args.network).get_symbol(args.num_classes)
+ # callback fuc
+ batch_end_callback = [mx.callback.Speedometer(args.batch_size, args.disp_batches)]
+ epoch_end_callback = mx.callback.do_checkpoint(args.model_prefix, args.save_frequency)
+ # kvstore
+ kv = mx.kvstore.create(args.kv_store)
+ # optimizer params
+ if 'lr_factor' not in args or args.lr_factor >= 1:
+ lr_scheduler = None
+ else:
+ steps = [int(l) for l in args.lr_steps.split(',')]
+ lr_scheduler = mx.lr_scheduler.MultiFactorScheduler(step=steps, factor=args.lr_factor)
+ optimizer_params = {'learning_rate': args.lr,
+ 'multi_precision': True,
+ 'momentum' : args.mom,
+ 'wd' : args.wd,
+ 'lr_scheduler': lr_scheduler}
+ # evaluation metrices
+ eval_metrics = [contrib_metrics.Accuracy(ignore_label=11),
+ contrib_metrics.CrossEntropy()]
+ # parameter init fuc
+ softmax_weight = mx.nd.array(np.array([0.2595, 0.1826, 4.5640, 0.1417, 0.9051, 0.3826,
+ 9.6446, 1.8418, 0.6823, 6.2478, 7.3614]))
+ height = train_dataiter.data[0][1].shape[2]
+ width = train_dataiter.data[0][1].shape[3]
+ softmax_weight = softmax_weight.reshape((1, 11, 1, 1))
+ softmax_weight = softmax_weight.broadcast_to((args.batch_size/len(ctx), 11, height, width))
+ xavier = mx.init.Xavier(rnd_type='gaussian', factor_type="in", magnitude=2)
+ initializer = mx.init.Mixed(['softmax_weight', '.*'],
+ [mx.init.Load({'softmax_weight':softmax_weight}), xavier])
+ # load model
+ if 'load_epoch' not in args or args.load_epoch_for_vgg16 is not None:
+ sym, arg_params, aux_params = _load_vgg_model(args, kv.rank)
+ else:
+ sym, arg_params, aux_params = _load_model(args, kv.rank)
+
+ if sym is not None:
+ assert sym.tojson() == network.tojson()
+ # create model
+ model = mx.mod.Module(context=ctx, symbol=network, logger=logger)
+
+ model.fit(
+ train_data=train_dataiter,
+ begin_epoch=args.load_epoch if args.load_epoch else 0,
+ eval_data=val_dataiter,
+ num_epoch=args.num_epochs,
+ eval_metric=eval_metrics,
+ kvstore=kv,
+ optimizer=args.optimizer,
+ optimizer_params=optimizer_params,
+ initializer=initializer,
+ arg_params=arg_params,
+ aux_params=aux_params,
+ batch_end_callback=batch_end_callback,
+ epoch_end_callback=epoch_end_callback,
+ allow_missing=True)
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser(description='Convert vgg16 model to vgg16fc model.')
+ parser.add_argument('--gpus', type=str, default='0',
+ help='list of gpus to run, e.g. 0 or 0,2,5. empty means using cpu')
+ parser.add_argument('--kv_store', type=str, default='device',
+ help='key-value store type')
+ parser.add_argument('--batch_size', type=int, default=12,
+ help='the batch size')
+ parser.add_argument('--disp_batches', type=int, default=5,
+ help='show progress for every n batches')
+ parser.add_argument('--data_path', type=str, default="/data/CamVid",
+ help='data path')
+ parser.add_argument('--train_file', type=str, default="train.txt",
+ help='train data list file name in data path')
+ parser.add_argument('--val_file', type=str, default="test.txt",
+ help='val data list file name in data path')
+ parser.add_argument('--num_classes', type=int, default=11,
+ help='num classes to clissufy')
+ parser.add_argument('--model', default='fcnxs',
+ help='The type of fcn-xs model, e.g. fcnxs, fcn16s, fcn8s.')
+ parser.add_argument('--model_prefix', type=str, default='models/segnet',
+ help='model prefix')
+ parser.add_argument('--load_epoch', type=int,
+ help='load the model on an epoch using the model-load-prefix')
+ parser.add_argument('--model_prefix_for_vgg16', type=str, default='vgg16',
+ help='model prefix')
+ parser.add_argument('--load_epoch_for_vgg16', type=int,
+ help='load the model on an epoch using the model-load-prefix')
+ parser.add_argument('--init-type', default="vgg16",
+ help='the init type of fcn-xs model, e.g. vgg16, fcnxs')
+ parser.add_argument('--retrain', action='store_true', default=False,
+ help='true means continue training.')
+ parser.add_argument('--num_epochs', type=int, default=250,
+ help='number of epochs to train')
+ parser.add_argument('--lr', type=float, default=0.005,
+ help='initial learning rate')
+ parser.add_argument('--lr_factor', type=float, default=0.1,
+ help='the ratio to reduce lr on each step')
+ parser.add_argument('--lr_steps', type=str, default='5000, 7000',
+ help='the epochs to reduce the lr, e.g. 30,60')
+ parser.add_argument('--save_frequency', type=int, default=5,
+ help='the frequency to save model')
+ parser.add_argument('--mom', type=float, default=0.9,
+ help='momentum for sgd')
+ parser.add_argument('--wd', type=float, default=0.0005,
+ help='weight decay for sgd')
+ parser.add_argument('--optimizer', type=str, default='sgd',
+ help='the optimizer type')
+ parser.add_argument('--log_file', type=str, default="log.txt",
+ help='the name of log file')
+ parser.add_argument('--rgb_mean', type=str, default='123.68, 116.779, 103.939',
+ help='rgb mean')
+ parser.add_argument('--network', type=str, required=True,
+ help='the neural network to use')
+ args = parser.parse_args()
+ logging.info(args)
+ train(args)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services