You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/03/23 20:38:00 UTC
[GitHub] lanking520 closed pull request #10179: [MXNET-116] object detector class added
lanking520 closed pull request #10179: [MXNET-116] object detector class added
URL: https://github.com/apache/incubator-mxnet/pull/10179
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/ci/docker/install/ubuntu_onnx.sh b/ci/docker/install/ubuntu_onnx.sh
index 72613cd5788..07acba01908 100755
--- a/ci/docker/install/ubuntu_onnx.sh
+++ b/ci/docker/install/ubuntu_onnx.sh
@@ -30,5 +30,5 @@ echo "Installing libprotobuf-dev and protobuf-compiler ..."
apt-get install -y libprotobuf-dev protobuf-compiler
echo "Installing pytest, pytest-cov, protobuf, Pillow, ONNX and tabulate ..."
-pip2 install pytest==3.4.0 pytest-cov==2.5.1 protobuf==3.0.0 onnx==1.0.1 Pillow==5.0.0 tabulate==0.7.5
-pip3 install pytest==3.4.0 pytest-cov==2.5.1 protobuf==3.0.0 onnx==1.0.1 Pillow==5.0.0 tabulate==0.7.5
+pip2 install pytest==3.4.0 pytest-cov==2.5.1 protobuf==3.5.2 onnx==1.1.1 Pillow==5.0.0 tabulate==0.7.5
+pip3 install pytest==3.4.0 pytest-cov==2.5.1 protobuf==3.5.2 onnx==1.1.1 Pillow==5.0.0 tabulate==0.7.5
diff --git a/docs/install/index.md b/docs/install/index.md
index 44395dcb7b4..d9d78dd3693 100644
--- a/docs/install/index.md
+++ b/docs/install/index.md
@@ -994,11 +994,11 @@ Refer to [#8671](https://github.com/apache/incubator-mxnet/issues/8671) for stat
<br/>
To build and install MXNet yourself, you need the following dependencies. Install the required dependencies:
-1. If [Microsoft Visual Studio 2015](https://www.visualstudio.com/downloads/) is not already installed, download and install it. You can download and install the free community edition.
+1. If [Microsoft Visual Studio 2015](https://www.visualstudio.com/vs/older-downloads/) is not already installed, download and install it. You can download and install the free community edition.
2. Download and install [CMake](https://cmake.org/) if it is not already installed.
3. Download and install [OpenCV](http://sourceforge.net/projects/opencvlibrary/files/opencv-win/3.0.0/opencv-3.0.0.exe/download).
4. Unzip the OpenCV package.
-5. Set the environment variable ```OpenCV_DIR``` to point to the ```OpenCV build directory```.
+5. Set the environment variable ```OpenCV_DIR``` to point to the ```OpenCV build directory``` (```C:\opencv\build\x64\vc14``` for example). Also, you need to add the OpenCV bin directory (```C:\opencv\build\x64\vc14\bin``` for example) to the ``PATH`` variable.
6. If you don't have the Intel Math Kernel Library (MKL) installed, download and install [OpenBlas](http://sourceforge.net/projects/openblas/files/v0.2.14/).
7. Set the environment variable ```OpenBLAS_HOME``` to point to the ```OpenBLAS``` directory that contains the ```include``` and ```lib``` directories. Typically, you can find the directory in ```C:\Program files (x86)\OpenBLAS\```.
8. Download and install [CUDA](https://developer.nvidia.com/cuda-downloads?target_os=Windows&target_arch=x86_64) and [cuDNN](https://developer.nvidia.com/cudnn). To get access to the download link, register as an NVIDIA community user.
diff --git a/docs/install/windows_setup.md b/docs/install/windows_setup.md
index 598a12fc4cc..09a39e2c469 100755
--- a/docs/install/windows_setup.md
+++ b/docs/install/windows_setup.md
@@ -21,11 +21,11 @@ This produces a library called ```libmxnet.dll```.
To build and install MXNet yourself, you need the following dependencies. Install the required dependencies:
-1. If [Microsoft Visual Studio 2015](https://www.visualstudio.com/downloads/) is not already installed, download and install it. You can download and install the free community edition.
+1. If [Microsoft Visual Studio 2015](https://www.visualstudio.com/vs/older-downloads/) is not already installed, download and install it. You can download and install the free community edition.
2. Download and Install [CMake](https://cmake.org/) if it is not already installed.
3. Download and install [OpenCV](http://sourceforge.net/projects/opencvlibrary/files/opencv-win/3.0.0/opencv-3.0.0.exe/download).
4. Unzip the OpenCV package.
-5. Set the environment variable ```OpenCV_DIR``` to point to the ```OpenCV build directory``` (```c:\utils\opencv\build``` for example).
+5. Set the environment variable ```OpenCV_DIR``` to point to the ```OpenCV build directory``` (```C:\opencv\build\x64\vc14``` for example). Also, you need to add the OpenCV bin directory (```C:\opencv\build\x64\vc14\bin``` for example) to the ``PATH`` variable.
6. If you have Intel Math Kernel Library (MKL) installed, set ```MKL_ROOT``` to point to ```MKL``` directory that contains the ```include``` and ```lib```. Typically, you can find the directory in
```C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2018\windows\mkl```.
7. If you don't have the Intel Math Kernel Library (MKL) installed, download and install [OpenBlas](http://sourceforge.net/projects/openblas/files/v0.2.14/).
diff --git a/include/mxnet/base.h b/include/mxnet/base.h
index 0e58765aaf4..c0eb97aa0b3 100644
--- a/include/mxnet/base.h
+++ b/include/mxnet/base.h
@@ -365,6 +365,10 @@ inline std::ostream& operator<<(std::ostream &out, const Context &ctx) {
#define MXNET_DESCRIBE(...) describe(__VA_ARGS__ "\n\nFrom:" __FILE__ ":" STRINGIZE(__LINE__))
#define ADD_FILELINE "\n\nDefined in " __FILE__ ":L" STRINGIZE(__LINE__)
+#if MXNET_USE_MKLDNN == 1
+constexpr size_t kMKLDNNAlign = 64;
+#endif
+
} // namespace mxnet
#include "./tensor_blob.h"
diff --git a/plugin/caffe/caffe_common.h b/plugin/caffe/caffe_common.h
index ba7b9ad7b8d..211d8c44d51 100644
--- a/plugin/caffe/caffe_common.h
+++ b/plugin/caffe/caffe_common.h
@@ -91,7 +91,7 @@ class LayerRegistry {
/*! \brief override type_name for caffe::LayerParameter */
namespace dmlc {
- DMLC_DECLARE_TYPE_NAME(::caffe::LayerParameter, "caffe-layer-parameter")
+ DMLC_DECLARE_TYPE_NAME(::caffe::LayerParameter, "caffe-layer-parameter");
}
#endif // PLUGIN_CAFFE_CAFFE_COMMON_H_
diff --git a/scala-package/examples/scripts/inferexample/objectdetector/get_ssd_data.sh b/scala-package/examples/scripts/inferexample/objectdetector/get_ssd_data.sh
new file mode 100755
index 00000000000..824a96b2733
--- /dev/null
+++ b/scala-package/examples/scripts/inferexample/objectdetector/get_ssd_data.sh
@@ -0,0 +1,45 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+set -e
+
+MXNET_ROOT=$(cd "$(dirname $0)/../../../.."; pwd)
+
+data_path=$MXNET_ROOT/examples/scripts/inferexample/models/resnet50_ssd
+
+image_path=$MXNET_ROOT/examples/scripts/inferexample/images
+
+if [ ! -d "$data_path" ]; then
+ mkdir -p "$data_path"
+fi
+
+if [ ! -d "$image_path" ]; then
+ mkdir -p "$image_path"
+fi
+
+if [ ! -f "$data_path" ]; then
+ wget https://s3.amazonaws.com/model-server/models/resnet50_ssd/resnet50_ssd_model-symbol.json -P $data_path
+ wget https://s3.amazonaws.com/model-server/models/resnet50_ssd/resnet50_ssd_model-0000.params -P $data_path
+ wget https://raw.githubusercontent.com/awslabs/mxnet-model-server/master/examples/ssd/synset.txt -P $data_path
+ cd $image_path
+ wget https://cloud.githubusercontent.com/assets/3307514/20012566/cbb53c76-a27d-11e6-9aaa-91939c9a1cd5.jpg -O 000001.jpg
+ wget https://cloud.githubusercontent.com/assets/3307514/20012567/cbb60336-a27d-11e6-93ff-cbc3f09f5c9e.jpg -O dog.jpg
+ wget https://cloud.githubusercontent.com/assets/3307514/20012563/cbb41382-a27d-11e6-92a9-18dab4fd1ad3.jpg -O person.jpg
+fi
\ No newline at end of file
diff --git a/scala-package/examples/scripts/inferexample/objectdetector/run_ssd_example.sh b/scala-package/examples/scripts/inferexample/objectdetector/run_ssd_example.sh
new file mode 100755
index 00000000000..90a6a9c0168
--- /dev/null
+++ b/scala-package/examples/scripts/inferexample/objectdetector/run_ssd_example.sh
@@ -0,0 +1,35 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+MXNET_ROOT=$(cd "$(dirname $0)/../../../../../"; pwd)
+CLASS_PATH=$MXNET_ROOT/scala-package/assembly/osx-x86_64-cpu/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/*:$MXNET_ROOT/scala-package/infer/target/*
+
+# model dir and prefix
+MODEL_DIR=$1
+# input image
+INPUT_IMG=$2
+# which input image dir
+INPUT_DIR=$3
+
+java -Xmx8G -cp $CLASS_PATH \
+ ml.dmlc.mxnetexamples.inferexample.objectdetector.SSDClassifierExample \
+ --model-dir $MODEL_DIR \
+ --input-image $INPUT_IMG \
+ --input-dir $INPUT_DIR
\ No newline at end of file
diff --git a/scala-package/examples/src/main/scala/ml/dmlc/mxnetexamples/inferexample/objectdetector/README.md b/scala-package/examples/src/main/scala/ml/dmlc/mxnetexamples/inferexample/objectdetector/README.md
new file mode 100644
index 00000000000..03b3bdc0847
--- /dev/null
+++ b/scala-package/examples/src/main/scala/ml/dmlc/mxnetexamples/inferexample/objectdetector/README.md
@@ -0,0 +1,116 @@
+# Single Shot Multi Object Detection using Scala Inference API
+
+In this example, you will learn how to use Scala Inference API to import pre-trained Single Shot Multi Object Detection (SSD) MXNet model.
+
+The pre-trained model is trained on the [Pascal VOC 2012 dataset](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/index.html). The network is a SSD model built on Resnet50 as base network to extract image features. The model is trained to detect the following entities (classes): ['aeroplane', 'bicycle', 'bird', 'boat', 'bottle', 'bus', 'car', 'cat', 'chair', 'cow', 'diningtable', 'dog', 'horse', 'motorbike', 'person', 'pottedplant', 'sheep', 'sofa', 'train', 'tvmonitor']. For more details about the model, you can refer to the [MXNet SSD example](https://github.com/apache/incubator-mxnet/tree/master/example/ssd).
+
+
+## Contents
+
+1. [Prerequisites](#prerequisites)
+2. [Download artifacts](#download-artifacts)
+3. [Setup datapath and parameters](#setup-datapath-and-parameters)
+4. [Run the image inference example](#run-the-image-inference-example)
+5. [Infer APIs](#infer-api-details)
+6. [Next steps](#next-steps)
+
+
+## Prerequisites
+
+1. MXNet
+2. MXNet Scala Package
+3. [IntelliJ IDE (or alternative IDE) project setup](http://mxnet.incubator.apache.org/tutorials/scala/mxnet_scala_on_intellij.html) with the MXNet Scala Package
+4. wget
+
+
+## Setup Guide
+
+### Download Artifacts
+#### Step 1
+You can download the files using the script `get_ssd_data.sh`. It will download and place the model files in a `model` folder and the test image files in a `image` folder in the current directory.
+From the `scala-package/examples/scripts/inferexample/imageclassifier/` folder run:
+
+```bash
+./get_resnet_data.sh
+```
+
+**Note**: You may need to run `chmod +x get_resnet_data.sh` before running this script.
+
+Alternatively use the following links to download the Symbol and Params files via your browser:
+- [resnet50_ssd_model-symbol.json](https://s3.amazonaws.com/model-server/models/resnet50_ssd/resnet50_ssd_model-symbol.json)
+- [resnet50_ssd_model-0000.params](https://s3.amazonaws.com/model-server/models/resnet50_ssd/resnet50_ssd_model-0000.params)
+- [synset.txt](https://github.com/awslabs/mxnet-model-server/blob/master/examples/ssd/synset.txt)
+
+In the pre-trained model, the `input_name` is `data` and shape is `(1, 3, 512, 512)`.
+This shape translates to: a batch of `1` image, the image has color and uses `3` channels (RGB), and the image has the dimensions of `512` pixels in height by `512` pixels in width.
+
+The signature also specifies `image/jpeg` as the expected input type, since this example's image pre-processor only supports the handling of binary JPEG images.
+
+The signature specifies the output shape is `(1, 6132, 6)`. As with the input, the `1` is the number of images. `6132` is the number of prediction results, and `6` is for the size of each prediction. Each prediction contains the following components:
+- `Class`
+- `Accuracy`
+- `Xmin`
+- `Ymin`
+- `Xmax`
+- `Ymax`
+
+
+### Setup Datapath and Parameters
+#### Step 2
+The code `Line 31: val baseDir = System.getProperty("user.dir")` in the example will automatically searches the work directory you have defined. Please put the files in your [work directory](https://stackoverflow.com/questions/16239130/java-user-dir-property-what-exactly-does-it-mean). <!-- how do you define the work directory? -->
+
+Alternatively, if you would like to use your own path, please change line 31 into your own path
+```scala
+val baseDir = <Your Own Path>
+```
+
+The followings is the parameters defined for this example, you can find more information in the `class SSDClassifierExample`.
+
+| Argument | Comments |
+| ----------------------------- | ---------------------------------------- |
+| `model-dir` | Folder path with prefix to the model (including json, params, and any synset file). |
+| `input-image` | The image to run inference on. |
+| `input-dir` | The directory of images to run inference on. |
+
+
+## How to Run Inference
+After the previous steps, you should be able to run the code using the following script that will pass all of the required parameters to the Infer API.
+
+From the `scala-package/examples/scripts/inferexample/objectdetector/` folder run:
+
+```bash
+./run_ssd_example.sh ../model/resnet50_ssd_model ../image/dog.jpg ../image
+```
+
+**Notes**:
+* These are relative paths to this script.
+* You may need to run `chmod +x run_ssd_example.sh` before running this script.
+
+The example should give expected output as shown below:
+```
+Class: car
+Probabilties: 0.99847263
+(Coord:,312.21335,72.0291,456.01443,150.66176)
+Class: bicycle
+Probabilties: 0.90473825
+(Coord:,155.95807,149.96362,383.8369,418.94513)
+Class: dog
+Probabilties: 0.8226818
+(Coord:,83.82353,179.13998,206.63783,476.7875)
+```
+The first output came from the image `dog.jpg`, with top3 prediction picked.
+
+
+## Infer API Details
+This example uses ObjectDetector class provided by MXNet's scala package Infer APIs. It provides methods to load the images, create NDArray out of BufferedImage and run prediction using Classifier and Predictor APIs.
+
+
+## References
+This documentation used the model and inference setup guide from the [MXNet Model Server SSD example](https://github.com/awslabs/mxnet-model-server/blob/master/examples/ssd/README.md).
+
+
+## Next Steps
+
+Check out the following related tutorials and examples for the Infer API:
+
+* [Image Classification with the MXNet Scala Infer API](../imageclassifier/README.md)
\ No newline at end of file
diff --git a/scala-package/examples/src/main/scala/ml/dmlc/mxnetexamples/inferexample/objectdetector/SSDClassifierExample.scala b/scala-package/examples/src/main/scala/ml/dmlc/mxnetexamples/inferexample/objectdetector/SSDClassifierExample.scala
new file mode 100644
index 00000000000..e9e0f6a70a7
--- /dev/null
+++ b/scala-package/examples/src/main/scala/ml/dmlc/mxnetexamples/inferexample/objectdetector/SSDClassifierExample.scala
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnetexamples.inferexample.objectdetector
+
+import ml.dmlc.mxnet.{DType, Shape, DataDesc}
+import ml.dmlc.mxnet.infer._
+import org.kohsuke.args4j.{CmdLineParser, Option}
+import org.slf4j.LoggerFactory
+
+import scala.collection.JavaConverters._
+import java.nio.file.{Files, Paths}
+
+class SSDClassifierExample {
+ @Option(name = "--model-dir", usage = "the input model directory and prefix of the model")
+ private val modelPathPrefix: String = "/model/ssd_resnet50_512"
+ @Option(name = "--input-image", usage = "the input image")
+ private val inputImagePath: String = "/images/dog.jpg"
+ @Option(name = "--input-dir", usage = "the input batch of images directory")
+ private val inputImageDir: String = "/images/"
+}
+
+object SSDClassifierExample {
+
+ private val logger = LoggerFactory.getLogger(classOf[SSDClassifierExample])
+ private type SSDOut = (String, Array[Float])
+
+ def runObjectDetectionSingle(modelPathPrefix: String, inputImagePath: String):
+ IndexedSeq[IndexedSeq[(String, Array[Float])]] = {
+ val dType = DType.Float32
+ val inputShape = Shape(1, 3, 512, 512)
+ // ssd detections, numpy.array([[id, score, x1, y1, x2, y2]...])
+ val outputShape = Shape(1, 6132, 6)
+ val inputDescriptors = IndexedSeq(DataDesc("data", inputShape, dType, "NCHW"))
+ val img = ImageClassifier.loadImageFromFile(inputImagePath)
+ val objDetector = new ObjectDetector(modelPathPrefix, inputDescriptors)
+ val output = objDetector.imageObjectDetect(img, Some(3))
+
+ output
+ }
+
+ def runObjectDetectionBatch(modelPathPrefix: String, inputImageDir: String):
+ IndexedSeq[IndexedSeq[(String, Array[Float])]] = {
+ val dType = DType.Float32
+ val inputShape = Shape(1, 3, 512, 512)
+ // ssd detections, numpy.array([[id, score, x1, y1, x2, y2]...])
+ val outputShape = Shape(1, 6132, 6)
+ val inputDescriptors = IndexedSeq(DataDesc("data", inputShape, dType, "NCHW"))
+ val imgList = ImageClassifier.loadInputBatch(inputImageDir)
+ val objDetector = new ObjectDetector(modelPathPrefix, inputDescriptors)
+ val outputList = objDetector.imageBatchObjectDetect(imgList, Some(1))
+ outputList
+ }
+
+ def main(args: Array[String]): Unit = {
+ val inst = new SSDClassifierExample
+ val parser : CmdLineParser = new CmdLineParser(inst)
+ parser.parseArgument(args.toList.asJava)
+ val baseDir = System.getProperty("user.dir")
+ val mdprefixDir = baseDir + inst.modelPathPrefix
+ val imgPath = baseDir + inst.inputImagePath
+ val imgDir = baseDir + inst.inputImageDir
+ if (!checkExist(Array(mdprefixDir + "-symbol.json", imgDir, imgPath))) {
+ logger.error("Model or input image path does not exist")
+ sys.exit(1)
+ }
+
+ try {
+ val inputShape = Shape(1, 3, 512, 512)
+ val outputShape = Shape(1, 6132, 6)
+
+ val width = inputShape(2)
+ val height = inputShape(3)
+ var outputStr : String = "\n"
+
+ val output = runObjectDetectionSingle(mdprefixDir, imgPath)
+
+
+ for (ele <- output) {
+ for (i <- ele) {
+ outputStr += "Class: " + i._1 + "\n"
+ val arr = i._2
+ outputStr += "Probabilties: " + arr(0) + "\n"
+ val coord = Array[Float](
+ arr(1) * width, arr(2) * height,
+ arr(3) * width, arr(4) * height
+ )
+ outputStr += "Coord:" + coord.mkString(",") + "\n"
+ }
+ }
+ logger.info(outputStr)
+
+ val outputList = runObjectDetectionBatch(mdprefixDir, imgDir)
+
+ outputStr = "\n"
+ for (idx <- outputList.indices) {
+ outputStr += "*** Image " + (idx + 1) + "***" + "\n"
+ for (i <- outputList(idx)) {
+ outputStr += "Class: " + i._1 + "\n"
+ val arr = i._2
+ outputStr += "Probabilties: " + arr(0) + "\n"
+ val coord = Array[Float](
+ arr(1) * width, arr(2) * height,
+ arr(3) * width, arr(4) * height
+ )
+ outputStr += "Coord:" + coord.mkString(",") + "\n"
+ }
+ }
+ logger.info(outputStr)
+
+ } catch {
+ case ex: Exception => {
+ logger.error(ex.getMessage, ex)
+ parser.printUsage(System.err)
+ sys.exit(1)
+ }
+ }
+ sys.exit(0)
+ }
+
+
+ def checkExist(arr : Array[String]) : Boolean = {
+ var exist : Boolean = true
+ for (item <- arr) {
+ exist = Files.exists(Paths.get(item)) && exist
+ if (!exist) {
+ logger.error("Cannot find: " + item)
+ }
+ }
+ exist
+ }
+
+}
diff --git a/scala-package/examples/src/test/scala/ml/dmlc/mxnetexamples/inferexample/objectdetector/ObjectDetectorExampleSuite.scala b/scala-package/examples/src/test/scala/ml/dmlc/mxnetexamples/inferexample/objectdetector/ObjectDetectorExampleSuite.scala
new file mode 100644
index 00000000000..67480babf47
--- /dev/null
+++ b/scala-package/examples/src/test/scala/ml/dmlc/mxnetexamples/inferexample/objectdetector/ObjectDetectorExampleSuite.scala
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnetexamples.inferexample.objectdetector
+
+import org.scalatest.{BeforeAndAfterAll, FunSuite}
+import org.slf4j.LoggerFactory
+import scala.sys.process.Process
+
+class ObjectDetectorExampleSuite extends FunSuite with BeforeAndAfterAll {
+ private val logger = LoggerFactory.getLogger(classOf[ObjectDetectorExampleSuite])
+
+ test("testObjectDetectionExample") {
+ logger.info("Downloading resnetssd model")
+ val tempDirPath = System.getProperty("java.io.tmpdir")
+
+ logger.info("tempDirPath: %s".format(tempDirPath))
+
+ val modelBase = "https://s3.amazonaws.com/model-server/models/resnet50_ssd/"
+ val synsetBase = "https://raw.githubusercontent.com/awslabs/mxnet-model-server/master/examples/"
+ val imageBase = "https://cloud.githubusercontent.com/assets/3307514/"
+
+ Process("wget " + modelBase + "resnet50_ssd_model-symbol.json " + "-P " +
+ tempDirPath + "resnetssd/ -q") !
+
+
+ Process("wget " + modelBase + "resnet50_ssd_model-0000.params " +
+ "-P " + tempDirPath + "resnetssd/ -q") !
+
+
+ Process("wget " + synsetBase + "ssd/synset.txt " + "-P" +
+ tempDirPath + "resnetssd/ -q") !
+
+ Process("wget " +
+ imageBase + "20012567/cbb60336-a27d-11e6-93ff-cbc3f09f5c9e.jpg " +
+ "-P " + tempDirPath + "inputImages/") !
+
+
+ val modelDirPath = tempDirPath + "resnetssd/"
+ val inputImagePath = tempDirPath +
+ "inputImages/cbb60336-a27d-11e6-93ff-cbc3f09f5c9e.jpg"
+ val inputImageDir = tempDirPath + "inputImages/"
+
+ val output = SSDClassifierExample.runObjectDetectionSingle(modelDirPath + "resnet50_ssd_model",
+ inputImagePath)
+
+ assert(output(0)(0)._1 === "car")
+
+ val outputList = SSDClassifierExample.runObjectDetectionBatch(
+ modelDirPath + "resnet50_ssd_model",
+ inputImageDir)
+
+ assert(output(0)(0)._1 === "car")
+
+ Process("rm -rf " + modelDirPath + " " + inputImageDir) !
+ }
+}
diff --git a/scala-package/infer/src/main/scala/ml/dmlc/mxnet/infer/ObjectDetector.scala b/scala-package/infer/src/main/scala/ml/dmlc/mxnet/infer/ObjectDetector.scala
new file mode 100644
index 00000000000..fe373a927ae
--- /dev/null
+++ b/scala-package/infer/src/main/scala/ml/dmlc/mxnet/infer/ObjectDetector.scala
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnet.infer
+// scalastyle:off
+import java.awt.image.BufferedImage
+// scalastyle:on
+import ml.dmlc.mxnet.NDArray
+import ml.dmlc.mxnet.DataDesc
+import scala.collection.mutable.ListBuffer
+/**
+ * A class for object detection tasks
+ *
+ * @param modelPathPrefix PathPrefix from where to load the symbol, parameters and synset.txt
+ * Example: file://model-dir/ssd_resnet50_512
+ * (will resolve both ssd_resnet50_512-symbol.json
+ * and ssd_resnet50_512-0000.params)
+ * file://model-dir/synset.txt
+ * @param inputDescriptors Descriptors defining the input node names, shape,
+ * layout and Type parameters
+ */
+class ObjectDetector(modelPathPrefix: String,
+ inputDescriptors: IndexedSeq[DataDesc]) {
+
+ val classifier: Classifier = getClassifier(modelPathPrefix, inputDescriptors)
+
+ val inputLayout = inputDescriptors(0).layout
+
+ val inputShape = inputDescriptors(0).shape
+
+ val handler = classifier.handler
+
+ val predictor = classifier.predictor
+
+ val synset = classifier.synset
+
+ // Considering 'NCHW' as default layout when not provided
+ // Else get axis according to the layout
+ val height = inputShape(if (inputLayout.indexOf('H')<0) 2 else inputLayout.indexOf('H'))
+ val width = inputShape(if (inputLayout.indexOf('W')<0) 3 else inputLayout.indexOf('W'))
+
+ /**
+ * To Detect bounding boxes and corresponding labels
+ *
+ * @param inputImage : PathPrefix of the input image
+ * @param topK : Get top k elements with maximum probability
+ * @return List of List of tuples of (class, [probability, xmin, ymin, xmax, ymax])
+ */
+ def imageObjectDetect(inputImage: BufferedImage,
+ topK: Option[Int] = None)
+ : IndexedSeq[IndexedSeq[(String, Array[Float])]] = {
+
+ val scaledImage = ImageClassifier.reshapeImage(inputImage, width, height)
+ val pixelsNDArray = ImageClassifier.bufferedImageToPixels(scaledImage, inputShape)
+ val output = objectDetectWithNDArray(IndexedSeq(pixelsNDArray), topK)
+ handler.execute(pixelsNDArray.dispose())
+ output
+ }
+
+ /**
+ * Takes input images as NDArrays. Useful when you want to perform multiple operations on
+ * the input Array, or when you want to pass a batch of input images.
+ * @param input : Indexed Sequence of NDArrays
+ * @param topK : (Optional) How many top_k(sorting will be based on the last axis)
+ * elements to return. If not passed, returns all unsorted output.
+ * @return List of List of tuples of (class, [probability, xmin, ymin, xmax, ymax])
+ */
+ def objectDetectWithNDArray(input: IndexedSeq[NDArray], topK: Option[Int])
+ : IndexedSeq[IndexedSeq[(String, Array[Float])]] = {
+
+ val predictResult = predictor.predictWithNDArray(input)(0)
+ var batchResult = ListBuffer[IndexedSeq[(String, Array[Float])]]()
+ for (i <- 0 until predictResult.shape(0)) {
+ val r = predictResult.at(i)
+ batchResult += sortAndReformat(r, topK)
+ handler.execute(r.dispose())
+ }
+ handler.execute(predictResult.dispose())
+ batchResult.toIndexedSeq
+ }
+
+ private def sortAndReformat(predictResultND : NDArray, topK: Option[Int])
+ : IndexedSeq[(String, Array[Float])] = {
+ val predictResult: ListBuffer[Array[Float]] = ListBuffer[Array[Float]]()
+ val accuracy : ListBuffer[Float] = ListBuffer[Float]()
+
+ // iterating over the all the predictions
+ val length = predictResultND.shape(0)
+
+ for (i <- 0 until length) {
+ val r = predictResultND.at(i)
+ val tempArr = r.toArray
+ if (tempArr(0) != -1.0) {
+ predictResult += tempArr
+ accuracy += tempArr(1)
+ } else {
+ // Ignore the minus 1 part
+ }
+ handler.execute(r.dispose())
+ }
+ var result = IndexedSeq[(String, Array[Float])]()
+ if(topK.isDefined) {
+ var sortedIndices = accuracy.zipWithIndex.sortBy(-_._1).map(_._2)
+ sortedIndices = sortedIndices.take(topK.get)
+ // takeRight(5) would provide the output as Array[Accuracy, Xmin, Ymin, Xmax, Ymax
+ result = sortedIndices.map(idx
+ => (synset(predictResult(idx)(0).toInt),
+ predictResult(idx).takeRight(5))).toIndexedSeq
+ } else {
+ result = predictResult.map(ele
+ => (synset(ele(0).toInt), ele.takeRight(5))).toIndexedSeq
+ }
+
+ result
+ }
+
+ /**
+ * To classify batch of input images according to the provided model
+ * @param inputBatch Input batch of Buffered images
+ * @param topK Get top k elements with maximum probability
+ * @return List of list of tuples of (class, probability)
+ */
+ def imageBatchObjectDetect(inputBatch: Traversable[BufferedImage], topK: Option[Int] = None):
+ IndexedSeq[IndexedSeq[(String, Array[Float])]] = {
+
+ val imageBatch = ListBuffer[NDArray]()
+ for (image <- inputBatch) {
+ val scaledImage = ImageClassifier.reshapeImage(image, width, height)
+ val pixelsNdarray = ImageClassifier.bufferedImageToPixels(scaledImage, inputShape)
+ imageBatch += pixelsNdarray
+ }
+ val op = NDArray.concatenate(imageBatch)
+
+ val result = objectDetectWithNDArray(IndexedSeq(op), topK)
+ handler.execute(op.dispose())
+ for (ele <- imageBatch) {
+ handler.execute(ele.dispose())
+ }
+ result
+ }
+
+ def getClassifier(modelPathPrefix: String, inputDescriptors: IndexedSeq[DataDesc]): Classifier = {
+ new Classifier(modelPathPrefix, inputDescriptors)
+ }
+
+}
diff --git a/scala-package/infer/src/test/scala/ml/dmlc/mxnet/infer/ObjectDetectorSuite.scala b/scala-package/infer/src/test/scala/ml/dmlc/mxnet/infer/ObjectDetectorSuite.scala
new file mode 100644
index 00000000000..afb26063ac6
--- /dev/null
+++ b/scala-package/infer/src/test/scala/ml/dmlc/mxnet/infer/ObjectDetectorSuite.scala
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnet.infer
+
+
+// scalastyle:off
+import java.awt.image.BufferedImage
+// scalastyle:on
+import ml.dmlc.mxnet.Context
+import ml.dmlc.mxnet.DataDesc
+import ml.dmlc.mxnet.{NDArray, Shape}
+import org.mockito.Matchers.any
+import org.mockito.Mockito
+import org.scalatest.BeforeAndAfterAll
+
+
+class ObjectDetectorSuite extends ClassifierSuite with BeforeAndAfterAll {
+
+ class MyObjectDetector(modelPathPrefix: String,
+ inputDescriptors: IndexedSeq[DataDesc])
+ extends ObjectDetector(modelPathPrefix, inputDescriptors) {
+
+ override def getClassifier(modelPathPrefix: String, inputDescriptors:
+ IndexedSeq[DataDesc]): Classifier = {
+ new MyClassifier(modelPathPrefix, inputDescriptors)
+ }
+
+ }
+
+ class MyClassifier(modelPathPrefix: String,
+ protected override val inputDescriptors: IndexedSeq[DataDesc])
+ extends Classifier(modelPathPrefix, inputDescriptors, Context.cpu(), Some(0)) {
+
+ override def getPredictor(): MyClassyPredictor = {
+ Mockito.mock(classOf[MyClassyPredictor])
+ }
+ def getSynset(): IndexedSeq[String] = synset
+ }
+
+ test("objectDetectWithInputImage") {
+ val inputDescriptor = IndexedSeq[DataDesc](new DataDesc(modelPath, Shape(1, 3, 512, 512)))
+ val inputImage = new BufferedImage(512, 512, BufferedImage.TYPE_INT_RGB)
+ val testObjectDetector: ObjectDetector =
+ new MyObjectDetector(modelPath, inputDescriptor)
+
+ val predictRaw: IndexedSeq[Array[Array[Float]]] =
+ IndexedSeq[Array[Array[Float]]](Array(
+ Array(1.0f, 0.42f, 0.45f, 0.66f, 0.72f, 0.88f),
+ Array(2.0f, 0.88f, 0.21f, 0.33f, 0.45f, 0.66f),
+ Array(3.0f, 0.62f, 0.50f, 0.42f, 0.68f, 0.99f)
+ ))
+ val predictResultND: NDArray =
+ NDArray.array(predictRaw.flatten.flatten.toArray, Shape(1, 3, 6))
+
+ val synset = testObjectDetector.synset
+
+ val predictResult: IndexedSeq[IndexedSeq[(String, Array[Float])]] =
+ IndexedSeq[IndexedSeq[(String, Array[Float])]](
+ IndexedSeq[(String, Array[Float])](
+ (synset(2), Array(0.88f, 0.21f, 0.33f, 0.45f, 0.66f)),
+ (synset(3), Array(0.62f, 0.50f, 0.42f, 0.68f, 0.99f)),
+ (synset(1), Array(0.42f, 0.45f, 0.66f, 0.72f, 0.88f))
+ )
+ )
+
+ Mockito.doReturn(IndexedSeq(predictResultND)).when(testObjectDetector.predictor)
+ .predictWithNDArray(any(classOf[IndexedSeq[NDArray]]))
+
+ val result: IndexedSeq[IndexedSeq[(String, Array[Float])]] =
+ testObjectDetector.imageObjectDetect(inputImage, Some(3))
+
+ for (idx <- predictResult(0).indices) {
+ assert(predictResult(0)(idx)._1 == result(0)(idx)._1)
+ for (arridx <- predictResult(0)(idx)._2.indices) {
+ assert(predictResult(0)(idx)._2(arridx) == result(0)(idx)._2(arridx))
+ }
+ }
+ }
+
+ test("objectDetectWithBatchImages") {
+ val inputDescriptor = IndexedSeq[DataDesc](new DataDesc(modelPath, Shape(1, 3, 512, 512)))
+ val inputImage = new BufferedImage(224, 224, BufferedImage.TYPE_INT_RGB)
+ val imageBatch = IndexedSeq[BufferedImage](inputImage, inputImage)
+
+ val testObjectDetector: ObjectDetector =
+ new MyObjectDetector(modelPath, inputDescriptor)
+
+ val predictRaw: IndexedSeq[Array[Array[Float]]] =
+ IndexedSeq[Array[Array[Float]]](
+ Array(
+ Array(1.0f, 0.42f, 0.45f, 0.66f, 0.72f, 0.88f),
+ Array(2.0f, 0.88f, 0.21f, 0.33f, 0.45f, 0.66f),
+ Array(3.0f, 0.62f, 0.50f, 0.42f, 0.68f, 0.99f)
+ ),
+ Array(
+ Array(0.0f, 0.42f, 0.45f, 0.66f, 0.72f, 0.88f),
+ Array(2.0f, 0.23f, 0.21f, 0.33f, 0.45f, 0.66f),
+ Array(2.0f, 0.94f, 0.50f, 0.42f, 0.68f, 0.99f)
+ )
+ )
+ val predictResultND: NDArray =
+ NDArray.array(predictRaw.flatten.flatten.toArray, Shape(2, 3, 6))
+
+ val synset = testObjectDetector.synset
+
+ val predictResult: IndexedSeq[IndexedSeq[(String, Array[Float])]] =
+ IndexedSeq[IndexedSeq[(String, Array[Float])]](
+ IndexedSeq[(String, Array[Float])](
+ (synset(2), Array(0.88f, 0.21f, 0.33f, 0.45f, 0.66f)),
+ (synset(3), Array(0.62f, 0.50f, 0.42f, 0.68f, 0.99f)),
+ (synset(1), Array(0.42f, 0.45f, 0.66f, 0.72f, 0.88f))
+ ),
+ IndexedSeq[(String, Array[Float])](
+ (synset(2), Array(0.94f, 0.50f, 0.42f, 0.68f, 0.99f)),
+ (synset(0), Array(0.42f, 0.45f, 0.66f, 0.72f, 0.88f)),
+ (synset(2), Array(0.23f, 0.21f, 0.33f, 0.45f, 0.66f))
+ )
+ )
+
+ Mockito.doReturn(IndexedSeq(predictResultND)).when(testObjectDetector.predictor)
+ .predictWithNDArray(any(classOf[IndexedSeq[NDArray]]))
+
+ val result: IndexedSeq[IndexedSeq[(String, Array[Float])]] =
+ testObjectDetector.imageBatchObjectDetect(imageBatch, Some(3))
+ for (preidx <- predictResult.indices) {
+ for (idx <- predictResult(preidx).indices) {
+ assert(predictResult(preidx)(idx)._1 == result(preidx)(idx)._1)
+ for (arridx <- predictResult(preidx)(idx)._2.indices) {
+ assert(predictResult(preidx)(idx)._2(arridx) == result(preidx)(idx)._2(arridx))
+ }
+ }
+ }
+
+ }
+
+}
diff --git a/src/operator/nn/mkldnn/mkldnn_base-inl.h b/src/operator/nn/mkldnn/mkldnn_base-inl.h
index 362f5fbde72..61bef117a88 100644
--- a/src/operator/nn/mkldnn/mkldnn_base-inl.h
+++ b/src/operator/nn/mkldnn/mkldnn_base-inl.h
@@ -227,7 +227,7 @@ class TmpMemMgr {
size_t curr_size;
// This estimate the required temp memory size in an operator.
size_t est_size;
- const size_t alignment = 4096;
+ const size_t alignment = kMKLDNNAlign;
public:
static TmpMemMgr *Get() {
diff --git a/src/storage/cpu_device_storage.h b/src/storage/cpu_device_storage.h
index 6b85ba8dabe..43e98fe04a1 100644
--- a/src/storage/cpu_device_storage.h
+++ b/src/storage/cpu_device_storage.h
@@ -55,9 +55,9 @@ class CPUDeviceStorage {
* \brief Alignment of allocation.
*/
#if MXNET_USE_MKLDNN == 1
- // MKLDNN requires special alignment. 4096 is used by the MKLDNN library in
+ // MKLDNN requires special alignment. 64 is used by the MKLDNN library in
// memory allocation.
- static constexpr size_t alignment_ = 4096;
+ static constexpr size_t alignment_ = kMKLDNNAlign;
#else
static constexpr size_t alignment_ = 16;
#endif
diff --git a/tests/python/unittest/test_image.py b/tests/python/unittest/test_image.py
index 124c94c5eb3..78c3ce14eb4 100644
--- a/tests/python/unittest/test_image.py
+++ b/tests/python/unittest/test_image.py
@@ -110,9 +110,9 @@ def test_resize_short(self):
for _ in range(3):
new_size = np.random.randint(1, 1000)
if h > w:
- new_h, new_w = new_size * h / w, new_size
+ new_h, new_w = new_size * h // w, new_size
else:
- new_h, new_w = new_size, new_size * w / h
+ new_h, new_w = new_size, new_size * w // h
for interp in range(0, 2):
# area-based/lanczos don't match with cv2?
cv_resized = cv2.resize(cv_img, (new_w, new_h), interpolation=interp)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services