You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/10/09 16:28:11 UTC
[GitHub] yzhliu closed pull request #12758: [MXNET-716][MIRROR #12723] Scala
Benchmark Extension pack
yzhliu closed pull request #12758: [MXNET-716][MIRROR #12723] Scala Benchmark Extension pack
URL: https://github.com/apache/incubator-mxnet/pull/12758
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/scala-package/examples/scripts/benchmark/run_text_charrnn_bm.sh b/scala-package/examples/scripts/benchmark/run_text_charrnn_bm.sh
new file mode 100755
index 00000000000..d2c1ee60d41
--- /dev/null
+++ b/scala-package/examples/scripts/benchmark/run_text_charrnn_bm.sh
@@ -0,0 +1,57 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+hw_type=cpu
+if [ "$1" = "gpu" ]
+then
+ hw_type=gpu
+fi
+
+platform=linux-x86_64
+
+if [ "$OSTYPE" == "darwin"* ]
+then
+ platform=osx-x86_64
+fi
+
+MXNET_ROOT=$(cd "$(dirname $0)/../../../.."; pwd)
+CLASS_PATH=$MXNET_ROOT/scala-package/assembly/$platform-$hw_type/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/*:$MXNET_ROOT/scala-package/infer/target/*
+
+MODEL_NAME=$2
+
+RUNS=$3
+
+# model dir
+MODEL_PATH_PREFIX=$4
+# input image
+DATA_PATH=$5
+
+# feel free to change the starter sentence
+STARTER_SENTENCE="The joke"
+
+java -Xmx8G -Dmxnet.traceLeakedObjects=false -cp $CLASS_PATH \
+ org.apache.mxnetexamples.benchmark.ScalaInferenceBenchmark \
+ --example $MODEL_NAME \
+ --count $RUNS \
+ --model-prefix $MODEL_PATH_PREFIX \
+ --data-path $DATA_PATH \
+ --starter-sentence "$STARTER_SENTENCE"
+
diff --git a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/benchmark/README.md b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/benchmark/README.md
index cfff93397be..753cb312541 100644
--- a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/benchmark/README.md
+++ b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/benchmark/README.md
@@ -46,4 +46,38 @@ You may need to run ```chmod u+x run_image_inference_bm.sh``` before running thi
batch_inference_latency p99 4241, batch_inference_p50 4241, batch_inference_average 4241.00
```
-More examples to be added soon.
\ No newline at end of file
+* *Object Detection Example*
+<br> The following shows an example of running SSDClassifier under the benchmark script. The script takes in the number of iterations for inference calls, the batch size for batch inference calls, the model path, input file, and input directory.
+For more details to run SSDClassifierExample as a standalone file, refer to the [README](https://github.com/apache/incubator-mxnet/blob/master/scala-package/examples/src/main/scala/org/apache/mxnetexamples/infer/objectdetector/README.md) for SSDClassifierExample.
+You may need to run ```chmod u+x run_image_inference_bm.sh``` before running this script.
+ ```bash
+ cd <Path-To-MXNET-Repo>/scala-package/examples/scripts/infer/objectdetector
+ ./get_ssd_data.sh
+ cd <Path-To-MXNET-Repo>/scala-package/examples/scripts/benchmark
+ ./run_image_inference_bm.sh cpu ObjectDetectionExample 100 10 ../infer/models/resnet50_ssd/resnet50_ssd_model ../infer/images/dog.jpg ../infer/images/
+ ```
+ Upon running this script, you might see an output like this :
+ ```
+ [main] INFO org.apache.mxnetexamples.benchmark.CLIParserBase -
+ single_inference_latency p99 1663, single_inference_p50 729, single_inference_average 755.17
+ ...
+
+ INFO org.apache.mxnetexamples.benchmark.CLIParserBase -
+ batch_inference_latency p99 4241, batch_inference_p50 4241, batch_inference_average 4241.00
+ ```
+
+* *Text Generation through RNNs*
+<br>The following shows an example of running TestCharRnn under the benchmark script. The script takes in the number of iterations for inference calls, the model path and the input text file.
+For more details to run TestCharRnn as a standalone file, refer to the [README](https://github.com/apache/incubator-mxnet/blob/master/scala-package/examples/src/main/scala/org/apache/mxnetexamples/rnn/README.md) for TextCharRnn.
+You may need to run ```chmod u+x run_text_charrnn_bm.sh``` before running this script.
+ ```bash
+ wget https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/RNN/obama.zip
+ unzip obama.zip
+ cd <Path-To-MXNET-Repo>/scala-package/examples/scripts/benchmark
+ ./run_text_charrnn_bm.sh cpu CharRnn 100 <path-to-model>/obama <path-to-model>/obama.txt
+ ```
+ Upon running this script, you might see an output like this :
+ ```
+ [main] INFO org.apache.mxnetexamples.benchmark.CLIParserBase -
+ single_inference_latency p99 4097, single_inference_p50 2560, single_inference_average 2673.720000
+ ```
diff --git a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/benchmark/ScalaInferenceBenchmark.scala b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/benchmark/ScalaInferenceBenchmark.scala
index 9ae50dc9d12..dad2b53e73a 100644
--- a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/benchmark/ScalaInferenceBenchmark.scala
+++ b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/benchmark/ScalaInferenceBenchmark.scala
@@ -20,6 +20,8 @@ package org.apache.mxnetexamples.benchmark
import org.apache.mxnetexamples.InferBase
import org.apache.mxnetexamples.infer.imageclassifier.ImageClassifierExample
import org.apache.mxnet._
+import org.apache.mxnetexamples.infer.objectdetector.SSDClassifierExample
+import org.apache.mxnetexamples.rnn.TestCharRnn
import org.kohsuke.args4j.{CmdLineParser, Option}
import org.slf4j.LoggerFactory
@@ -89,10 +91,11 @@ object ScalaInferenceBenchmark {
val times: Seq[Long] = inferenceTimes
val p50 = percentile(50, times)
val p99 = percentile(99, times)
+ val p90 = percentile(90, times)
val average = times.sum / (times.length * 1.0)
- logger.info("\n%s_p99 %d, %s_p50 %d, %s_average %1.2f".format(metricsPrefix,
- p99, metricsPrefix, p50, metricsPrefix, average))
+ logger.info("\n%s_p99 %d, %s_p90 %d, %s_p50 %d, %s_average %1.2f".format(metricsPrefix,
+ p99, metricsPrefix, p90, metricsPrefix, p50, metricsPrefix, average))
}
@@ -113,6 +116,18 @@ object ScalaInferenceBenchmark {
val parsedVals = new CmdLineParser(imParser).parseArgument(args.toList.asJava)
new ImageClassifierExample(imParser)
}
+ case "ObjectDetectionExample" => {
+ val imParser = new org.apache.mxnetexamples.infer.objectdetector.CLIParser
+ baseCLI = imParser
+ val parsedVals = new CmdLineParser(imParser).parseArgument(args.toList.asJava)
+ new SSDClassifierExample(imParser)
+ }
+ case "CharRnn" => {
+ val imParser = new org.apache.mxnetexamples.rnn.CLIParser
+ baseCLI = imParser
+ val parsedVals = new CmdLineParser(imParser).parseArgument(args.toList.asJava)
+ new TestCharRnn(imParser)
+ }
case _ => throw new Exception("Invalid example name to run")
}
diff --git a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/infer/objectdetector/SSDClassifierExample.scala b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/infer/objectdetector/SSDClassifierExample.scala
index 53c4d367048..a8d40c9ef6a 100644
--- a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/infer/objectdetector/SSDClassifierExample.scala
+++ b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/infer/objectdetector/SSDClassifierExample.scala
@@ -16,7 +16,11 @@
*/
package org.apache.mxnetexamples.infer.objectdetector
+// scalastyle:off
+import java.awt.image.BufferedImage
+import org.apache.mxnetexamples.benchmark.CLIParserBase
+// scalastyle:on
import java.io.File
import org.apache.mxnet._
@@ -27,6 +31,8 @@ import org.slf4j.LoggerFactory
import scala.collection.JavaConverters._
import java.nio.file.{Files, Paths}
+import org.apache.mxnetexamples.InferBase
+
import scala.collection.mutable.ListBuffer
// scalastyle:off
@@ -37,15 +43,6 @@ import scala.collection.mutable.ListBuffer
* @see <a href="https://github.com/apache/incubator-mxnet/tree/master/scala-package/examples/src/main/scala/org/apache/mxnetexamples/infer/objectdetector" target="_blank">Instructions to run this example</a>
*/
// scalastyle:on
-class SSDClassifierExample {
- @Option(name = "--model-path-prefix", usage = "the input model directory and prefix of the model")
- private val modelPathPrefix: String = "/model/ssd_resnet50_512"
- @Option(name = "--input-image", usage = "the input image")
- private val inputImagePath: String = "/images/dog.jpg"
- @Option(name = "--input-dir", usage = "the input batch of images directory")
- private val inputImageDir: String = "/images/"
-}
-
object SSDClassifierExample {
private val logger = LoggerFactory.getLogger(classOf[SSDClassifierExample])
@@ -111,7 +108,7 @@ object SSDClassifierExample {
}
def main(args: Array[String]): Unit = {
- val inst = new SSDClassifierExample
+ val inst = new CLIParser
val parser : CmdLineParser = new CmdLineParser(inst)
parser.parseArgument(args.toList.asJava)
val mdprefixDir = inst.modelPathPrefix
@@ -193,4 +190,64 @@ object SSDClassifierExample {
exist
}
-}
\ No newline at end of file
+}
+
+class CLIParser extends CLIParserBase {
+ @Option(name = "--model-path-prefix", usage = "the input model directory and prefix of the model")
+ val modelPathPrefix: String = "/model/ssd_resnet50_512"
+ @Option(name = "--input-image", usage = "the input image")
+ val inputImagePath: String = "/images/dog.jpg"
+ @Option(name = "--input-dir", usage = "the input batch of images directory")
+ val inputImageDir: String = "/images/"
+}
+
+class SSDClassifierExample(CLIParser: CLIParser)
+ extends InferBase {
+ override def loadModel(context: Array[Context]): Any = {
+ val dType = DType.Float32
+ val inputShape = Shape(1, 3, 512, 512)
+ val inputDescriptors = IndexedSeq(DataDesc("data", inputShape, dType, "NCHW"))
+ new ObjectDetector(CLIParser.modelPathPrefix, inputDescriptors, context)
+ }
+ override def loadSingleData(): Any = {
+ val img = ImageClassifier.loadImageFromFile(CLIParser.inputImagePath)
+ img
+ }
+
+ override def runSingleInference(loadedModel: Any, input: Any): Any = {
+ val detector = loadedModel.asInstanceOf[ObjectDetector]
+ val imgInput = input.asInstanceOf[BufferedImage]
+ detector.imageObjectDetect(imgInput)
+ }
+
+ override def loadInputBatch(inputPaths: Any): Any = {
+ val batchFile = inputPaths.asInstanceOf[List[String]]
+ ImageClassifier.loadInputBatch(batchFile)
+ }
+
+ override def loadBatchFileList(batchSize: Int): List[Any] = {
+ val dir = new File(CLIParser.inputImageDir)
+ require(dir.exists && dir.isDirectory,
+ "input image directory: %s not found".format(CLIParser.inputImageDir))
+ val output = ListBuffer[List[String]]()
+ var batch = ListBuffer[String]()
+ for (imgFile: File <- dir.listFiles()){
+ batch += imgFile.getPath
+ if (batch.length == batchSize) {
+ output += batch.toList
+ batch = ListBuffer[String]()
+ }
+ }
+ if (batch.length > 0) {
+ output += batch.toList
+ }
+ output.toList
+ }
+
+ override def runBatchInference(loadedModel: Any, input: Any): Any = {
+ val model = loadedModel.asInstanceOf[ObjectDetector]
+ val imgInput = input.asInstanceOf[Traversable[BufferedImage]]
+ val output = model.imageBatchObjectDetect(imgInput, Some(5))
+ output
+ }
+}
diff --git a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/rnn/TestCharRnn.scala b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/rnn/TestCharRnn.scala
index 5eb3ab41805..0fbdf7d918d 100644
--- a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/rnn/TestCharRnn.scala
+++ b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/rnn/TestCharRnn.scala
@@ -18,14 +18,17 @@
package org.apache.mxnetexamples.rnn
import org.apache.mxnet._
+import org.apache.mxnetexamples.InferBase
+import org.apache.mxnetexamples.benchmark.CLIParserBase
import org.kohsuke.args4j.{CmdLineParser, Option}
import org.slf4j.LoggerFactory
+
import scala.collection.JavaConverters._
/**
- * Follows the demo, to test the char rnn:
- * https://github.com/dmlc/mxnet/blob/master/example/rnn/char-rnn.ipynb
- */
+ * Follows the demo, to test the char rnn:
+ * https://github.com/dmlc/mxnet/blob/master/example/rnn/char-rnn.ipynb
+ */
object TestCharRnn {
private val logger = LoggerFactory.getLogger(classOf[TrainCharRnn])
@@ -83,7 +86,7 @@ object TestCharRnn {
}
def main(args: Array[String]): Unit = {
- val stcr = new TestCharRnn
+ val stcr = new CLIParser
val parser: CmdLineParser = new CmdLineParser(stcr)
try {
parser.parseArgument(args.toList.asJava)
@@ -99,11 +102,63 @@ object TestCharRnn {
}
}
-class TestCharRnn {
+class CLIParser extends CLIParserBase {
@Option(name = "--data-path", usage = "the input train data file")
- private val dataPath: String = "./data/obama.txt"
+ val dataPath: String = "./data/obama.txt"
@Option(name = "--model-prefix", usage = "the model prefix")
- private val modelPrefix: String = "./model/obama"
+ val modelPrefix: String = "./model/obama"
@Option(name = "--starter-sentence", usage = "the starter sentence")
- private val starterSentence: String = "The joke"
+ val starterSentence: String = "The joke"
+}
+
+class TestCharRnn(CLIParser: CLIParser) extends InferBase {
+
+ private var vocab : Map[String, Int] = null
+
+ override def loadModel(context: Array[Context]): Any = {
+ val batchSize = 32
+ val buckets = List(129)
+ val numHidden = 512
+ val numEmbed = 256
+ val numLstmLayer = 3
+ val (_, argParams, _) = Model.loadCheckpoint(CLIParser.modelPrefix, 75)
+ this.vocab = Utils.buildVocab(CLIParser.dataPath)
+ val model = new RnnModel.LSTMInferenceModel(numLstmLayer, vocab.size + 1,
+ numHidden = numHidden, numEmbed = numEmbed,
+ numLabel = vocab.size + 1, argParams = argParams, dropout = 0.2f)
+ model
+ }
+
+ override def loadSingleData(): Any = {
+ val revertVocab = Utils.makeRevertVocab(vocab)
+ revertVocab
+ }
+
+ override def runSingleInference(loadedModel: Any, input: Any): Any = {
+ val model = loadedModel.asInstanceOf[RnnModel.LSTMInferenceModel]
+ val revertVocab = input.asInstanceOf[Map[Int, String]]
+ // generate a sequence of 1200 chars
+ val seqLength = 1200
+ val inputNdarray = NDArray.zeros(1)
+ // Feel free to change the starter sentence
+ var output = CLIParser.starterSentence
+ val randomSample = true
+ var newSentence = true
+ val ignoreLength = output.length()
+
+ for (i <- 0 until seqLength) {
+ if (i <= ignoreLength - 1) Utils.makeInput(output(i), vocab, inputNdarray)
+ else Utils.makeInput(output.takeRight(1)(0), vocab, inputNdarray)
+ val prob = model.forward(inputNdarray, newSentence)
+ newSentence = false
+ val nextChar = Utils.makeOutput(prob, revertVocab, randomSample)
+ if (nextChar == "") newSentence = true
+ if (i >= ignoreLength) output = output ++ nextChar
+ }
+ output
+ }
+
+ override def loadBatchFileList(batchSize: Int): List[Any] = null
+ override def loadInputBatch(source: Any): Any = null
+ override def runBatchInference(loadedModel: Any, input: Any): Any = null
}
diff --git a/scala-package/examples/src/test/scala/org/apache/mxnetexamples/benchmark/ScalaInferenceBenchmarkSuite.scala b/scala-package/examples/src/test/scala/org/apache/mxnetexamples/benchmark/ScalaInferenceBenchmarkSuite.scala
index 8786e63efbb..0b7f4693c5f 100644
--- a/scala-package/examples/src/test/scala/org/apache/mxnetexamples/benchmark/ScalaInferenceBenchmarkSuite.scala
+++ b/scala-package/examples/src/test/scala/org/apache/mxnetexamples/benchmark/ScalaInferenceBenchmarkSuite.scala
@@ -57,4 +57,56 @@ class ScalaInferenceBenchmarkSuite extends FunSuite with BeforeAndAfterAll {
ScalaInferenceBenchmark.main(args)
}
+ test("Testing Benchmark -- Object Detection") {
+ logger.info("Downloading resnetssd model")
+ val tempDirPath = System.getProperty("java.io.tmpdir")
+ logger.info("tempDirPath: %s".format(tempDirPath))
+ val modelBase = "https://s3.amazonaws.com/model-server/models/resnet50_ssd/"
+ val imageBase = "https://s3.amazonaws.com/model-server/inputs/"
+ Util.downloadUrl(modelBase + "resnet50_ssd_model-symbol.json",
+ tempDirPath + "/resnetssd/resnet50_ssd_model-symbol.json")
+ Util.downloadUrl(modelBase + "resnet50_ssd_model-0000.params",
+ tempDirPath + "/resnetssd/resnet50_ssd_model-0000.params")
+ Util.downloadUrl(modelBase + "synset.txt",
+ tempDirPath + "/resnetssd/synset.txt")
+ Util.downloadUrl(imageBase + "dog-ssd.jpg",
+ tempDirPath + "/inputImages/resnetssd/dog-ssd.jpg")
+ val modelDirPath = tempDirPath + File.separator + "resnetssd/"
+ val inputImagePath = tempDirPath + File.separator +
+ "inputImages/resnetssd/dog-ssd.jpg"
+ val inputImageDir = tempDirPath + File.separator + "inputImages/resnetssd/"
+ val args = Array(
+ "--example", "ObjectDetectionExample",
+ "--count", "1",
+ "--batchSize", "10",
+ "--model-path-prefix", s"$modelDirPath/resnet50_ssd_model",
+ "--input-image", inputImagePath,
+ "--input-dir", inputImageDir
+ )
+ ScalaInferenceBenchmark.main(args)
+ }
+
+ test("Testing Benchmark -- charRNN Model") {
+ logger.info("Downloading LSTM model")
+ val tempDirPath = System.getProperty("java.io.tmpdir")
+ logger.info("tempDirPath: %s".format(tempDirPath))
+ val baseUrl = "https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/RNN/"
+ Util.downloadUrl(baseUrl + "obama.zip", tempDirPath + "/RNN/obama.zip")
+ Util.downloadUrl(baseUrl + "sherlockholmes.train.txt",
+ tempDirPath + "/RNN/sherlockholmes.train.txt")
+ Util.downloadUrl(baseUrl + "sherlockholmes.valid.txt",
+ tempDirPath + "/RNN/sherlockholmes.valid.txt")
+ // TODO: Need to confirm with Windows
+ Process(s"unzip $tempDirPath/RNN/obama.zip -d $tempDirPath/RNN/") !
+
+ val args = Array(
+ "--example", "CharRnn",
+ "--count", "1",
+ "--data-path", s"$tempDirPath/RNN/obama.txt",
+ "--model-prefix", s"$tempDirPath/RNN/obama",
+ "--starter-sentence", "The joke"
+ )
+ ScalaInferenceBenchmark.main(args)
+ }
+
}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services