You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by me...@apache.org on 2016/02/16 05:17:24 UTC
spark git commit: [SPARK-13018][DOCS] Replace example code in
mllib-pmml-model-export.md using include_example
Repository: spark
Updated Branches:
refs/heads/master cbeb006f2 -> e4675c240
[SPARK-13018][DOCS] Replace example code in mllib-pmml-model-export.md using include_example
Replace example code in mllib-pmml-model-export.md using include_example
https://issues.apache.org/jira/browse/SPARK-13018
The example code in the user guide is embedded in the markdown and hence it is not easy to test. It would be nice to automatically test them. This JIRA is to discuss options to automate example code testing and see what we can do in Spark 1.6.
Goal is to move actual example code to spark/examples and test compilation in Jenkins builds. Then in the markdown, we can reference part of the code to show in the user guide. This requires adding a Jekyll tag that is similar to https://github.com/jekyll/jekyll/blob/master/lib/jekyll/tags/include.rb, e.g., called include_example.
`{% include_example scala/org/apache/spark/examples/mllib/PMMLModelExportExample.scala %}`
Jekyll will find `examples/src/main/scala/org/apache/spark/examples/mllib/PMMLModelExportExample.scala` and pick code blocks marked "example" and replace code block in
`{% highlight %}`
in the markdown.
See more sub-tasks in parent ticket: https://issues.apache.org/jira/browse/SPARK-11337
Author: Xin Ren <ia...@126.com>
Closes #11126 from keypointt/SPARK-13018.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e4675c24
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/e4675c24
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e4675c24
Branch: refs/heads/master
Commit: e4675c240255207c5dd812fa657e6aca2dc9cfeb
Parents: cbeb006
Author: Xin Ren <ia...@126.com>
Authored: Mon Feb 15 20:17:21 2016 -0800
Committer: Xiangrui Meng <me...@databricks.com>
Committed: Mon Feb 15 20:17:21 2016 -0800
----------------------------------------------------------------------
docs/mllib-pmml-model-export.md | 35 +-----------
.../examples/mllib/PMMLModelExportExample.scala | 59 ++++++++++++++++++++
2 files changed, 62 insertions(+), 32 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/e4675c24/docs/mllib-pmml-model-export.md
----------------------------------------------------------------------
diff --git a/docs/mllib-pmml-model-export.md b/docs/mllib-pmml-model-export.md
index b532ad9..58ed5a0 100644
--- a/docs/mllib-pmml-model-export.md
+++ b/docs/mllib-pmml-model-export.md
@@ -45,41 +45,12 @@ The table below outlines the `spark.mllib` models that can be exported to PMML a
<div data-lang="scala" markdown="1">
To export a supported `model` (see table above) to PMML, simply call `model.toPMML`.
+As well as exporting the PMML model to a String (`model.toPMML` as in the example above), you can export the PMML model to other formats.
+
Refer to the [`KMeans` Scala docs](api/scala/index.html#org.apache.spark.mllib.clustering.KMeans) and [`Vectors` Scala docs](api/scala/index.html#org.apache.spark.mllib.linalg.Vectors) for details on the API.
Here a complete example of building a KMeansModel and print it out in PMML format:
-{% highlight scala %}
-import org.apache.spark.mllib.clustering.KMeans
-import org.apache.spark.mllib.linalg.Vectors
-
-// Load and parse the data
-val data = sc.textFile("data/mllib/kmeans_data.txt")
-val parsedData = data.map(s => Vectors.dense(s.split(' ').map(_.toDouble))).cache()
-
-// Cluster the data into two classes using KMeans
-val numClusters = 2
-val numIterations = 20
-val clusters = KMeans.train(parsedData, numClusters, numIterations)
-
-// Export to PMML
-println("PMML Model:\n" + clusters.toPMML)
-{% endhighlight %}
-
-As well as exporting the PMML model to a String (`model.toPMML` as in the example above), you can export the PMML model to other formats:
-
-{% highlight scala %}
-// Export the model to a String in PMML format
-clusters.toPMML
-
-// Export the model to a local file in PMML format
-clusters.toPMML("/tmp/kmeans.xml")
-
-// Export the model to a directory on a distributed file system in PMML format
-clusters.toPMML(sc,"/tmp/kmeans")
-
-// Export the model to the OutputStream in PMML format
-clusters.toPMML(System.out)
-{% endhighlight %}
+{% include_example scala/org/apache/spark/examples/mllib/PMMLModelExportExample.scala %}
For unsupported models, either you will not find a `.toPMML` method or an `IllegalArgumentException` will be thrown.
http://git-wip-us.apache.org/repos/asf/spark/blob/e4675c24/examples/src/main/scala/org/apache/spark/examples/mllib/PMMLModelExportExample.scala
----------------------------------------------------------------------
diff --git a/examples/src/main/scala/org/apache/spark/examples/mllib/PMMLModelExportExample.scala b/examples/src/main/scala/org/apache/spark/examples/mllib/PMMLModelExportExample.scala
new file mode 100644
index 0000000..d74d74a
--- /dev/null
+++ b/examples/src/main/scala/org/apache/spark/examples/mllib/PMMLModelExportExample.scala
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.clustering.KMeans
+import org.apache.spark.mllib.linalg.Vectors
+// $example off$
+
+object PMMLModelExportExample {
+
+ def main(args: Array[String]): Unit = {
+ val conf = new SparkConf().setAppName("PMMLModelExportExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ // Load and parse the data
+ val data = sc.textFile("data/mllib/kmeans_data.txt")
+ val parsedData = data.map(s => Vectors.dense(s.split(' ').map(_.toDouble))).cache()
+
+ // Cluster the data into two classes using KMeans
+ val numClusters = 2
+ val numIterations = 20
+ val clusters = KMeans.train(parsedData, numClusters, numIterations)
+
+ // Export to PMML to a String in PMML format
+ println("PMML Model:\n" + clusters.toPMML)
+
+ // Export the model to a local file in PMML format
+ clusters.toPMML("/tmp/kmeans.xml")
+
+ // Export the model to a directory on a distributed file system in PMML format
+ clusters.toPMML(sc, "/tmp/kmeans")
+
+ // Export the model to the OutputStream in PMML format
+ clusters.toPMML(System.out)
+ // $example off$
+
+ sc.stop()
+ }
+}
+// scalastyle:on println
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org