You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by jk...@apache.org on 2015/12/13 02:47:06 UTC
spark git commit: [SPARK-12199][DOC] Follow-up: Refine example code
in ml-features.md
Repository: spark
Updated Branches:
refs/heads/master 03138b67d -> 98b212d36
[SPARK-12199][DOC] Follow-up: Refine example code in ml-features.md
https://issues.apache.org/jira/browse/SPARK-12199
Follow-up PR of SPARK-11551. Fix some errors in ml-features.md
mengxr
Author: Xusen Yin <yi...@gmail.com>
Closes #10193 from yinxusen/SPARK-12199.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/98b212d3
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/98b212d3
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/98b212d3
Branch: refs/heads/master
Commit: 98b212d36b34ab490c391ea2adf5b141e4fb9289
Parents: 03138b6
Author: Xusen Yin <yi...@gmail.com>
Authored: Sat Dec 12 17:47:01 2015 -0800
Committer: Joseph K. Bradley <jo...@databricks.com>
Committed: Sat Dec 12 17:47:01 2015 -0800
----------------------------------------------------------------------
docs/ml-features.md | 22 ++++-----
.../spark/examples/ml/JavaBinarizerExample.java | 2 +-
.../python/ml/polynomial_expansion_example.py | 6 +--
.../examples/ml/ElementWiseProductExample.scala | 52 --------------------
.../examples/ml/ElementwiseProductExample.scala | 52 ++++++++++++++++++++
5 files changed, 67 insertions(+), 67 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/98b212d3/docs/ml-features.md
----------------------------------------------------------------------
diff --git a/docs/ml-features.md b/docs/ml-features.md
index 8b00cc6..158f3f2 100644
--- a/docs/ml-features.md
+++ b/docs/ml-features.md
@@ -63,7 +63,7 @@ the [IDF Python docs](api/python/pyspark.ml.html#pyspark.ml.feature.IDF) for mor
`Word2VecModel`. The model maps each word to a unique fixed-size vector. The `Word2VecModel`
transforms each document into a vector using the average of all words in the document; this vector
can then be used for as features for prediction, document similarity calculations, etc.
-Please refer to the [MLlib user guide on Word2Vec](mllib-feature-extraction.html#Word2Vec) for more
+Please refer to the [MLlib user guide on Word2Vec](mllib-feature-extraction.html#word2Vec) for more
details.
In the following code segment, we start with a set of documents, each of which is represented as a sequence of words. For each document, we transform it into a feature vector. This feature vector could then be passed to a learning algorithm.
@@ -411,7 +411,7 @@ for more details on the API.
Refer to the [DCT Java docs](api/java/org/apache/spark/ml/feature/DCT.html)
for more details on the API.
-{% include_example java/org/apache/spark/examples/ml/JavaDCTExample.java %}}
+{% include_example java/org/apache/spark/examples/ml/JavaDCTExample.java %}
</div>
</div>
@@ -669,7 +669,7 @@ for more details on the API.
The following example demonstrates how to load a dataset in libsvm format and then normalize each row to have unit $L^2$ norm and unit $L^\infty$ norm.
<div class="codetabs">
-<div data-lang="scala">
+<div data-lang="scala" markdown="1">
Refer to the [Normalizer Scala docs](api/scala/index.html#org.apache.spark.ml.feature.Normalizer)
for more details on the API.
@@ -677,7 +677,7 @@ for more details on the API.
{% include_example scala/org/apache/spark/examples/ml/NormalizerExample.scala %}
</div>
-<div data-lang="java">
+<div data-lang="java" markdown="1">
Refer to the [Normalizer Java docs](api/java/org/apache/spark/ml/feature/Normalizer.html)
for more details on the API.
@@ -685,7 +685,7 @@ for more details on the API.
{% include_example java/org/apache/spark/examples/ml/JavaNormalizerExample.java %}
</div>
-<div data-lang="python">
+<div data-lang="python" markdown="1">
Refer to the [Normalizer Python docs](api/python/pyspark.ml.html#pyspark.ml.feature.Normalizer)
for more details on the API.
@@ -709,7 +709,7 @@ Note that if the standard deviation of a feature is zero, it will return default
The following example demonstrates how to load a dataset in libsvm format and then normalize each feature to have unit standard deviation.
<div class="codetabs">
-<div data-lang="scala">
+<div data-lang="scala" markdown="1">
Refer to the [StandardScaler Scala docs](api/scala/index.html#org.apache.spark.ml.feature.StandardScaler)
for more details on the API.
@@ -717,7 +717,7 @@ for more details on the API.
{% include_example scala/org/apache/spark/examples/ml/StandardScalerExample.scala %}
</div>
-<div data-lang="java">
+<div data-lang="java" markdown="1">
Refer to the [StandardScaler Java docs](api/java/org/apache/spark/ml/feature/StandardScaler.html)
for more details on the API.
@@ -725,7 +725,7 @@ for more details on the API.
{% include_example java/org/apache/spark/examples/ml/JavaStandardScalerExample.java %}
</div>
-<div data-lang="python">
+<div data-lang="python" markdown="1">
Refer to the [StandardScaler Python docs](api/python/pyspark.ml.html#pyspark.ml.feature.StandardScaler)
for more details on the API.
@@ -788,7 +788,7 @@ More details can be found in the API docs for [Bucketizer](api/scala/index.html#
The following example demonstrates how to bucketize a column of `Double`s into another index-wised column.
<div class="codetabs">
-<div data-lang="scala">
+<div data-lang="scala" markdown="1">
Refer to the [Bucketizer Scala docs](api/scala/index.html#org.apache.spark.ml.feature.Bucketizer)
for more details on the API.
@@ -796,7 +796,7 @@ for more details on the API.
{% include_example scala/org/apache/spark/examples/ml/BucketizerExample.scala %}
</div>
-<div data-lang="java">
+<div data-lang="java" markdown="1">
Refer to the [Bucketizer Java docs](api/java/org/apache/spark/ml/feature/Bucketizer.html)
for more details on the API.
@@ -804,7 +804,7 @@ for more details on the API.
{% include_example java/org/apache/spark/examples/ml/JavaBucketizerExample.java %}
</div>
-<div data-lang="python">
+<div data-lang="python" markdown="1">
Refer to the [Bucketizer Python docs](api/python/pyspark.ml.html#pyspark.ml.feature.Bucketizer)
for more details on the API.
http://git-wip-us.apache.org/repos/asf/spark/blob/98b212d3/examples/src/main/java/org/apache/spark/examples/ml/JavaBinarizerExample.java
----------------------------------------------------------------------
diff --git a/examples/src/main/java/org/apache/spark/examples/ml/JavaBinarizerExample.java b/examples/src/main/java/org/apache/spark/examples/ml/JavaBinarizerExample.java
index 9698cac..1eda1f6 100644
--- a/examples/src/main/java/org/apache/spark/examples/ml/JavaBinarizerExample.java
+++ b/examples/src/main/java/org/apache/spark/examples/ml/JavaBinarizerExample.java
@@ -59,7 +59,7 @@ public class JavaBinarizerExample {
DataFrame binarizedDataFrame = binarizer.transform(continuousDataFrame);
DataFrame binarizedFeatures = binarizedDataFrame.select("binarized_feature");
for (Row r : binarizedFeatures.collect()) {
- Double binarized_value = r.getDouble(0);
+ Double binarized_value = r.getDouble(0);
System.out.println(binarized_value);
}
// $example off$
http://git-wip-us.apache.org/repos/asf/spark/blob/98b212d3/examples/src/main/python/ml/polynomial_expansion_example.py
----------------------------------------------------------------------
diff --git a/examples/src/main/python/ml/polynomial_expansion_example.py b/examples/src/main/python/ml/polynomial_expansion_example.py
index 3d4fafd..89f5cbe 100644
--- a/examples/src/main/python/ml/polynomial_expansion_example.py
+++ b/examples/src/main/python/ml/polynomial_expansion_example.py
@@ -30,9 +30,9 @@ if __name__ == "__main__":
# $example on$
df = sqlContext\
- .createDataFrame([(Vectors.dense([-2.0, 2.3]), ),
- (Vectors.dense([0.0, 0.0]), ),
- (Vectors.dense([0.6, -1.1]), )],
+ .createDataFrame([(Vectors.dense([-2.0, 2.3]),),
+ (Vectors.dense([0.0, 0.0]),),
+ (Vectors.dense([0.6, -1.1]),)],
["features"])
px = PolynomialExpansion(degree=2, inputCol="features", outputCol="polyFeatures")
polyDF = px.transform(df)
http://git-wip-us.apache.org/repos/asf/spark/blob/98b212d3/examples/src/main/scala/org/apache/spark/examples/ml/ElementWiseProductExample.scala
----------------------------------------------------------------------
diff --git a/examples/src/main/scala/org/apache/spark/examples/ml/ElementWiseProductExample.scala b/examples/src/main/scala/org/apache/spark/examples/ml/ElementWiseProductExample.scala
deleted file mode 100644
index 872de51..0000000
--- a/examples/src/main/scala/org/apache/spark/examples/ml/ElementWiseProductExample.scala
+++ /dev/null
@@ -1,52 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-// scalastyle:off println
-package org.apache.spark.examples.ml
-
-// $example on$
-import org.apache.spark.ml.feature.ElementwiseProduct
-import org.apache.spark.mllib.linalg.Vectors
-// $example off$
-import org.apache.spark.sql.SQLContext
-import org.apache.spark.{SparkConf, SparkContext}
-
-object ElementwiseProductExample {
- def main(args: Array[String]): Unit = {
- val conf = new SparkConf().setAppName("ElementwiseProductExample")
- val sc = new SparkContext(conf)
- val sqlContext = new SQLContext(sc)
-
- // $example on$
- // Create some vector data; also works for sparse vectors
- val dataFrame = sqlContext.createDataFrame(Seq(
- ("a", Vectors.dense(1.0, 2.0, 3.0)),
- ("b", Vectors.dense(4.0, 5.0, 6.0)))).toDF("id", "vector")
-
- val transformingVector = Vectors.dense(0.0, 1.0, 2.0)
- val transformer = new ElementwiseProduct()
- .setScalingVec(transformingVector)
- .setInputCol("vector")
- .setOutputCol("transformedVector")
-
- // Batch transform the vectors to create new column:
- transformer.transform(dataFrame).show()
- // $example off$
- sc.stop()
- }
-}
-// scalastyle:on println
http://git-wip-us.apache.org/repos/asf/spark/blob/98b212d3/examples/src/main/scala/org/apache/spark/examples/ml/ElementwiseProductExample.scala
----------------------------------------------------------------------
diff --git a/examples/src/main/scala/org/apache/spark/examples/ml/ElementwiseProductExample.scala b/examples/src/main/scala/org/apache/spark/examples/ml/ElementwiseProductExample.scala
new file mode 100644
index 0000000..872de51
--- /dev/null
+++ b/examples/src/main/scala/org/apache/spark/examples/ml/ElementwiseProductExample.scala
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.ml
+
+// $example on$
+import org.apache.spark.ml.feature.ElementwiseProduct
+import org.apache.spark.mllib.linalg.Vectors
+// $example off$
+import org.apache.spark.sql.SQLContext
+import org.apache.spark.{SparkConf, SparkContext}
+
+object ElementwiseProductExample {
+ def main(args: Array[String]): Unit = {
+ val conf = new SparkConf().setAppName("ElementwiseProductExample")
+ val sc = new SparkContext(conf)
+ val sqlContext = new SQLContext(sc)
+
+ // $example on$
+ // Create some vector data; also works for sparse vectors
+ val dataFrame = sqlContext.createDataFrame(Seq(
+ ("a", Vectors.dense(1.0, 2.0, 3.0)),
+ ("b", Vectors.dense(4.0, 5.0, 6.0)))).toDF("id", "vector")
+
+ val transformingVector = Vectors.dense(0.0, 1.0, 2.0)
+ val transformer = new ElementwiseProduct()
+ .setScalingVec(transformingVector)
+ .setInputCol("vector")
+ .setOutputCol("transformedVector")
+
+ // Batch transform the vectors to create new column:
+ transformer.transform(dataFrame).show()
+ // $example off$
+ sc.stop()
+ }
+}
+// scalastyle:on println
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org