You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by keypointt <gi...@git.apache.org> on 2016/02/07 08:25:32 UTC
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
GitHub user keypointt opened a pull request:
https://github.com/apache/spark/pull/11108
[SPARK-13019][Docs] Replace example code in mllib-statistics.md using include_example
https://issues.apache.org/jira/browse/SPARK-13019
Hi, here I have two doubts for this PR:
1. should I include "print" code or "SparkContext" code into example? I found it's not consistent in current code
2. the initialization of RDDs, I'm using some very primitive and simple ones (eg: `val seriesX: RDD[Double] = sc.parallelize(Array(1, 2, 3, 3, 5))` ), and not sure I'm doing it right
Thanks for reviewing :smile:
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/keypointt/spark SPARK-13019
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11108.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11108
----
commit 49b7012e7b67d3fc7db031d0604d76d4150034eb
Author: Xin Ren <ia...@126.com>
Date: 2016-02-02T01:51:25Z
[SPARK-13019] raplce for summary staticstics, scala code
commit 83592bcafa553bf9439da7db72552868b3ed967a
Author: Xin Ren <ia...@126.com>
Date: 2016-02-02T05:01:20Z
[SPARK-13019] test out on/off, for import part
commit 069341b3d22c147fcf84e338e9b1b7d1f9fdae8e
Author: Xin Ren <ia...@126.com>
Date: 2016-02-03T01:33:33Z
[SPARK-13019] create separate example files, but cannot compile yet
commit 2058b16bc3c097b204bb6c94226e1a8f85017679
Author: Xin Ren <ia...@126.com>
Date: 2016-02-03T23:25:59Z
[SPARK-13019] move new files into mllib folder
commit b3285423a625812f64968767b8063ca83ab58f3c
Author: Xin Ren <ia...@126.com>
Date: 2016-02-03T23:29:04Z
[SPARK-13019] remote python init files
commit 12fda2be3033fb0477951c276366215ed2ed2736
Author: Xin Ren <ia...@126.com>
Date: 2016-02-03T23:39:48Z
[SPARK-13019] comment broken code to pass complie process
commit 2abfaa93b9dee4a86d304a54f59143a5fa0ee401
Author: Xin Ren <ia...@126.com>
Date: 2016-02-04T01:18:33Z
[SPARK-13019] remove code block tag
commit 157da53ca1c6ceb13eb720e968d2e09fed44571d
Author: Xin Ren <ia...@126.com>
Date: 2016-02-04T01:26:36Z
[SPARK-13019] make commented code explicit in html content
commit 323304fe2aa3033a88429cf3a0e5adef345d2c24
Author: Xin Ren <ia...@126.com>
Date: 2016-02-04T22:55:51Z
[SPARK-13019] Stratified Sampling working
commit 3692d30bfec83e53883a017339d08cfaf3223266
Author: Xin Ren <ia...@126.com>
Date: 2016-02-05T01:14:01Z
[SPARK-13019] hypothesis testing working
commit 89c3d2ed7d0134d27ecd5c2077e45c12e1702552
Author: Xin Ren <ia...@126.com>
Date: 2016-02-05T01:33:25Z
[SPARK-13019] Hypothesis Testing Kolmogorov Smirnov Test Example is working
commit 4dbbc6d32ed3c045e5b947df77f11195adb1255f
Author: Xin Ren <ia...@126.com>
Date: 2016-02-05T01:37:43Z
[SPARK-13019] remove empty lines
commit f024fc3c5a6021f2d320c72f9ded78adc3347fbb
Author: Xin Ren <ia...@126.com>
Date: 2016-02-05T02:03:26Z
[SPARK-13019] random data generation example working
commit 6f949cde4f18829b2a1c7f946373fc008f4a5bb1
Author: Xin Ren <ia...@126.com>
Date: 2016-02-07T03:19:47Z
[SPARK-13019] Kernel Density Estimation Example is working
commit a4dd0fb5781dbdc162fd8652001b8de09b225297
Author: Xin Ren <ia...@126.com>
Date: 2016-02-07T07:16:33Z
[SPARK-13019] code style check
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-186669448
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133734
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/StratifiedSamplingExample.scala ---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+// $example on$
+import org.apache.spark.{SparkConf, SparkContext}
+// $example off$
+
+object StratifiedSamplingExample {
+
+ def main(args: Array[String]) {
+
+ val conf = new SparkConf().setAppName("StratifiedSamplingExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ // an RDD[(K, V)] of any key value pairs
+ val data = sc.parallelize(
+ Seq((1, 'a'), (1, 'b'), (2, 'c'), (2, 'd'), (2, 'e'), (3, 'f')))
+
+ // specify the exact fraction desired from each key
+ val fractions = Map(1 -> 0.1, 2 -> 0.6, 3 -> 0.3)
+
+ // Get an exact sample from each stratum
+ val approxSample = data.sampleByKey(withReplacement = false, fractions)
+ val exactSample = data.sampleByKeyExact(withReplacement = false, fractions)
+
--- End diff --
remove it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55145522
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaHypothesisTestingKolmogorovSmirnovTestExample.java ---
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
+
+import org.apache.spark.SparkConf;
+// $example on$
+import org.apache.spark.api.java.JavaDoubleRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.mllib.stat.Statistics;
+import org.apache.spark.mllib.stat.test.KolmogorovSmirnovTestResult;
+// $example off$
+
+public class JavaHypothesisTestingKolmogorovSmirnovTestExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaHypothesisTestingKolmogorovSmirnovTestExample");
--- End diff --
exceeds 100
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55145725
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/KernelDensityEstimationExample.scala ---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.stat.KernelDensity
+import org.apache.spark.rdd.RDD
+// $example off$
+
+object KernelDensityEstimationExample {
+
+ def main(args: Array[String]) {
+
+ val conf = new SparkConf().setAppName("KernelDensityEstimationExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ // an RDD of sample data
+ val data: RDD[Double] = sc.parallelize(Seq(1, 1, 1, 2, 3, 4, 5, 5, 6, 7, 8, 9, 9))
+
+ // Construct the density estimator with the sample data and a standard deviation
+ // for the Gaussian kernels
+ val kd = new KernelDensity()
+ .setSample(data)
+ .setBandwidth(3.0)
+
+ // Find density estimates for the given values
+ val densities = kd.estimate(Array(-1.0, 2.0, 5.0))
+ // $example off$
+
+ densities.foreach(print)
--- End diff --
print => println
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r56435017
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/HypothesisTestingExample.scala ---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.mllib.regression.LabeledPoint
+import org.apache.spark.mllib.stat.Statistics
+import org.apache.spark.mllib.stat.test.ChiSqTestResult
+import org.apache.spark.rdd.RDD
+// $example off$
+
+object HypothesisTestingExample {
+
+ def main(args: Array[String]) {
+
+ val conf = new SparkConf().setAppName("HypothesisTestingExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ // a vector composed of the frequencies of events
+ val vec: Vector = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25)
+
+ // compute the goodness of fit. If a second vector to test against is not supplied
+ // as a parameter, the test runs against a uniform distribution.
+ val goodnessOfFitTestResult = Statistics.chiSqTest(vec)
+ // summary of the test including the p-value, degrees of freedom, test statistic, the method
+ // used, and the null hypothesis.
+ println(goodnessOfFitTestResult)
+ println()
+
+ // a contingency matrix. Create a dense matrix ((1.0, 2.0), (3.0, 4.0), (5.0, 6.0))
+ val mat: Matrix = Matrices.dense(3, 2, Array(1.0, 3.0, 5.0, 2.0, 4.0, 6.0))
+
+ // conduct Pearson's independence test on the input contingency matrix
+ val independenceTestResult = Statistics.chiSqTest(mat)
+ // summary of the test including the p-value, degrees of freedom
+ println(independenceTestResult)
+ println()
+
+ val p1 = LabeledPoint(1.0, Vectors.dense(1.0, 0.0, 3.0))
+ val p2 = LabeledPoint(1.0, Vectors.dense(1.0, 2.0, 0.0))
+ val p3 = LabeledPoint(-1.0, Vectors.dense(-1.0, 0.0, -0.5))
+ val obs: RDD[LabeledPoint] = sc.parallelize(Seq(p1, p2, p3)) // (feature, label) pairs.
+
+ // The contingency table is constructed from the raw (feature, label) pairs and used to conduct
+ // the independence test. Returns an array containing the ChiSquaredTestResult for every feature
+ // against the label.
+ val featureTestResults: Array[ChiSqTestResult] = Statistics.chiSqTest(obs)
+ featureTestResults.zipWithIndex.foreach { result =>
+ println(s"Column " + (result._2 + 1).toString + ":")
--- End diff --
You are not using `s"..."`. Put all variables inside quotes would make the code easier to read.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by keypointt <gi...@git.apache.org>.
Github user keypointt commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-197375927
cc @mengxr
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133908
--- Diff: examples/src/main/python/mllib/hypothesis_testing_example.py ---
@@ -0,0 +1,60 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import print_function
+
+from pyspark import SparkContext
+# $example on$
+from pyspark.mllib.linalg import Vectors, Matrices
--- End diff --
from pyspark.mllib.linalg import Matrices, Vectors
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133670
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaCorrelationsExample.java ---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
+
+import org.apache.spark.SparkConf;
+// $example on$
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaDoubleRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.mllib.linalg.Matrix;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.stat.Statistics;
+// $example off$
+
+public class JavaCorrelationsExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaCorrelationsExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ JavaDoubleRDD seriesX = jsc.parallelizeDoubles(
+ Arrays.asList(new Double[]{1.0, 2.0, 3.0, 3.0, 5.0})); // a series
+
+ // must have the same number of partitions and cardinality as seriesX
+ JavaDoubleRDD seriesY = jsc.parallelizeDoubles(
+ Arrays.asList(new Double[]{11.0, 22.0, 33.0, 33.0, 555.0}));
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-183101942
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-180969684
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-193012434
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52532/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133874
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaHypothesisTestingExample.java ---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
+
+import org.apache.spark.SparkConf;
+// $example on$
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.mllib.linalg.Matrices;
+import org.apache.spark.mllib.linalg.Matrix;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.stat.Statistics;
+import org.apache.spark.mllib.stat.test.ChiSqTestResult;
+// $example off$
+
+public class JavaHypothesisTestingExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaHypothesisTestingExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ Vector vec = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25); // a vector composed of the frequencies of events
+
+ // compute the goodness of fit. If a second vector to test against is not supplied as a parameter,
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133759
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaStratifiedSamplingExample.java ---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.HashMap;
+// $example on$
+import java.util.Map;
+// $example off$
+
+import scala.Tuple2;
+
+import org.apache.spark.api.java.function.VoidFunction;
+// $example on$
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.api.java.JavaPairRDD;
+// $example off$
+import org.apache.spark.SparkConf;
+
+public class JavaStratifiedSamplingExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaStratifiedSamplingExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ List<Tuple2<Integer, Character>> list = new ArrayList<>();
+ list.add(new Tuple2(1, 'a'));
+ list.add(new Tuple2(1, 'b'));
+ list.add(new Tuple2(2, 'c'));
+ list.add(new Tuple2(2, 'd'));
+ list.add(new Tuple2(2, 'e'));
+ list.add(new Tuple2(3, 'f'));
+
+ // an RDD of any key value pairs JavaPairRDD<K, V>
+ JavaPairRDD<Integer, Character> data = jsc.parallelizePairs(list);
+
+ // specify the exact fraction desired from each key Map<K, Object>
+ Map<Integer, Object> fractions = new HashMap<>();
+
+ fractions.put(1, 0.1);
+ fractions.put(2, 0.6);
+ fractions.put(3, 0.3);
+
+ // Get an exact sample from each stratum
+ JavaPairRDD<Integer, Character> approxSample = data.sampleByKey(false, fractions); // JavaPairRDD<K, V>
--- End diff --
execeeds 100
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r56433888
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaStratifiedSamplingExample.java ---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaSparkContext;
+
+// $example on$
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import scala.Tuple2;
+
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.function.VoidFunction;
+// $example off$
+
+public class JavaStratifiedSamplingExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaStratifiedSamplingExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ List<Tuple2<Integer, Character>> list = new ArrayList<>();
--- End diff --
Use `Arrays.list`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by keypointt <gi...@git.apache.org>.
Github user keypointt commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-199571476
I've add keyword argument at my forked repo: https://github.com/keypointt/spark/commit/892fe600e48b49b26a29120c99d171db02c659ab
should I open a separate PR? since new commit is not syncing to this PR
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55145536
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaHypothesisTestingKolmogorovSmirnovTestExample.java ---
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
--- End diff --
```java
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
// $example on$
import java.util.Arrays;
import org.apache.spark.api.java.JavaDoubleRDD;
import org.apache.spark.mllib.stat.Statistics;
import org.apache.spark.mllib.stat.test.KolmogorovSmirnovTestResult;
// $example off$
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-193472148
**[Test build #52598 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52598/consoleFull)** for PR 11108 at commit [`acf7096`](https://github.com/apache/spark/commit/acf7096e750b9150fbe309fc6c90aecb27b1102d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55145734
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaKernelDensityEstimationExample.java ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
+
+import org.apache.spark.SparkConf;
+// $example on$
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.mllib.stat.KernelDensity;
+// $example off$
+
+public class JavaKernelDensityEstimationExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaKernelDensityEstimationExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ JavaRDD<Double> data = jsc.parallelize(
+ Arrays.asList(1.0, 1.0, 1.0, 2.0, 3.0, 4.0, 5.0, 5.0, 6.0, 7.0, 8.0, 9.0, 9.0)); // an RDD of sample data
+
+ // Construct the density estimator with the sample data and a standard deviation for the Gaussian kernels
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-184576324
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51352/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133567
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/SummaryStatisticsExample.scala ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.linalg.Vectors
+import org.apache.spark.mllib.stat.{MultivariateStatisticalSummary, Statistics}
+// $example off$
+
+object SummaryStatisticsExample {
+
+ def main(args: Array[String]) {
--- End diff --
`def main(args: Array[String]):Unit = {`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55145679
--- Diff: docs/mllib-statistics.md ---
@@ -562,18 +277,7 @@ distribution `N(0, 1)`, and then map it to `N(1, 4)`.
Refer to the [`RandomRDDs` Scala docs](api/scala/index.html#org.apache.spark.mllib.random.RandomRDDs) for details on the API.
-{% highlight scala %}
-import org.apache.spark.SparkContext
-import org.apache.spark.mllib.random.RandomRDDs._
-
-val sc: SparkContext = ...
-
-// Generate a random double RDD that contains 1 million i.i.d. values drawn from the
-// standard normal distribution `N(0, 1)`, evenly distributed in 10 partitions.
-val u = normalRDD(sc, 1000000L, 10)
-// Apply a transform to get a random double RDD following `N(1, 4)`.
-val v = u.map(x => 1.0 + 2.0 * x)
-{% endhighlight %}
+{% include_example scala/org/apache/spark/examples/mllib/RandomDataGenerationExample.scala %}
--- End diff --
Let's keep the random data generation examples here without substitution, since the example is quite simple. Same for `JavaRandomDataGenerationExample` and `random_data_generation_example`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-183108603
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-193437107
@keypointt It looks good to me except for those trivials.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-199563705
I reverted the change in master to unblock other builds. @keypointt Could you test it on Scala 2.10? It seems that Scala 2.10 doesn't resolve keyword arguments nicely:
~~~scala
def f(a: String, b: String, c: String = "xxx") = {
println(a, b, c)
}
f(a = "abc", "def")
<console>:34: error: not enough arguments for method f: (a: String, b: String, c: String)Unit.
Unspecified value parameter b.
f(a = "abc", "def")
~~~
We should either fix the seed or use keyword argument for `fractions`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-199538981
Seems this one breaks scala 2.10 compilation (https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/spark-master-compile-sbt-scala-2.10/547/console). Can you fix it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133753
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaStratifiedSamplingExample.java ---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.ArrayList;
--- End diff --
```java
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
// $example on$
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import scala.Tuple2;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.function.VoidFunction;
// $example off$
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133728
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/StratifiedSamplingExample.scala ---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+// $example on$
--- End diff --
remove it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r56434426
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaSummaryStatisticsExample.java ---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaSparkContext;
+// $example on$
+import java.util.Arrays;
+
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.stat.MultivariateStatisticalSummary;
+import org.apache.spark.mllib.stat.Statistics;
+// $example off$
+
+public class JavaSummaryStatisticsExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaSummaryStatisticsExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ Vector v1 = Vectors.dense(1.0, 10.0, 100.0);
+ Vector v2 = Vectors.dense(2.0, 20.0, 200.0);
+ Vector v3 = Vectors.dense(3.0, 30.0, 300.0);
+
+ JavaRDD<Vector> mat = jsc.parallelize(Arrays.asList(v1, v2, v3)); // an RDD of Vectors
--- End diff --
Embed values in `Arras.asList`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-186930988
**[Test build #51646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51646/consoleFull)** for PR 11108 at commit [`f945222`](https://github.com/apache/spark/commit/f945222ad1cfd1f4756258a49b372aa7bd32d9fc).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-183108594
**[Test build #51145 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51145/consoleFull)** for PR 11108 at commit [`a4dd0fb`](https://github.com/apache/spark/commit/a4dd0fb5781dbdc162fd8652001b8de09b225297).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133729
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/StratifiedSamplingExample.scala ---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+// $example on$
+import org.apache.spark.{SparkConf, SparkContext}
+// $example off$
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r56434815
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/HypothesisTestingExample.scala ---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.mllib.regression.LabeledPoint
+import org.apache.spark.mllib.stat.Statistics
+import org.apache.spark.mllib.stat.test.ChiSqTestResult
+import org.apache.spark.rdd.RDD
+// $example off$
+
+object HypothesisTestingExample {
+
+ def main(args: Array[String]) {
+
+ val conf = new SparkConf().setAppName("HypothesisTestingExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ // a vector composed of the frequencies of events
+ val vec: Vector = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25)
+
+ // compute the goodness of fit. If a second vector to test against is not supplied
+ // as a parameter, the test runs against a uniform distribution.
+ val goodnessOfFitTestResult = Statistics.chiSqTest(vec)
+ // summary of the test including the p-value, degrees of freedom, test statistic, the method
+ // used, and the null hypothesis.
+ println(goodnessOfFitTestResult)
+ println()
--- End diff --
This is not really necessary as part of the example code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r56434972
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/HypothesisTestingExample.scala ---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.mllib.regression.LabeledPoint
+import org.apache.spark.mllib.stat.Statistics
+import org.apache.spark.mllib.stat.test.ChiSqTestResult
+import org.apache.spark.rdd.RDD
+// $example off$
+
+object HypothesisTestingExample {
+
+ def main(args: Array[String]) {
+
+ val conf = new SparkConf().setAppName("HypothesisTestingExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ // a vector composed of the frequencies of events
+ val vec: Vector = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25)
+
+ // compute the goodness of fit. If a second vector to test against is not supplied
+ // as a parameter, the test runs against a uniform distribution.
+ val goodnessOfFitTestResult = Statistics.chiSqTest(vec)
+ // summary of the test including the p-value, degrees of freedom, test statistic, the method
+ // used, and the null hypothesis.
+ println(goodnessOfFitTestResult)
+ println()
+
+ // a contingency matrix. Create a dense matrix ((1.0, 2.0), (3.0, 4.0), (5.0, 6.0))
+ val mat: Matrix = Matrices.dense(3, 2, Array(1.0, 3.0, 5.0, 2.0, 4.0, 6.0))
+
+ // conduct Pearson's independence test on the input contingency matrix
+ val independenceTestResult = Statistics.chiSqTest(mat)
+ // summary of the test including the p-value, degrees of freedom
+ println(independenceTestResult)
+ println()
+
+ val p1 = LabeledPoint(1.0, Vectors.dense(1.0, 0.0, 3.0))
+ val p2 = LabeledPoint(1.0, Vectors.dense(1.0, 2.0, 0.0))
+ val p3 = LabeledPoint(-1.0, Vectors.dense(-1.0, 0.0, -0.5))
+ val obs: RDD[LabeledPoint] = sc.parallelize(Seq(p1, p2, p3)) // (feature, label) pairs.
+
+ // The contingency table is constructed from the raw (feature, label) pairs and used to conduct
+ // the independence test. Returns an array containing the ChiSquaredTestResult for every feature
+ // against the label.
+ val featureTestResults: Array[ChiSqTestResult] = Statistics.chiSqTest(obs)
+ featureTestResults.zipWithIndex.foreach { result =>
--- End diff --
use `case (k, v) =>`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by keypointt <gi...@git.apache.org>.
Github user keypointt commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-199567084
Oh yes I just tried 2.10 and 2.11, where 2.11 build succeeded but 2.10 failed.
Should I just explicitly add val seed?
// specify the exact fraction desired from each key
val fractions = Map(1 -> 0.1, 2 -> 0.6, 3 -> 0.3)
val seed = Utils.random.nextLong
// Get an approximate sample from each stratum
val approxSample = data.sampleByKey(withReplacement = false, fractions, seed)
// Get an exact sample from each stratum
val exactSample = data.sampleByKeyExact(withReplacement = false, fractions, seed)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133731
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/StratifiedSamplingExample.scala ---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+// $example on$
+import org.apache.spark.{SparkConf, SparkContext}
+// $example off$
+
+object StratifiedSamplingExample {
+
+ def main(args: Array[String]) {
--- End diff --
def main(args: Array[String]): Unit = {
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-187928181
**[Test build #51800 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51800/consoleFull)** for PR 11108 at commit [`aec10ca`](https://github.com/apache/spark/commit/aec10cac879f29a3edb03e0855e60a772ca30c18).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55145531
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaHypothesisTestingKolmogorovSmirnovTestExample.java ---
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
+
+import org.apache.spark.SparkConf;
+// $example on$
+import org.apache.spark.api.java.JavaDoubleRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.mllib.stat.Statistics;
+import org.apache.spark.mllib.stat.test.KolmogorovSmirnovTestResult;
+// $example off$
+
+public class JavaHypothesisTestingKolmogorovSmirnovTestExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaHypothesisTestingKolmogorovSmirnovTestExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ JavaDoubleRDD data = jsc.parallelizeDoubles(Arrays.asList(0.1, 0.15, 0.2, 0.3, 0.25));
+ KolmogorovSmirnovTestResult testResult = Statistics.kolmogorovSmirnovTest(data, "norm", 0.0, 1.0);
--- End diff --
exceeds 100
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by keypointt <gi...@git.apache.org>.
Github user keypointt commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-199406018
Hi @mengxr how do you like this change?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-193013801
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133801
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/HypothesisTestingExample.scala ---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.mllib.regression.LabeledPoint
+import org.apache.spark.mllib.stat.Statistics
+import org.apache.spark.mllib.stat.test.ChiSqTestResult
+import org.apache.spark.rdd.RDD
+// $example off$
+
+object HypothesisTestingExample {
+
+ def main(args: Array[String]) {
+
+ val conf = new SparkConf().setAppName("HypothesisTestingExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ // a vector composed of the frequencies of events
+ val vec: Vector = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25)
+
+ // compute the goodness of fit. If a second vector to test against is not supplied
+ // as a parameter, the test runs against a uniform distribution.
+ val goodnessOfFitTestResult = Statistics.chiSqTest(vec)
+ println(goodnessOfFitTestResult) // summary of the test including the p-value,
+ // degrees of freedom, test statistic, the method used, and the null hypothesis.
+
+ // a contingency matrix. Create a dense matrix ((1.0, 2.0), (3.0, 4.0), (5.0, 6.0))
+ val mat: Matrix = Matrices.dense(3, 2, Array(1.0, 3.0, 5.0, 2.0, 4.0, 6.0))
+
+ // conduct Pearson's independence test on the input contingency matrix
+ val independenceTestResult = Statistics.chiSqTest(mat)
+ println(independenceTestResult) // summary of the test including the p-value, degrees of freedom
--- End diff --
change it into
```Scala
// summary of the test including the p-value, degrees of freedom
println(independenceTestResult)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55145417
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/HypothesisTestingKolmogorovSmirnovTestExample.scala ---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.stat.Statistics
+import org.apache.spark.rdd.RDD
+// $example off$
+
+object HypothesisTestingKolmogorovSmirnovTestExample {
+
+ def main(args: Array[String]) {
+
+ val conf = new SparkConf().setAppName("HypothesisTestingKolmogorovSmirnovTestExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ val data: RDD[Double] = sc.parallelize(Seq(0.1, 0.15, 0.2, 0.3, 0.25)) // an RDD of sample data
+
+ // run a KS test for the sample versus a standard normal distribution
+ val testResult = Statistics.kolmogorovSmirnovTest(data, "norm", 0, 1)
+ println(testResult) // summary of the test including the p-value, test statistic,
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by keypointt <gi...@git.apache.org>.
Github user keypointt commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-193010119
thanks a lot @yinxusen , I'm fixing the import format in the other PRs, will commit soon
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55265391
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/CorrelationsExample.scala ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.mllib.stat.Statistics
+import org.apache.spark.rdd.RDD
+// $example off$
+
+object CorrelationsExample {
+
+ def main(args: Array[String]): Unit = {
+
+ val conf = new SparkConf().setAppName("CorrelationsExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ val seriesX: RDD[Double] = sc.parallelize(Array(1, 2, 3, 3, 5)) // a series
+ val seriesY: RDD[Double] = sc.parallelize(Array(11, 22, 33, 33, 555))
+ // must have the same number of partitions and cardinality as seriesX
--- End diff --
It's strange to have a comment below a code. Let's move it up.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133797
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/HypothesisTestingExample.scala ---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.mllib.regression.LabeledPoint
+import org.apache.spark.mllib.stat.Statistics
+import org.apache.spark.mllib.stat.test.ChiSqTestResult
+import org.apache.spark.rdd.RDD
+// $example off$
+
+object HypothesisTestingExample {
+
+ def main(args: Array[String]) {
+
+ val conf = new SparkConf().setAppName("HypothesisTestingExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ // a vector composed of the frequencies of events
+ val vec: Vector = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25)
+
+ // compute the goodness of fit. If a second vector to test against is not supplied
+ // as a parameter, the test runs against a uniform distribution.
+ val goodnessOfFitTestResult = Statistics.chiSqTest(vec)
+ println(goodnessOfFitTestResult) // summary of the test including the p-value,
--- End diff --
Change the two lines into
```Scala
// summary of the test including the p-value, degrees of freedom, test statistic, the method
// used, and the null hypothesis.
println(goodnessOfFitTestResult)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133648
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/CorrelationsExample.scala ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.mllib.stat.Statistics
+// $example off$
+import org.apache.spark.rdd.RDD
+
+object CorrelationsExample {
+
+ def main(args: Array[String]) {
--- End diff --
def main(args: Array[String]): Unit = {
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-186669449
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51602/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133631
--- Diff: examples/src/main/python/mllib/summary_statistics_example.py ---
@@ -0,0 +1,43 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import print_function
+
+import numpy as np
--- End diff --
```python
from pyspark import SparkContext
# $example on$
import numpy as np
from pyspark.mllib.stat import Statistics
# $example off$
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-183108605
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51145/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133630
--- Diff: examples/src/main/python/mllib/summary_statistics_example.py ---
@@ -0,0 +1,43 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import print_function
+
+import numpy as np
+
+from pyspark import SparkContext
+# $example on$
+from pyspark.mllib.stat import Statistics
+# $example off$
+
+if __name__ == "__main__":
+ sc = SparkContext(appName="SummaryStatisticsExample") # SparkContext
+
+ # $example on$
+ v1 = np.array([1.0, 2.0, 3.0])
--- End diff --
change it the same with its scala/java companions
```python
v1 = np.array([1.0, 10.0, 100.0])
v2 = np.array([2.0, 20.0, 200.0])
v3 = np.array([3.0, 30.0, 300.0])
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-186669407
**[Test build #51602 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51602/consoleFull)** for PR 11108 at commit [`d817d0b`](https://github.com/apache/spark/commit/d817d0bbbea4913688bd6c3c66cecf95b9dbe198).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55273882
--- Diff: docs/mllib-statistics.md ---
@@ -584,24 +299,24 @@ distribution `N(0, 1)`, and then map it to `N(1, 4)`.
Refer to the [`RandomRDDs` Java docs](api/java/org/apache/spark/mllib/random/RandomRDDs) for details on the API.
-{% highlight java %}
-import org.apache.spark.SparkContext;
-import org.apache.spark.api.JavaDoubleRDD;
-import static org.apache.spark.mllib.random.RandomRDDs.*;
-
-JavaSparkContext jsc = ...
-
-// Generate a random double RDD that contains 1 million i.i.d. values drawn from the
-// standard normal distribution `N(0, 1)`, evenly distributed in 10 partitions.
-JavaDoubleRDD u = normalJavaRDD(jsc, 1000000L, 10);
-// Apply a transform to get a random double RDD following `N(1, 4)`.
-JavaDoubleRDD v = u.map(
- new Function<Double, Double>() {
- public Double call(Double x) {
- return 1.0 + 2.0 * x;
- }
- });
-{% endhighlight %}
+-{% highlight java %}
+ -import org.apache.spark.SparkContext;
+ -import org.apache.spark.api.JavaDoubleRDD;
+ -import static org.apache.spark.mllib.random.RandomRDDs.*;
+ -
+ -JavaSparkContext jsc = ...
+ -
+ -// Generate a random double RDD that contains 1 million i.i.d. values drawn from the
+ -// standard normal distribution `N(0, 1)`, evenly distributed in 10 partitions.
+ -JavaDoubleRDD u = normalJavaRDD(jsc, 1000000L, 10);
+ -// Apply a transform to get a random double RDD following `N(1, 4)`.
+ -JavaDoubleRDD v = u.map(
+ - new Function<Double, Double>() {
+ - public Double call(Double x) {
+ - return 1.0 + 2.0 * x;
+ - }
+ - });
+ -{% endhighlight %}
--- End diff --
No I mean you need to remove those minus signs in the front of each line.
On Monday, March 7, 2016, Xin Ren <no...@github.com> wrote:
> In docs/mllib-statistics.md
> <https://github.com/apache/spark/pull/11108#discussion_r55272379>:
>
> > + -import org.apache.spark.api.JavaDoubleRDD;
> > + -import static org.apache.spark.mllib.random.RandomRDDs.*;
> > + -
> > + -JavaSparkContext jsc = ...
> > + -
> > + -// Generate a random double RDD that contains 1 million i.i.d. values drawn from the
> > + -// standard normal distribution `N(0, 1)`, evenly distributed in 10 partitions.
> > + -JavaDoubleRDD u = normalJavaRDD(jsc, 1000000L, 10);
> > + -// Apply a transform to get a random double RDD following `N(1, 4)`.
> > + -JavaDoubleRDD v = u.map(
> > + - new Function<Double, Double>() {
> > + - public Double call(Double x) {
> > + - return 1.0 + 2.0 * x;
> > + - }
> > + - });
> > + -{% endhighlight %}
>
> you mean putting a comment here, explaining why didn't create a separate
> example file?
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/11108/files#r55272379>.
>
--
Cheers,
Xusen Yin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-199979915
* See https://github.com/apache/spark/blob/master/docs/building-spark.md#building-for-scala-210.
* We don't need to specify seed. Just use keyword argument for `fractions`.
* You can either re-open this PR or create a new one. Just need to make it clear what you changed on top of previously merged.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-198044618
**[Test build #53460 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53460/consoleFull)** for PR 11108 at commit [`a4eb28d`](https://github.com/apache/spark/commit/a4eb28d07a99e559132160f8ae9ac993d47d8fa3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-197634056
@mengxr If there is no empty line, we will get a bulk of results which are hard to tell and recognize. I think it's a good experience for users that they can use `./bin/run-example ml.xxxExample` to see some tidy outputs directly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-187928340
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51800/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55145468
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/HypothesisTestingKolmogorovSmirnovTestExample.scala ---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.stat.Statistics
+import org.apache.spark.rdd.RDD
+// $example off$
+
+object HypothesisTestingKolmogorovSmirnovTestExample {
+
+ def main(args: Array[String]) {
+
+ val conf = new SparkConf().setAppName("HypothesisTestingKolmogorovSmirnovTestExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ val data: RDD[Double] = sc.parallelize(Seq(0.1, 0.15, 0.2, 0.3, 0.25)) // an RDD of sample data
+
+ // run a KS test for the sample versus a standard normal distribution
+ val testResult = Statistics.kolmogorovSmirnovTest(data, "norm", 0, 1)
+ println(testResult) // summary of the test including the p-value, test statistic,
--- End diff --
add a println
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133875
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaHypothesisTestingExample.java ---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
+
+import org.apache.spark.SparkConf;
+// $example on$
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.mllib.linalg.Matrices;
+import org.apache.spark.mllib.linalg.Matrix;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.stat.Statistics;
+import org.apache.spark.mllib.stat.test.ChiSqTestResult;
+// $example off$
+
+public class JavaHypothesisTestingExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaHypothesisTestingExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ Vector vec = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25); // a vector composed of the frequencies of events
+
+ // compute the goodness of fit. If a second vector to test against is not supplied as a parameter,
+ // the test runs against a uniform distribution.
+ ChiSqTestResult goodnessOfFitTestResult = Statistics.chiSqTest(vec);
+ // summary of the test including the p-value, degrees of freedom, test statistic, the method used,
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55145730
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaKernelDensityEstimationExample.java ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
+
+import org.apache.spark.SparkConf;
+// $example on$
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.mllib.stat.KernelDensity;
+// $example off$
+
+public class JavaKernelDensityEstimationExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaKernelDensityEstimationExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ JavaRDD<Double> data = jsc.parallelize(
+ Arrays.asList(1.0, 1.0, 1.0, 2.0, 3.0, 4.0, 5.0, 5.0, 6.0, 7.0, 8.0, 9.0, 9.0)); // an RDD of sample data
--- End diff --
exceeds 100
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-198049131
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53460/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133661
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaCorrelationsExample.java ---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
--- End diff --
```java
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
// $example on$
import java.util.Arrays;
import org.apache.spark.api.java.JavaDoubleRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.mllib.linalg.Matrix;
import org.apache.spark.mllib.linalg.Vector;
import org.apache.spark.mllib.linalg.Vectors;
import org.apache.spark.mllib.stat.Statistics;
// $example off$
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-183102012
cc @yinxusen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133644
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/CorrelationsExample.scala ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
--- End diff --
```scala
import org.apache.spark.{SparkConf, SparkContext}
// $example on$
import org.apache.spark.mllib.linalg._
import org.apache.spark.mllib.stat.Statistics
import org.apache.spark.rdd.RDD
// $example off$
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/11108
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by keypointt <gi...@git.apache.org>.
Github user keypointt commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55272379
--- Diff: docs/mllib-statistics.md ---
@@ -584,24 +299,24 @@ distribution `N(0, 1)`, and then map it to `N(1, 4)`.
Refer to the [`RandomRDDs` Java docs](api/java/org/apache/spark/mllib/random/RandomRDDs) for details on the API.
-{% highlight java %}
-import org.apache.spark.SparkContext;
-import org.apache.spark.api.JavaDoubleRDD;
-import static org.apache.spark.mllib.random.RandomRDDs.*;
-
-JavaSparkContext jsc = ...
-
-// Generate a random double RDD that contains 1 million i.i.d. values drawn from the
-// standard normal distribution `N(0, 1)`, evenly distributed in 10 partitions.
-JavaDoubleRDD u = normalJavaRDD(jsc, 1000000L, 10);
-// Apply a transform to get a random double RDD following `N(1, 4)`.
-JavaDoubleRDD v = u.map(
- new Function<Double, Double>() {
- public Double call(Double x) {
- return 1.0 + 2.0 * x;
- }
- });
-{% endhighlight %}
+-{% highlight java %}
+ -import org.apache.spark.SparkContext;
+ -import org.apache.spark.api.JavaDoubleRDD;
+ -import static org.apache.spark.mllib.random.RandomRDDs.*;
+ -
+ -JavaSparkContext jsc = ...
+ -
+ -// Generate a random double RDD that contains 1 million i.i.d. values drawn from the
+ -// standard normal distribution `N(0, 1)`, evenly distributed in 10 partitions.
+ -JavaDoubleRDD u = normalJavaRDD(jsc, 1000000L, 10);
+ -// Apply a transform to get a random double RDD following `N(1, 4)`.
+ -JavaDoubleRDD v = u.map(
+ - new Function<Double, Double>() {
+ - public Double call(Double x) {
+ - return 1.0 + 2.0 * x;
+ - }
+ - });
+ -{% endhighlight %}
--- End diff --
you mean putting a comment here, explaining why didn't create a separate example file?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r56435241
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/SummaryStatisticsExample.scala ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.linalg.Vectors
+import org.apache.spark.mllib.stat.{MultivariateStatisticalSummary, Statistics}
+// $example off$
+
+object SummaryStatisticsExample {
+
+ def main(args: Array[String]): Unit = {
+
+ val conf = new SparkConf().setAppName("SummaryStatisticsExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ val v1 = Vectors.dense(1.0, 10.0, 100.0)
+ val v2 = Vectors.dense(2.0, 20.0, 200.0)
+ val v3 = Vectors.dense(3.0, 30.0, 300.0)
+
+ val observations = sc.parallelize(Seq(v1, v2, v3))
--- End diff --
Embed v1, v2, v3.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133763
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaStratifiedSamplingExample.java ---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.HashMap;
+// $example on$
+import java.util.Map;
+// $example off$
+
+import scala.Tuple2;
+
+import org.apache.spark.api.java.function.VoidFunction;
+// $example on$
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.api.java.JavaPairRDD;
+// $example off$
+import org.apache.spark.SparkConf;
+
+public class JavaStratifiedSamplingExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaStratifiedSamplingExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ List<Tuple2<Integer, Character>> list = new ArrayList<>();
+ list.add(new Tuple2(1, 'a'));
+ list.add(new Tuple2(1, 'b'));
+ list.add(new Tuple2(2, 'c'));
+ list.add(new Tuple2(2, 'd'));
+ list.add(new Tuple2(2, 'e'));
+ list.add(new Tuple2(3, 'f'));
+
+ // an RDD of any key value pairs JavaPairRDD<K, V>
+ JavaPairRDD<Integer, Character> data = jsc.parallelizePairs(list);
+
+ // specify the exact fraction desired from each key Map<K, Object>
+ Map<Integer, Object> fractions = new HashMap<>();
+
+ fractions.put(1, 0.1);
+ fractions.put(2, 0.6);
+ fractions.put(3, 0.3);
+
+ // Get an exact sample from each stratum
+ JavaPairRDD<Integer, Character> approxSample = data.sampleByKey(false, fractions); // JavaPairRDD<K, V>
+ JavaPairRDD<Integer, Character> exactSample = data.sampleByKeyExact(false, fractions); // JavaPairRDD<K, V>
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r56434392
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaStratifiedSamplingExample.java ---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaSparkContext;
+
+// $example on$
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import scala.Tuple2;
+
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.function.VoidFunction;
+// $example off$
+
+public class JavaStratifiedSamplingExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaStratifiedSamplingExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ List<Tuple2<Integer, Character>> list = new ArrayList<>();
+ list.add(new Tuple2(1, 'a'));
+ list.add(new Tuple2(1, 'b'));
+ list.add(new Tuple2(2, 'c'));
+ list.add(new Tuple2(2, 'd'));
+ list.add(new Tuple2(2, 'e'));
+ list.add(new Tuple2(3, 'f'));
+
+ // an RDD of any key value pairs JavaPairRDD<K, V>
+ JavaPairRDD<Integer, Character> data = jsc.parallelizePairs(list);
+
+ // specify the exact fraction desired from each key Map<K, Object>
+ Map<Integer, Object> fractions = new HashMap<>();
+
+ fractions.put(1, 0.1);
+ fractions.put(2, 0.6);
+ fractions.put(3, 0.3);
+
+ // Get an exact sample from each stratum
+ JavaPairRDD<Integer, Character> approxSample =
+ data.sampleByKey(false, fractions); // JavaPairRDD<K, V>
+ JavaPairRDD<Integer, Character> exactSample =
+ data.sampleByKeyExact(false, fractions); // JavaPairRDD<K, V>
+ // $example off$
+
+ approxSample.foreach(new VoidFunction<Tuple2<Integer, Character>>() {
--- End diff --
* Okay, I see why we don't want to have `println` lines in the example now. If it is just one line of code, I prefer putting it inside the example.
* This won't work on a distributed server. We need to collect them first.
~~~java
for(Tuple2<Integer, Character> t: approxSample.collect()) {
...
}
~~~
Please also try saving some type info by using `<>` if feasible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r56434589
--- Diff: examples/src/main/python/mllib/hypothesis_testing_example.py ---
@@ -0,0 +1,63 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import print_function
+
+from pyspark import SparkContext
+# $example on$
+from pyspark.mllib.linalg import Matrices, Vectors
+from pyspark.mllib.regression import LabeledPoint
+from pyspark.mllib.stat import Statistics
+# $example off$
+
+if __name__ == "__main__":
+ sc = SparkContext(appName="HypothesisTestingExample") # SparkContext
+
+ # $example on$
+ vec = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25) # a vector composed of the frequencies of events
+
+ # compute the goodness of fit. If a second vector to test against
+ # is not supplied as a parameter, the test runs against a uniform distribution.
+ goodnessOfFitTestResult = Statistics.chiSqTest(vec)
+ print(goodnessOfFitTestResult) # summary of the test including the p-value, degrees of freedom,
+ # test statistic, the method used, and the null hypothesis.
+ print()
+
+ mat = Matrices.dense(3, 2, [1.0, 3.0, 5.0, 2.0, 4.0, 6.0]) # a contingency matrix
+
+ # conduct Pearson's independence test on the input contingency matrix
+ independenceTestResult = Statistics.chiSqTest(mat)
+ print(independenceTestResult) # summary of the test including the p-value, degrees of freedom
+ print()
+
+ p1 = LabeledPoint(1.0, [1.0, 0.0, 3.0])
+ p2 = LabeledPoint(1.0, [1.0, 2.0, 0.0])
+ p3 = LabeledPoint(1.0, [-1.0, 0.0, -0.5])
+ obs = sc.parallelize([p1, p2, p3]) # LabeledPoint(feature, label)
+
+ # The contingency table is constructed from an RDD of LabeledPoint and used to conduct
+ # the independence test. Returns an array containing the ChiSquaredTestResult for every feature
+ # against the label.
+ featureTestResults = Statistics.chiSqTest(obs)
+
+ for i, result in enumerate(featureTestResults):
+ print("Column: " + str(i + 1))
--- End diff --
`print("Column %d:\n%s" % (i+1, result))` (index should be before `:`)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133852
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/HypothesisTestingExample.scala ---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.mllib.regression.LabeledPoint
+import org.apache.spark.mllib.stat.Statistics
+import org.apache.spark.mllib.stat.test.ChiSqTestResult
+import org.apache.spark.rdd.RDD
+// $example off$
+
+object HypothesisTestingExample {
+
+ def main(args: Array[String]) {
+
+ val conf = new SparkConf().setAppName("HypothesisTestingExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ // a vector composed of the frequencies of events
+ val vec: Vector = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25)
+
+ // compute the goodness of fit. If a second vector to test against is not supplied
+ // as a parameter, the test runs against a uniform distribution.
+ val goodnessOfFitTestResult = Statistics.chiSqTest(vec)
+ println(goodnessOfFitTestResult) // summary of the test including the p-value,
+ // degrees of freedom, test statistic, the method used, and the null hypothesis.
+
+ // a contingency matrix. Create a dense matrix ((1.0, 2.0), (3.0, 4.0), (5.0, 6.0))
+ val mat: Matrix = Matrices.dense(3, 2, Array(1.0, 3.0, 5.0, 2.0, 4.0, 6.0))
+
+ // conduct Pearson's independence test on the input contingency matrix
+ val independenceTestResult = Statistics.chiSqTest(mat)
+ println(independenceTestResult) // summary of the test including the p-value, degrees of freedom
+
+ val p1 = LabeledPoint(1.0, Vectors.dense(1.0, 0.0, 3.0))
+ val p2 = LabeledPoint(1.0, Vectors.dense(1.0, 2.0, 0.0))
+ val p3 = LabeledPoint(-1.0, Vectors.dense(-1.0, 0.0, -0.5))
+ val obs: RDD[LabeledPoint] = sc.parallelize(Seq(p1, p2, p3)) // (feature, label) pairs.
+
+ // The contingency table is constructed from the raw (feature, label) pairs and used to conduct
+ // the independence test. Returns an array containing the ChiSquaredTestResult for every feature
+ // against the label.
+ val featureTestResults: Array[ChiSqTestResult] = Statistics.chiSqTest(obs)
+ var i = 1
+ featureTestResults.foreach { result =>
+ println(s"Column $i:\n$result")
--- End diff --
add a "\n" for each output line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133872
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaHypothesisTestingExample.java ---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
+
+import org.apache.spark.SparkConf;
+// $example on$
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.mllib.linalg.Matrices;
+import org.apache.spark.mllib.linalg.Matrix;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.stat.Statistics;
+import org.apache.spark.mllib.stat.test.ChiSqTestResult;
+// $example off$
+
+public class JavaHypothesisTestingExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaHypothesisTestingExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ Vector vec = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25); // a vector composed of the frequencies of events
--- End diff --
exceeds 100
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-193472866
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-193012216
**[Test build #52532 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52532/consoleFull)** for PR 11108 at commit [`e2737ee`](https://github.com/apache/spark/commit/e2737eedd6c45c82f25045442b1d811ab2c395ec).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-197601829
I'm making a pass now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55145748
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaKernelDensityEstimationExample.java ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
--- End diff --
```java
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
// $example on$
import java.util.Arrays;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.mllib.stat.KernelDensity;
// $example off$
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55267567
--- Diff: docs/mllib-statistics.md ---
@@ -612,17 +327,17 @@ distribution `N(0, 1)`, and then map it to `N(1, 4)`.
Refer to the [`RandomRDDs` Python docs](api/python/pyspark.mllib.html#pyspark.mllib.random.RandomRDDs) for more details on the API.
-{% highlight python %}
-from pyspark.mllib.random import RandomRDDs
-
-sc = ... # SparkContext
-
-# Generate a random double RDD that contains 1 million i.i.d. values drawn from the
-# standard normal distribution `N(0, 1)`, evenly distributed in 10 partitions.
-u = RandomRDDs.normalRDD(sc, 1000000L, 10)
-# Apply a transform to get a random double RDD following `N(1, 4)`.
-v = u.map(lambda x: 1.0 + 2.0 * x)
-{% endhighlight %}
+-{% highlight python %}
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55145411
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/HypothesisTestingKolmogorovSmirnovTestExample.scala ---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.stat.Statistics
+import org.apache.spark.rdd.RDD
+// $example off$
+
+object HypothesisTestingKolmogorovSmirnovTestExample {
+
+ def main(args: Array[String]) {
+
+ val conf = new SparkConf().setAppName("HypothesisTestingKolmogorovSmirnovTestExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ val data: RDD[Double] = sc.parallelize(Seq(0.1, 0.15, 0.2, 0.3, 0.25)) // an RDD of sample data
--- End diff --
2-indent
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-193472871
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52598/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by keypointt <gi...@git.apache.org>.
Github user keypointt commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-199539470
Oh sure I'll try to fix it asap
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55267756
--- Diff: docs/mllib-statistics.md ---
@@ -584,24 +299,24 @@ distribution `N(0, 1)`, and then map it to `N(1, 4)`.
Refer to the [`RandomRDDs` Java docs](api/java/org/apache/spark/mllib/random/RandomRDDs) for details on the API.
-{% highlight java %}
-import org.apache.spark.SparkContext;
-import org.apache.spark.api.JavaDoubleRDD;
-import static org.apache.spark.mllib.random.RandomRDDs.*;
-
-JavaSparkContext jsc = ...
-
-// Generate a random double RDD that contains 1 million i.i.d. values drawn from the
-// standard normal distribution `N(0, 1)`, evenly distributed in 10 partitions.
-JavaDoubleRDD u = normalJavaRDD(jsc, 1000000L, 10);
-// Apply a transform to get a random double RDD following `N(1, 4)`.
-JavaDoubleRDD v = u.map(
- new Function<Double, Double>() {
- public Double call(Double x) {
- return 1.0 + 2.0 * x;
- }
- });
-{% endhighlight %}
+-{% highlight java %}
+ -import org.apache.spark.SparkContext;
+ -import org.apache.spark.api.JavaDoubleRDD;
+ -import static org.apache.spark.mllib.random.RandomRDDs.*;
+ -
+ -JavaSparkContext jsc = ...
+ -
+ -// Generate a random double RDD that contains 1 million i.i.d. values drawn from the
+ -// standard normal distribution `N(0, 1)`, evenly distributed in 10 partitions.
+ -JavaDoubleRDD u = normalJavaRDD(jsc, 1000000L, 10);
+ -// Apply a transform to get a random double RDD following `N(1, 4)`.
+ -JavaDoubleRDD v = u.map(
+ - new Function<Double, Double>() {
+ - public Double call(Double x) {
+ - return 1.0 + 2.0 * x;
+ - }
+ - });
+ -{% endhighlight %}
--- End diff --
Put comment here makes it more clear. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-184571237
**[Test build #51352 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51352/consoleFull)** for PR 11108 at commit [`0df3e65`](https://github.com/apache/spark/commit/0df3e65b7ea64165ec4e9301ddc8e91c1abcd082).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-184575949
**[Test build #51352 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51352/consoleFull)** for PR 11108 at commit [`0df3e65`](https://github.com/apache/spark/commit/0df3e65b7ea64165ec4e9301ddc8e91c1abcd082).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r56433747
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaHypothesisTestingExample.java ---
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaSparkContext;
+
+// $example on$
+import java.util.Arrays;
+
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.mllib.linalg.Matrices;
+import org.apache.spark.mllib.linalg.Matrix;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.stat.Statistics;
+import org.apache.spark.mllib.stat.test.ChiSqTestResult;
+// $example off$
+
+public class JavaHypothesisTestingExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaHypothesisTestingExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ // a vector composed of the frequencies of events
+ Vector vec = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25);
+
+ // compute the goodness of fit. If a second vector to test against is not supplied
+ // as a parameter, the test runs against a uniform distribution.
+ ChiSqTestResult goodnessOfFitTestResult = Statistics.chiSqTest(vec);
+ // summary of the test including the p-value, degrees of freedom, test statistic,
+ // the method used, and the null hypothesis.
+ System.out.println(goodnessOfFitTestResult);
+ System.out.println();
+
+ // Create a contingency matrix ((1.0, 2.0), (3.0, 4.0), (5.0, 6.0))
+ Matrix mat = Matrices.dense(3, 2, new double[]{1.0, 3.0, 5.0, 2.0, 4.0, 6.0});
+
+ // conduct Pearson's independence test on the input contingency matrix
+ ChiSqTestResult independenceTestResult = Statistics.chiSqTest(mat);
+ // summary of the test including the p-value, degrees of freedom...
+ System.out.println(independenceTestResult);
+ System.out.println();
+
+ LabeledPoint p1 = new LabeledPoint(1.0, Vectors.dense(1.0, 0.0, 3.0));
+ LabeledPoint p2 = new LabeledPoint(1.0, Vectors.dense(1.0, 2.0, 0.0));
+ LabeledPoint p3 = new LabeledPoint(-1.0, Vectors.dense(-1.0, 0.0, -0.5));
+ // an RDD of labeled points
+ JavaRDD<LabeledPoint> obs = jsc.parallelize(Arrays.asList(p1, p2, p3));
--- End diff --
Same here. Embed `new LabelPoint` in `Arrays.asList`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133868
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaHypothesisTestingExample.java ---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
--- End diff --
```java
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
// $example on$
import java.util.Arrays;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.mllib.linalg.Matrices;
import org.apache.spark.mllib.linalg.Matrix;
import org.apache.spark.mllib.linalg.Vector;
import org.apache.spark.mllib.linalg.Vectors;
import org.apache.spark.mllib.regression.LabeledPoint;
import org.apache.spark.mllib.stat.Statistics;
import org.apache.spark.mllib.stat.test.ChiSqTestResult;
// $example off$
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133590
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaSummaryStatisticsExample.java ---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
+
+// $example on$
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.stat.MultivariateStatisticalSummary;
+import org.apache.spark.mllib.stat.Statistics;
+// $example off$
+import org.apache.spark.SparkConf;
+
+public class JavaSummaryStatisticsExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaSummaryStatisticsExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ Vector v1 = Vectors.dense(1.0, 10.0, 100.0);
+ Vector v2 = Vectors.dense(2.0, 20.0, 200.0);
+ Vector v3 = Vectors.dense(3.0, 30.0, 300.0);
+
+ JavaRDD<Vector> mat = jsc.parallelize(Arrays.asList(v1, v2, v3)); // an RDD of Vectors
+
+ // Compute column summary statistics.
+ MultivariateStatisticalSummary summary = Statistics.colStats(mat.rdd());
+ System.out.println(summary.mean()); // a dense vector containing the mean value for each column
+ System.out.println(summary.variance()); // column-wise variance
+ System.out.println(summary.numNonzeros()); // number of nonzeros in each column
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-198004940
I agree. But we don't really need lines just to print empty lines. For example:
~~~scala
println(goodnessOfFitTestResult)
println()
~~~
could be simply
~~~scala
println(s"$goodnessOfFitTestResult\n")
~~~
It is good to keep the example code short.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133891
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaHypothesisTestingExample.java ---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
+
+import org.apache.spark.SparkConf;
+// $example on$
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.mllib.linalg.Matrices;
+import org.apache.spark.mllib.linalg.Matrix;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.stat.Statistics;
+import org.apache.spark.mllib.stat.test.ChiSqTestResult;
+// $example off$
+
+public class JavaHypothesisTestingExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaHypothesisTestingExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ Vector vec = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25); // a vector composed of the frequencies of events
+
+ // compute the goodness of fit. If a second vector to test against is not supplied as a parameter,
+ // the test runs against a uniform distribution.
+ ChiSqTestResult goodnessOfFitTestResult = Statistics.chiSqTest(vec);
+ // summary of the test including the p-value, degrees of freedom, test statistic, the method used,
+ // and the null hypothesis.
+ System.out.println(goodnessOfFitTestResult);
+
+ // Create a dense matrix ((1.0, 2.0), (3.0, 4.0), (5.0, 6.0))
+ Matrix mat = Matrices.dense(3, 2, new double[]{1.0, 3.0, 5.0, 2.0, 4.0, 6.0}); // a contingency matrix
+
+ // conduct Pearson's independence test on the input contingency matrix
+ ChiSqTestResult independenceTestResult = Statistics.chiSqTest(mat);
+ // summary of the test including the p-value, degrees of freedom...
+ System.out.println(independenceTestResult);
+
+ LabeledPoint p1 = new LabeledPoint(1.0, Vectors.dense(1.0, 0.0, 3.0));
+ LabeledPoint p2 = new LabeledPoint(1.0, Vectors.dense(1.0, 2.0, 0.0));
+ LabeledPoint p3 = new LabeledPoint(-1.0, Vectors.dense(-1.0, 0.0, -0.5));
+ JavaRDD<LabeledPoint> obs = jsc.parallelize(Arrays.asList(p1, p2, p3)); // an RDD of labeled points
+
+ // The contingency table is constructed from the raw (feature, label) pairs and used to conduct
+ // the independence test. Returns an array containing the ChiSquaredTestResult for every feature
+ // against the label.
+ ChiSqTestResult[] featureTestResults = Statistics.chiSqTest(obs.rdd());
+ int i = 1;
+ for (ChiSqTestResult result : featureTestResults) {
+ System.out.println("Column " + i + ":");
+ System.out.println(result); // summary of the test
--- End diff --
add a println
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-193010336
**[Test build #52532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52532/consoleFull)** for PR 11108 at commit [`e2737ee`](https://github.com/apache/spark/commit/e2737eedd6c45c82f25045442b1d811ab2c395ec).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-193984966
@keypointt Thanks for working on this. @mengxr LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55145721
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/KernelDensityEstimationExample.scala ---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.stat.KernelDensity
+import org.apache.spark.rdd.RDD
+// $example off$
+
+object KernelDensityEstimationExample {
+
+ def main(args: Array[String]) {
--- End diff --
def main(args: Array[String]): Unit = {
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r56433742
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaCorrelationsExample.java ---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaSparkContext;
+// $example on$
+import java.util.Arrays;
+
+import org.apache.spark.api.java.JavaDoubleRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.mllib.linalg.Matrix;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.stat.Statistics;
+// $example off$
+
+public class JavaCorrelationsExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaCorrelationsExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ JavaDoubleRDD seriesX = jsc.parallelizeDoubles(
+ Arrays.asList(1.0, 2.0, 3.0, 3.0, 5.0)); // a series
+
+ // must have the same number of partitions and cardinality as seriesX
+ JavaDoubleRDD seriesY = jsc.parallelizeDoubles(
+ Arrays.asList(11.0, 22.0, 33.0, 33.0, 555.0));
+
+ // compute the correlation using Pearson's method. Enter "spearman" for Spearman's method.
+ // If a method is not specified, Pearson's method will be used by default.
+ Double correlation = Statistics.corr(seriesX.srdd(), seriesY.srdd(), "pearson");
+ System.out.println("Correlation is: " + correlation);
+
+ Vector v1 = Vectors.dense(1.0, 10.0, 100.0);
+ Vector v2 = Vectors.dense(2.0, 20.0, 200.0);
+ Vector v3 = Vectors.dense(5.0, 33.0, 366.0);
+
+ // note that each Vector is a row and not a column
+ JavaRDD<Vector> data = jsc.parallelize(Arrays.asList(v1, v2, v3));
--- End diff --
We can save some code by embedding `Vectors.dense(...)` in `Arrays.asList(...)`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55266994
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaStratifiedSamplingExample.java ---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaSparkContext;
+
+// $example on$
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import scala.Tuple2;
+
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.function.VoidFunction;
+// $example off$
+
+public class JavaStratifiedSamplingExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaStratifiedSamplingExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ List<Tuple2<Integer, Character>> list = new ArrayList<>();
+ list.add(new Tuple2(1, 'a'));
+ list.add(new Tuple2(1, 'b'));
+ list.add(new Tuple2(2, 'c'));
+ list.add(new Tuple2(2, 'd'));
+ list.add(new Tuple2(2, 'e'));
+ list.add(new Tuple2(3, 'f'));
+
+ // an RDD of any key value pairs JavaPairRDD<K, V>
+ JavaPairRDD<Integer, Character> data = jsc.parallelizePairs(list);
+
+ // specify the exact fraction desired from each key Map<K, Object>
+ Map<Integer, Object> fractions = new HashMap<>();
+
+ fractions.put(1, 0.1);
+ fractions.put(2, 0.6);
+ fractions.put(3, 0.3);
+
+ // Get an exact sample from each stratum
+ JavaPairRDD<Integer, Character> approxSample =
+ data.sampleByKey(false, fractions); // JavaPairRDD<K, V>
+ JavaPairRDD<Integer, Character> exactSample =
+ data.sampleByKeyExact(false, fractions); // JavaPairRDD<K, V>
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55265409
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/CorrelationsExample.scala ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.mllib.stat.Statistics
+import org.apache.spark.rdd.RDD
+// $example off$
+
+object CorrelationsExample {
+
+ def main(args: Array[String]): Unit = {
+
+ val conf = new SparkConf().setAppName("CorrelationsExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ val seriesX: RDD[Double] = sc.parallelize(Array(1, 2, 3, 3, 5)) // a series
+ val seriesY: RDD[Double] = sc.parallelize(Array(11, 22, 33, 33, 555))
+ // must have the same number of partitions and cardinality as seriesX
+
+ // compute the correlation using Pearson's method. Enter "spearman" for Spearman's method. If a
+ // method is not specified, Pearson's method will be used by default.
+ val correlation: Double = Statistics.corr(seriesX, seriesY, "pearson")
+ println(s"Correlation is: $correlation")
+
+ val data: RDD[Vector] = sc.parallelize(
+ Seq(
+ Vectors.dense(1.0, 10.0, 100.0),
+ Vectors.dense(2.0, 20.0, 200.0),
+ Vectors.dense(5.0, 33.0, 366.0))
+ ) // note that each Vector is a row and not a column
--- End diff --
2-indent here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-187919190
**[Test build #51800 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51800/consoleFull)** for PR 11108 at commit [`aec10ca`](https://github.com/apache/spark/commit/aec10cac879f29a3edb03e0855e60a772ca30c18).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-193012431
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-184576320
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133922
--- Diff: examples/src/main/python/mllib/hypothesis_testing_example.py ---
@@ -0,0 +1,60 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import print_function
+
+from pyspark import SparkContext
+# $example on$
+from pyspark.mllib.linalg import Vectors, Matrices
+from pyspark.mllib.regression import LabeledPoint
+from pyspark.mllib.stat import Statistics
+# $example off$
+
+if __name__ == "__main__":
+ sc = SparkContext(appName="HypothesisTestingExample") # SparkContext
+
+ # $example on$
+ vec = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25) # a vector composed of the frequencies of events
+
+ # compute the goodness of fit. If a second vector to test against
+ # is not supplied as a parameter, the test runs against a uniform distribution.
+ goodnessOfFitTestResult = Statistics.chiSqTest(vec)
+ print(goodnessOfFitTestResult) # summary of the test including the p-value, degrees of freedom,
+ # test statistic, the method used, and the null hypothesis.
+
+ mat = Matrices.dense(3, 2, [1.0, 3.0, 5.0, 2.0, 4.0, 6.0]) # a contingency matrix
+
+ # conduct Pearson's independence test on the input contingency matrix
+ independenceTestResult = Statistics.chiSqTest(mat)
+ print(independenceTestResult) # summary of the test including the p-value, degrees of freedom
--- End diff --
add a println
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133878
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaHypothesisTestingExample.java ---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
+
+import org.apache.spark.SparkConf;
+// $example on$
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.mllib.linalg.Matrices;
+import org.apache.spark.mllib.linalg.Matrix;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.stat.Statistics;
+import org.apache.spark.mllib.stat.test.ChiSqTestResult;
+// $example off$
+
+public class JavaHypothesisTestingExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaHypothesisTestingExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ Vector vec = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25); // a vector composed of the frequencies of events
+
+ // compute the goodness of fit. If a second vector to test against is not supplied as a parameter,
+ // the test runs against a uniform distribution.
+ ChiSqTestResult goodnessOfFitTestResult = Statistics.chiSqTest(vec);
+ // summary of the test including the p-value, degrees of freedom, test statistic, the method used,
+ // and the null hypothesis.
+ System.out.println(goodnessOfFitTestResult);
+
+ // Create a dense matrix ((1.0, 2.0), (3.0, 4.0), (5.0, 6.0))
+ Matrix mat = Matrices.dense(3, 2, new double[]{1.0, 3.0, 5.0, 2.0, 4.0, 6.0}); // a contingency matrix
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133654
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/CorrelationsExample.scala ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.mllib.stat.Statistics
+// $example off$
+import org.apache.spark.rdd.RDD
+
+object CorrelationsExample {
+
+ def main(args: Array[String]) {
+
+ val conf = new SparkConf().setAppName("CorrelationsExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ val seriesX: RDD[Double] = sc.parallelize(Array(1, 2, 3, 3, 5)) // a series
--- End diff --
2-indent for inline comment
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133883
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaHypothesisTestingExample.java ---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
+
+import org.apache.spark.SparkConf;
+// $example on$
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.mllib.linalg.Matrices;
+import org.apache.spark.mllib.linalg.Matrix;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.stat.Statistics;
+import org.apache.spark.mllib.stat.test.ChiSqTestResult;
+// $example off$
+
+public class JavaHypothesisTestingExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaHypothesisTestingExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ Vector vec = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25); // a vector composed of the frequencies of events
+
+ // compute the goodness of fit. If a second vector to test against is not supplied as a parameter,
+ // the test runs against a uniform distribution.
+ ChiSqTestResult goodnessOfFitTestResult = Statistics.chiSqTest(vec);
+ // summary of the test including the p-value, degrees of freedom, test statistic, the method used,
+ // and the null hypothesis.
+ System.out.println(goodnessOfFitTestResult);
+
+ // Create a dense matrix ((1.0, 2.0), (3.0, 4.0), (5.0, 6.0))
+ Matrix mat = Matrices.dense(3, 2, new double[]{1.0, 3.0, 5.0, 2.0, 4.0, 6.0}); // a contingency matrix
+
+ // conduct Pearson's independence test on the input contingency matrix
+ ChiSqTestResult independenceTestResult = Statistics.chiSqTest(mat);
+ // summary of the test including the p-value, degrees of freedom...
+ System.out.println(independenceTestResult);
+
+ LabeledPoint p1 = new LabeledPoint(1.0, Vectors.dense(1.0, 0.0, 3.0));
+ LabeledPoint p2 = new LabeledPoint(1.0, Vectors.dense(1.0, 2.0, 0.0));
+ LabeledPoint p3 = new LabeledPoint(-1.0, Vectors.dense(-1.0, 0.0, -0.5));
+ JavaRDD<LabeledPoint> obs = jsc.parallelize(Arrays.asList(p1, p2, p3)); // an RDD of labeled points
+
+ // The contingency table is constructed from the raw (feature, label) pairs and used to conduct
+ // the independence test. Returns an array containing the ChiSquaredTestResult for every feature
+ // against the label.
+ ChiSqTestResult[] featureTestResults = Statistics.chiSqTest(obs.rdd());
+ int i = 1;
+ for (ChiSqTestResult result : featureTestResults) {
+ System.out.println("Column " + i + ":");
+ System.out.println(result); // summary of the test
--- End diff --
2-indent
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133765
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaStratifiedSamplingExample.java ---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.HashMap;
+// $example on$
+import java.util.Map;
+// $example off$
+
+import scala.Tuple2;
+
+import org.apache.spark.api.java.function.VoidFunction;
+// $example on$
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.api.java.JavaPairRDD;
+// $example off$
+import org.apache.spark.SparkConf;
+
+public class JavaStratifiedSamplingExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaStratifiedSamplingExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ List<Tuple2<Integer, Character>> list = new ArrayList<>();
+ list.add(new Tuple2(1, 'a'));
+ list.add(new Tuple2(1, 'b'));
+ list.add(new Tuple2(2, 'c'));
+ list.add(new Tuple2(2, 'd'));
+ list.add(new Tuple2(2, 'e'));
+ list.add(new Tuple2(3, 'f'));
+
+ // an RDD of any key value pairs JavaPairRDD<K, V>
+ JavaPairRDD<Integer, Character> data = jsc.parallelizePairs(list);
+
+ // specify the exact fraction desired from each key Map<K, Object>
+ Map<Integer, Object> fractions = new HashMap<>();
+
+ fractions.put(1, 0.1);
+ fractions.put(2, 0.6);
+ fractions.put(3, 0.3);
+
+ // Get an exact sample from each stratum
+ JavaPairRDD<Integer, Character> approxSample = data.sampleByKey(false, fractions); // JavaPairRDD<K, V>
+ JavaPairRDD<Integer, Character> exactSample = data.sampleByKeyExact(false, fractions); // JavaPairRDD<K, V>
+ // $example off$
+
+ approxSample.foreach(new VoidFunction<Tuple2<Integer, Character>>() {
+ public void call(Tuple2<Integer, Character> t) throws Exception {
+ System.out.println(t._1() + " " + t._2());
+ }
+ });
+
--- End diff --
add a println here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55267527
--- Diff: docs/mllib-statistics.md ---
@@ -562,18 +277,18 @@ distribution `N(0, 1)`, and then map it to `N(1, 4)`.
Refer to the [`RandomRDDs` Scala docs](api/scala/index.html#org.apache.spark.mllib.random.RandomRDDs) for details on the API.
-{% highlight scala %}
-import org.apache.spark.SparkContext
-import org.apache.spark.mllib.random.RandomRDDs._
-
-val sc: SparkContext = ...
-
-// Generate a random double RDD that contains 1 million i.i.d. values drawn from the
-// standard normal distribution `N(0, 1)`, evenly distributed in 10 partitions.
-val u = normalRDD(sc, 1000000L, 10)
-// Apply a transform to get a random double RDD following `N(1, 4)`.
-val v = u.map(x => 1.0 + 2.0 * x)
-{% endhighlight %}
+-{% highlight scala %}
--- End diff --
Remove those "-"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-199527881
LGTM. Merged into master. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-193463121
**[Test build #52598 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52598/consoleFull)** for PR 11108 at commit [`acf7096`](https://github.com/apache/spark/commit/acf7096e750b9150fbe309fc6c90aecb27b1102d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-187928337
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-186935010
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51646/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133804
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/HypothesisTestingExample.scala ---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.mllib.regression.LabeledPoint
+import org.apache.spark.mllib.stat.Statistics
+import org.apache.spark.mllib.stat.test.ChiSqTestResult
+import org.apache.spark.rdd.RDD
+// $example off$
+
+object HypothesisTestingExample {
+
+ def main(args: Array[String]) {
+
+ val conf = new SparkConf().setAppName("HypothesisTestingExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ // a vector composed of the frequencies of events
+ val vec: Vector = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25)
+
+ // compute the goodness of fit. If a second vector to test against is not supplied
+ // as a parameter, the test runs against a uniform distribution.
+ val goodnessOfFitTestResult = Statistics.chiSqTest(vec)
+ println(goodnessOfFitTestResult) // summary of the test including the p-value,
+ // degrees of freedom, test statistic, the method used, and the null hypothesis.
+
+ // a contingency matrix. Create a dense matrix ((1.0, 2.0), (3.0, 4.0), (5.0, 6.0))
+ val mat: Matrix = Matrices.dense(3, 2, Array(1.0, 3.0, 5.0, 2.0, 4.0, 6.0))
+
+ // conduct Pearson's independence test on the input contingency matrix
+ val independenceTestResult = Statistics.chiSqTest(mat)
+ println(independenceTestResult) // summary of the test including the p-value, degrees of freedom
+
+ val p1 = LabeledPoint(1.0, Vectors.dense(1.0, 0.0, 3.0))
+ val p2 = LabeledPoint(1.0, Vectors.dense(1.0, 2.0, 0.0))
+ val p3 = LabeledPoint(-1.0, Vectors.dense(-1.0, 0.0, -0.5))
+ val obs: RDD[LabeledPoint] = sc.parallelize(Seq(p1, p2, p3)) // (feature, label) pairs.
--- End diff --
2-indent
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-192960056
@keypointt Finish one pass
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r56435214
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/StratifiedSamplingExample.scala ---
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+
+object StratifiedSamplingExample {
+
+ def main(args: Array[String]): Unit = {
+
+ val conf = new SparkConf().setAppName("StratifiedSamplingExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ // an RDD[(K, V)] of any key value pairs
+ val data = sc.parallelize(
+ Seq((1, 'a'), (1, 'b'), (2, 'c'), (2, 'd'), (2, 'e'), (3, 'f')))
+
+ // specify the exact fraction desired from each key
+ val fractions = Map(1 -> 0.1, 2 -> 0.6, 3 -> 0.3)
+
+ // Get an exact sample from each stratum
+ val approxSample = data.sampleByKey(withReplacement = false, fractions)
+ val exactSample = data.sampleByKeyExact(withReplacement = false, fractions)
+ // $example off$
+
+ approxSample.foreach(println)
+ println()
+ exactSample.foreach(println)
--- End diff --
(minor) To show that the sample size is exact. We can print its size.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r56435169
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/StratifiedSamplingExample.scala ---
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+
+object StratifiedSamplingExample {
+
+ def main(args: Array[String]): Unit = {
+
+ val conf = new SparkConf().setAppName("StratifiedSamplingExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ // an RDD[(K, V)] of any key value pairs
+ val data = sc.parallelize(
+ Seq((1, 'a'), (1, 'b'), (2, 'c'), (2, 'd'), (2, 'e'), (3, 'f')))
+
+ // specify the exact fraction desired from each key
+ val fractions = Map(1 -> 0.1, 2 -> 0.6, 3 -> 0.3)
+
+ // Get an exact sample from each stratum
+ val approxSample = data.sampleByKey(withReplacement = false, fractions)
+ val exactSample = data.sampleByKeyExact(withReplacement = false, fractions)
+ // $example off$
+
+ approxSample.foreach(println)
--- End diff --
It would be useful to print a header to say which one is approx and which is exact.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133879
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaHypothesisTestingExample.java ---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
+
+import org.apache.spark.SparkConf;
+// $example on$
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.mllib.linalg.Matrices;
+import org.apache.spark.mllib.linalg.Matrix;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.stat.Statistics;
+import org.apache.spark.mllib.stat.test.ChiSqTestResult;
+// $example off$
+
+public class JavaHypothesisTestingExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaHypothesisTestingExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ Vector vec = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25); // a vector composed of the frequencies of events
+
+ // compute the goodness of fit. If a second vector to test against is not supplied as a parameter,
+ // the test runs against a uniform distribution.
+ ChiSqTestResult goodnessOfFitTestResult = Statistics.chiSqTest(vec);
+ // summary of the test including the p-value, degrees of freedom, test statistic, the method used,
+ // and the null hypothesis.
+ System.out.println(goodnessOfFitTestResult);
+
+ // Create a dense matrix ((1.0, 2.0), (3.0, 4.0), (5.0, 6.0))
+ Matrix mat = Matrices.dense(3, 2, new double[]{1.0, 3.0, 5.0, 2.0, 4.0, 6.0}); // a contingency matrix
+
+ // conduct Pearson's independence test on the input contingency matrix
+ ChiSqTestResult independenceTestResult = Statistics.chiSqTest(mat);
+ // summary of the test including the p-value, degrees of freedom...
+ System.out.println(independenceTestResult);
+
+ LabeledPoint p1 = new LabeledPoint(1.0, Vectors.dense(1.0, 0.0, 3.0));
+ LabeledPoint p2 = new LabeledPoint(1.0, Vectors.dense(1.0, 2.0, 0.0));
+ LabeledPoint p3 = new LabeledPoint(-1.0, Vectors.dense(-1.0, 0.0, -0.5));
+ JavaRDD<LabeledPoint> obs = jsc.parallelize(Arrays.asList(p1, p2, p3)); // an RDD of labeled points
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-183130950
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51154/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-186668527
**[Test build #51602 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51602/consoleFull)** for PR 11108 at commit [`d817d0b`](https://github.com/apache/spark/commit/d817d0bbbea4913688bd6c3c66cecf95b9dbe198).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-183130947
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133889
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaHypothesisTestingExample.java ---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
+
+import org.apache.spark.SparkConf;
+// $example on$
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.mllib.linalg.Matrices;
+import org.apache.spark.mllib.linalg.Matrix;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.stat.Statistics;
+import org.apache.spark.mllib.stat.test.ChiSqTestResult;
+// $example off$
+
+public class JavaHypothesisTestingExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaHypothesisTestingExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ Vector vec = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25); // a vector composed of the frequencies of events
+
+ // compute the goodness of fit. If a second vector to test against is not supplied as a parameter,
+ // the test runs against a uniform distribution.
+ ChiSqTestResult goodnessOfFitTestResult = Statistics.chiSqTest(vec);
+ // summary of the test including the p-value, degrees of freedom, test statistic, the method used,
+ // and the null hypothesis.
+ System.out.println(goodnessOfFitTestResult);
+
+ // Create a dense matrix ((1.0, 2.0), (3.0, 4.0), (5.0, 6.0))
+ Matrix mat = Matrices.dense(3, 2, new double[]{1.0, 3.0, 5.0, 2.0, 4.0, 6.0}); // a contingency matrix
+
+ // conduct Pearson's independence test on the input contingency matrix
+ ChiSqTestResult independenceTestResult = Statistics.chiSqTest(mat);
+ // summary of the test including the p-value, degrees of freedom...
+ System.out.println(independenceTestResult);
--- End diff --
add a println
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r56434099
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaStratifiedSamplingExample.java ---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaSparkContext;
+
+// $example on$
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import scala.Tuple2;
+
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.function.VoidFunction;
+// $example off$
+
+public class JavaStratifiedSamplingExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaStratifiedSamplingExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ List<Tuple2<Integer, Character>> list = new ArrayList<>();
+ list.add(new Tuple2(1, 'a'));
+ list.add(new Tuple2(1, 'b'));
+ list.add(new Tuple2(2, 'c'));
+ list.add(new Tuple2(2, 'd'));
+ list.add(new Tuple2(2, 'e'));
+ list.add(new Tuple2(3, 'f'));
+
+ // an RDD of any key value pairs JavaPairRDD<K, V>
+ JavaPairRDD<Integer, Character> data = jsc.parallelizePairs(list);
+
+ // specify the exact fraction desired from each key Map<K, Object>
+ Map<Integer, Object> fractions = new HashMap<>();
+
+ fractions.put(1, 0.1);
+ fractions.put(2, 0.6);
+ fractions.put(3, 0.3);
+
+ // Get an exact sample from each stratum
+ JavaPairRDD<Integer, Character> approxSample =
+ data.sampleByKey(false, fractions); // JavaPairRDD<K, V>
--- End diff --
remove comment
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by keypointt <gi...@git.apache.org>.
Github user keypointt commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-199537847
Thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-198040141
Sure, that makes sense.
Sent from my iPhone
> On Mar 17, 2016, at 11:04, Xiangrui Meng <no...@github.com> wrote:
>
> I agree. But we don't really need lines just to print empty lines. For example:
>
> println(goodnessOfFitTestResult)
> println()
> could be simply
>
> println(s"$goodnessOfFitTestResult\n")
> It is good to keep the example code short.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133911
--- Diff: examples/src/main/python/mllib/hypothesis_testing_example.py ---
@@ -0,0 +1,60 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import print_function
+
+from pyspark import SparkContext
+# $example on$
+from pyspark.mllib.linalg import Vectors, Matrices
+from pyspark.mllib.regression import LabeledPoint
+from pyspark.mllib.stat import Statistics
+# $example off$
+
+if __name__ == "__main__":
+ sc = SparkContext(appName="HypothesisTestingExample") # SparkContext
+
+ # $example on$
+ vec = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25) # a vector composed of the frequencies of events
+
+ # compute the goodness of fit. If a second vector to test against
+ # is not supplied as a parameter, the test runs against a uniform distribution.
+ goodnessOfFitTestResult = Statistics.chiSqTest(vec)
+ print(goodnessOfFitTestResult) # summary of the test including the p-value, degrees of freedom,
+ # test statistic, the method used, and the null hypothesis.
+
+ mat = Matrices.dense(3, 2, [1.0, 3.0, 5.0, 2.0, 4.0, 6.0]) # a contingency matrix
+
+ # conduct Pearson's independence test on the input contingency matrix
+ independenceTestResult = Statistics.chiSqTest(mat)
+ print(independenceTestResult) # summary of the test including the p-value, degrees of freedom
+
+ p1 = LabeledPoint(1.0, [1.0, 0.0, 3.0])
+ p2 = LabeledPoint(1.0, [1.0, 2.0, 0.0])
+ p3 = LabeledPoint(1.0, [-1.0, 0.0, -0.5])
+ obs = sc.parallelize([p1, p2, p3]) # LabeledPoint(feature, label) .
--- End diff --
Remove the final dot
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r56434078
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaStratifiedSamplingExample.java ---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaSparkContext;
+
+// $example on$
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import scala.Tuple2;
+
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.function.VoidFunction;
+// $example off$
+
+public class JavaStratifiedSamplingExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaStratifiedSamplingExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ List<Tuple2<Integer, Character>> list = new ArrayList<>();
+ list.add(new Tuple2(1, 'a'));
+ list.add(new Tuple2(1, 'b'));
+ list.add(new Tuple2(2, 'c'));
+ list.add(new Tuple2(2, 'd'));
+ list.add(new Tuple2(2, 'e'));
+ list.add(new Tuple2(3, 'f'));
+
+ // an RDD of any key value pairs JavaPairRDD<K, V>
+ JavaPairRDD<Integer, Character> data = jsc.parallelizePairs(list);
+
+ // specify the exact fraction desired from each key Map<K, Object>
+ Map<Integer, Object> fractions = new HashMap<>();
+
+ fractions.put(1, 0.1);
+ fractions.put(2, 0.6);
+ fractions.put(3, 0.3);
+
+ // Get an exact sample from each stratum
--- End diff --
The comment is wrong.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r56434020
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaStratifiedSamplingExample.java ---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaSparkContext;
+
+// $example on$
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import scala.Tuple2;
+
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.function.VoidFunction;
+// $example off$
+
+public class JavaStratifiedSamplingExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaStratifiedSamplingExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ List<Tuple2<Integer, Character>> list = new ArrayList<>();
+ list.add(new Tuple2(1, 'a'));
+ list.add(new Tuple2(1, 'b'));
+ list.add(new Tuple2(2, 'c'));
+ list.add(new Tuple2(2, 'd'));
+ list.add(new Tuple2(2, 'e'));
+ list.add(new Tuple2(3, 'f'));
+
+ // an RDD of any key value pairs JavaPairRDD<K, V>
--- End diff --
The comment is not necessary.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-186934904
**[Test build #51646 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51646/consoleFull)** for PR 11108 at commit [`f945222`](https://github.com/apache/spark/commit/f945222ad1cfd1f4756258a49b372aa7bd32d9fc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by keypointt <gi...@git.apache.org>.
Github user keypointt commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-199960473
Hi @mengxr , I'm using below command to compile this branch (after merge in latest master branch) by scala-2.10
`build/sbt -Pyarn -Phadoop-2.3 -Pkinesis-asl -Pspark-ganglia-lgpl -Phive -Phive-thriftserver -Pscala-2.10 compile test:compile`
and I'm always getting this error `sbt.ResolveException: unresolved dependency: org.scala-lang#jline;2.11.7: not found` where the jline version is set at https://github.com/apache/spark/blob/48978abfa4d8f2cf79a4b053cc8bc7254cc2d61b/pom.xml#L157
I'm assuming that master branch now is using scala-2.11 and when building 2.10 there could be some resolve issues?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-198048953
**[Test build #53460 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53460/consoleFull)** for PR 11108 at commit [`a4eb28d`](https://github.com/apache/spark/commit/a4eb28d07a99e559132160f8ae9ac993d47d8fa3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r56434617
--- Diff: examples/src/main/python/mllib/hypothesis_testing_kolmogorov_smirnov_test_example.py ---
@@ -0,0 +1,40 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import print_function
+
+from pyspark import SparkContext
+# $example on$
+from pyspark.mllib.stat import Statistics
+# $example off$
+
+if __name__ == "__main__":
+ sc = SparkContext(appName="HypothesisTestingKolmogorovSmirnovTestExample") # SparkContext
--- End diff --
remove comment
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by keypointt <gi...@git.apache.org>.
Github user keypointt commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55274411
--- Diff: docs/mllib-statistics.md ---
@@ -584,24 +299,24 @@ distribution `N(0, 1)`, and then map it to `N(1, 4)`.
Refer to the [`RandomRDDs` Java docs](api/java/org/apache/spark/mllib/random/RandomRDDs) for details on the API.
-{% highlight java %}
-import org.apache.spark.SparkContext;
-import org.apache.spark.api.JavaDoubleRDD;
-import static org.apache.spark.mllib.random.RandomRDDs.*;
-
-JavaSparkContext jsc = ...
-
-// Generate a random double RDD that contains 1 million i.i.d. values drawn from the
-// standard normal distribution `N(0, 1)`, evenly distributed in 10 partitions.
-JavaDoubleRDD u = normalJavaRDD(jsc, 1000000L, 10);
-// Apply a transform to get a random double RDD following `N(1, 4)`.
-JavaDoubleRDD v = u.map(
- new Function<Double, Double>() {
- public Double call(Double x) {
- return 1.0 + 2.0 * x;
- }
- });
-{% endhighlight %}
+-{% highlight java %}
+ -import org.apache.spark.SparkContext;
+ -import org.apache.spark.api.JavaDoubleRDD;
+ -import static org.apache.spark.mllib.random.RandomRDDs.*;
+ -
+ -JavaSparkContext jsc = ...
+ -
+ -// Generate a random double RDD that contains 1 million i.i.d. values drawn from the
+ -// standard normal distribution `N(0, 1)`, evenly distributed in 10 partitions.
+ -JavaDoubleRDD u = normalJavaRDD(jsc, 1000000L, 10);
+ -// Apply a transform to get a random double RDD following `N(1, 4)`.
+ -JavaDoubleRDD v = u.map(
+ - new Function<Double, Double>() {
+ - public Double call(Double x) {
+ - return 1.0 + 2.0 * x;
+ - }
+ - });
+ -{% endhighlight %}
--- End diff --
got it, fixing them now :p
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133886
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaHypothesisTestingExample.java ---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
+
+import org.apache.spark.SparkConf;
+// $example on$
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.mllib.linalg.Matrices;
+import org.apache.spark.mllib.linalg.Matrix;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.regression.LabeledPoint;
+import org.apache.spark.mllib.stat.Statistics;
+import org.apache.spark.mllib.stat.test.ChiSqTestResult;
+// $example off$
+
+public class JavaHypothesisTestingExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaHypothesisTestingExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ Vector vec = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25); // a vector composed of the frequencies of events
+
+ // compute the goodness of fit. If a second vector to test against is not supplied as a parameter,
+ // the test runs against a uniform distribution.
+ ChiSqTestResult goodnessOfFitTestResult = Statistics.chiSqTest(vec);
+ // summary of the test including the p-value, degrees of freedom, test statistic, the method used,
+ // and the null hypothesis.
+ System.out.println(goodnessOfFitTestResult);
--- End diff --
add a println
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133709
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaCorrelationsExample.java ---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
+
+import org.apache.spark.SparkConf;
+// $example on$
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaDoubleRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.mllib.linalg.Matrix;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.stat.Statistics;
+// $example off$
+
+public class JavaCorrelationsExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaCorrelationsExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ JavaDoubleRDD seriesX = jsc.parallelizeDoubles(
+ Arrays.asList(new Double[]{1.0, 2.0, 3.0, 3.0, 5.0})); // a series
+
+ // must have the same number of partitions and cardinality as seriesX
+ JavaDoubleRDD seriesY = jsc.parallelizeDoubles(
+ Arrays.asList(new Double[]{11.0, 22.0, 33.0, 33.0, 555.0}));
+
+ // compute the correlation using Pearson's method. Enter "spearman" for Spearman's method.
+ // If a method is not specified, Pearson's method will be used by default.
+ Double correlation = Statistics.corr(seriesX.srdd(), seriesY.srdd(), "pearson");
+ System.out.println("correlation is: " + correlation);
--- End diff --
`"correlation is: "` --> `"Correlation is: "`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-193013802
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52533/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-186935006
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55145410
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/HypothesisTestingKolmogorovSmirnovTestExample.scala ---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.stat.Statistics
+import org.apache.spark.rdd.RDD
+// $example off$
+
+object HypothesisTestingKolmogorovSmirnovTestExample {
+
+ def main(args: Array[String]) {
--- End diff --
def main(args: Array[String]): Unit = {
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-183106586
**[Test build #51145 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51145/consoleFull)** for PR 11108 at commit [`a4dd0fb`](https://github.com/apache/spark/commit/a4dd0fb5781dbdc162fd8652001b8de09b225297).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133924
--- Diff: examples/src/main/python/mllib/hypothesis_testing_example.py ---
@@ -0,0 +1,60 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import print_function
+
+from pyspark import SparkContext
+# $example on$
+from pyspark.mllib.linalg import Vectors, Matrices
+from pyspark.mllib.regression import LabeledPoint
+from pyspark.mllib.stat import Statistics
+# $example off$
+
+if __name__ == "__main__":
+ sc = SparkContext(appName="HypothesisTestingExample") # SparkContext
+
+ # $example on$
+ vec = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25) # a vector composed of the frequencies of events
+
+ # compute the goodness of fit. If a second vector to test against
+ # is not supplied as a parameter, the test runs against a uniform distribution.
+ goodnessOfFitTestResult = Statistics.chiSqTest(vec)
+ print(goodnessOfFitTestResult) # summary of the test including the p-value, degrees of freedom,
+ # test statistic, the method used, and the null hypothesis.
+
+ mat = Matrices.dense(3, 2, [1.0, 3.0, 5.0, 2.0, 4.0, 6.0]) # a contingency matrix
+
+ # conduct Pearson's independence test on the input contingency matrix
+ independenceTestResult = Statistics.chiSqTest(mat)
+ print(independenceTestResult) # summary of the test including the p-value, degrees of freedom
+
+ p1 = LabeledPoint(1.0, [1.0, 0.0, 3.0])
+ p2 = LabeledPoint(1.0, [1.0, 2.0, 0.0])
+ p3 = LabeledPoint(1.0, [-1.0, 0.0, -0.5])
+ obs = sc.parallelize([p1, p2, p3]) # LabeledPoint(feature, label) .
+
+ # The contingency table is constructed from an RDD of LabeledPoint and used to conduct
+ # the independence test. Returns an array containing the ChiSquaredTestResult for every feature
+ # against the label.
+ featureTestResults = Statistics.chiSqTest(obs)
+
+ for i, result in enumerate(featureTestResults):
+ print("Column: " + str(i + 1))
+ print(result)
--- End diff --
add a println
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133572
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/SummaryStatisticsExample.scala ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.linalg.Vectors
+import org.apache.spark.mllib.stat.{MultivariateStatisticalSummary, Statistics}
+// $example off$
+
+object SummaryStatisticsExample {
+
+ def main(args: Array[String]) {
+
+ val conf = new SparkConf().setAppName("SummaryStatisticsExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ val v1 = Vectors.dense(1.0, 10.0, 100.0)
+ val v2 = Vectors.dense(2.0, 20.0, 200.0)
+ val v3 = Vectors.dense(3.0, 30.0, 300.0)
+
+ val observations = sc.parallelize(Seq(v1, v2, v3))
+
+ // Compute column summary statistics.
+ val summary: MultivariateStatisticalSummary = Statistics.colStats(observations)
+ println(summary.mean) // a dense vector containing the mean value for each column
--- End diff --
2-indent for inline comment
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by keypointt <gi...@git.apache.org>.
Github user keypointt commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-199549785
@yhuai I've just tested on my local machine, and there is no compile error and the example output results correctly.
Also it seems all the parameters are right according to method documentation http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.PairRDDFunctions
And the build before merging to master was also successful...I'm not quite sure why this master build failed...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133680
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/CorrelationsExample.scala ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.mllib.stat.Statistics
+// $example off$
+import org.apache.spark.rdd.RDD
+
+object CorrelationsExample {
+
+ def main(args: Array[String]) {
+
+ val conf = new SparkConf().setAppName("CorrelationsExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ val seriesX: RDD[Double] = sc.parallelize(Array(1, 2, 3, 3, 5)) // a series
+ val seriesY: RDD[Double] = sc.parallelize(Array(11, 22, 33, 33, 555))
+ // must have the same number of partitions and cardinality as seriesX
+
+ // compute the correlation using Pearson's method. Enter "spearman" for Spearman's method. If a
+ // method is not specified, Pearson's method will be used by default.
+ val correlation: Double = Statistics.corr(seriesX, seriesY, "pearson")
+ println(correlation)
--- End diff --
`println(s"Correlation is: $correlation")`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r56433947
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaStratifiedSamplingExample.java ---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaSparkContext;
+
+// $example on$
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import scala.Tuple2;
+
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.function.VoidFunction;
+// $example off$
+
+public class JavaStratifiedSamplingExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaStratifiedSamplingExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ List<Tuple2<Integer, Character>> list = new ArrayList<>();
+ list.add(new Tuple2(1, 'a'));
+ list.add(new Tuple2(1, 'b'));
+ list.add(new Tuple2(2, 'c'));
+ list.add(new Tuple2(2, 'd'));
+ list.add(new Tuple2(2, 'e'));
+ list.add(new Tuple2(3, 'f'));
+
+ // an RDD of any key value pairs JavaPairRDD<K, V>
+ JavaPairRDD<Integer, Character> data = jsc.parallelizePairs(list);
+
+ // specify the exact fraction desired from each key Map<K, Object>
+ Map<Integer, Object> fractions = new HashMap<>();
--- End diff --
(minor) You can use `ImmutableMap.of` from Guava to simplify the map construction.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133702
--- Diff: examples/src/main/python/mllib/correlations_example.py ---
@@ -0,0 +1,49 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import print_function
+
+import numpy as np
+
+from pyspark import SparkContext
+# $example on$
+from pyspark.mllib.stat import Statistics
+# $example off$
+
+if __name__ == "__main__":
+ sc = SparkContext(appName="CorrelationsExample") # SparkContext
+
+ # $example on$
+ seriesX = sc.parallelize([1.0, 2.0, 3.0, 3.0, 5.0]) # a series
+ # seriesY must have the same number of partitions and cardinality as seriesX
+ seriesY = sc.parallelize([11.0, 22.0, 33.0, 33.0, 555.0])
+
+ # Compute the correlation using Pearson's method. Enter "spearman" for Spearman's method.
+ # If a method is not specified, Pearson's method will be used by default.
+ print(Statistics.corr(seriesX, seriesY, method="pearson"))
--- End diff --
`print("Correlation is: " + str(Statistics.corr(seriesX, seriesY, method="pearson")))`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133573
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/SummaryStatisticsExample.scala ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.linalg.Vectors
+import org.apache.spark.mllib.stat.{MultivariateStatisticalSummary, Statistics}
+// $example off$
+
+object SummaryStatisticsExample {
+
+ def main(args: Array[String]) {
+
+ val conf = new SparkConf().setAppName("SummaryStatisticsExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ val v1 = Vectors.dense(1.0, 10.0, 100.0)
+ val v2 = Vectors.dense(2.0, 20.0, 200.0)
+ val v3 = Vectors.dense(3.0, 30.0, 300.0)
+
+ val observations = sc.parallelize(Seq(v1, v2, v3))
+
+ // Compute column summary statistics.
+ val summary: MultivariateStatisticalSummary = Statistics.colStats(observations)
+ println(summary.mean) // a dense vector containing the mean value for each column
+ println(summary.variance) // column-wise variance
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55267546
--- Diff: docs/mllib-statistics.md ---
@@ -584,24 +299,24 @@ distribution `N(0, 1)`, and then map it to `N(1, 4)`.
Refer to the [`RandomRDDs` Java docs](api/java/org/apache/spark/mllib/random/RandomRDDs) for details on the API.
-{% highlight java %}
-import org.apache.spark.SparkContext;
-import org.apache.spark.api.JavaDoubleRDD;
-import static org.apache.spark.mllib.random.RandomRDDs.*;
-
-JavaSparkContext jsc = ...
-
-// Generate a random double RDD that contains 1 million i.i.d. values drawn from the
-// standard normal distribution `N(0, 1)`, evenly distributed in 10 partitions.
-JavaDoubleRDD u = normalJavaRDD(jsc, 1000000L, 10);
-// Apply a transform to get a random double RDD following `N(1, 4)`.
-JavaDoubleRDD v = u.map(
- new Function<Double, Double>() {
- public Double call(Double x) {
- return 1.0 + 2.0 * x;
- }
- });
-{% endhighlight %}
+-{% highlight java %}
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-193013759
**[Test build #52533 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52533/consoleFull)** for PR 11108 at commit [`3329394`](https://github.com/apache/spark/commit/33293947bde90fd29014587cd42533df121bd783).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-197608555
@keypointt @yinxusen I made one pass and left some comments inline. One issue is whether we want to print empty lines. It doesn't seem necessary to me for example code appearing in the user guide. There are also some redundant comments, which we should remove. But overall, this look great!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r56434127
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaStratifiedSamplingExample.java ---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaSparkContext;
+
+// $example on$
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import scala.Tuple2;
+
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.function.VoidFunction;
+// $example off$
+
+public class JavaStratifiedSamplingExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaStratifiedSamplingExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ List<Tuple2<Integer, Character>> list = new ArrayList<>();
+ list.add(new Tuple2(1, 'a'));
+ list.add(new Tuple2(1, 'b'));
+ list.add(new Tuple2(2, 'c'));
+ list.add(new Tuple2(2, 'd'));
+ list.add(new Tuple2(2, 'e'));
+ list.add(new Tuple2(3, 'f'));
+
+ // an RDD of any key value pairs JavaPairRDD<K, V>
+ JavaPairRDD<Integer, Character> data = jsc.parallelizePairs(list);
+
+ // specify the exact fraction desired from each key Map<K, Object>
+ Map<Integer, Object> fractions = new HashMap<>();
+
+ fractions.put(1, 0.1);
+ fractions.put(2, 0.6);
+ fractions.put(3, 0.3);
+
+ // Get an exact sample from each stratum
+ JavaPairRDD<Integer, Character> approxSample =
+ data.sampleByKey(false, fractions); // JavaPairRDD<K, V>
+ JavaPairRDD<Integer, Character> exactSample =
+ data.sampleByKeyExact(false, fractions); // JavaPairRDD<K, V>
--- End diff --
remove comment
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55266976
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaStratifiedSamplingExample.java ---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaSparkContext;
+
+// $example on$
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import scala.Tuple2;
+
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.function.VoidFunction;
+// $example off$
+
+public class JavaStratifiedSamplingExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaStratifiedSamplingExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ List<Tuple2<Integer, Character>> list = new ArrayList<>();
+ list.add(new Tuple2(1, 'a'));
+ list.add(new Tuple2(1, 'b'));
+ list.add(new Tuple2(2, 'c'));
+ list.add(new Tuple2(2, 'd'));
+ list.add(new Tuple2(2, 'e'));
+ list.add(new Tuple2(3, 'f'));
+
+ // an RDD of any key value pairs JavaPairRDD<K, V>
+ JavaPairRDD<Integer, Character> data = jsc.parallelizePairs(list);
+
+ // specify the exact fraction desired from each key Map<K, Object>
+ Map<Integer, Object> fractions = new HashMap<>();
+
+ fractions.put(1, 0.1);
+ fractions.put(2, 0.6);
+ fractions.put(3, 0.3);
+
+ // Get an exact sample from each stratum
+ JavaPairRDD<Integer, Character> approxSample =
+ data.sampleByKey(false, fractions); // JavaPairRDD<K, V>
--- End diff --
2-indent
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133669
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaCorrelationsExample.java ---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
+
+import org.apache.spark.SparkConf;
+// $example on$
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaDoubleRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.mllib.linalg.Matrix;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.stat.Statistics;
+// $example off$
+
+public class JavaCorrelationsExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaCorrelationsExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ JavaDoubleRDD seriesX = jsc.parallelizeDoubles(
+ Arrays.asList(new Double[]{1.0, 2.0, 3.0, 3.0, 5.0})); // a series
--- End diff --
Simplify it to `Arrays.asList(1.0, 2.0, 3.0, 3.0, 5.0)); // a series` with 2-indent for inline comment
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133920
--- Diff: examples/src/main/python/mllib/hypothesis_testing_example.py ---
@@ -0,0 +1,60 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import print_function
+
+from pyspark import SparkContext
+# $example on$
+from pyspark.mllib.linalg import Vectors, Matrices
+from pyspark.mllib.regression import LabeledPoint
+from pyspark.mllib.stat import Statistics
+# $example off$
+
+if __name__ == "__main__":
+ sc = SparkContext(appName="HypothesisTestingExample") # SparkContext
+
+ # $example on$
+ vec = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25) # a vector composed of the frequencies of events
+
+ # compute the goodness of fit. If a second vector to test against
+ # is not supplied as a parameter, the test runs against a uniform distribution.
+ goodnessOfFitTestResult = Statistics.chiSqTest(vec)
+ print(goodnessOfFitTestResult) # summary of the test including the p-value, degrees of freedom,
--- End diff --
add a println
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r56434528
--- Diff: examples/src/main/python/mllib/hypothesis_testing_example.py ---
@@ -0,0 +1,63 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import print_function
+
+from pyspark import SparkContext
+# $example on$
+from pyspark.mllib.linalg import Matrices, Vectors
+from pyspark.mllib.regression import LabeledPoint
+from pyspark.mllib.stat import Statistics
+# $example off$
+
+if __name__ == "__main__":
+ sc = SparkContext(appName="HypothesisTestingExample") # SparkContext
+
+ # $example on$
+ vec = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25) # a vector composed of the frequencies of events
+
+ # compute the goodness of fit. If a second vector to test against
+ # is not supplied as a parameter, the test runs against a uniform distribution.
+ goodnessOfFitTestResult = Statistics.chiSqTest(vec)
+ print(goodnessOfFitTestResult) # summary of the test including the p-value, degrees of freedom,
+ # test statistic, the method used, and the null hypothesis.
+ print()
--- End diff --
It is weird to have a `print()` line right after a comment. We can do `print("%s\n" % goodnessOfFitTestResult)`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133806
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/HypothesisTestingExample.scala ---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.mllib.regression.LabeledPoint
+import org.apache.spark.mllib.stat.Statistics
+import org.apache.spark.mllib.stat.test.ChiSqTestResult
+import org.apache.spark.rdd.RDD
+// $example off$
+
+object HypothesisTestingExample {
+
+ def main(args: Array[String]) {
+
+ val conf = new SparkConf().setAppName("HypothesisTestingExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ // a vector composed of the frequencies of events
+ val vec: Vector = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25)
+
+ // compute the goodness of fit. If a second vector to test against is not supplied
+ // as a parameter, the test runs against a uniform distribution.
+ val goodnessOfFitTestResult = Statistics.chiSqTest(vec)
+ println(goodnessOfFitTestResult) // summary of the test including the p-value,
+ // degrees of freedom, test statistic, the method used, and the null hypothesis.
+
+ // a contingency matrix. Create a dense matrix ((1.0, 2.0), (3.0, 4.0), (5.0, 6.0))
+ val mat: Matrix = Matrices.dense(3, 2, Array(1.0, 3.0, 5.0, 2.0, 4.0, 6.0))
+
+ // conduct Pearson's independence test on the input contingency matrix
+ val independenceTestResult = Statistics.chiSqTest(mat)
+ println(independenceTestResult) // summary of the test including the p-value, degrees of freedom
+
+ val p1 = LabeledPoint(1.0, Vectors.dense(1.0, 0.0, 3.0))
+ val p2 = LabeledPoint(1.0, Vectors.dense(1.0, 2.0, 0.0))
+ val p3 = LabeledPoint(-1.0, Vectors.dense(-1.0, 0.0, -0.5))
+ val obs: RDD[LabeledPoint] = sc.parallelize(Seq(p1, p2, p3)) // (feature, label) pairs.
+
+ // The contingency table is constructed from the raw (feature, label) pairs and used to conduct
+ // the independence test. Returns an array containing the ChiSquaredTestResult for every feature
+ // against the label.
+ val featureTestResults: Array[ChiSqTestResult] = Statistics.chiSqTest(obs)
+ var i = 1
+ featureTestResults.foreach { result =>
+ println(s"Column $i:\n$result")
+ i += 1
+ } // summary of the test
--- End diff --
2-indent
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r56433851
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaKernelDensityEstimationExample.java ---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaSparkContext;
+// $example on$
+import java.util.Arrays;
+
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.mllib.stat.KernelDensity;
+// $example off$
+
+public class JavaKernelDensityEstimationExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaKernelDensityEstimationExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ // an RDD of sample data
+ JavaRDD<Double> data = jsc.parallelize(
+ Arrays.asList(1.0, 1.0, 1.0, 2.0, 3.0, 4.0, 5.0, 5.0, 6.0, 7.0, 8.0, 9.0, 9.0));
+
+ // Construct the density estimator with the sample data
+ // and a standard deviation for the Gaussian kernels
+ KernelDensity kd = new KernelDensity().setSample(data).setBandwidth(3.0);
+
+ // Find density estimates for the given values
+ double[] densities = kd.estimate(new double[]{-1.0, 2.0, 5.0});
+ // $example off$
+
+ System.out.println(Arrays.toString(densities));
--- End diff --
Move `println` line inside `$example`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133588
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaSummaryStatisticsExample.java ---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
+
+// $example on$
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.stat.MultivariateStatisticalSummary;
+import org.apache.spark.mllib.stat.Statistics;
+// $example off$
+import org.apache.spark.SparkConf;
+
+public class JavaSummaryStatisticsExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaSummaryStatisticsExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ Vector v1 = Vectors.dense(1.0, 10.0, 100.0);
+ Vector v2 = Vectors.dense(2.0, 20.0, 200.0);
+ Vector v3 = Vectors.dense(3.0, 30.0, 300.0);
+
+ JavaRDD<Vector> mat = jsc.parallelize(Arrays.asList(v1, v2, v3)); // an RDD of Vectors
+
+ // Compute column summary statistics.
+ MultivariateStatisticalSummary summary = Statistics.colStats(mat.rdd());
+ System.out.println(summary.mean()); // a dense vector containing the mean value for each column
+ System.out.println(summary.variance()); // column-wise variance
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r56434630
--- Diff: examples/src/main/python/mllib/hypothesis_testing_example.py ---
@@ -0,0 +1,63 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import print_function
+
+from pyspark import SparkContext
+# $example on$
+from pyspark.mllib.linalg import Matrices, Vectors
+from pyspark.mllib.regression import LabeledPoint
+from pyspark.mllib.stat import Statistics
+# $example off$
+
+if __name__ == "__main__":
+ sc = SparkContext(appName="HypothesisTestingExample") # SparkContext
--- End diff --
remove comment
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133737
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/StratifiedSamplingExample.scala ---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+// $example on$
+import org.apache.spark.{SparkConf, SparkContext}
+// $example off$
+
+object StratifiedSamplingExample {
+
+ def main(args: Array[String]) {
+
+ val conf = new SparkConf().setAppName("StratifiedSamplingExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ // an RDD[(K, V)] of any key value pairs
+ val data = sc.parallelize(
+ Seq((1, 'a'), (1, 'b'), (2, 'c'), (2, 'd'), (2, 'e'), (3, 'f')))
+
+ // specify the exact fraction desired from each key
+ val fractions = Map(1 -> 0.1, 2 -> 0.6, 3 -> 0.3)
+
+ // Get an exact sample from each stratum
+ val approxSample = data.sampleByKey(withReplacement = false, fractions)
+ val exactSample = data.sampleByKeyExact(withReplacement = false, fractions)
+
+ // $example off$
+
+ approxSample.foreach(println)
--- End diff --
add a println() here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133583
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaSummaryStatisticsExample.java ---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
+
+// $example on$
--- End diff --
```java
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
// $example on$
import java.util.Arrays;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.mllib.linalg.Vector;
import org.apache.spark.mllib.linalg.Vectors;
import org.apache.spark.mllib.stat.MultivariateStatisticalSummary;
import org.apache.spark.mllib.stat.Statistics;
// $example off$
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133829
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/HypothesisTestingExample.scala ---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.mllib.regression.LabeledPoint
+import org.apache.spark.mllib.stat.Statistics
+import org.apache.spark.mllib.stat.test.ChiSqTestResult
+import org.apache.spark.rdd.RDD
+// $example off$
+
+object HypothesisTestingExample {
+
+ def main(args: Array[String]) {
+
+ val conf = new SparkConf().setAppName("HypothesisTestingExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ // a vector composed of the frequencies of events
+ val vec: Vector = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25)
+
+ // compute the goodness of fit. If a second vector to test against is not supplied
+ // as a parameter, the test runs against a uniform distribution.
+ val goodnessOfFitTestResult = Statistics.chiSqTest(vec)
+ println(goodnessOfFitTestResult) // summary of the test including the p-value,
+ // degrees of freedom, test statistic, the method used, and the null hypothesis.
+
+ // a contingency matrix. Create a dense matrix ((1.0, 2.0), (3.0, 4.0), (5.0, 6.0))
+ val mat: Matrix = Matrices.dense(3, 2, Array(1.0, 3.0, 5.0, 2.0, 4.0, 6.0))
+
+ // conduct Pearson's independence test on the input contingency matrix
+ val independenceTestResult = Statistics.chiSqTest(mat)
+ println(independenceTestResult) // summary of the test including the p-value, degrees of freedom
--- End diff --
add a println here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-193011222
**[Test build #52533 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52533/consoleFull)** for PR 11108 at commit [`3329394`](https://github.com/apache/spark/commit/33293947bde90fd29014587cd42533df121bd783).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133812
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/HypothesisTestingExample.scala ---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.mllib.regression.LabeledPoint
+import org.apache.spark.mllib.stat.Statistics
+import org.apache.spark.mllib.stat.test.ChiSqTestResult
+import org.apache.spark.rdd.RDD
+// $example off$
+
+object HypothesisTestingExample {
+
+ def main(args: Array[String]) {
+
+ val conf = new SparkConf().setAppName("HypothesisTestingExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ // a vector composed of the frequencies of events
+ val vec: Vector = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25)
+
+ // compute the goodness of fit. If a second vector to test against is not supplied
+ // as a parameter, the test runs against a uniform distribution.
+ val goodnessOfFitTestResult = Statistics.chiSqTest(vec)
+ println(goodnessOfFitTestResult) // summary of the test including the p-value,
+ // degrees of freedom, test statistic, the method used, and the null hypothesis.
+
+ // a contingency matrix. Create a dense matrix ((1.0, 2.0), (3.0, 4.0), (5.0, 6.0))
+ val mat: Matrix = Matrices.dense(3, 2, Array(1.0, 3.0, 5.0, 2.0, 4.0, 6.0))
+
+ // conduct Pearson's independence test on the input contingency matrix
+ val independenceTestResult = Statistics.chiSqTest(mat)
+ println(independenceTestResult) // summary of the test including the p-value, degrees of freedom
+
+ val p1 = LabeledPoint(1.0, Vectors.dense(1.0, 0.0, 3.0))
+ val p2 = LabeledPoint(1.0, Vectors.dense(1.0, 2.0, 0.0))
+ val p3 = LabeledPoint(-1.0, Vectors.dense(-1.0, 0.0, -0.5))
+ val obs: RDD[LabeledPoint] = sc.parallelize(Seq(p1, p2, p3)) // (feature, label) pairs.
+
+ // The contingency table is constructed from the raw (feature, label) pairs and used to conduct
+ // the independence test. Returns an array containing the ChiSquaredTestResult for every feature
+ // against the label.
+ val featureTestResults: Array[ChiSqTestResult] = Statistics.chiSqTest(obs)
+ var i = 1
--- End diff --
Use `featureTestResults.zipWithIndex` instead of `var`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133827
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/HypothesisTestingExample.scala ---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.mllib.regression.LabeledPoint
+import org.apache.spark.mllib.stat.Statistics
+import org.apache.spark.mllib.stat.test.ChiSqTestResult
+import org.apache.spark.rdd.RDD
+// $example off$
+
+object HypothesisTestingExample {
+
+ def main(args: Array[String]) {
+
+ val conf = new SparkConf().setAppName("HypothesisTestingExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ // a vector composed of the frequencies of events
+ val vec: Vector = Vectors.dense(0.1, 0.15, 0.2, 0.3, 0.25)
+
+ // compute the goodness of fit. If a second vector to test against is not supplied
+ // as a parameter, the test runs against a uniform distribution.
+ val goodnessOfFitTestResult = Statistics.chiSqTest(vec)
+ println(goodnessOfFitTestResult) // summary of the test including the p-value,
--- End diff --
add a println here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11108#issuecomment-198049129
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133576
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/SummaryStatisticsExample.scala ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.mllib
+
+import org.apache.spark.{SparkConf, SparkContext}
+// $example on$
+import org.apache.spark.mllib.linalg.Vectors
+import org.apache.spark.mllib.stat.{MultivariateStatisticalSummary, Statistics}
+// $example off$
+
+object SummaryStatisticsExample {
+
+ def main(args: Array[String]) {
+
+ val conf = new SparkConf().setAppName("SummaryStatisticsExample")
+ val sc = new SparkContext(conf)
+
+ // $example on$
+ val v1 = Vectors.dense(1.0, 10.0, 100.0)
+ val v2 = Vectors.dense(2.0, 20.0, 200.0)
+ val v3 = Vectors.dense(3.0, 30.0, 300.0)
+
+ val observations = sc.parallelize(Seq(v1, v2, v3))
+
+ // Compute column summary statistics.
+ val summary: MultivariateStatisticalSummary = Statistics.colStats(observations)
+ println(summary.mean) // a dense vector containing the mean value for each column
+ println(summary.variance) // column-wise variance
+ println(summary.numNonzeros) // number of nonzeros in each column
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13019][Docs] Replace example code in ml...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11108#discussion_r55133587
--- Diff: examples/src/main/java/org/apache/spark/examples/mllib/JavaSummaryStatisticsExample.java ---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib;
+
+import java.util.Arrays;
+
+// $example on$
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.mllib.linalg.Vector;
+import org.apache.spark.mllib.linalg.Vectors;
+import org.apache.spark.mllib.stat.MultivariateStatisticalSummary;
+import org.apache.spark.mllib.stat.Statistics;
+// $example off$
+import org.apache.spark.SparkConf;
+
+public class JavaSummaryStatisticsExample {
+ public static void main(String[] args) {
+
+ SparkConf conf = new SparkConf().setAppName("JavaSummaryStatisticsExample");
+ JavaSparkContext jsc = new JavaSparkContext(conf);
+
+ // $example on$
+ Vector v1 = Vectors.dense(1.0, 10.0, 100.0);
+ Vector v2 = Vectors.dense(2.0, 20.0, 200.0);
+ Vector v3 = Vectors.dense(3.0, 30.0, 300.0);
+
+ JavaRDD<Vector> mat = jsc.parallelize(Arrays.asList(v1, v2, v3)); // an RDD of Vectors
+
+ // Compute column summary statistics.
+ MultivariateStatisticalSummary summary = Statistics.colStats(mat.rdd());
+ System.out.println(summary.mean()); // a dense vector containing the mean value for each column
--- End diff --
2-indent inline comment
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org