You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by shahidki31 <gi...@git.apache.org> on 2018/05/05 21:31:54 UTC
[GitHub] spark pull request #21248: Example code for Power Iteration Clustering
GitHub user shahidki31 opened a pull request:
https://github.com/apache/spark/pull/21248
Example code for Power Iteration Clustering
## What changes were proposed in this pull request?
Added example code for Power Iteration Clustering in Spark ML examples
## How was this patch tested?
Locally ran
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shahidki31/spark sparkCommit
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21248.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21248
----
commit 16f0e22c844a281ab4b0ecc2e70f858745db398c
Author: Shahid <sh...@...>
Date: 2018-05-05T20:58:33Z
Example code for Power Iteration Clustering
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21248: [SPARK-24191][ML]Scala Example code for Power Iteration ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21248
**[Test build #4196 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4196/testReport)** for PR 21248 at commit [`d005493`](https://github.com/apache/spark/commit/d0054939562ff822c2a0581af2a616f61a82e131).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21248: [SPARK-24191][ML]Scala Example code for Power Ite...
Posted by hhbyyh <gi...@git.apache.org>.
Github user hhbyyh commented on a diff in the pull request:
https://github.com/apache/spark/pull/21248#discussion_r189466147
--- Diff: examples/src/main/scala/org/apache/spark/examples/ml/PowerIterationClusteringExample.scala ---
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.ml
+
+import org.apache.log4j.{Level, Logger}
+
+// $example on$
+import org.apache.spark.ml.clustering.PowerIterationClustering
+// $example off$
+import org.apache.spark.sql.{DataFrame, Row, SparkSession}
+
+
+ /**
+ * An example demonstrating power iteration clustering.
+ * Run with
+ * {{{
+ * bin/run-example ml.PowerIterationClusteringExample
+ * }}}
+ */
+
+object PowerIterationClusteringExample {
+
+ def main(args: Array[String]): Unit = {
+ val spark = SparkSession
+ .builder
+ .appName(s"${this.getClass.getSimpleName}")
+ .getOrCreate()
+
+ Logger.getRootLogger.setLevel(Level.WARN)
+
+ // $example on$
+
+ // Generates data.
+ val radius1 = 1.0
+ val numPoints1 = 5
+ val radius2 = 4.0
+ val numPoints2 = 20
+
+ val dataset = generatePICData(spark, radius1, radius2, numPoints1, numPoints2)
+
+ // Trains a PIC model.
+ val model = new PowerIterationClustering().
+ setK(2).
+ setInitMode("degree").
+ setMaxIter(20)
+
+ val prediction = model.transform(dataset).select("id", "prediction")
+
+ // Shows the result.
+ // println("Cluster Assignment: ")
+ val result = prediction.collect().map {
+ row => (row(1), row(0))
+ }.groupBy(_._1).mapValues(_.map(_._2))
+
+ result.foreach {
+ case (cluster, points) => println(s"$cluster -> [${points.mkString(",")}]")
+ }
--- End diff --
This can be achieved by DataFrame API, groupBy.. collect_set
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21248: [SPARK-24191][ML]Scala Example code for Power Ite...
Posted by shahidki31 <gi...@git.apache.org>.
Github user shahidki31 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21248#discussion_r189704948
--- Diff: examples/src/main/scala/org/apache/spark/examples/ml/PowerIterationClusteringExample.scala ---
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.ml
+
+import org.apache.log4j.{Level, Logger}
+
+// $example on$
+import org.apache.spark.ml.clustering.PowerIterationClustering
+// $example off$
+import org.apache.spark.sql.{DataFrame, Row, SparkSession}
+
+
+ /**
+ * An example demonstrating power iteration clustering.
+ * Run with
+ * {{{
+ * bin/run-example ml.PowerIterationClusteringExample
+ * }}}
+ */
+
+object PowerIterationClusteringExample {
+
+ def main(args: Array[String]): Unit = {
+ val spark = SparkSession
+ .builder
+ .appName(s"${this.getClass.getSimpleName}")
+ .getOrCreate()
+
+ Logger.getRootLogger.setLevel(Level.WARN)
+
+ // $example on$
+
+ // Generates data.
+ val radius1 = 1.0
+ val numPoints1 = 5
+ val radius2 = 4.0
+ val numPoints2 = 20
+
+ val dataset = generatePICData(spark, radius1, radius2, numPoints1, numPoints2)
+
+ // Trains a PIC model.
+ val model = new PowerIterationClustering().
+ setK(2).
+ setInitMode("degree").
+ setMaxIter(20)
+
+ val prediction = model.transform(dataset).select("id", "prediction")
+
+ // Shows the result.
+ // println("Cluster Assignment: ")
+ val result = prediction.collect().map {
+ row => (row(1), row(0))
+ }.groupBy(_._1).mapValues(_.map(_._2))
+
+ result.foreach {
+ case (cluster, points) => println(s"$cluster -> [${points.mkString(",")}]")
+ }
--- End diff --
prediction.show() displays the cluster assignment properly. So this may not be required. current output based on the new dataset will be,
| id|prediction|
+---+----------+
| 4| 1|
| 0| 0|
| 1| 0|
| 3| 1|
| 2| 0|
+---+----------+
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21248: [SPARK-24191][ML]Scala Example code for Power Iteration ...
Posted by shahidki31 <gi...@git.apache.org>.
Github user shahidki31 commented on the issue:
https://github.com/apache/spark/pull/21248
Thanks @srowen
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21248: [SPARK-24191]Example code for Power Iteration Clustering
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21248
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21248: [SPARK-24191][ML]Scala Example code for Power Iteration ...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/21248
OK, I think this could have been combined with https://github.com/apache/spark/pull/21283, but whatever.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21248: [SPARK-24191][ML]Example code for Power Iteration Cluste...
Posted by shahidki31 <gi...@git.apache.org>.
Github user shahidki31 commented on the issue:
https://github.com/apache/spark/pull/21248
cc @mengxr @WeichenXu123 @felixcheung. Can you please verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21248: [SPARK-24191][ML]Scala Example code for Power Ite...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21248
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21248: [SPARK-24191][ML]Scala Example code for Power Ite...
Posted by hhbyyh <gi...@git.apache.org>.
Github user hhbyyh commented on a diff in the pull request:
https://github.com/apache/spark/pull/21248#discussion_r189466112
--- Diff: examples/src/main/scala/org/apache/spark/examples/ml/PowerIterationClusteringExample.scala ---
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.ml
+
+import org.apache.log4j.{Level, Logger}
+
+// $example on$
+import org.apache.spark.ml.clustering.PowerIterationClustering
+// $example off$
+import org.apache.spark.sql.{DataFrame, Row, SparkSession}
+
+
+ /**
+ * An example demonstrating power iteration clustering.
+ * Run with
+ * {{{
+ * bin/run-example ml.PowerIterationClusteringExample
+ * }}}
+ */
+
+object PowerIterationClusteringExample {
+
+ def main(args: Array[String]): Unit = {
+ val spark = SparkSession
+ .builder
+ .appName(s"${this.getClass.getSimpleName}")
+ .getOrCreate()
+
+ Logger.getRootLogger.setLevel(Level.WARN)
+
+ // $example on$
+
+ // Generates data.
+ val radius1 = 1.0
+ val numPoints1 = 5
+ val radius2 = 4.0
+ val numPoints2 = 20
+
+ val dataset = generatePICData(spark, radius1, radius2, numPoints1, numPoints2)
+
+ // Trains a PIC model.
+ val model = new PowerIterationClustering().
+ setK(2).
+ setInitMode("degree").
+ setMaxIter(20)
+
+ val prediction = model.transform(dataset).select("id", "prediction")
+
+ // Shows the result.
+ // println("Cluster Assignment: ")
--- End diff --
Maybe add `prediction.show()` first to demo the result of the `transform`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21248: [SPARK-24191][ML]Scala Example code for Power Iteration ...
Posted by shahidki31 <gi...@git.apache.org>.
Github user shahidki31 commented on the issue:
https://github.com/apache/spark/pull/21248
Hi @hhbyyh @srowen ,
PowerIterationClustering API has some modifications based on the discussion in the JIRA, https://issues.apache.org/jira/browse/SPARK-15784.
The examples also have modified based on the latest PR of Power Iteration Clustering, https://github.com/apache/spark/pull/21493
Can you please review the modified changes?
Thanks
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21248: [SPARK-24191][ML]Scala Example code for Power Ite...
Posted by hhbyyh <gi...@git.apache.org>.
Github user hhbyyh commented on a diff in the pull request:
https://github.com/apache/spark/pull/21248#discussion_r189465979
--- Diff: examples/src/main/scala/org/apache/spark/examples/ml/PowerIterationClusteringExample.scala ---
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.ml
+
+import org.apache.log4j.{Level, Logger}
+
+// $example on$
+import org.apache.spark.ml.clustering.PowerIterationClustering
+// $example off$
+import org.apache.spark.sql.{DataFrame, Row, SparkSession}
+
+
+ /**
+ * An example demonstrating power iteration clustering.
+ * Run with
+ * {{{
+ * bin/run-example ml.PowerIterationClusteringExample
+ * }}}
+ */
+
+object PowerIterationClusteringExample {
+
+ def main(args: Array[String]): Unit = {
+ val spark = SparkSession
+ .builder
+ .appName(s"${this.getClass.getSimpleName}")
+ .getOrCreate()
+
+ Logger.getRootLogger.setLevel(Level.WARN)
--- End diff --
This may not be necessary.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21248: [SPARK-24191][ML]Scala Example code for Power Ite...
Posted by shahidki31 <gi...@git.apache.org>.
Github user shahidki31 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21248#discussion_r189704094
--- Diff: examples/src/main/scala/org/apache/spark/examples/ml/PowerIterationClusteringExample.scala ---
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.ml
+
+import org.apache.log4j.{Level, Logger}
+
+// $example on$
+import org.apache.spark.ml.clustering.PowerIterationClustering
+// $example off$
+import org.apache.spark.sql.{DataFrame, Row, SparkSession}
+
+
+ /**
+ * An example demonstrating power iteration clustering.
+ * Run with
+ * {{{
+ * bin/run-example ml.PowerIterationClusteringExample
+ * }}}
+ */
+
+object PowerIterationClusteringExample {
+
+ def main(args: Array[String]): Unit = {
+ val spark = SparkSession
+ .builder
+ .appName(s"${this.getClass.getSimpleName}")
+ .getOrCreate()
+
+ Logger.getRootLogger.setLevel(Level.WARN)
--- End diff --
Yes. I have removed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21248: [SPARK-24191][ML]Scala Example code for Power Ite...
Posted by shahidki31 <gi...@git.apache.org>.
Github user shahidki31 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21248#discussion_r189704384
--- Diff: examples/src/main/scala/org/apache/spark/examples/ml/PowerIterationClusteringExample.scala ---
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.ml
+
+import org.apache.log4j.{Level, Logger}
+
+// $example on$
+import org.apache.spark.ml.clustering.PowerIterationClustering
+// $example off$
+import org.apache.spark.sql.{DataFrame, Row, SparkSession}
+
+
+ /**
+ * An example demonstrating power iteration clustering.
+ * Run with
+ * {{{
+ * bin/run-example ml.PowerIterationClusteringExample
+ * }}}
+ */
+
+object PowerIterationClusteringExample {
+
+ def main(args: Array[String]): Unit = {
+ val spark = SparkSession
+ .builder
+ .appName(s"${this.getClass.getSimpleName}")
+ .getOrCreate()
+
+ Logger.getRootLogger.setLevel(Level.WARN)
+
+ // $example on$
+
+ // Generates data.
+ val radius1 = 1.0
+ val numPoints1 = 5
+ val radius2 = 4.0
+ val numPoints2 = 20
+
+ val dataset = generatePICData(spark, radius1, radius2, numPoints1, numPoints2)
+
+ // Trains a PIC model.
+ val model = new PowerIterationClustering().
+ setK(2).
+ setInitMode("degree").
+ setMaxIter(20)
+
+ val prediction = model.transform(dataset).select("id", "prediction")
+
+ // Shows the result.
+ // println("Cluster Assignment: ")
--- End diff --
Yes. prediction.show() would be better in the new dataset.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21248: [SPARK-24191][ML]Scala Example code for Power Iteration ...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/21248
Merged to master
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21248: [SPARK-24191][ML]Scala Example code for Power Iteration ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21248
**[Test build #4196 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4196/testReport)** for PR 21248 at commit [`d005493`](https://github.com/apache/spark/commit/d0054939562ff822c2a0581af2a616f61a82e131).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21248: [SPARK-24191]Example code for Power Iteration Clustering
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21248
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21248: [SPARK-24191][ML]Scala Example code for Power Iteration ...
Posted by shahidki31 <gi...@git.apache.org>.
Github user shahidki31 commented on the issue:
https://github.com/apache/spark/pull/21248
Thank you @hhbyyh for the review. I have created a new dataset for the example code, instead of function generated dataset. I have addressed your review comments.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org