You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by shahidki31 <gi...@git.apache.org> on 2018/05/05 21:31:54 UTC

[GitHub] spark pull request #21248: Example code for Power Iteration Clustering

GitHub user shahidki31 opened a pull request:

    https://github.com/apache/spark/pull/21248

    Example code for Power Iteration Clustering

    ## What changes were proposed in this pull request?
    
    Added example code for Power Iteration Clustering in Spark ML examples
    
    ## How was this patch tested?
    Locally ran
    
    Please review http://spark.apache.org/contributing.html before opening a pull request.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/shahidki31/spark sparkCommit

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/21248.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21248
    
----
commit 16f0e22c844a281ab4b0ecc2e70f858745db398c
Author: Shahid <sh...@...>
Date:   2018-05-05T20:58:33Z

    Example code for Power Iteration Clustering

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21248: [SPARK-24191][ML]Scala Example code for Power Iteration ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21248
  
    **[Test build #4196 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4196/testReport)** for PR 21248 at commit [`d005493`](https://github.com/apache/spark/commit/d0054939562ff822c2a0581af2a616f61a82e131).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21248: [SPARK-24191][ML]Scala Example code for Power Ite...

Posted by hhbyyh <gi...@git.apache.org>.
Github user hhbyyh commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21248#discussion_r189466147
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/ml/PowerIterationClusteringExample.scala ---
    @@ -0,0 +1,114 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +// scalastyle:off println
    +package org.apache.spark.examples.ml
    +
    +import org.apache.log4j.{Level, Logger}
    +
    +// $example on$
    +import org.apache.spark.ml.clustering.PowerIterationClustering
    +// $example off$
    +import org.apache.spark.sql.{DataFrame, Row, SparkSession}
    +
    +
    + /**
    +  * An example demonstrating power iteration clustering.
    +  * Run with
    +  * {{{
    +  * bin/run-example ml.PowerIterationClusteringExample
    +  * }}}
    +  */
    +
    +object PowerIterationClusteringExample {
    +
    +  def main(args: Array[String]): Unit = {
    +    val spark = SparkSession
    +      .builder
    +      .appName(s"${this.getClass.getSimpleName}")
    +      .getOrCreate()
    +
    +    Logger.getRootLogger.setLevel(Level.WARN)
    +
    +    // $example on$
    +
    +    // Generates data.
    +    val radius1 = 1.0
    +    val numPoints1 = 5
    +    val radius2 = 4.0
    +    val numPoints2 = 20
    +
    +    val dataset = generatePICData(spark, radius1, radius2, numPoints1, numPoints2)
    +
    +    // Trains a PIC model.
    +    val model = new PowerIterationClustering().
    +      setK(2).
    +      setInitMode("degree").
    +      setMaxIter(20)
    +
    +    val prediction = model.transform(dataset).select("id", "prediction")
    +
    +    //  Shows the result.
    +    //  println("Cluster Assignment: ")
    +    val result = prediction.collect().map {
    +      row => (row(1), row(0))
    +    }.groupBy(_._1).mapValues(_.map(_._2))
    +
    +    result.foreach {
    +      case (cluster, points) => println(s"$cluster -> [${points.mkString(",")}]")
    +    }
    --- End diff --
    
    This can be achieved by DataFrame API, groupBy.. collect_set


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21248: [SPARK-24191][ML]Scala Example code for Power Ite...

Posted by shahidki31 <gi...@git.apache.org>.
Github user shahidki31 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21248#discussion_r189704948
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/ml/PowerIterationClusteringExample.scala ---
    @@ -0,0 +1,114 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +// scalastyle:off println
    +package org.apache.spark.examples.ml
    +
    +import org.apache.log4j.{Level, Logger}
    +
    +// $example on$
    +import org.apache.spark.ml.clustering.PowerIterationClustering
    +// $example off$
    +import org.apache.spark.sql.{DataFrame, Row, SparkSession}
    +
    +
    + /**
    +  * An example demonstrating power iteration clustering.
    +  * Run with
    +  * {{{
    +  * bin/run-example ml.PowerIterationClusteringExample
    +  * }}}
    +  */
    +
    +object PowerIterationClusteringExample {
    +
    +  def main(args: Array[String]): Unit = {
    +    val spark = SparkSession
    +      .builder
    +      .appName(s"${this.getClass.getSimpleName}")
    +      .getOrCreate()
    +
    +    Logger.getRootLogger.setLevel(Level.WARN)
    +
    +    // $example on$
    +
    +    // Generates data.
    +    val radius1 = 1.0
    +    val numPoints1 = 5
    +    val radius2 = 4.0
    +    val numPoints2 = 20
    +
    +    val dataset = generatePICData(spark, radius1, radius2, numPoints1, numPoints2)
    +
    +    // Trains a PIC model.
    +    val model = new PowerIterationClustering().
    +      setK(2).
    +      setInitMode("degree").
    +      setMaxIter(20)
    +
    +    val prediction = model.transform(dataset).select("id", "prediction")
    +
    +    //  Shows the result.
    +    //  println("Cluster Assignment: ")
    +    val result = prediction.collect().map {
    +      row => (row(1), row(0))
    +    }.groupBy(_._1).mapValues(_.map(_._2))
    +
    +    result.foreach {
    +      case (cluster, points) => println(s"$cluster -> [${points.mkString(",")}]")
    +    }
    --- End diff --
    
    prediction.show() displays the cluster assignment properly. So this may not be required. current output based on the new dataset will be,
    | id|prediction|
    +---+----------+
    |  4|            1|
    |  0|            0|
    |  1|            0|
    |  3|            1|
    |  2|            0|
    +---+----------+


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21248: [SPARK-24191][ML]Scala Example code for Power Iteration ...

Posted by shahidki31 <gi...@git.apache.org>.
Github user shahidki31 commented on the issue:

    https://github.com/apache/spark/pull/21248
  
    Thanks @srowen 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21248: [SPARK-24191]Example code for Power Iteration Clustering

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21248
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21248: [SPARK-24191][ML]Scala Example code for Power Iteration ...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/21248
  
    OK, I think this could have been combined with https://github.com/apache/spark/pull/21283, but whatever.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21248: [SPARK-24191][ML]Example code for Power Iteration Cluste...

Posted by shahidki31 <gi...@git.apache.org>.
Github user shahidki31 commented on the issue:

    https://github.com/apache/spark/pull/21248
  
    cc @mengxr @WeichenXu123 @felixcheung. Can you please verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21248: [SPARK-24191][ML]Scala Example code for Power Ite...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/21248


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21248: [SPARK-24191][ML]Scala Example code for Power Ite...

Posted by hhbyyh <gi...@git.apache.org>.
Github user hhbyyh commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21248#discussion_r189466112
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/ml/PowerIterationClusteringExample.scala ---
    @@ -0,0 +1,114 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +// scalastyle:off println
    +package org.apache.spark.examples.ml
    +
    +import org.apache.log4j.{Level, Logger}
    +
    +// $example on$
    +import org.apache.spark.ml.clustering.PowerIterationClustering
    +// $example off$
    +import org.apache.spark.sql.{DataFrame, Row, SparkSession}
    +
    +
    + /**
    +  * An example demonstrating power iteration clustering.
    +  * Run with
    +  * {{{
    +  * bin/run-example ml.PowerIterationClusteringExample
    +  * }}}
    +  */
    +
    +object PowerIterationClusteringExample {
    +
    +  def main(args: Array[String]): Unit = {
    +    val spark = SparkSession
    +      .builder
    +      .appName(s"${this.getClass.getSimpleName}")
    +      .getOrCreate()
    +
    +    Logger.getRootLogger.setLevel(Level.WARN)
    +
    +    // $example on$
    +
    +    // Generates data.
    +    val radius1 = 1.0
    +    val numPoints1 = 5
    +    val radius2 = 4.0
    +    val numPoints2 = 20
    +
    +    val dataset = generatePICData(spark, radius1, radius2, numPoints1, numPoints2)
    +
    +    // Trains a PIC model.
    +    val model = new PowerIterationClustering().
    +      setK(2).
    +      setInitMode("degree").
    +      setMaxIter(20)
    +
    +    val prediction = model.transform(dataset).select("id", "prediction")
    +
    +    //  Shows the result.
    +    //  println("Cluster Assignment: ")
    --- End diff --
    
    Maybe add `prediction.show()` first to demo the result of the `transform`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21248: [SPARK-24191][ML]Scala Example code for Power Iteration ...

Posted by shahidki31 <gi...@git.apache.org>.
Github user shahidki31 commented on the issue:

    https://github.com/apache/spark/pull/21248
  
    Hi @hhbyyh  @srowen  ,
    PowerIterationClustering API has some modifications based on the discussion in the JIRA, https://issues.apache.org/jira/browse/SPARK-15784.
    The examples also have modified based on the latest PR of Power Iteration Clustering, https://github.com/apache/spark/pull/21493
    
    Can you please review the modified changes?
    Thanks


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21248: [SPARK-24191][ML]Scala Example code for Power Ite...

Posted by hhbyyh <gi...@git.apache.org>.
Github user hhbyyh commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21248#discussion_r189465979
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/ml/PowerIterationClusteringExample.scala ---
    @@ -0,0 +1,114 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +// scalastyle:off println
    +package org.apache.spark.examples.ml
    +
    +import org.apache.log4j.{Level, Logger}
    +
    +// $example on$
    +import org.apache.spark.ml.clustering.PowerIterationClustering
    +// $example off$
    +import org.apache.spark.sql.{DataFrame, Row, SparkSession}
    +
    +
    + /**
    +  * An example demonstrating power iteration clustering.
    +  * Run with
    +  * {{{
    +  * bin/run-example ml.PowerIterationClusteringExample
    +  * }}}
    +  */
    +
    +object PowerIterationClusteringExample {
    +
    +  def main(args: Array[String]): Unit = {
    +    val spark = SparkSession
    +      .builder
    +      .appName(s"${this.getClass.getSimpleName}")
    +      .getOrCreate()
    +
    +    Logger.getRootLogger.setLevel(Level.WARN)
    --- End diff --
    
    This may not be necessary.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21248: [SPARK-24191][ML]Scala Example code for Power Ite...

Posted by shahidki31 <gi...@git.apache.org>.
Github user shahidki31 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21248#discussion_r189704094
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/ml/PowerIterationClusteringExample.scala ---
    @@ -0,0 +1,114 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +// scalastyle:off println
    +package org.apache.spark.examples.ml
    +
    +import org.apache.log4j.{Level, Logger}
    +
    +// $example on$
    +import org.apache.spark.ml.clustering.PowerIterationClustering
    +// $example off$
    +import org.apache.spark.sql.{DataFrame, Row, SparkSession}
    +
    +
    + /**
    +  * An example demonstrating power iteration clustering.
    +  * Run with
    +  * {{{
    +  * bin/run-example ml.PowerIterationClusteringExample
    +  * }}}
    +  */
    +
    +object PowerIterationClusteringExample {
    +
    +  def main(args: Array[String]): Unit = {
    +    val spark = SparkSession
    +      .builder
    +      .appName(s"${this.getClass.getSimpleName}")
    +      .getOrCreate()
    +
    +    Logger.getRootLogger.setLevel(Level.WARN)
    --- End diff --
    
    Yes. I have removed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21248: [SPARK-24191][ML]Scala Example code for Power Ite...

Posted by shahidki31 <gi...@git.apache.org>.
Github user shahidki31 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21248#discussion_r189704384
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/ml/PowerIterationClusteringExample.scala ---
    @@ -0,0 +1,114 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +// scalastyle:off println
    +package org.apache.spark.examples.ml
    +
    +import org.apache.log4j.{Level, Logger}
    +
    +// $example on$
    +import org.apache.spark.ml.clustering.PowerIterationClustering
    +// $example off$
    +import org.apache.spark.sql.{DataFrame, Row, SparkSession}
    +
    +
    + /**
    +  * An example demonstrating power iteration clustering.
    +  * Run with
    +  * {{{
    +  * bin/run-example ml.PowerIterationClusteringExample
    +  * }}}
    +  */
    +
    +object PowerIterationClusteringExample {
    +
    +  def main(args: Array[String]): Unit = {
    +    val spark = SparkSession
    +      .builder
    +      .appName(s"${this.getClass.getSimpleName}")
    +      .getOrCreate()
    +
    +    Logger.getRootLogger.setLevel(Level.WARN)
    +
    +    // $example on$
    +
    +    // Generates data.
    +    val radius1 = 1.0
    +    val numPoints1 = 5
    +    val radius2 = 4.0
    +    val numPoints2 = 20
    +
    +    val dataset = generatePICData(spark, radius1, radius2, numPoints1, numPoints2)
    +
    +    // Trains a PIC model.
    +    val model = new PowerIterationClustering().
    +      setK(2).
    +      setInitMode("degree").
    +      setMaxIter(20)
    +
    +    val prediction = model.transform(dataset).select("id", "prediction")
    +
    +    //  Shows the result.
    +    //  println("Cluster Assignment: ")
    --- End diff --
    
    Yes. prediction.show() would be better in the new dataset.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21248: [SPARK-24191][ML]Scala Example code for Power Iteration ...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/21248
  
    Merged to master


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21248: [SPARK-24191][ML]Scala Example code for Power Iteration ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21248
  
    **[Test build #4196 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4196/testReport)** for PR 21248 at commit [`d005493`](https://github.com/apache/spark/commit/d0054939562ff822c2a0581af2a616f61a82e131).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21248: [SPARK-24191]Example code for Power Iteration Clustering

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21248
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21248: [SPARK-24191][ML]Scala Example code for Power Iteration ...

Posted by shahidki31 <gi...@git.apache.org>.
Github user shahidki31 commented on the issue:

    https://github.com/apache/spark/pull/21248
  
    Thank you @hhbyyh for the review. I have created a new dataset for the example code, instead of function generated dataset. I have addressed your review comments. 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org