You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jkbradley <gi...@git.apache.org> on 2018/04/17 20:20:07 UTC
[GitHub] spark pull request #21090: [SPARK-15784][ML] Add Power Iteration Clustering ...
GitHub user jkbradley opened a pull request:
https://github.com/apache/spark/pull/21090
[SPARK-15784][ML] Add Power Iteration Clustering to spark.ml
## What changes were proposed in this pull request?
This PR adds PowerIterationClustering as a Transformer to spark.ml. In the transform method, it calls spark.mllib's PowerIterationClustering.run() method and transforms the return value assignments (the Kmeans output of the pseudo-eigenvector) as a DataFrame (id: LongType, cluster: IntegerType).
This PR is copied and modified from https://github.com/apache/spark/pull/15770 The primary author is @wangmiao1981
## How was this patch tested?
This PR has 2 types of tests:
* Copies of tests from spark.mllib's PIC tests
* New tests specific to the spark.ml APIs
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jkbradley/spark wangmiao1981-pic
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21090.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21090
----
commit e4492a64b74b0bbbbccc2da8f13353d37bb9bb0c
Author: wm624@hotmail.com <wm...@...>
Date: 2016-06-13T19:47:42Z
add pic framework (model, class etc)
commit 70862491e5b86ce4add500a0c96ae5220733b35d
Author: wm624@hotmail.com <wm...@...>
Date: 2016-06-13T23:28:09Z
change a comment
commit b73d8a78fa69f83c278996feb1b19502ef871c5b
Author: wm624@hotmail.com <wm...@...>
Date: 2016-06-17T17:27:55Z
add missing functions fit predict load save etc.
commit 022fe523f735c5519f948b175871489f79434fb5
Author: wm624@hotmail.com <wm...@...>
Date: 2016-06-18T01:12:41Z
add unit test flie
commit 552cf54fb03f88af023f080e60fa50f1f39060fc
Author: wm624@hotmail.com <wm...@...>
Date: 2016-06-20T17:35:05Z
add test cases part 1
commit 0b4954d55b4d344794d3c47366220c67f07d0d43
Author: wm624@hotmail.com <wm...@...>
Date: 2016-06-20T20:29:54Z
add unit test part 2: test fit, parameters etc.
commit f22b01e06eaaf5951befcebdffc18c8e519183d2
Author: wm624@hotmail.com <wm...@...>
Date: 2016-06-20T21:22:59Z
fix a type issue
commit 305b194dae40eaff990c18837c3f2bc8d469e60c
Author: wm624@hotmail.com <wm...@...>
Date: 2016-06-21T20:07:27Z
add more unit tests
commit 4b32cbf02965c5c1a0c094fa144836dab0dfd543
Author: wm624@hotmail.com <wm...@...>
Date: 2016-06-21T21:46:25Z
delete unused import and add comments
commit f6eda88a6c0af416b988a2c37f46c8b08e5e99cf
Author: wm624@hotmail.com <wm...@...>
Date: 2016-10-25T21:28:12Z
change version to 2.1.0
commit 45c4b1cd1afa28c775c666b57ecee614ed9a41d0
Author: wm624@hotmail.com <wm...@...>
Date: 2016-11-03T23:26:01Z
change PIC as a Transformer
commit e8d7ed37138909d010a812fba7d03ef30a4f6e05
Author: wm624@hotmail.com <wm...@...>
Date: 2016-11-04T17:28:26Z
add LabelCol
commit e4e1e055a9b3ab54b83331ac7dc56d6b792dcf7b
Author: wm624@hotmail.com <wm...@...>
Date: 2016-11-04T18:36:09Z
change col implementation
commit 8384422ec0e7192cc8ce53df02ddb4ae0401fd0b
Author: wm624@hotmail.com <wm...@...>
Date: 2017-02-17T22:20:00Z
address some of the comments
commit d6a199c48ff940861d80caf275da29d99375ce33
Author: wm624@hotmail.com <wm...@...>
Date: 2017-02-21T22:37:51Z
add additional test with dataset having more data
commit b0c3aff4a76ace99c104c2b2c10c9485a028bfd6
Author: wm624@hotmail.com <wm...@...>
Date: 2017-03-14T23:13:45Z
change input data format
commit 091225dd2f1c353edc28dc4299034a018a92bc81
Author: wm624@hotmail.com <wm...@...>
Date: 2017-03-15T22:49:45Z
resolve warnings
commit 8bb99567556ce29c75d5f395157d0161dff695bc
Author: wm624@hotmail.com <wm...@...>
Date: 2017-03-16T18:33:47Z
add neighbor and weight cols
commit 8ba82e8392e6d607ab750ed8eb3caaf386e1352a
Author: wangmiao1981 <wm...@...>
Date: 2017-08-15T21:13:55Z
address review comments 1
commit 468a94741efe6530c9acfbb1af4f46499550ee1f
Author: wangmiao1981 <wm...@...>
Date: 2017-08-15T21:23:39Z
fix style
commit ec10f24336ff51354a1657c7ceadb9ada8cd1484
Author: wangmiao1981 <wm...@...>
Date: 2017-08-15T22:30:28Z
remove unused comments
commit 5710cfcf2e3596c95f353ce043f7358a030d70a0
Author: wangmiao1981 <wm...@...>
Date: 2017-08-15T23:43:14Z
add Since
commit 88654b3055ebd863e3b3c5774abdce28f3cda184
Author: wangmiao1981 <wm...@...>
Date: 2017-08-17T00:12:12Z
fix missing >
commit 804adc6fece91e7264f315ee965faa40c5e334c5
Author: wangmiao1981 <wm...@...>
Date: 2017-08-17T17:26:40Z
fix doc
commit 4a6dd79a9c37f71ea4378692438f19b3247b7913
Author: wangmiao1981 <wm...@...>
Date: 2017-10-25T23:16:55Z
address review comments
commit 5cb8ed6de3865f58719b3b30888b3bc4542905d4
Author: wangmiao1981 <wm...@...>
Date: 2017-10-30T21:44:24Z
fix unit test
commit 6abf6023868d944068a26186cde3fbadffd83a74
Author: Joseph K. Bradley <jo...@...>
Date: 2018-04-03T23:46:40Z
cleanups to docs
commit d9270876797153d7660843fc621e707b4dff71ca
Author: Joseph K. Bradley <jo...@...>
Date: 2018-04-03T23:52:36Z
typo
commit d2157489770a79fe443d567bfc03d61f72fbe161
Author: Joseph K. Bradley <jo...@...>
Date: 2018-04-17T20:17:15Z
final updates for PIC PR
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21090: [SPARK-15784][ML] Add Power Iteration Clustering to spar...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the issue:
https://github.com/apache/spark/pull/21090
Thanks for reviewing this and for the LGTM @wangmiao1981 ! I'll merge with master now, with you as the primary author.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21090: [SPARK-15784][ML] Add Power Iteration Clustering to spar...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21090
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2458/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21090: [SPARK-15784][ML] Add Power Iteration Clustering to spar...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21090
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21090: [SPARK-15784][ML] Add Power Iteration Clustering to spar...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21090
**[Test build #89472 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89472/testReport)** for PR 21090 at commit [`d215748`](https://github.com/apache/spark/commit/d2157489770a79fe443d567bfc03d61f72fbe161).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21090: [SPARK-15784][ML] Add Power Iteration Clustering to spar...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the issue:
https://github.com/apache/spark/pull/21090
@wangmiao1981 and @WeichenXu123 would you mind taking a look? Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21090: [SPARK-24026][ML] Add Power Iteration Clustering ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21090
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21090: [SPARK-15784][ML] Add Power Iteration Clustering to spar...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21090
**[Test build #89536 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89536/testReport)** for PR 21090 at commit [`375e150`](https://github.com/apache/spark/commit/375e150e2f317e8bfc46992c13f60ca74ed04c00).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21090: [SPARK-24026][ML] Add Power Iteration Clustering ...
Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21090#discussion_r182931610
--- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/PowerIterationClustering.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.clustering
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.Transformer
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared._
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.clustering.{PowerIterationClustering => MLlibPowerIterationClustering}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{DataFrame, Dataset, Row}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for PowerIterationClustering
+ */
+private[clustering] trait PowerIterationClusteringParams extends Params with HasMaxIter
+ with HasPredictionCol {
+
+ /**
+ * The number of clusters to create (k). Must be > 1. Default: 2.
+ * @group param
+ */
+ @Since("2.4.0")
+ final val k = new IntParam(this, "k", "The number of clusters to create. " +
+ "Must be > 1.", ParamValidators.gt(1))
+
+ /** @group getParam */
+ @Since("2.4.0")
+ def getK: Int = $(k)
+
+ /**
+ * Param for the initialization algorithm. This can be either "random" to use a random vector
+ * as vertex properties, or "degree" to use a normalized sum of similarities with other vertices.
+ * Default: random.
+ * @group expertParam
+ */
+ @Since("2.4.0")
+ final val initMode = {
+ val allowedParams = ParamValidators.inArray(Array("random", "degree"))
+ new Param[String](this, "initMode", "The initialization algorithm. This can be either " +
+ "'random' to use a random vector as vertex properties, or 'degree' to use a normalized sum " +
+ "of similarities with other vertices. Supported options: 'random' and 'degree'.",
+ allowedParams)
+ }
+
+ /** @group expertGetParam */
+ @Since("2.4.0")
+ def getInitMode: String = $(initMode)
+
+ /**
+ * Param for the name of the input column for vertex IDs.
+ * Default: "id"
+ * @group param
+ */
+ @Since("2.4.0")
+ val idCol = new Param[String](this, "idCol", "Name of the input column for vertex IDs.",
+ (value: String) => value.nonEmpty)
+
+ setDefault(idCol, "id")
+
+ /** @group getParam */
+ @Since("2.4.0")
+ def getIdCol: String = getOrDefault(idCol)
+
+ /**
+ * Param for the name of the input column for neighbors in the adjacency list representation.
+ * Default: "neighbors"
+ * @group param
+ */
+ @Since("2.4.0")
+ val neighborsCol = new Param[String](this, "neighborsCol",
+ "Name of the input column for neighbors in the adjacency list representation.",
+ (value: String) => value.nonEmpty)
+
+ setDefault(neighborsCol, "neighbors")
+
+ /** @group getParam */
+ @Since("2.4.0")
+ def getNeighborsCol: String = $(neighborsCol)
+
+ /**
+ * Param for the name of the input column for neighbors in the adjacency list representation.
+ * Default: "similarities"
+ * @group param
+ */
+ @Since("2.4.0")
+ val similaritiesCol = new Param[String](this, "similaritiesCol",
+ "Name of the input column for neighbors in the adjacency list representation.",
+ (value: String) => value.nonEmpty)
+
--- End diff --
Seems similaritiesCol is exactly the same as neighborsCol. Is this right?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21090: [SPARK-15784][ML] Add Power Iteration Clustering to spar...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on the issue:
https://github.com/apache/spark/pull/21090
Take a quick look. Despite of the style failure and a minor format issue, LGTM.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21090: [SPARK-24026][ML] Add Power Iteration Clustering ...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/21090#discussion_r188813735
--- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/PowerIterationClustering.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.clustering
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.Transformer
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared._
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.clustering.{PowerIterationClustering => MLlibPowerIterationClustering}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{DataFrame, Dataset, Row}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for PowerIterationClustering
+ */
+private[clustering] trait PowerIterationClusteringParams extends Params with HasMaxIter
+ with HasPredictionCol {
+
+ /**
+ * The number of clusters to create (k). Must be > 1. Default: 2.
+ * @group param
+ */
+ @Since("2.4.0")
+ final val k = new IntParam(this, "k", "The number of clusters to create. " +
+ "Must be > 1.", ParamValidators.gt(1))
+
+ /** @group getParam */
+ @Since("2.4.0")
+ def getK: Int = $(k)
+
+ /**
+ * Param for the initialization algorithm. This can be either "random" to use a random vector
+ * as vertex properties, or "degree" to use a normalized sum of similarities with other vertices.
+ * Default: random.
+ * @group expertParam
+ */
+ @Since("2.4.0")
+ final val initMode = {
+ val allowedParams = ParamValidators.inArray(Array("random", "degree"))
+ new Param[String](this, "initMode", "The initialization algorithm. This can be either " +
+ "'random' to use a random vector as vertex properties, or 'degree' to use a normalized sum " +
+ "of similarities with other vertices. Supported options: 'random' and 'degree'.",
+ allowedParams)
+ }
+
+ /** @group expertGetParam */
+ @Since("2.4.0")
+ def getInitMode: String = $(initMode)
+
+ /**
+ * Param for the name of the input column for vertex IDs.
+ * Default: "id"
+ * @group param
+ */
+ @Since("2.4.0")
+ val idCol = new Param[String](this, "idCol", "Name of the input column for vertex IDs.",
+ (value: String) => value.nonEmpty)
+
+ setDefault(idCol, "id")
+
+ /** @group getParam */
+ @Since("2.4.0")
+ def getIdCol: String = getOrDefault(idCol)
+
+ /**
+ * Param for the name of the input column for neighbors in the adjacency list representation.
+ * Default: "neighbors"
+ * @group param
+ */
+ @Since("2.4.0")
+ val neighborsCol = new Param[String](this, "neighborsCol",
+ "Name of the input column for neighbors in the adjacency list representation.",
+ (value: String) => value.nonEmpty)
+
+ setDefault(neighborsCol, "neighbors")
+
+ /** @group getParam */
+ @Since("2.4.0")
+ def getNeighborsCol: String = $(neighborsCol)
+
+ /**
+ * Param for the name of the input column for neighbors in the adjacency list representation.
+ * Default: "similarities"
+ * @group param
+ */
+ @Since("2.4.0")
+ val similaritiesCol = new Param[String](this, "similaritiesCol",
+ "Name of the input column for neighbors in the adjacency list representation.",
+ (value: String) => value.nonEmpty)
+
--- End diff --
No, it's meant to be an adjacency list representation of the graph: neighborsCol has the set of neighbor vertex IDs, and similaritiesCol has the corresponding set of edge weights.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21090: [SPARK-15784][ML] Add Power Iteration Clustering to spar...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21090
**[Test build #89472 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89472/testReport)** for PR 21090 at commit [`d215748`](https://github.com/apache/spark/commit/d2157489770a79fe443d567bfc03d61f72fbe161).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21090: [SPARK-15784][ML] Add Power Iteration Clustering ...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/21090#discussion_r182606716
--- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/PowerIterationClustering.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.clustering
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.Transformer
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared._
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.clustering.{PowerIterationClustering => MLlibPowerIterationClustering}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{DataFrame, Dataset, Row}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for PowerIterationClustering
+ */
+private[clustering] trait PowerIterationClusteringParams extends Params with HasMaxIter
+ with HasPredictionCol {
+
+ /**
+ * The number of clusters to create (k). Must be > 1. Default: 2.
+ * @group param
+ */
+ @Since("2.4.0")
+ final val k = new IntParam(this, "k", "The number of clusters to create. " +
+ "Must be > 1.", ParamValidators.gt(1))
+
+ /** @group getParam */
+ @Since("2.4.0")
+ def getK: Int = $(k)
+
+ /**
+ * Param for the initialization algorithm. This can be either "random" to use a random vector
+ * as vertex properties, or "degree" to use a normalized sum of similarities with other vertices.
+ * Default: random.
+ * @group expertParam
+ */
+ @Since("2.4.0")
+ final val initMode = {
+ val allowedParams = ParamValidators.inArray(Array("random", "degree"))
+ new Param[String](this, "initMode", "The initialization algorithm. This can be either " +
+ "'random' to use a random vector as vertex properties, or 'degree' to use a normalized sum " +
+ "of similarities with other vertices. Supported options: 'random' and 'degree'.",
+ allowedParams)
+ }
+
+ /** @group expertGetParam */
+ @Since("2.4.0")
+ def getInitMode: String = $(initMode)
+
+ /**
+ * Param for the name of the input column for vertex IDs.
+ * Default: "id"
+ * @group param
+ */
+ @Since("2.4.0")
+ val idCol = new Param[String](this, "idCol", "Name of the input column for vertex IDs.",
+ (value: String) => value.nonEmpty)
+
+ setDefault(idCol, "id")
+
+ /** @group getParam */
+ @Since("2.4.0")
+ def getIdCol: String = getOrDefault(idCol)
+
+ /**
+ * Param for the name of the input column for neighbors in the adjacency list representation.
+ * Default: "neighbors"
+ * @group param
+ */
+ @Since("2.4.0")
+ val neighborsCol = new Param[String](this, "neighborsCol",
+ "Name of the input column for neighbors in the adjacency list representation.",
+ (value: String) => value.nonEmpty)
+
+ setDefault(neighborsCol, "neighbors")
+
+ /** @group getParam */
+ @Since("2.4.0")
+ def getNeighborsCol: String = $(neighborsCol)
+
+ /**
+ * Param for the name of the input column for neighbors in the adjacency list representation.
+ * Default: "similarities"
+ * @group param
+ */
+ @Since("2.4.0")
+ val similaritiesCol = new Param[String](this, "similaritiesCol",
+ "Name of the input column for neighbors in the adjacency list representation.",
+ (value: String) => value.nonEmpty)
+
+ setDefault(similaritiesCol, "similarities")
+
+ /** @group getParam */
+ @Since("2.4.0")
+ def getSimilaritiesCol: String = $(similaritiesCol)
+
+ protected def validateAndTransformSchema(schema: StructType): StructType = {
+ SchemaUtils.checkColumnTypes(schema, $(idCol), Seq(IntegerType, LongType))
+ SchemaUtils.checkColumnTypes(schema, $(neighborsCol),
+ Seq(ArrayType(IntegerType, containsNull = false),
+ ArrayType(LongType, containsNull = false)))
+ SchemaUtils.checkColumnTypes(schema, $(similaritiesCol),
+ Seq(ArrayType(FloatType, containsNull = false),
+ ArrayType(DoubleType, containsNull = false)))
+ SchemaUtils.appendColumn(schema, $(predictionCol), IntegerType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * Power Iteration Clustering (PIC), a scalable graph clustering algorithm developed by
+ * <a href=http://www.icml2010.org/papers/387.pdf>Lin and Cohen</a>. From the abstract:
+ * PIC finds a very low-dimensional embedding of a dataset using truncated power
+ * iteration on a normalized pair-wise similarity matrix of the data.
+ *
+ * PIC takes an affinity matrix between items (or vertices) as input. An affinity matrix
+ * is a symmetric matrix whose entries are non-negative similarities between items.
+ * PIC takes this matrix (or graph) as an adjacency matrix. Specifically, each input row includes:
+ * - `idCol`: vertex ID
+ * - `neighborsCol`: neighbors of vertex in `idCol`
+ * - `similaritiesCol`: non-negative weights (similarities) of edges between the vertex
+ * in `idCol` and each neighbor in `neighborsCol`
+ * PIC returns a cluster assignment for each input vertex. It appends a new column `predictionCol`
+ * containing the cluster assignment in `[0,k)` for each row (vertex).
+ *
+ * Notes:
+ * - [[PowerIterationClustering]] is a transformer with an expensive [[transform]] operation.
+ * Transform runs the iterative PIC algorithm to cluster the whole input dataset.
+ * - Input validation: This validates that similarities are non-negative but does NOT validate
+ * that the input matrix is symmetric.
+ *
+ * @see <a href=http://en.wikipedia.org/wiki/Spectral_clustering>
+ * Spectral clustering (Wikipedia)</a>
+ */
+@Since("2.4.0")
+@Experimental
+class PowerIterationClustering private[clustering] (
+ @Since("2.4.0") override val uid: String)
+ extends Transformer with PowerIterationClusteringParams with DefaultParamsWritable {
+
+ setDefault(
+ k -> 2,
+ maxIter -> 20,
+ initMode -> "random")
+
+ @Since("2.4.0")
+ def this() = this(Identifiable.randomUID("PowerIterationClustering"))
+
+ /** @group setParam */
+ @Since("2.4.0")
+ def setPredictionCol(value: String): this.type = set(predictionCol, value)
+
+ /** @group setParam */
+ @Since("2.4.0")
+ def setK(value: Int): this.type = set(k, value)
+
+ /** @group expertSetParam */
+ @Since("2.4.0")
+ def setInitMode(value: String): this.type = set(initMode, value)
+
+ /** @group setParam */
+ @Since("2.4.0")
+ def setMaxIter(value: Int): this.type = set(maxIter, value)
+
+ /** @group setParam */
+ @Since("2.4.0")
+ def setIdCol(value: String): this.type = set(idCol, value)
+
+ /** @group setParam */
+ @Since("2.4.0")
+ def setNeighborsCol(value: String): this.type = set(neighborsCol, value)
+
+ /** @group setParam */
+ @Since("2.4.0")
+ def setSimilaritiesCol(value: String): this.type = set(similaritiesCol, value)
+
+ @Since("2.4.0")
+ override def transform(dataset: Dataset[_]): DataFrame = {
+ transformSchema(dataset.schema, logging = true)
+
+ val sparkSession = dataset.sparkSession
+ val idColValue = $(idCol)
+ val rdd: RDD[(Long, Long, Double)] =
+ dataset.select(
+ col($(idCol)).cast(LongType),
+ col($(neighborsCol)).cast(ArrayType(LongType, containsNull = false)),
+ col($(similaritiesCol)).cast(ArrayType(DoubleType, containsNull = false))
+ ).rdd.flatMap {
+ case Row(id: Long, nbrs: Seq[_], sims: Seq[_]) =>
+ require(nbrs.size == sims.size, s"The length of the neighbor ID list must be " +
+ s"equal to the the length of the neighbor similarity list. Row for ID " +
+ s"$idColValue=$id has neighbor ID list of length ${nbrs.length} but similarity list " +
+ s"of length ${sims.length}.")
+ nbrs.asInstanceOf[Seq[Long]].zip(sims.asInstanceOf[Seq[Double]]).map {
+ case (nbr, similarity) => (id, nbr, similarity)
+ }
+ }
--- End diff --
Thanks for reminding me! I'll do it now.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21090: [SPARK-15784][ML] Add Power Iteration Clustering to spar...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21090
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89536/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21090: [SPARK-15784][ML] Add Power Iteration Clustering ...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21090#discussion_r182254888
--- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/PowerIterationClustering.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.clustering
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.Transformer
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared._
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.clustering.{PowerIterationClustering => MLlibPowerIterationClustering}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{DataFrame, Dataset, Row}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for PowerIterationClustering
+ */
+private[clustering] trait PowerIterationClusteringParams extends Params with HasMaxIter
+ with HasPredictionCol {
+
+ /**
+ * The number of clusters to create (k). Must be > 1. Default: 2.
+ * @group param
+ */
+ @Since("2.4.0")
+ final val k = new IntParam(this, "k", "The number of clusters to create. " +
+ "Must be > 1.", ParamValidators.gt(1))
+
+ /** @group getParam */
+ @Since("2.4.0")
+ def getK: Int = $(k)
+
+ /**
+ * Param for the initialization algorithm. This can be either "random" to use a random vector
+ * as vertex properties, or "degree" to use a normalized sum of similarities with other vertices.
+ * Default: random.
+ * @group expertParam
+ */
+ @Since("2.4.0")
+ final val initMode = {
+ val allowedParams = ParamValidators.inArray(Array("random", "degree"))
+ new Param[String](this, "initMode", "The initialization algorithm. This can be either " +
+ "'random' to use a random vector as vertex properties, or 'degree' to use a normalized sum " +
+ "of similarities with other vertices. Supported options: 'random' and 'degree'.",
+ allowedParams)
+ }
+
+ /** @group expertGetParam */
+ @Since("2.4.0")
+ def getInitMode: String = $(initMode)
+
+ /**
+ * Param for the name of the input column for vertex IDs.
+ * Default: "id"
+ * @group param
+ */
+ @Since("2.4.0")
+ val idCol = new Param[String](this, "idCol", "Name of the input column for vertex IDs.",
+ (value: String) => value.nonEmpty)
+
+ setDefault(idCol, "id")
+
+ /** @group getParam */
+ @Since("2.4.0")
+ def getIdCol: String = getOrDefault(idCol)
+
+ /**
+ * Param for the name of the input column for neighbors in the adjacency list representation.
+ * Default: "neighbors"
+ * @group param
+ */
+ @Since("2.4.0")
+ val neighborsCol = new Param[String](this, "neighborsCol",
+ "Name of the input column for neighbors in the adjacency list representation.",
+ (value: String) => value.nonEmpty)
+
+ setDefault(neighborsCol, "neighbors")
+
+ /** @group getParam */
+ @Since("2.4.0")
+ def getNeighborsCol: String = $(neighborsCol)
+
+ /**
+ * Param for the name of the input column for neighbors in the adjacency list representation.
+ * Default: "similarities"
+ * @group param
+ */
+ @Since("2.4.0")
+ val similaritiesCol = new Param[String](this, "similaritiesCol",
+ "Name of the input column for neighbors in the adjacency list representation.",
+ (value: String) => value.nonEmpty)
+
+ setDefault(similaritiesCol, "similarities")
+
+ /** @group getParam */
+ @Since("2.4.0")
+ def getSimilaritiesCol: String = $(similaritiesCol)
+
+ protected def validateAndTransformSchema(schema: StructType): StructType = {
+ SchemaUtils.checkColumnTypes(schema, $(idCol), Seq(IntegerType, LongType))
+ SchemaUtils.checkColumnTypes(schema, $(neighborsCol),
+ Seq(ArrayType(IntegerType, containsNull = false),
+ ArrayType(LongType, containsNull = false)))
+ SchemaUtils.checkColumnTypes(schema, $(similaritiesCol),
+ Seq(ArrayType(FloatType, containsNull = false),
+ ArrayType(DoubleType, containsNull = false)))
+ SchemaUtils.appendColumn(schema, $(predictionCol), IntegerType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * Power Iteration Clustering (PIC), a scalable graph clustering algorithm developed by
+ * <a href=http://www.icml2010.org/papers/387.pdf>Lin and Cohen</a>. From the abstract:
+ * PIC finds a very low-dimensional embedding of a dataset using truncated power
+ * iteration on a normalized pair-wise similarity matrix of the data.
+ *
+ * PIC takes an affinity matrix between items (or vertices) as input. An affinity matrix
+ * is a symmetric matrix whose entries are non-negative similarities between items.
+ * PIC takes this matrix (or graph) as an adjacency matrix. Specifically, each input row includes:
+ * - `idCol`: vertex ID
+ * - `neighborsCol`: neighbors of vertex in `idCol`
+ * - `similaritiesCol`: non-negative weights (similarities) of edges between the vertex
+ * in `idCol` and each neighbor in `neighborsCol`
+ * PIC returns a cluster assignment for each input vertex. It appends a new column `predictionCol`
+ * containing the cluster assignment in `[0,k)` for each row (vertex).
+ *
+ * Notes:
+ * - [[PowerIterationClustering]] is a transformer with an expensive [[transform]] operation.
+ * Transform runs the iterative PIC algorithm to cluster the whole input dataset.
+ * - Input validation: This validates that similarities are non-negative but does NOT validate
+ * that the input matrix is symmetric.
+ *
+ * @see <a href=http://en.wikipedia.org/wiki/Spectral_clustering>
+ * Spectral clustering (Wikipedia)</a>
+ */
+@Since("2.4.0")
+@Experimental
+class PowerIterationClustering private[clustering] (
+ @Since("2.4.0") override val uid: String)
+ extends Transformer with PowerIterationClusteringParams with DefaultParamsWritable {
+
+ setDefault(
+ k -> 2,
+ maxIter -> 20,
+ initMode -> "random")
+
+ @Since("2.4.0")
+ def this() = this(Identifiable.randomUID("PowerIterationClustering"))
+
+ /** @group setParam */
+ @Since("2.4.0")
+ def setPredictionCol(value: String): this.type = set(predictionCol, value)
+
+ /** @group setParam */
+ @Since("2.4.0")
+ def setK(value: Int): this.type = set(k, value)
+
+ /** @group expertSetParam */
+ @Since("2.4.0")
+ def setInitMode(value: String): this.type = set(initMode, value)
+
+ /** @group setParam */
+ @Since("2.4.0")
+ def setMaxIter(value: Int): this.type = set(maxIter, value)
+
+ /** @group setParam */
+ @Since("2.4.0")
+ def setIdCol(value: String): this.type = set(idCol, value)
+
+ /** @group setParam */
+ @Since("2.4.0")
+ def setNeighborsCol(value: String): this.type = set(neighborsCol, value)
+
+ /** @group setParam */
+ @Since("2.4.0")
+ def setSimilaritiesCol(value: String): this.type = set(similaritiesCol, value)
+
+ @Since("2.4.0")
+ override def transform(dataset: Dataset[_]): DataFrame = {
+ transformSchema(dataset.schema, logging = true)
+
+ val sparkSession = dataset.sparkSession
+ val idColValue = $(idCol)
+ val rdd: RDD[(Long, Long, Double)] =
+ dataset.select(
+ col($(idCol)).cast(LongType),
+ col($(neighborsCol)).cast(ArrayType(LongType, containsNull = false)),
+ col($(similaritiesCol)).cast(ArrayType(DoubleType, containsNull = false))
+ ).rdd.flatMap {
+ case Row(id: Long, nbrs: Seq[_], sims: Seq[_]) =>
+ require(nbrs.size == sims.size, s"The length of the neighbor ID list must be " +
+ s"equal to the the length of the neighbor similarity list. Row for ID " +
+ s"$idColValue=$id has neighbor ID list of length ${nbrs.length} but similarity list " +
+ s"of length ${sims.length}.")
+ nbrs.asInstanceOf[Seq[Long]].zip(sims.asInstanceOf[Seq[Double]]).map {
+ case (nbr, similarity) => (id, nbr, similarity)
+ }
+ }
--- End diff --
Add `Instrumentation`? Or I can add it in a separate PR.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21090: [SPARK-15784][ML] Add Power Iteration Clustering to spar...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21090
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21090: [SPARK-15784][ML] Add Power Iteration Clustering to spar...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21090
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21090: [SPARK-15784][ML] Add Power Iteration Clustering ...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21090#discussion_r182243819
--- Diff: mllib/src/test/scala/org/apache/spark/ml/clustering/PowerIterationClusteringSuite.scala ---
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.clustering
+
+import scala.collection.mutable
+
+import org.apache.spark.ml.util.DefaultReadWriteTest
+import org.apache.spark.mllib.util.MLlibTestSparkContext
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.{DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.{SparkException, SparkFunSuite}
+
+
+class PowerIterationClusteringSuite extends SparkFunSuite
+ with MLlibTestSparkContext with DefaultReadWriteTest {
+
+ @transient var data: Dataset[_] = _
+ final val r1 = 1.0
+ final val n1 = 10
+ final val r2 = 4.0
+ final val n2 = 40
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+
+ data = PowerIterationClusteringSuite.generatePICData(spark, r1, r2, n1, n2)
+ }
+
+ test("default parameters") {
+ val pic = new PowerIterationClustering()
+
+ assert(pic.getK === 2)
+ assert(pic.getMaxIter === 20)
+ assert(pic.getInitMode === "random")
+ assert(pic.getPredictionCol === "prediction")
+ assert(pic.getIdCol === "id")
+ assert(pic.getNeighborsCol === "neighbors")
+ assert(pic.getSimilaritiesCol === "similarities")
+ }
+
+ test("parameter validation") {
+ intercept[IllegalArgumentException] {
+ new PowerIterationClustering().setK(1)
+ }
+ intercept[IllegalArgumentException] {
+ new PowerIterationClustering().setInitMode("no_such_a_mode")
+ }
+ intercept[IllegalArgumentException] {
+ new PowerIterationClustering().setIdCol("")
+ }
+ intercept[IllegalArgumentException] {
+ new PowerIterationClustering().setNeighborsCol("")
+ }
+ intercept[IllegalArgumentException] {
+ new PowerIterationClustering().setSimilaritiesCol("")
+ }
+ }
+
+ test("power iteration clustering") {
+ val n = n1 + n2
+
+ val model = new PowerIterationClustering()
+ .setK(2)
+ .setMaxIter(40)
+ val result = model.transform(data)
+
+ val predictions = Array.fill(2)(mutable.Set.empty[Long])
+ result.select("id", "prediction").collect().foreach {
+ case Row(id: Long, cluster: Integer) => predictions(cluster) += id
+ }
+ assert(predictions.toSet == Set((1 until n1).toSet, (n1 until n).toSet))
+
+ val result2 = new PowerIterationClustering()
+ .setK(2)
+ .setMaxIter(10)
+ .setInitMode("degree")
+ .transform(data)
+ val predictions2 = Array.fill(2)(mutable.Set.empty[Long])
+ result2.select("id", "prediction").collect().foreach {
+ case Row(id: Long, cluster: Integer) => predictions2(cluster) += id
+ }
+ assert(predictions2.toSet == Set((1 until n1).toSet, (n1 until n).toSet))
+ }
+
+ test("supported input types") {
+ val model = new PowerIterationClustering()
+ .setK(2)
+ .setMaxIter(1)
+
+ def runTest(idType: DataType, neighborType: DataType, similarityType: DataType): Unit = {
+ val typedData = data.select(
+ col("id").cast(idType).alias("id"),
+ col("neighbors").cast(ArrayType(neighborType, containsNull = false)).alias("neighbors"),
+ col("similarities").cast(ArrayType(similarityType, containsNull = false))
+ .alias("similarities")
+ )
+ model.transform(typedData).collect()
+ }
+
+ for (idType <- Seq(IntegerType, LongType)) {
+ runTest(idType, LongType, DoubleType)
+ }
+ for (neighborType <- Seq(IntegerType, LongType)) {
+ runTest(LongType, neighborType, DoubleType)
+ }
+ for (similarityType <- Seq(FloatType, DoubleType)) {
+ runTest(LongType, LongType, similarityType)
+ }
+ }
+
+ test("invalid input: wrong types") {
+ val model = new PowerIterationClustering()
+ .setK(2)
+ .setMaxIter(1)
+ intercept[IllegalArgumentException] {
+ val typedData = data.select(
+ col("id").cast(DoubleType).alias("id"),
+ col("neighbors"),
+ col("similarities")
+ )
+ model.transform(typedData)
+ }
+ intercept[IllegalArgumentException] {
+ val typedData = data.select(
+ col("id"),
+ col("neighbors").cast(ArrayType(DoubleType, containsNull = false)).alias("neighbors"),
+ col("similarities")
+ )
+ model.transform(typedData)
+ }
+ intercept[IllegalArgumentException] {
+
--- End diff --
remove blank line or add blank line after line 139 for consistence?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21090: [SPARK-15784][ML] Add Power Iteration Clustering to spar...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21090
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89472/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21090: [SPARK-15784][ML] Add Power Iteration Clustering to spar...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the issue:
https://github.com/apache/spark/pull/21090
**To review this PR**: This was copied from https://github.com/apache/spark/pull/15770 with the following changes:
* Addressed comments in original PR (See my review comments there)
* Added Param validators for required input columns
* Renamed “weights” column to “similarities”
* Made algorithm take more types of inputs: Long/Int and Double/Float
* Removed test("set parameters") since setters are already tested in the read/write test.
If you saw the previous PR, you should be able to review this one based on the last 3 commits, viewable in this diff: https://github.com/jkbradley/spark/compare/5cb8ed6...wangmiao1981-pic
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21090: [SPARK-15784][ML] Add Power Iteration Clustering to spar...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21090
**[Test build #89536 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89536/testReport)** for PR 21090 at commit [`375e150`](https://github.com/apache/spark/commit/375e150e2f317e8bfc46992c13f60ca74ed04c00).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21090: [SPARK-15784][ML] Add Power Iteration Clustering to spar...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21090
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2404/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21090: [SPARK-15784][ML] Add Power Iteration Clustering to spar...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21090
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org