You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yinxusen <gi...@git.apache.org> on 2015/03/09 15:16:38 UTC
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
GitHub user yinxusen opened a pull request:
https://github.com/apache/spark/pull/4951
[SPARK-5986][MLLib] Add save/load for k-means
This PR adds save/load for K-means as described in SPARK-5986. Python version will be added in another PR.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yinxusen/spark SPARK-5986
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4951.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4951
----
commit dce70553cb0e5c25d1bb0a415929eb5066af964a
Author: Xusen Yin <yi...@gmail.com>
Date: 2015-03-09T14:12:59Z
add save/load for k-means for SPARK-5986
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4951#issuecomment-77877303
[Test build #28393 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28393/consoleFull) for PR 4951 at commit [`dce7055`](https://github.com/apache/spark/commit/dce70553cb0e5c25d1bb0a415929eb5066af964a).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class KMeansModel (val clusterCenters: Array[Vector]) extends Saveable with Serializable `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4951#issuecomment-78190584
[Test build #28453 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28453/consoleFull) for PR 4951 at commit [`6dd74a0`](https://github.com/apache/spark/commit/6dd74a0f57678bdbfc6654433047e96ff1801429).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class KMeansModel (val clusterCenters: Array[Vector]) extends Saveable with Serializable `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4951#discussion_r26173136
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala ---
@@ -17,15 +17,23 @@
package org.apache.spark.mllib.clustering
+import org.json4s._
+import org.json4s.JsonDSL._
+import org.json4s.jackson.JsonMethods._
+
import org.apache.spark.api.java.JavaRDD
+import org.apache.spark.mllib.linalg._
--- End diff --
Be specific about the imports. You only need `Vector`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4951#issuecomment-77958821
[Test build #28408 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28408/consoleFull) for PR 4951 at commit [`cd390fd`](https://github.com/apache/spark/commit/cd390fd294f65d465abee8a066fc29d7958ae9ec).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4951#discussion_r26173149
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala ---
@@ -58,4 +66,59 @@ class KMeansModel (val clusterCenters: Array[Vector]) extends Serializable {
private def clusterCentersWithNorm: Iterable[VectorWithNorm] =
clusterCenters.map(new VectorWithNorm(_))
+
+ override def save(sc: SparkContext, path: String): Unit = {
+ KMeansModel.SaveLoadV1_0.save(sc, this, path)
+ }
+
+ override protected def formatVersion: String = "1.0"
+}
+
+object KMeansModel extends Loader[KMeansModel] {
+ override def load(sc: SparkContext, path: String): KMeansModel = {
+ KMeansModel.SaveLoadV1_0.load(sc, path)
+ }
+
+ case class IndexedPoint(id: Int, point: Vector)
+
+ object IndexedPoint {
--- End diff --
make this private
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/4951
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4951#issuecomment-77969784
[Test build #28408 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28408/consoleFull) for PR 4951 at commit [`cd390fd`](https://github.com/apache/spark/commit/cd390fd294f65d465abee8a066fc29d7958ae9ec).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class KMeansModel (val clusterCenters: Array[Vector]) extends Saveable with Serializable `
* ` case class IndexedPoint(id: Int, point: Vector)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4951#discussion_r26083656
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala ---
@@ -58,4 +66,59 @@ class KMeansModel (val clusterCenters: Array[Vector]) extends Serializable {
private def clusterCentersWithNorm: Iterable[VectorWithNorm] =
clusterCenters.map(new VectorWithNorm(_))
+
+ override def save(sc: SparkContext, path: String): Unit = {
+ KMeansModel.SaveLoadV1_0.save(sc, this, path)
+ }
+
+ override protected def formatVersion: String = "1.0"
+}
+
+object KMeansModel extends Loader[KMeansModel] {
+ override def load(sc: SparkContext, path: String): KMeansModel = {
+ KMeansModel.SaveLoadV1_0.load(sc, path)
+ }
+
+ case class IndexedPoint(id: Int, point: Vector)
--- End diff --
How about using `Int` here to represent indexes? I think there is no need to use `Long`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4951#discussion_r26173140
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala ---
@@ -17,15 +17,23 @@
package org.apache.spark.mllib.clustering
+import org.json4s._
+import org.json4s.JsonDSL._
+import org.json4s.jackson.JsonMethods._
+
import org.apache.spark.api.java.JavaRDD
+import org.apache.spark.mllib.linalg._
+import org.apache.spark.mllib.util.{Loader, Saveable}
+import org.apache.spark.mllib.util.Loader._
--- End diff --
Better use `Loader.abc` in the code instead of importing them globally.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4951#discussion_r26088743
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala ---
@@ -58,4 +66,59 @@ class KMeansModel (val clusterCenters: Array[Vector]) extends Serializable {
private def clusterCentersWithNorm: Iterable[VectorWithNorm] =
clusterCenters.map(new VectorWithNorm(_))
+
+ override def save(sc: SparkContext, path: String): Unit = {
+ KMeansModel.SaveLoadV1_0.save(sc, this, path)
+ }
+
+ override protected def formatVersion: String = "1.0"
+}
+
+object KMeansModel extends Loader[KMeansModel] {
+ override def load(sc: SparkContext, path: String): KMeansModel = {
+ KMeansModel.SaveLoadV1_0.load(sc, path)
+ }
+
+ case class IndexedPoint(id: Int, point: Vector)
--- End diff --
Yes, `Int` is sufficient. This class should be private. Please also check the scope of other classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4951#issuecomment-77861051
[Test build #28393 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28393/consoleFull) for PR 4951 at commit [`dce7055`](https://github.com/apache/spark/commit/dce70553cb0e5c25d1bb0a415929eb5066af964a).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4951#issuecomment-77861968
[Test build #28394 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28394/consoleFull) for PR 4951 at commit [`b144216`](https://github.com/apache/spark/commit/b144216f741776fdfe4c8e95d63650bd46c659d5).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4951#issuecomment-78190589
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28453/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4951#issuecomment-78183339
[Test build #28453 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28453/consoleFull) for PR 4951 at commit [`6dd74a0`](https://github.com/apache/spark/commit/6dd74a0f57678bdbfc6654433047e96ff1801429).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/4951#issuecomment-78216664
LGTM. Merged into master. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4951#issuecomment-77878875
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28394/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4951#issuecomment-77877321
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28393/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4951#issuecomment-77878863
[Test build #28394 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28394/consoleFull) for PR 4951 at commit [`b144216`](https://github.com/apache/spark/commit/b144216f741776fdfe4c8e95d63650bd46c659d5).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class KMeansModel (val clusterCenters: Array[Vector]) extends Saveable with Serializable `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4951#issuecomment-77969794
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28408/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4951#discussion_r26173145
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala ---
@@ -58,4 +66,59 @@ class KMeansModel (val clusterCenters: Array[Vector]) extends Serializable {
private def clusterCentersWithNorm: Iterable[VectorWithNorm] =
clusterCenters.map(new VectorWithNorm(_))
+
+ override def save(sc: SparkContext, path: String): Unit = {
+ KMeansModel.SaveLoadV1_0.save(sc, this, path)
+ }
+
+ override protected def formatVersion: String = "1.0"
+}
+
+object KMeansModel extends Loader[KMeansModel] {
+ override def load(sc: SparkContext, path: String): KMeansModel = {
+ KMeansModel.SaveLoadV1_0.load(sc, path)
+ }
+
+ case class IndexedPoint(id: Int, point: Vector)
--- End diff --
make this private. The class name could be changed to `case class Cluster(id: Int, center: Vector)`. `IndexedPoint` is too general.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4951#discussion_r26173165
--- Diff: mllib/src/test/scala/org/apache/spark/mllib/clustering/KMeansSuite.scala ---
@@ -257,6 +258,54 @@ class KMeansSuite extends FunSuite with MLlibTestSparkContext {
assert(predicts(0) != predicts(3))
}
}
+
+ test("model save/load") {
+ val tempDir = Utils.createTempDir()
+ val path = tempDir.toURI.toString
+
+ Array(true, false).foreach { case selector =>
+ val model = KMeansSuite.createModel(10, 3, selector)
+ // Save model, load it back, and compare.
+ try {
+ model.save(sc, path)
+ val sameModel = KMeansModel.load(sc, path)
+ KMeansSuite.checkEqual(model, sameModel, selector)
+ } finally {
+ Utils.deleteRecursively(tempDir)
+ }
+ }
+ }
+}
+
+object KMeansSuite extends FunSuite {
+ def createModel(dim: Int, k: Int, isSparse: Boolean): KMeansModel = {
+ val singlePoint = isSparse match {
+ case true =>
+ Vectors.sparse(dim, Array.empty[Int], Array.empty[Double])
+ case _ =>
+ Vectors.dense(Array.fill[Double](dim)(0.0))
+ }
+ new KMeansModel(Array.fill[Vector](k)(singlePoint))
+ }
+
+ def checkEqual(a: KMeansModel, b: KMeansModel, isSparse: Boolean): Unit = {
+ assert(a.k === b.k)
+ isSparse match {
+ case true =>
--- End diff --
I suggest removing `isSparse` and using the following:
~~~scala
a.clusterCenters.zip(b.clusterCenters).foreach {
case (ca: SparseVector, cb: SparseVector) =>
assert(ca === cb)
case (ca: DenseVector, cb: DenseVector) =>
assert(ca === cb)
case _ =>
throw ...
}
~~~
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5986][MLLib] Add save/load for k-means
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4951#discussion_r26062595
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala ---
@@ -58,4 +65,49 @@ class KMeansModel (val clusterCenters: Array[Vector]) extends Serializable {
private def clusterCentersWithNorm: Iterable[VectorWithNorm] =
clusterCenters.map(new VectorWithNorm(_))
+
+ override def save(sc: SparkContext, path: String): Unit = {
+ KMeansModel.SaveLoadV1_0.save(sc, this, path)
+ }
+
+ override protected def formatVersion: String = "1.0"
+}
+
+object KMeansModel extends Loader[KMeansModel] {
+ override def load(sc: SparkContext, path: String): KMeansModel = {
+ KMeansModel.SaveLoadV1_0.load(sc, path)
+ }
+
+ private[clustering]
+ object SaveLoadV1_0 {
+
+ private val thisFormatVersion = "1.0"
+
+ private[clustering]
+ val thisClassName = "org.apache.spark.mllib.clustering.KMeansModel"
+
+ def save(sc: SparkContext, model: KMeansModel, path: String): Unit = {
+ val sqlContext = new SQLContext(sc)
+ val wrapper = new VectorUDT()
+ val metadata = compact(render(
+ ("class" -> thisClassName) ~ ("version" -> thisFormatVersion) ~ ("k" -> model.k)))
+ sc.parallelize(Seq(metadata), 1).saveAsTextFile(Loader.metadataPath(path))
+ val dataRDD = sc.parallelize(model.clusterCenters).map(wrapper.serialize)
--- End diff --
* We don't need `wrapper.serialize` for vectors.
* We need to store cluster indices. If the centers are saved to more than one partitions, we cannot easily load them back in the original order.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org