You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Joseph K. Bradley (JIRA)" <ji...@apache.org> on 2015/02/28 00:00:07 UTC
[jira] [Commented] (SPARK-6068) KMeans Parallel test may fail
[ https://issues.apache.org/jira/browse/SPARK-6068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14340980#comment-14340980 ]
Joseph K. Bradley commented on SPARK-6068:
------------------------------------------
This kind of failure seems very unlikely in practice. For the test suite, it would be reasonable to use a few fixed random seeds as in some other test suites: [https://github.com/apache/spark/blob/master/mllib/src/test/scala/org/apache/spark/mllib/clustering/GaussianMixtureSuite.scala#L40]
If this is really an issue in practice, then (1) seems best. However, I would vote against it as not worth it for realistic settings, especially since kmeans Parallel is only used for initialization.
> KMeans Parallel test may fail
> -----------------------------
>
> Key: SPARK-6068
> URL: https://issues.apache.org/jira/browse/SPARK-6068
> Project: Spark
> Issue Type: Bug
> Components: MLlib
> Affects Versions: 1.2.1
> Reporter: Derrick Burns
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The test "k-means|| initialization in KMeansSuite can fail when the random number generator is truly random.
> The test is predicated on the assumption that each round of K-Means || will add at least one new cluster center. The current implementation of K-Means || adds 2*k cluster centers with high probability. However, there is no deterministic lower bound on the number of cluster centers added.
> Choices are:
> 1) change the KMeans || implementation to iterate on selecting points until it has satisfied a lower bound on the number of points chosen.
> 2) eliminate the test
> 3) ignore the problem and depend on the random number generator to sample the space in a lucky manner.
> Option (1) is most in keeping with the contract that KMeans || should provide a precise number of cluster centers when possible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org