You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by WeichenXu123 <gi...@git.apache.org> on 2016/08/13 07:42:53 UTC
[GitHub] spark pull request #14628: [SPARK-17033][Follow-up][ML][MLLib] Improve kmean...
GitHub user WeichenXu123 opened a pull request:
https://github.com/apache/spark/pull/14628
[SPARK-17033][Follow-up][ML][MLLib] Improve kmean aggregate to treeAggregate
## What changes were proposed in this pull request?
The kmean use `aggregate` to compute points cost. As the data RDD is huge so it is better to use `treeAggregate`.
## How was this patch tested?
Existing test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/WeichenXu123/spark improve_kmean_aggregate
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14628.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14628
----
commit 3edc7e0b9bed770a595f54c83f2e7a4195319a12
Author: WeichenXu <we...@outlook.com>
Date: 2016-08-12T08:20:53Z
update kmean aggregate to treeAggregate
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14628: [SPARK-17033][Follow-up][ML][MLLib] Improve kmean aggreg...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14628
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63718/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14628: [SPARK-17033][Follow-up][ML][MLLib] Improve kmean aggreg...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14628
**[Test build #63718 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63718/consoleFull)** for PR 14628 at commit [`3edc7e0`](https://github.com/apache/spark/commit/3edc7e0b9bed770a595f54c83f2e7a4195319a12).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14628: [SPARK-17050][ML][MLLib] Improve kmean rdd.aggregate to ...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/14628
Awesome thanks for taking the time to do this. A few follow up questions:
1) So this is happening with the default tree depth (2) did you try it with other depths?
2) Have you had a chance to run it with spark-perf or otherwise benchmark the change to validate that it does actually result in the expected performance improvement/does not lead to a regression?
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14628: [SPARK-17033][Follow-up][ML][MLLib] Improve kmean aggreg...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14628
**[Test build #63718 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63718/consoleFull)** for PR 14628 at commit [`3edc7e0`](https://github.com/apache/spark/commit/3edc7e0b9bed770a595f54c83f2e7a4195319a12).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14628: [SPARK-17033][Follow-up][ML][MLLib] Improve kmean aggreg...
Posted by WeichenXu123 <gi...@git.apache.org>.
Github user WeichenXu123 commented on the issue:
https://github.com/apache/spark/pull/14628
@lins05 Ok, give me some time to check whether the one in LDAModel is also proper to use treeAggregate....
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14628: [SPARK-17050][ML][MLLib] Improve kmean rdd.aggregate to ...
Posted by WeichenXu123 <gi...@git.apache.org>.
Github user WeichenXu123 commented on the issue:
https://github.com/apache/spark/pull/14628
@holdenk
I think depth (2) is enough to handle large RDD and bigger depth may add cost. I'll append test result later. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14628: [SPARK-17050][ML][MLLib] Improve kmean rdd.aggreg...
Posted by WeichenXu123 <gi...@git.apache.org>.
Github user WeichenXu123 closed the pull request at:
https://github.com/apache/spark/pull/14628
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14628: [SPARK-17033][Follow-up][ML][MLLib] Improve kmean aggreg...
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on the issue:
https://github.com/apache/spark/pull/14628
A grep shows there is also call to `RDD.aggregate` in `LDAModel`, should we fix it here as well?
```sh
mllib/src/main/scala/org/apache/spark/mllib/clustering]$ git grep -E '\<aggregate\>' |grep -v //
KMeans.scala:417: .aggregate(new Array[Double](runs))(
LDAModel.scala:756: graph.vertices.aggregate(0.0)(seqOp, _ + _)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14628: [SPARK-17033][Follow-up][ML][MLLib] Improve kmean aggreg...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14628
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14628: [SPARK-17050][ML][MLLib] Improve kmean rdd.aggregate to ...
Posted by WeichenXu123 <gi...@git.apache.org>.
Github user WeichenXu123 commented on the issue:
https://github.com/apache/spark/pull/14628
@lins05 I think the LDA model data size usually will be much smaller than training data, so that here it seems no need to change to treeAggregate ( for DistributedLDAModel ) . thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14628: [SPARK-17050][ML][MLLib] Improve kmean rdd.aggregate to ...
Posted by WeichenXu123 <gi...@git.apache.org>.
Github user WeichenXu123 commented on the issue:
https://github.com/apache/spark/pull/14628
because KMeans algo is being optimized by another task I close this PR for now and when that one merged I'll check for whether this need to be optimized.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org