You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by WeichenXu123 <gi...@git.apache.org> on 2016/09/03 16:21:08 UTC
[GitHub] spark pull request #14950: [SPARK-17390][ML][MLLib] Optimize MultivariantOnl...
GitHub user WeichenXu123 opened a pull request:
https://github.com/apache/spark/pull/14950
[SPARK-17390][ML][MLLib] Optimize MultivariantOnlineSummerizer by making the summarized target configurable
## What changes were proposed in this pull request?
add a mask parameter to `MultivariantOnlineSummerizer` constructor.
it can be the following values now:
meanMask
varianceMask
minMask
maxMask
numNonZerosMask
so that we can config the summarized targets in the following way:
`new MultivariantOnlineSummerizer(meanMask|varianceMask)`
it represent this summarizer will only compute mean and variance.
## How was this patch tested?
unit test added.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/WeichenXu123/spark optimize_MultivariantOnlineSummerizer
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14950.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14950
----
commit 834bce992bf89e6e2e5886e6ab9e34c3752d8af4
Author: WeichenXu <we...@outlook.com>
Date: 2016-09-01T01:41:03Z
update.
commit dc44bb974f3969ab70032e947dd9e06b87e317db
Author: WeichenXu <we...@outlook.com>
Date: 2016-09-02T19:56:25Z
update
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14950: [SPARK-17390][ML][MLLib] Optimize MultivariantOnlineSumm...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14950
**[Test build #64904 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64904/consoleFull)** for PR 14950 at commit [`be286eb`](https://github.com/apache/spark/commit/be286eb630fbed17ad55f97d0b53d1f25794ed37).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class MultivariateOnlineSummarizer(mask: Int)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14950: [SPARK-17390][ML][MLLib] Optimize MultivariantOnlineSumm...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14950
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14950: [SPARK-17390][ML][MLLib] Optimize MultivariantOnlineSumm...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14950
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64904/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14950: [SPARK-17390][ML][MLLib] Optimize MultivariantOnlineSumm...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14950
**[Test build #64905 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64905/consoleFull)** for PR 14950 at commit [`3029468`](https://github.com/apache/spark/commit/302946821db75117b4ab2346b4b445472ed50eb4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14950: [SPARK-17390][ML][MLLib] Optimize MultivariantOnlineSumm...
Posted by WeichenXu123 <gi...@git.apache.org>.
Github user WeichenXu123 commented on the issue:
https://github.com/apache/spark/pull/14950
@srowen not only cpu cost, if data dimension is big, serialization cost will be big, such as https://github.com/apache/spark/pull/14109
and compute all target seems not proper if we may add more summary targets in the future ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14950: [WIP][SPARK-17390][ML][MLLib] Optimize Multivaria...
Posted by WeichenXu123 <gi...@git.apache.org>.
Github user WeichenXu123 closed the pull request at:
https://github.com/apache/spark/pull/14950
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14950: [SPARK-17390][ML][MLLib] Optimize MultivariantOnlineSumm...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14950
**[Test build #64902 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64902/consoleFull)** for PR 14950 at commit [`dc44bb9`](https://github.com/apache/spark/commit/dc44bb974f3969ab70032e947dd9e06b87e317db).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class MultivariateOnlineSummarizer(mask: Int)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14950: [SPARK-17390][ML][MLLib] Optimize MultivariantOnlineSumm...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14950
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14950: [SPARK-17390][ML][MLLib] Optimize MultivariantOnlineSumm...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14950
**[Test build #64904 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64904/consoleFull)** for PR 14950 at commit [`be286eb`](https://github.com/apache/spark/commit/be286eb630fbed17ad55f97d0b53d1f25794ed37).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14950: [SPARK-17390][ML][MLLib] Optimize MultivariantOnlineSumm...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14950
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14950: [SPARK-17390][ML][MLLib] Optimize MultivariantOnlineSumm...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14950
**[Test build #64902 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64902/consoleFull)** for PR 14950 at commit [`dc44bb9`](https://github.com/apache/spark/commit/dc44bb974f3969ab70032e947dd9e06b87e317db).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14950: [SPARK-17390][ML][MLLib] Optimize MultivariantOnlineSumm...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14950
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64902/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14950: [WIP][SPARK-17390][ML][MLLib] Optimize MultivariantOnlin...
Posted by WeichenXu123 <gi...@git.apache.org>.
Github user WeichenXu123 commented on the issue:
https://github.com/apache/spark/pull/14950
when benchmark is done I will reopen it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14950: [SPARK-17390][ML][MLLib] Optimize MultivariantOnlineSumm...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/14950
Hm how much does this really save? these are pretty cheap sufficient statistics. Now you have to know whether your particular object was configured or not to return the answer you want.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14950: [SPARK-17390][ML][MLLib] Optimize MultivariantOnlineSumm...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14950
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64905/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14950: [SPARK-17390][ML][MLLib] Optimize MultivariantOnlineSumm...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14950
**[Test build #64905 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64905/consoleFull)** for PR 14950 at commit [`3029468`](https://github.com/apache/spark/commit/302946821db75117b4ab2346b4b445472ed50eb4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class MultivariateOnlineSummarizer(mask: Int)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org