You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by chenghao-intel <gi...@git.apache.org> on 2015/01/26 07:01:26 UTC
[GitHub] spark pull request: [SPARK-5404] [SQL] update the default statisti...
GitHub user chenghao-intel opened a pull request:
https://github.com/apache/spark/pull/4199
[SPARK-5404] [SQL] update the default statistic number
The default statistic of a logical plan node is too aggressive for most of case, and probably doesn't work correctly for most of join order optimization.
e.g. A (100k) JOIN B(50K) RIGHT JOIN C(1G)
Ideally, we expect to get 2 broadcast nested loop joins, however, the statistic of the logical `A JOIN B` is 5 G (=100K * 50K).
This implementation still cannot get the accurate statistic, but perhaps useful for most of join cases.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/chenghao-intel/spark statistic
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4199.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4199
----
commit 52c8fb1b01b1262f86b355d653e3eab1e6296302
Author: Cheng Hao <ha...@intel.com>
Date: 2015-01-26T05:31:09Z
update the default statistic number
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5404] [SQL] update the default statisti...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/4199#issuecomment-72400547
I don't think that I agree with this change. In general it is always safe to do a shuffle join where as a broadcast join could possible cause the driver to OOM. However, I'm worried that this change will make us faster for some workloads but possibly also unstable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5404] [SQL] update the default statisti...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4199#issuecomment-71421339
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26078/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5404] [SQL] update the default statisti...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/4199#issuecomment-72576028
ok, I am closing this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5404] [SQL] update the default statisti...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4199#issuecomment-71417879
[Test build #26078 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26078/consoleFull) for PR 4199 at commit [`52c8fb1`](https://github.com/apache/spark/commit/52c8fb1b01b1262f86b355d653e3eab1e6296302).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5404] [SQL] update the default statisti...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4199#issuecomment-71421334
[Test build #26078 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26078/consoleFull) for PR 4199 at commit [`52c8fb1`](https://github.com/apache/spark/commit/52c8fb1b01b1262f86b355d653e3eab1e6296302).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5404] [SQL] update the default statisti...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel closed the pull request at:
https://github.com/apache/spark/pull/4199
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org