You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/19 04:00:30 UTC
[GitHub] [spark] AngersZhuuuu opened a new pull request #31240: [SPARK-34121][SQL] Intersect operator missing rowCount when CBO enabled
AngersZhuuuu opened a new pull request #31240:
URL: https://github.com/apache/spark/pull/31240
### What changes were proposed in this pull request?
### Why are the changes needed?
Improve query performance, [JoinEstimation.estimateInnerOuterJoin](https://github.com/apache/spark/blob/d6a68e0b67ff7de58073c176dd097070e88ac831/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/JoinEstimation.scala#L55-L156) need the row count.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31240: [SPARK-34121][SQL] Intersect operator missing rowCount when CBO enabled
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31240:
URL: https://github.com/apache/spark/pull/31240#issuecomment-762804220
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38811/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31240: [SPARK-34121][SQL] Intersect operator missing rowCount when CBO enabled
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31240:
URL: https://github.com/apache/spark/pull/31240#issuecomment-762824348
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38811/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31240: [SPARK-34121][SQL] Intersect operator missing rowCount when CBO enabled
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31240:
URL: https://github.com/apache/spark/pull/31240#issuecomment-762605575
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38799/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31240: [SPARK-34121][SQL] Intersect operator missing rowCount when CBO enabled
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31240:
URL: https://github.com/apache/spark/pull/31240#issuecomment-762699395
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134212/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31240: [SPARK-34121][SQL] Intersect operator missing rowCount when CBO enabled
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31240:
URL: https://github.com/apache/spark/pull/31240#issuecomment-763272275
late LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31240: [SPARK-34121][SQL] Intersect operator missing rowCount when CBO enabled
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31240:
URL: https://github.com/apache/spark/pull/31240#issuecomment-762824348
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38811/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #31240: [SPARK-34121][SQL] Intersect operator missing rowCount when CBO enabled
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #31240:
URL: https://github.com/apache/spark/pull/31240#discussion_r560068147
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/BasicStatsPlanVisitor.scala
##########
@@ -57,7 +57,15 @@ object BasicStatsPlanVisitor extends LogicalPlanVisitor[Statistics] {
override def visitGlobalLimit(p: GlobalLimit): Statistics = fallback(p)
- override def visitIntersect(p: Intersect): Statistics = fallback(p)
+ override def visitIntersect(p: Intersect): Statistics = {
+ val stats = p.children.map(_.stats)
+ val rowCount = if (stats.exists(_.rowCount.isEmpty)) {
+ None
+ } else {
+ Some(stats.map(_.rowCount.get).min)
+ }
+ Statistics(sizeInBytes = stats.map(_.sizeInBytes).min, rowCount = rowCount)
Review comment:
We should avoid change the logic, maybe it should be:
```scala
val leftStats = p.left.stats
val rightStats = p.right.stats
val leftSize = leftStats.sizeInBytes
val rightSize = rightStats.sizeInBytes
if (leftSize < rightSize) {
Statistics(sizeInBytes = leftSize, rowCount = leftStats.rowCount)
} else {
Statistics(sizeInBytes = rightSize, rowCount = rightStats.rowCount)
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31240: [SPARK-34121][SQL] Intersect operator missing rowCount when CBO enabled
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #31240:
URL: https://github.com/apache/spark/pull/31240#discussion_r560085505
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/BasicStatsPlanVisitor.scala
##########
@@ -57,7 +57,15 @@ object BasicStatsPlanVisitor extends LogicalPlanVisitor[Statistics] {
override def visitGlobalLimit(p: GlobalLimit): Statistics = fallback(p)
- override def visitIntersect(p: Intersect): Statistics = fallback(p)
+ override def visitIntersect(p: Intersect): Statistics = {
+ val stats = p.children.map(_.stats)
+ val rowCount = if (stats.exists(_.rowCount.isEmpty)) {
+ None
+ } else {
+ Some(stats.map(_.rowCount.get).min)
+ }
+ Statistics(sizeInBytes = stats.map(_.sizeInBytes).min, rowCount = rowCount)
Review comment:
yea, done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31240: [SPARK-34121][SQL] Intersect operator missing rowCount when CBO enabled
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31240:
URL: https://github.com/apache/spark/pull/31240#issuecomment-762589102
**[Test build #134212 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134212/testReport)** for PR 31240 at commit [`0575c51`](https://github.com/apache/spark/commit/0575c512551a058069f46d59462e409b566179cb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31240: [SPARK-34121][SQL] Intersect operator missing rowCount when CBO enabled
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31240:
URL: https://github.com/apache/spark/pull/31240#issuecomment-762768259
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134224/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #31240: [SPARK-34121][SQL] Intersect operator missing rowCount when CBO enabled
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #31240:
URL: https://github.com/apache/spark/pull/31240
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31240: [SPARK-34121][SQL] Intersect operator missing rowCount when CBO enabled
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31240:
URL: https://github.com/apache/spark/pull/31240#issuecomment-762597699
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38799/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31240: [SPARK-34121][SQL] Intersect operator missing rowCount when CBO enabled
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31240:
URL: https://github.com/apache/spark/pull/31240#issuecomment-762790865
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38811/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31240: [SPARK-34121][SQL] Intersect operator missing rowCount when CBO enabled
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31240:
URL: https://github.com/apache/spark/pull/31240#issuecomment-762699395
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134212/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31240: [SPARK-34121][SQL] Intersect operator missing rowCount when CBO enabled
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31240:
URL: https://github.com/apache/spark/pull/31240#issuecomment-762678690
**[Test build #134212 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134212/testReport)** for PR 31240 at commit [`0575c51`](https://github.com/apache/spark/commit/0575c512551a058069f46d59462e409b566179cb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31240: [SPARK-34121][SQL] Intersect operator missing rowCount when CBO enabled
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31240:
URL: https://github.com/apache/spark/pull/31240#issuecomment-762768259
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134224/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31240: [SPARK-34121][SQL] Intersect operator missing rowCount when CBO enabled
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31240:
URL: https://github.com/apache/spark/pull/31240#issuecomment-762619709
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38799/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31240: [SPARK-34121][SQL] Intersect operator missing rowCount when CBO enabled
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31240:
URL: https://github.com/apache/spark/pull/31240#issuecomment-763250899
Merged to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31240: [SPARK-34121][SQL] Intersect operator missing rowCount when CBO enabled
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31240:
URL: https://github.com/apache/spark/pull/31240#issuecomment-762589102
**[Test build #134212 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134212/testReport)** for PR 31240 at commit [`0575c51`](https://github.com/apache/spark/commit/0575c512551a058069f46d59462e409b566179cb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31240: [SPARK-34121][SQL] Intersect operator missing rowCount when CBO enabled
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31240:
URL: https://github.com/apache/spark/pull/31240#issuecomment-762619709
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38799/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org