You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/05/30 23:34:19 UTC
[GitHub] [spark] shahidki31 opened a new pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
shahidki31 opened a new pull request #32704:
URL: https://github.com/apache/spark/pull/32704
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error message, please read the guideline first:
https://spark.apache.org/error-message-guidelines.html
-->
### What changes were proposed in this pull request?
Explain cost command in spark currently doesn't show statistics for all the nodes. It misses some nodes in almost all the TPCDS queries.
In this PR, we are collecting all the plan nodes including the subqueries and computing the statistics for each node, if it doesn't exists in stats cache,
### Why are the changes needed?
**Before Fix**
For eg: Query1,
Query15, Aggregate node doesn't have statistics
**After Fix:**
Query1:
Query 15:
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manual testing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32704:
URL: https://github.com/apache/spark/pull/32704#discussion_r642172184
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
##########
@@ -256,13 +255,9 @@ class QueryExecution(
// trigger to compute stats for logical plans
try {
- optimizedPlan.foreach(_.expressions.foreach(_.foreach {
- case subqueryExpression: SubqueryExpression =>
- // trigger subquery's child plan stats propagation
- subqueryExpression.plan.stats
- case _ =>
- }))
- optimizedPlan.stats
+ optimizedPlan.collectWithSubqueries {
Review comment:
cc @cloud-fan @maryannxue FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] shahidki31 commented on a change in pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
shahidki31 commented on a change in pull request #32704:
URL: https://github.com/apache/spark/pull/32704#discussion_r642279415
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
##########
@@ -256,13 +255,9 @@ class QueryExecution(
// trigger to compute stats for logical plans
try {
- optimizedPlan.foreach(_.expressions.foreach(_.foreach {
- case subqueryExpression: SubqueryExpression =>
- // trigger subquery's child plan stats propagation
- subqueryExpression.plan.stats
- case _ =>
- }))
- optimizedPlan.stats
+ optimizedPlan.collectWithSubqueries {
+ case plan => plan.stats
Review comment:
Yes, right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851403230
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43627/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851337265
**[Test build #139103 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139103/testReport)** for PR 32704 at commit [`b78a9c9`](https://github.com/apache/spark/commit/b78a9c96d342f4f2c38929ce2dc4860a98a7f984).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #32704:
URL: https://github.com/apache/spark/pull/32704
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851080849
I don't know enough to review this, sorry
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851332155
**[Test build #139103 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139103/testReport)** for PR 32704 at commit [`b78a9c9`](https://github.com/apache/spark/commit/b78a9c96d342f4f2c38929ce2dc4860a98a7f984).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851435601
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43623/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] shahidki31 commented on a change in pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
shahidki31 commented on a change in pull request #32704:
URL: https://github.com/apache/spark/pull/32704#discussion_r642265510
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
##########
@@ -256,13 +255,9 @@ class QueryExecution(
// trigger to compute stats for logical plans
try {
- optimizedPlan.foreach(_.expressions.foreach(_.foreach {
Review comment:
@cloud-fan I have made changes like above and I think this doesn't fix the issue. Issue here is not all the nodes have stats in statsCache.
This is from query3, which doesn't have subqueries, The aggregate node doesn't show stats with the above changes, but fix with the PR change
```
GlobalLimit 100, Statistics(sizeInBytes=2.9 KiB, rowCount=68)
+- LocalLimit 100, Statistics(sizeInBytes=3.5 KiB, rowCount=68)
+- Sort [d_year#9 ASC NULLS FIRST, sum_agg#89 DESC NULLS LAST, brand_id#87 ASC NULLS FIRST], true, Statistics(sizeInBytes=3.5 KiB, rowCount=68)
+- Aggregate [d_year#9, i_brand#62, i_brand_id#61], [d_year#9, i_brand_id#61 AS brand_id#87, i_brand#62 AS brand#88, MakeDecimal(sum(UnscaledValue(ss_ext_sales_price#45)),17,2) AS sum_agg#89]
+- Project [d_year#9, ss_ext_sales_price#45, i_brand_id#61, i_brand#62], Statistics(sizeInBytes=14.0 MiB, rowCount=2.77E+5)
+- Join Inner, (d_date_sk#3 = ss_sold_date_sk#53), Statistics(sizeInBytes=16.1 MiB, rowCount=2.77E+5)
:- Project [ss_ext_sales_price#45, ss_sold_date_sk#53, i_brand_id#61, i_brand#62], Statistics(sizeInBytes=14.0 MiB, rowCount=2.77E+5)
......
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851097981
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43607/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851097981
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43607/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851403230
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43627/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851149791
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139086/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851428027
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43623/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] shahidki31 commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
shahidki31 commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851346445
Retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851435601
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43623/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851337299
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139103/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] shahidki31 commented on a change in pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
shahidki31 commented on a change in pull request #32704:
URL: https://github.com/apache/spark/pull/32704#discussion_r642307807
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
##########
@@ -256,13 +255,9 @@ class QueryExecution(
// trigger to compute stats for logical plans
try {
- optimizedPlan.foreach(_.expressions.foreach(_.foreach {
- case subqueryExpression: SubqueryExpression =>
- // trigger subquery's child plan stats propagation
- subqueryExpression.plan.stats
- case _ =>
- }))
- optimizedPlan.stats
+ optimizedPlan.collectWithSubqueries {
+ case plan => plan.stats
Review comment:
Yes, I added a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851395499
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43627/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851081580
**[Test build #139086 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139086/testReport)** for PR 32704 at commit [`f1eced2`](https://github.com/apache/spark/commit/f1eced2782ec742d2dd04a122f15a4bf47ef237d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851092855
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43607/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan edited a comment on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
cloud-fan edited a comment on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851597160
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851178214
Does this fix nested subqueries?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] shahidki31 commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
shahidki31 commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851608659
Thanks a lot @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851507041
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139107/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851507041
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139107/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851597160
thanks, merging to master/3.1!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851092136
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43607/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32704:
URL: https://github.com/apache/spark/pull/32704#discussion_r642244548
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
##########
@@ -256,13 +255,9 @@ class QueryExecution(
// trigger to compute stats for logical plans
try {
- optimizedPlan.foreach(_.expressions.foreach(_.foreach {
Review comment:
how about
```
// trigger subquery's child plan stats propagation
optimizedPlan.subqueriesAll.foreah(_.stats)
optimizedPlan.stats
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851371944
**[Test build #139107 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139107/testReport)** for PR 32704 at commit [`b78a9c9`](https://github.com/apache/spark/commit/b78a9c96d342f4f2c38929ce2dc4860a98a7f984).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32704:
URL: https://github.com/apache/spark/pull/32704#discussion_r642277656
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
##########
@@ -256,13 +255,9 @@ class QueryExecution(
// trigger to compute stats for logical plans
try {
- optimizedPlan.foreach(_.expressions.foreach(_.foreach {
- case subqueryExpression: SubqueryExpression =>
- // trigger subquery's child plan stats propagation
- subqueryExpression.plan.stats
- case _ =>
- }))
- optimizedPlan.stats
+ optimizedPlan.collectWithSubqueries {
+ case plan => plan.stats
Review comment:
I see, so this basically triggers stats calculation for every node in the plan tree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] shahidki31 commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
shahidki31 commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851080011
cc @HyukjinKwon @maropu @cloud-fan @srowen Kindly review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851332155
**[Test build #139103 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139103/testReport)** for PR 32704 at commit [`b78a9c9`](https://github.com/apache/spark/commit/b78a9c96d342f4f2c38929ce2dc4860a98a7f984).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851505443
**[Test build #139107 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139107/testReport)** for PR 32704 at commit [`b78a9c9`](https://github.com/apache/spark/commit/b78a9c96d342f4f2c38929ce2dc4860a98a7f984).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] shahidki31 commented on a change in pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
shahidki31 commented on a change in pull request #32704:
URL: https://github.com/apache/spark/pull/32704#discussion_r642268386
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
##########
@@ -256,13 +255,9 @@ class QueryExecution(
// trigger to compute stats for logical plans
try {
- optimizedPlan.foreach(_.expressions.foreach(_.foreach {
Review comment:
Basically performing `optimizedPlan.stats` doesn't update `statsCache` for all the nodes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851337299
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139103/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851081580
**[Test build #139086 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139086/testReport)** for PR 32704 at commit [`f1eced2`](https://github.com/apache/spark/commit/f1eced2782ec742d2dd04a122f15a4bf47ef237d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851149228
**[Test build #139086 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139086/testReport)** for PR 32704 at commit [`f1eced2`](https://github.com/apache/spark/commit/f1eced2782ec742d2dd04a122f15a4bf47ef237d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] shahidki31 commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
shahidki31 commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851181036
@cloud-fan Yes, `collectWithSubqueries` will include nested subqueries as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851376629
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43623/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851371944
**[Test build #139107 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139107/testReport)** for PR 32704 at commit [`b78a9c9`](https://github.com/apache/spark/commit/b78a9c96d342f4f2c38929ce2dc4860a98a7f984).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32704:
URL: https://github.com/apache/spark/pull/32704#issuecomment-851149791
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139086/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32704:
URL: https://github.com/apache/spark/pull/32704#discussion_r642288216
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
##########
@@ -256,13 +255,9 @@ class QueryExecution(
// trigger to compute stats for logical plans
try {
- optimizedPlan.foreach(_.expressions.foreach(_.foreach {
- case subqueryExpression: SubqueryExpression =>
- // trigger subquery's child plan stats propagation
- subqueryExpression.plan.stats
- case _ =>
- }))
- optimizedPlan.stats
+ optimizedPlan.collectWithSubqueries {
+ case plan => plan.stats
Review comment:
Can we add a comment to explain why we need to do it? It's not that obvious to many reviewers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] shahidki31 commented on a change in pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes
Posted by GitBox <gi...@apache.org>.
shahidki31 commented on a change in pull request #32704:
URL: https://github.com/apache/spark/pull/32704#discussion_r642265510
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
##########
@@ -256,13 +255,9 @@ class QueryExecution(
// trigger to compute stats for logical plans
try {
- optimizedPlan.foreach(_.expressions.foreach(_.foreach {
Review comment:
@cloud-fan I have made changes like above and I think this doesn't fix the issue. Issue here is, not all the nodes have stats in statsCache.
This is from query3, which doesn't have subqueries, The aggregate node doesn't show stats with the above changes, but fix with the PR change
```
GlobalLimit 100, Statistics(sizeInBytes=2.9 KiB, rowCount=68)
+- LocalLimit 100, Statistics(sizeInBytes=3.5 KiB, rowCount=68)
+- Sort [d_year#9 ASC NULLS FIRST, sum_agg#89 DESC NULLS LAST, brand_id#87 ASC NULLS FIRST], true, Statistics(sizeInBytes=3.5 KiB, rowCount=68)
+- Aggregate [d_year#9, i_brand#62, i_brand_id#61], [d_year#9, i_brand_id#61 AS brand_id#87, i_brand#62 AS brand#88, MakeDecimal(sum(UnscaledValue(ss_ext_sales_price#45)),17,2) AS sum_agg#89]
+- Project [d_year#9, ss_ext_sales_price#45, i_brand_id#61, i_brand#62], Statistics(sizeInBytes=14.0 MiB, rowCount=2.77E+5)
+- Join Inner, (d_date_sk#3 = ss_sold_date_sk#53), Statistics(sizeInBytes=16.1 MiB, rowCount=2.77E+5)
:- Project [ss_ext_sales_price#45, ss_sold_date_sk#53, i_brand_id#61, i_brand#62], Statistics(sizeInBytes=14.0 MiB, rowCount=2.77E+5)
......
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org