You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/01 04:48:45 UTC
[GitHub] [spark] gaoyajun02 opened a new pull request #33884: [SPARK-36339] Add the option to use physicalStats in AQE
gaoyajun02 opened a new pull request #33884:
URL: https://github.com/apache/spark/pull/33884
### What changes were proposed in this pull request?
Add the option to use physicalStats in AQE
### Why are the changes needed?
Currently when AQE's queryStage is not materialized, it uses the stats of the logical plan to estimate whether the plan can be converted to BHJ, and in some scenarios the estimated value is several orders of magnitude larger than the actual broadcast data, which can lead to large tables being broadcast
### Does this PR introduce _any_ user-facing change?
Yes, a new config spark.sql.adaptive.broadcastJoin.usePhysicalStats.enabled added.
### How was this patch tested?
(TODO)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaoyajun02 closed pull request #33884: [SPARK-36339] Add the option to use physicalStats in AQE
Posted by GitBox <gi...@apache.org>.
gaoyajun02 closed pull request #33884:
URL: https://github.com/apache/spark/pull/33884
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaoyajun02 closed pull request #33884: [SPARK-36339] Add the option to use physicalStats in AQE
Posted by GitBox <gi...@apache.org>.
gaoyajun02 closed pull request #33884:
URL: https://github.com/apache/spark/pull/33884
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org