You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "chenfengbin2009 (via GitHub)" <gi...@apache.org> on 2024/08/28 11:50:10 UTC
[PR] To solve the issue of generating excessively large execution plans when encountering multiple levels of subqueries while enabling DynamicPartitionPruning. [spark]
chenfengbin2009 opened a new pull request, #47911:
URL: https://github.com/apache/spark/pull/47911
[SPARK-48486][CORE] Instead of PartitionPruning#prune is expected
### What changes were proposed in this pull request?
The code of method: prune in class: org.apache.spark.sql.execution.dynamicpruning.PartitionPruning is modified. And class: org.apache.spark.sql.internal.SQLConf has added a property value: DYNAMIC_PARTITION_PRUNING_MAX_LENGTH and a dynamicPartitionPruningMaxLength method.
### Why are the changes needed?
In order to address the issue of Spark's DynamicPartitionPruning generating plans with excessive character length and causing excessive memory usage leading to OOM errors when encountering too many levels of subqueries.
### Does this PR introduce _any_ user-facing change?
yes
When DYNAMIC_PARTITION_PRUNING_ENABLED is set to true and DPP optimization is enabled, the generated execution plan may trigger a potential issue.
### How was this patch tested?
I have provided test cases as much as possible.
### Was this patch authored or co-authored using generative AI tooling?
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] To solve the issue of generating excessively large execution plans when encountering multiple levels of subqueries while enabling DynamicPartitionPruning. [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #47911:
URL: https://github.com/apache/spark/pull/47911#issuecomment-2316972323
Mind filing a JIRA please? See also https://spark.apache.org/contributing.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48486][SQL] To solve the issue of generating excessively large execution plans when encountering multiple levels of subqueries while enabling DynamicPartitionPruning. [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #47911:
URL: https://github.com/apache/spark/pull/47911#issuecomment-2526544187
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] To solve the issue of generating excessively large execution plans when encountering multiple levels of subqueries while enabling DynamicPartitionPruning. [spark]
Posted by "chenfengbin2009 (via GitHub)" <gi...@apache.org>.
chenfengbin2009 commented on code in PR #47911:
URL: https://github.com/apache/spark/pull/47911#discussion_r1737590126
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -351,6 +351,14 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val DYNAMIC_PARTITION_PRUNING_MAX_LENGTH =
+ buildConf("spark.sql.optimizer.dynamicPartitionPruning.max.length")
+ .doc("The maximum character length of the plan for the nested subquery.When the set length " +
+ "is too large, it may cause the final plan to exceed the limit or memory overflow.")
+ .version("3.3.0")
Review Comment:
The adapted version has been changed from 3.3.0 to 4.0.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] To solve the issue of generating excessively large execution plans when encountering multiple levels of subqueries while enabling DynamicPartitionPruning. [spark]
Posted by "chenfengbin2009 (via GitHub)" <gi...@apache.org>.
chenfengbin2009 commented on code in PR #47911:
URL: https://github.com/apache/spark/pull/47911#discussion_r1736425963
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -351,6 +351,14 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val DYNAMIC_PARTITION_PRUNING_MAX_LENGTH =
+ buildConf("spark.sql.optimizer.dynamicPartitionPruning.max.length")
+ .doc("The maximum character length of the plan for the nested subquery.When the set length " +
+ "is too large, it may cause the final plan to exceed the limit or memory overflow.")
+ .version("3.3.0")
Review Comment:
Your suggestion is very good. I will modify the relevant content and submit it again. Do you have any better suggestions for the modified code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48486][SQL] To solve the issue of generating excessively large execution plans when encountering multiple levels of subqueries while enabling DynamicPartitionPruning. [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #47911: [SPARK-48486][SQL] To solve the issue of generating excessively large execution plans when encountering multiple levels of subqueries while enabling DynamicPartitionPruning.
URL: https://github.com/apache/spark/pull/47911
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] To solve the issue of generating excessively large execution plans when encountering multiple levels of subqueries while enabling DynamicPartitionPruning. [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #47911:
URL: https://github.com/apache/spark/pull/47911#discussion_r1735763223
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -351,6 +351,14 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val DYNAMIC_PARTITION_PRUNING_MAX_LENGTH =
+ buildConf("spark.sql.optimizer.dynamicPartitionPruning.max.length")
+ .doc("The maximum character length of the plan for the nested subquery.When the set length " +
+ "is too large, it may cause the final plan to exceed the limit or memory overflow.")
+ .version("3.3.0")
Review Comment:
Let's make it 4.0.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48486][SQL] To solve the issue of generating excessively large execution plans when encountering multiple levels of subqueries while enabling DynamicPartitionPruning. [spark]
Posted by "nikhil-zlai (via GitHub)" <gi...@apache.org>.
nikhil-zlai commented on PR #47911:
URL: https://github.com/apache/spark/pull/47911#issuecomment-4735214312
any updates on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org