You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "mihailoale-db (via GitHub)" <gi...@apache.org> on 2025/01/15 14:51:32 UTC
[PR] [SPARK-50830] [SQL] Return single-pass result as the dual run analysis result [spark]
mihailoale-db opened a new pull request, #49509:
URL: https://github.com/apache/spark/pull/49509
### What changes were proposed in this pull request?
Right now fixed-point resolved plan is return as a result of analysis of the unresolved plan. In this task we add a flag in order to guard returning of the single-pass result in specific cases so it can be reverted more easily + return single-pass resolver result as the result of the dual-run analysis.
### Why are the changes needed?
Goal is to have single-pass resolver result as the global one (analysis result of an unresolved plan). We are introducing it now but also adding a flag so it can be reverted easily.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests with the `ANALYZER_DUAL_RUN_LEGACY_AND_SINGLE_PASS_RESOLVER` value set to true.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50830] [SQL] Return single-pass result as the dual run analysis result [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #49509:
URL: https://github.com/apache/spark/pull/49509#discussion_r1917521347
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -276,6 +276,17 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val ANALYZER_DUAL_RUN_RETURN_SINGLE_PASS_RESULT =
+ buildConf("spark.sql.analyzer.singlePassResolver.returnSinglePassResultInDualRun")
+ .internal()
+ .doc(
+ "When true, return the result of the single-pass resolver as the result of the dual run" +
+ "analysis. Otherwise, return the result of the fixed-point analyzer."
Review Comment:
This is only effective when the dual run mode is enabled, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50830] [SQL] Return single-pass result as the dual run analysis result [spark]
Posted by "mihailoale-db (via GitHub)" <gi...@apache.org>.
mihailoale-db commented on code in PR #49509:
URL: https://github.com/apache/spark/pull/49509#discussion_r1918084982
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -276,6 +276,17 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val ANALYZER_DUAL_RUN_RETURN_SINGLE_PASS_RESULT =
+ buildConf("spark.sql.analyzer.singlePassResolver.returnSinglePassResultInDualRun")
+ .internal()
+ .doc(
+ "When true, return the result of the single-pass resolver as the result of the dual run" +
+ "analysis. Otherwise, return the result of the fixed-point analyzer."
Review Comment:
I updated the comment. Please check it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50830] [SQL] Return single-pass result as the dual run analysis result [spark]
Posted by "vladimirg-db (via GitHub)" <gi...@apache.org>.
vladimirg-db commented on PR #49509:
URL: https://github.com/apache/spark/pull/49509#issuecomment-2595125771
LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50830] [SQL] Return single-pass result as the dual run analysis result [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #49509: [SPARK-50830] [SQL] Return single-pass result as the dual run analysis result
URL: https://github.com/apache/spark/pull/49509
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50830] [SQL] Return single-pass result as the dual run analysis result [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #49509:
URL: https://github.com/apache/spark/pull/49509#discussion_r1918032701
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -276,6 +276,17 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val ANALYZER_DUAL_RUN_RETURN_SINGLE_PASS_RESULT =
+ buildConf("spark.sql.analyzer.singlePassResolver.returnSinglePassResultInDualRun")
+ .internal()
+ .doc(
+ "When true, return the result of the single-pass resolver as the result of the dual run" +
Review Comment:
```suggestion
"When true, return the result of the single-pass resolver as the result of the dual run " +
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50830] [SQL] Return single-pass result as the dual run analysis result [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #49509:
URL: https://github.com/apache/spark/pull/49509#issuecomment-2595411566
The Spark Connect failure is unrelated, thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org