You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "anchovYu (via GitHub)" <gi...@apache.org> on 2024/08/20 05:00:04 UTC

[PR] [WIP][SQL] Preempt low priority error on LCA until end of check analysis to improve error experience [spark]

anchovYu opened a new pull request, #47813:
URL: https://github.com/apache/spark/pull/47813

   ### What changes were proposed in this pull request?
   This PR allows high-priority analysis errors to be thrown first when a low-priority error on LCA (lateral column alias) occurs in the same query. 
   It preempts and stores the LCA error during check analysis. If another error happens, it will be thrown first; If not, the LCA error will be thrown at the end of the analysis.
   
   ### Why are the changes needed?
   This helps improve the user experience on errors, to provide higher priority information and the root cause to users. 
   The test in the PR is one case that UNRESOLVED_COLUMN should be thrown, instead of the LCA internal error, which was thrown without the change because the segment sits in the CTE definition that is traversed first in CheckAnalysis.
   
   ### Does this PR introduce _any_ user-facing change?
   No.
   
   
   ### How was this patch tested?
   A new test case.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [WIP][SPARK-49318][SQL] Preempt low priority error on LCA until end of check analysis to improve error experience [spark]

Posted by "anchovYu (via GitHub)" <gi...@apache.org>.
anchovYu commented on code in PR #47813:
URL: https://github.com/apache/spark/pull/47813#discussion_r1724480707


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala:
##########
@@ -57,6 +57,11 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog with QueryErrorsB
   val DATA_TYPE_MISMATCH_ERROR = TreeNodeTag[Unit]("dataTypeMismatchError")
   val INVALID_FORMAT_ERROR = TreeNodeTag[Unit]("invalidFormatError")
 
+  // Error that has a lower priority, that are not supposed to throw immediately on triggering,
+  // e.g. certain internal errors. These errors will be thrown at the end of the whole check
+  // analysis process, if no other error occurs.
+  var preemptedError: Option[SparkException] = None

Review Comment:
   I generalize it with a new class that records the 'heap' top. I only keep the top element since at most one error that has the highest priority gets thrown eventually. PTAL!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49318][SQL] Preempt low priority error on LCA until end of check analysis to improve error experience [spark]

Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #47813:
URL: https://github.com/apache/spark/pull/47813#issuecomment-2302920878

   Thanks, merging to master


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49318][SQL] Preempt low priority error on LCA until end of check analysis to improve error experience [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #47813:
URL: https://github.com/apache/spark/pull/47813#discussion_r1730245230


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala:
##########
@@ -114,10 +119,15 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog with QueryErrorsB
   private def checkNotContainingLCA(exprs: Seq[Expression], plan: LogicalPlan): Unit = {

Review Comment:
   What if we just check LCA at the very end of `CheckAnalysis`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [WIP][SPARK-49318][SQL] Preempt low priority error on LCA until end of check analysis to improve error experience [spark]

Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #47813:
URL: https://github.com/apache/spark/pull/47813#discussion_r1723983477


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala:
##########
@@ -57,6 +57,11 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog with QueryErrorsB
   val DATA_TYPE_MISMATCH_ERROR = TreeNodeTag[Unit]("dataTypeMismatchError")
   val INVALID_FORMAT_ERROR = TreeNodeTag[Unit]("invalidFormatError")
 
+  // Error that has a lower priority, that are not supposed to throw immediately on triggering,
+  // e.g. certain internal errors. These errors will be thrown at the end of the whole check
+  // analysis process, if no other error occurs.
+  var preemptedError: Option[SparkException] = None

Review Comment:
   Shall we make this a priority queue?  If there are more such errors, developers can set the priorities.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49318][SQL] Preempt low priority error on LCA until end of check analysis to improve error experience [spark]

Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang closed pull request #47813: [SPARK-49318][SQL] Preempt low priority error on LCA until end of check analysis to improve error experience
URL: https://github.com/apache/spark/pull/47813


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org