[GitHub] [spark] yeshengm opened a new pull request #25249: [SPARK-28237][SQL] Enforce Idempotence for Once batches in RuleExecutor

[GitBox <gi...@apache.org>]

yeshengm opened a new pull request #25249: [SPARK-28237][SQL] Enforce Idempotence for Once batches in RuleExecutor
URL: https://github.com/apache/spark/pull/25249
 
 
   ## What changes were proposed in this pull request?
   Since in adaptive query processing (AQE), query plans are optimized on the fly during execution, a few `Once` rules can be problematic for such optimization since they can either generate wrong plan/unnecessary intermediate plan nodes.
   
   This PR enforces idempotence for "Once" batches that are supposed to run once. This is a key enabler for AQE re-optimization and can improve robustness for existing optimizer rules.
   
   Once batches that are currently not idempotent are marked in a blacklist. We will submit followup PRs to fix idempotence of these rules.
   
   ## How was this patch tested?
   Existing UTs. Failing Once rules are temporarily blacklisted.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org