Re: [I] Physical optimizer re-runs rules on plans they have already settled [datafusion]

["alamb (via GitHub)" <gi...@apache.org>]

alamb commented on issue #25355:
URL: https://github.com/apache/datafusion/issues/25355#issuecomment-5762888518

   > The physical optimizer runs its rule list exactly once, in a hand-ordered sequence, with no iteration, no convergence detection, and no way to recognize that a rule is being handed a plan it has already settled. 
   
   I think the core problem is that some PhysicalOptimizer passes such as EnforceRequirements as are not actually "optimizers" (make the query faster) they are "enforcers" using you terminology -- that ensure certain invariants are upheld.
   
   The LogicalPlan has a split for the two types of passes
   - [`OptimizerRule`](https://docs.rs/datafusion/latest/datafusion/optimizer/trait.OptimizerRule.html):  passes to make the plan correct (e.g. type coercion)
   - [`AnalyzerRule`](https://docs.rs/datafusion/latest/datafusion/optimizer/trait.AnalyzerRule.html) : passes to make the plan faster (e.g. push filters down)
   
   > AnalyzerRules are different than an [OptimizerRule](https://docs.rs/datafusion/latest/datafusion/optimizer/trait.OptimizerRule.html)s which must preserve the semantics of the LogicalPlan, while computing results in a more optimal way.
   
   
   So thus, I would propose an alternate strategy here:
   1. Pull the "enforcement" rules into a new trait (perhaps `PhysicalAnalyzerRule` to mirror the Logical plan terminology)
   
   Then we would be in a position to rerun the PhysicalOptimizer (not analyzer) rules multiple times, checking for convergence, etc. 
   
   And our optimizers would be consistent and easy to explain
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org