[GitHub] [arrow-datafusion] Dandandan commented on issue #1972: DataFusion Optimizer framework discussion

[GitBox <gi...@apache.org>]

Dandandan commented on issue #1972:
URL: https://github.com/apache/arrow-datafusion/issues/1972#issuecomment-1069557904


   I believe ithat we really should continue experimenting with egg as a optimization framework. This doesn't mean we should move everything to the egg framework, but we can have one or multiple egg-based passes.
   
   Some nice things that make `egg` (besides written in Rust) a nice candidate for DataFusion:
   
   * Optimized algorithm hard to match with manual written optimization passes
   * Easier and less verbose to add simple rules
   * Plugin framework to add slightly more complex optimizations
   * Does not depend on rule order and combined with being able to apply multiple rules in one pass and until convergence, can optimize further than currently is possible (or what would be possible using a optimization strategy like Apache Spark)
   * Cost-based optimization easy to add (this is a native feature)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org