[GitHub] [arrow-datafusion] isidentical commented on pull request #3787: Join cardinality computation for cost-based nested join optimizations

[GitBox <gi...@apache.org>]

isidentical commented on PR #3787:
URL: https://github.com/apache/arrow-datafusion/pull/3787#issuecomment-1274703585

   Definitely agree with all the points @alamb! The same is true for most of the systems (e.g. compilers/interpreters); once you have the ability to determine a path with real data (JIT compilers), it is always going to be superior / more reliable than a profile-guided static analysis. But as you have already mentioned, doing adaptive (dynamical optimizations) optimizations is something that needs to be planned more carefully (in terms of the overall design of the execution process) and this is a lot more complex than simpler cost based analyzers.
   
   I'd be happy to also prepare a more general document after this PR about the remaining cost estimations, and how we can have a more reliable computation for other foundational operations (filter selectivity, aggregates, etc.). But that doesn't necessarily need to block the work on the adaptiveness part (e.g. one of the things that I'd love to see [and even maybe work on, if I can find some time] is #816, which is a mix of both a CBO and an adaptive optimization which we can (and should) actually do at the runtime).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org