You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Jefffrey (via GitHub)" <gi...@apache.org> on 2024/04/07 07:56:58 UTC
Re: [I] Improve create_initial_plan to get rid of "stack overflow" issues on complex queries [arrow-datafusion]
Jefffrey commented on issue #9573:
URL: https://github.com/apache/arrow-datafusion/issues/9573#issuecomment-2041357274
I took at shot at implementing something that might fix this, but one roadblock I am facing is currently this logic for the joins:
https://github.com/apache/arrow-datafusion/blob/1a002bccd420ff91ec149ee1ba9c42061510f906/datafusion/core/src/physical_planner.rs#L975-L989
It seems like this is a special case where join could be mutated before it is then transformed into its physical form, which doesn't sit right with what I'm attempting, and I feel doesn't sit right with the rest of the logic happening in the existing function.
I wonder if we should work on extracting this out first (maybe to the optimizer?) before then proceeding with a refactor of `create_initial_plan` :thinking:
This way `create_initial_plan` can assume the passed in `LogicalPlan` is final and simply needs conversion to physical form, and not need to deal with mutating the input plan any more than is required for this 1-1 conversion :thinking:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org