You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "askalt (via GitHub)" <gi...@apache.org> on 2025/02/18 07:40:39 UTC

Re: [I] Prepared physical plan reusage [datafusion]

askalt commented on issue #14342:
URL: https://github.com/apache/datafusion/issues/14342#issuecomment-2664835375

   @alamb 
   
   Sorry for delay from my side. I investigated plan re-use question again and what I noticed.
   
   - There is also a problem with nodes state `shared across partitions` and for example, initialized once per execution. For example, take a look at:
   https://github.com/apache/datafusion/blob/19fe44cf2f30cbdd63d4a4f52c74055163c6cc38/datafusion/physical-plan/src/joins/hash_join.rs#L340
   Join node keeps a future that collects a build side. 
   
   And, for example, recursive query executor solves this problem by copying a recursive term physical plan each iteration. 
   
   https://github.com/apache/datafusion/blob/19fe44cf2f30cbdd63d4a4f52c74055163c6cc38/datafusion/physical-plan/src/recursive_query.rs#L371-L389
   
   So, there a 3 problems that need to be solved to reuse plans and make a reusing theoretically useful (physical placeholders):
   
   1) How `ExecutionPlan` will resolve placeholders?
   
   As there is a single `params` set we can keep it inside `TaskContext` and fill when executions begins. Each `*Exec` is responsible to resolve placeholders in itself physical expressions.   
   For example:
   - We project a "x + $1" from t
   - `ProjectionExec` on `execute(self, task_ctx)` stage takes a param values from `task_ctx` and rewrite itself expressions to pass already resolved expressions to the stream. 
   - Code to look https://github.com/tarantool/datafusion/blob/5eede25b64a333dd991f7d27f301c03993ce760d/datafusion/physical-plan/src/projection.rs#L214-L222
   
   
   2) Where metrics will be stored?
   3) Where `plan state` (like join build side future) will be stored?
   
   To answer these questions I check how pg sql solves these problems. There is an additional tree which nodes called `PlanState`:
   https://github.com/postgres/postgres/blob/217919dd0954f54402e8d0a38cd203a740754077/src/include/nodes/execnodes.h#L1144-L1236
   
   Each `PlanState` node keeps a pointer to the corresponding plan node (in our situtation is `Arc<dyn ExecutionPlan>`) and an optional runtime stats (`ExecutionPlanMetricSet in our situation). We can create a similar structure that will repeat a form of `ExecutionPlan` and keeps Arcs to the nodes. 
   
   ```
      AggregateExecState ------->    AggregateExec
                 |                          |                  
      ProjectionExecState  -------->   ProjectionExec
                 |                          |
                ...                       ...
   ```
   
   So we can keep `*ExecState` cheap and no think about a weight of `*Exec` cloning. 
   
   When execute an `*Exec` we firstly build a "state" tree and then pass pointers to the corresponding `*ExecState` nodes to the `execute(...)` of the node. 
   
   To collect metrics we visit a state tree. In this state tree we can keep any additional per execution state.
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org