You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "ameyc (via GitHub)" <gi...@apache.org> on 2024/07/09 18:46:45 UTC

[I] Deterministic IDs for ExecutionPlan [datafusion]

ameyc opened a new issue, #11364:
URL: https://github.com/apache/datafusion/issues/11364

   ### Is your feature request related to a problem or challenge?
   
   Currently execution plans do not have an id associated with them this makes comparison of metrics across the runs. Additionally we would like to add a UI to our project to display the metrics as well snapshot our plans.
   
   Looking at the code, it seems to me that the the `physical_planner` [does record the node index](https://github.com/apache/datafusion/blob/main/datafusion/core/src/physical_planner.rs#L555C17-L555C30) which is not [passed down to `map_logical_node_to_physical`](https://github.com/apache/datafusion/blob/main/datafusion/core/src/physical_planner.rs#L697-L702).
   
   
   
   ### Describe the solution you'd like
   
   It would quite trivial to pass the node index to ExecutionPlan creation step and add additional field `operator_id`  `plan_id` to ExecutionPlanProperties.
   
   ### Describe alternatives you've considered
   
   We looked into using `task_id` field, but it always seems to be set to None.
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Deterministic IDs for ExecutionPlan [datafusion]

Posted by "ameyc (via GitHub)" <gi...@apache.org>.
ameyc commented on issue #11364:
URL: https://github.com/apache/datafusion/issues/11364#issuecomment-2307954266

   @ozankabak @alamb taking a look at this again as we are working on snapshotting and UI for Denoramlized. It seems to me that the ideal place to add these is in the `PlanProperties` with something like
   
   `pub struct PlanProperties {
       /// See [ExecutionPlanProperties::equivalence_properties]
       pub eq_properties: EquivalenceProperties,
       /// See [ExecutionPlanProperties::output_partitioning]
       pub partitioning: Partitioning,
       /// See [ExecutionPlanProperties::execution_mode]
       pub execution_mode: ExecutionMode,
       /// See [ExecutionPlanProperties::output_ordering]
       output_ordering: Option<LexOrdering>,
       /// See [ExecutionPlanProperties::node_id]
       node_id: Option<usize>,
   }`
   
   with a method on `ExecutionPlan` to set the node_id (as this isnt available when the `compute_properties()` is called on the node at creation time. 
   `
       /// Sets the node_id for this `ExecutionPlan` node if it supports node_ids
       fn with_node_id(&mut self, _node_id: usize) {}
   }
   `
   Once the final physical plan is generated in `create_physical_plan()` after all the optimizer passes, we can then visit all the nodes and call "with_node_id" on them. Does this sound like a reasonable approach?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Deterministic IDs for ExecutionPlan [datafusion]

Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak commented on issue #11364:
URL: https://github.com/apache/datafusion/issues/11364#issuecomment-2308848324

   Overall sounds reasonable, I will circle back tomorrow after discussing with Synnada folks. Maybe we can upstream some code to help.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Deterministic IDs for ExecutionPlan [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #11364:
URL: https://github.com/apache/datafusion/issues/11364#issuecomment-2308790111

   > Once the final physical plan is generated in create_physical_plan() after all the optimizer passes, we can then visit all the nodes and call "with_node_id" on them. Does this sound like a reasonable approach?
   
   I do think this sounds reasonable
   
   Some questions and thoughts:
   1. What would be the default value of id (if `with_node_id` wasn't called)?
   2. You might want to consider an API like `pub fn with_node_id(self: Arc<...>, node_id: usize` if you can as the ExcutionPlan nodes are almost always `Arc`d)
   3.  I would recommend not showing this id in the explain plan if they change / are not deterministic 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Deterministic IDs for ExecutionPlan [datafusion]

Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #11364:
URL: https://github.com/apache/datafusion/issues/11364#issuecomment-2693678321

   Just noticed the related PR was closed, maybe we can continue to discuss the final/general way here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Deterministic IDs for ExecutionPlan [datafusion]

Posted by "ameyc (via GitHub)" <gi...@apache.org>.
ameyc commented on issue #11364:
URL: https://github.com/apache/datafusion/issues/11364#issuecomment-2311635573

   Just cut a PR based on feedback from @alamb 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Deterministic IDs for ExecutionPlan [datafusion]

Posted by "berkaysynnada (via GitHub)" <gi...@apache.org>.
berkaysynnada commented on issue #11364:
URL: https://github.com/apache/datafusion/issues/11364#issuecomment-2696488555

   We'd love to resume iterations on https://github.com/apache/datafusion/pull/12186#issuecomment-2554359314


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Deterministic IDs for ExecutionPlan [datafusion]

Posted by "ameyc (via GitHub)" <gi...@apache.org>.
ameyc commented on issue #11364:
URL: https://github.com/apache/datafusion/issues/11364#issuecomment-2695713929

   would love to actually get this merged in @xudong963 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Deterministic IDs for ExecutionPlan [datafusion]

Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak commented on issue #11364:
URL: https://github.com/apache/datafusion/issues/11364#issuecomment-2311741785

   We have been thinking about this, I will reply in the PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Deterministic IDs for ExecutionPlan [datafusion]

Posted by "mustafasrepo (via GitHub)" <gi...@apache.org>.
mustafasrepo commented on issue #11364:
URL: https://github.com/apache/datafusion/issues/11364#issuecomment-2220122107

   During planning there are couple of physical optimization rules which modify the `ExecutionPlan` (rules can be found in [the link](https://github.com/apache/datafusion/blob/146b679aa19c7749cc73d0c27440419d6498142b/datafusion/core/src/physical_optimizer/optimizer.rs#L78)).
   Hence, I think proper place to generate id for an `ExecutionPlan` is the end of all optimizations where `ExecutionPlan` is finalized (doing this during the conversion from logical plan to physical plan may be premature).
   
   I guess, once `ExecutionPlan` is finalized, we can generate an id for the plan using `TreeNode` API. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Deterministic IDs for ExecutionPlan [datafusion]

Posted by "ameyc (via GitHub)" <gi...@apache.org>.
ameyc commented on issue #11364:
URL: https://github.com/apache/datafusion/issues/11364#issuecomment-2311030720

   > * What would be the default value of id (if `with_node_id` wasn't called)?
   
   so was thinking if this isn't set, simply letting it be None in the default. this will be mostly to account for UserDefined ExecutionPlans that haven't overridden that method.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Deterministic IDs for ExecutionPlan [datafusion]

Posted by "ameyc (via GitHub)" <gi...@apache.org>.
ameyc closed issue #11364: Deterministic IDs for ExecutionPlan
URL: https://github.com/apache/datafusion/issues/11364


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org