You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2025/08/04 04:51:50 UTC
[PR] Add ExecutionPlan::reset_state [datafusion]
adriangb opened a new pull request, #17028:
URL: https://github.com/apache/datafusion/pull/17028
Fixes #16998. Closes #17016.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add ExecutionPlan::reset_state [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #17028:
URL: https://github.com/apache/datafusion/pull/17028#issuecomment-3161676175
done! https://github.com/apache/datafusion/issues/17060
should we wait for anything else before merging this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add ExecutionPlan::reset_state [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #17028:
URL: https://github.com/apache/datafusion/pull/17028#discussion_r2252606364
##########
datafusion/physical-plan/src/execution_plan.rs:
##########
@@ -194,6 +194,31 @@ pub trait ExecutionPlan: Debug + DisplayAs + Send + Sync {
children: Vec<Arc<dyn ExecutionPlan>>,
) -> Result<Arc<dyn ExecutionPlan>>;
+ /// Reset any internal state within this [`ExecutionPlan`].
+ ///
+ /// This method is called when an [`ExecutionPlan`] needs to be re-executed,
+ /// such as in recursive queries. Unlike [`ExecutionPlan::with_new_children`], this method
+ /// ensures that any stateful components (e.g., [`DynamicFilterPhysicalExpr`])
+ /// are reset to their initial state.
+ ///
+ /// The default implementation simply calls [`ExecutionPlan::with_new_children`] with the existing children,
+ /// effectively creating a new instance of the [`ExecutionPlan`] with the same children but without
+ /// necessarily resetting any internal state. Implementations that require resetting of some
+ /// internal state should override this method to provide the necessary logic.
+ ///
+ /// This method should *not* reset state recursively for children, as it is expected that
+ /// it will be called from within a walk of the execution plan tree so that it will be called on each child later
+ /// or was already called on each child.
+ ///
+ /// Note to implementers: unlike [`ExecutionPlan::with_new_children`] this method does not accept new children as an argument,
+ /// thus it is expected that any cached plan properties will remain valid after the reset.
+ ///
+ /// [`DynamicFilterPhysicalExpr`]: datafusion_physical_expr::expressions::DynamicFilterPhysicalExpr
Review Comment:
I wonder if we should also add a note to DynamicFilterPhysicalExpr saying any ExecutionPlan that uses them should also implement `reset_state`
##########
datafusion/physical-plan/src/sorts/sort.rs:
##########
@@ -1116,10 +1127,46 @@ impl ExecutionPlan for SortExec {
self: Arc<Self>,
children: Vec<Arc<dyn ExecutionPlan>>,
) -> Result<Arc<dyn ExecutionPlan>> {
- let mut new_sort = SortExec::new(self.expr.clone(), Arc::clone(&children[0]))
- .with_fetch(self.fetch)
- .with_preserve_partitioning(self.preserve_partitioning);
- new_sort.filter = self.filter.clone();
+ let mut new_sort = self.cloned();
+ assert!(
+ children.len() == 1,
+ "SortExec should have exactly one child"
+ );
+ new_sort.input = Arc::clone(&children[0]);
+ // Recompute the properties based on the new input since they may have changed.
+ let (cache, sort_prefix) = Self::compute_properties(
+ &new_sort.input,
+ new_sort.expr.clone(),
+ new_sort.preserve_partitioning,
+ )
+ .expect(concat!(
+ "Safety: we had already been calling `compute_properties(...).unwrap()` in `new()` ",
+ "and it seems to be okay",
+ "\n",
+ "We assumed that doing the same thing here directly instead ",
+ "of calling `new()` (as we did before this commit) is also okay but it's possible that ",
+ "implementations have drifted and this is no longer safe even if `new()` still works, ",
+ "for example if `new()` now does something different than just calling `compute_properties(...).unwrap()`",
+ "\n",
+ "This is clearly a bug, please report it!"
Review Comment:
I think we should return [`DatafusionError::Internal`](https://docs.rs/datafusion/latest/datafusion/common/enum.DataFusionError.html#variant.Internal) here rather than panic'ing as it is a better UX (if you want fail fast in debug builds, perhaps you could add a `debug_assert`)
I also recommend converting the explanation into comments and leaving the panic message like "Internal inconsistency in SortExec"
The rationale is that if a user sees this message it is not going to mean anything to them and they can't fix it, and this text will obscure the conclusion (this is a bug they can not do anything to fix). A developer will come to the source location and can read the comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add ExecutionPlan::reset_state [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on code in PR #17028:
URL: https://github.com/apache/datafusion/pull/17028#discussion_r2254131582
##########
datafusion/physical-plan/src/sorts/sort.rs:
##########
@@ -1116,10 +1127,46 @@ impl ExecutionPlan for SortExec {
self: Arc<Self>,
children: Vec<Arc<dyn ExecutionPlan>>,
) -> Result<Arc<dyn ExecutionPlan>> {
- let mut new_sort = SortExec::new(self.expr.clone(), Arc::clone(&children[0]))
- .with_fetch(self.fetch)
- .with_preserve_partitioning(self.preserve_partitioning);
- new_sort.filter = self.filter.clone();
+ let mut new_sort = self.cloned();
+ assert!(
+ children.len() == 1,
+ "SortExec should have exactly one child"
+ );
+ new_sort.input = Arc::clone(&children[0]);
+ // Recompute the properties based on the new input since they may have changed.
+ let (cache, sort_prefix) = Self::compute_properties(
+ &new_sort.input,
+ new_sort.expr.clone(),
+ new_sort.preserve_partitioning,
+ )
Review Comment:
Why don't we put the logic into `reset_state` if the changes are due to the `reset_state`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add ExecutionPlan::reset_state [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #17028:
URL: https://github.com/apache/datafusion/pull/17028#discussion_r2252648667
##########
datafusion/physical-plan/src/execution_plan.rs:
##########
@@ -194,6 +194,31 @@ pub trait ExecutionPlan: Debug + DisplayAs + Send + Sync {
children: Vec<Arc<dyn ExecutionPlan>>,
) -> Result<Arc<dyn ExecutionPlan>>;
+ /// Reset any internal state within this [`ExecutionPlan`].
+ ///
+ /// This method is called when an [`ExecutionPlan`] needs to be re-executed,
+ /// such as in recursive queries. Unlike [`ExecutionPlan::with_new_children`], this method
+ /// ensures that any stateful components (e.g., [`DynamicFilterPhysicalExpr`])
+ /// are reset to their initial state.
+ ///
+ /// The default implementation simply calls [`ExecutionPlan::with_new_children`] with the existing children,
+ /// effectively creating a new instance of the [`ExecutionPlan`] with the same children but without
+ /// necessarily resetting any internal state. Implementations that require resetting of some
+ /// internal state should override this method to provide the necessary logic.
+ ///
+ /// This method should *not* reset state recursively for children, as it is expected that
+ /// it will be called from within a walk of the execution plan tree so that it will be called on each child later
+ /// or was already called on each child.
+ ///
+ /// Note to implementers: unlike [`ExecutionPlan::with_new_children`] this method does not accept new children as an argument,
+ /// thus it is expected that any cached plan properties will remain valid after the reset.
+ ///
+ /// [`DynamicFilterPhysicalExpr`]: datafusion_physical_expr::expressions::DynamicFilterPhysicalExpr
Review Comment:
98b5a7240
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add ExecutionPlan::reset_state [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #17028:
URL: https://github.com/apache/datafusion/pull/17028#discussion_r2254951604
##########
datafusion/physical-plan/src/sorts/sort.rs:
##########
@@ -1116,10 +1127,46 @@ impl ExecutionPlan for SortExec {
self: Arc<Self>,
children: Vec<Arc<dyn ExecutionPlan>>,
) -> Result<Arc<dyn ExecutionPlan>> {
- let mut new_sort = SortExec::new(self.expr.clone(), Arc::clone(&children[0]))
- .with_fetch(self.fetch)
- .with_preserve_partitioning(self.preserve_partitioning);
- new_sort.filter = self.filter.clone();
+ let mut new_sort = self.cloned();
+ assert!(
+ children.len() == 1,
+ "SortExec should have exactly one child"
+ );
+ new_sort.input = Arc::clone(&children[0]);
+ // Recompute the properties based on the new input since they may have changed.
+ let (cache, sort_prefix) = Self::compute_properties(
+ &new_sort.input,
+ new_sort.expr.clone(),
+ new_sort.preserve_partitioning,
+ )
Review Comment:
Sorry I missed your original point
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add ExecutionPlan::reset_state [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #17028:
URL: https://github.com/apache/datafusion/pull/17028#issuecomment-3168366880
It seems like we should probably backport this to branch-49 for the 49.0.1 release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add ExecutionPlan::reset_state [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #17028:
URL: https://github.com/apache/datafusion/pull/17028#discussion_r2254867354
##########
datafusion/physical-plan/src/sorts/sort.rs:
##########
@@ -1116,10 +1127,46 @@ impl ExecutionPlan for SortExec {
self: Arc<Self>,
children: Vec<Arc<dyn ExecutionPlan>>,
) -> Result<Arc<dyn ExecutionPlan>> {
- let mut new_sort = SortExec::new(self.expr.clone(), Arc::clone(&children[0]))
- .with_fetch(self.fetch)
- .with_preserve_partitioning(self.preserve_partitioning);
- new_sort.filter = self.filter.clone();
+ let mut new_sort = self.cloned();
+ assert!(
+ children.len() == 1,
+ "SortExec should have exactly one child"
+ );
+ new_sort.input = Arc::clone(&children[0]);
+ // Recompute the properties based on the new input since they may have changed.
+ let (cache, sort_prefix) = Self::compute_properties(
+ &new_sort.input,
+ new_sort.expr.clone(),
+ new_sort.preserve_partitioning,
+ )
Review Comment:
this function returns a Result though, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add ExecutionPlan::reset_state [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #17028:
URL: https://github.com/apache/datafusion/pull/17028
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add ExecutionPlan::reset_state [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #17028:
URL: https://github.com/apache/datafusion/pull/17028#issuecomment-3155913347
> I recommend putting the PR into upgrading doc.
Good suggestion, added!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add ExecutionPlan::reset_state [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #17028:
URL: https://github.com/apache/datafusion/pull/17028#issuecomment-3156102223
Are we concerned that `reset_state` will disable dynamic the dynamic filter optimization? I wasn't able to cook up an example query, I don't do much recursive queries myself, but I imagine that unless the optimizer runs again after `reset_state` to re-connect dynamic filters to a data source they can't possibly work. Right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add ExecutionPlan::reset_state [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #17028:
URL: https://github.com/apache/datafusion/pull/17028#discussion_r2254932508
##########
datafusion/physical-plan/src/sorts/sort.rs:
##########
@@ -1116,10 +1127,46 @@ impl ExecutionPlan for SortExec {
self: Arc<Self>,
children: Vec<Arc<dyn ExecutionPlan>>,
) -> Result<Arc<dyn ExecutionPlan>> {
- let mut new_sort = SortExec::new(self.expr.clone(), Arc::clone(&children[0]))
- .with_fetch(self.fetch)
- .with_preserve_partitioning(self.preserve_partitioning);
- new_sort.filter = self.filter.clone();
+ let mut new_sort = self.cloned();
+ assert!(
+ children.len() == 1,
+ "SortExec should have exactly one child"
+ );
+ new_sort.input = Arc::clone(&children[0]);
+ // Recompute the properties based on the new input since they may have changed.
+ let (cache, sort_prefix) = Self::compute_properties(
+ &new_sort.input,
+ new_sort.expr.clone(),
+ new_sort.preserve_partitioning,
+ )
Review Comment:
Ah good point yep I'll do that!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add ExecutionPlan::reset_state [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #17028:
URL: https://github.com/apache/datafusion/pull/17028#discussion_r2252643695
##########
datafusion/physical-plan/src/sorts/sort.rs:
##########
@@ -1116,10 +1127,46 @@ impl ExecutionPlan for SortExec {
self: Arc<Self>,
children: Vec<Arc<dyn ExecutionPlan>>,
) -> Result<Arc<dyn ExecutionPlan>> {
- let mut new_sort = SortExec::new(self.expr.clone(), Arc::clone(&children[0]))
- .with_fetch(self.fetch)
- .with_preserve_partitioning(self.preserve_partitioning);
- new_sort.filter = self.filter.clone();
+ let mut new_sort = self.cloned();
+ assert!(
+ children.len() == 1,
+ "SortExec should have exactly one child"
+ );
+ new_sort.input = Arc::clone(&children[0]);
+ // Recompute the properties based on the new input since they may have changed.
+ let (cache, sort_prefix) = Self::compute_properties(
+ &new_sort.input,
+ new_sort.expr.clone(),
+ new_sort.preserve_partitioning,
+ )
+ .expect(concat!(
+ "Safety: we had already been calling `compute_properties(...).unwrap()` in `new()` ",
+ "and it seems to be okay",
+ "\n",
+ "We assumed that doing the same thing here directly instead ",
+ "of calling `new()` (as we did before this commit) is also okay but it's possible that ",
+ "implementations have drifted and this is no longer safe even if `new()` still works, ",
+ "for example if `new()` now does something different than just calling `compute_properties(...).unwrap()`",
+ "\n",
+ "This is clearly a bug, please report it!"
Review Comment:
@alamb this already had the possibility to panic in it because it called `SortExec::new()`:
https://github.com/apache/datafusion/blob/eb2b8c07b2f564fbd9046c67de41ac4ebfe7ae84/datafusion/physical-plan/src/sorts/sort.rs#L865-L872
Returning a `Result` would be a major breaking change to the `ExecutionPlan::with_new_children` API which I don't think we should do in this PR.
I do think `ExecutionPlan::with_new_children` returning a Result would be a good thing. In general I think trait methods should err on the side of returning a result in case some implementation needs to. If none of them do I'd expect compilation to make it pretty much a non issue for performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add ExecutionPlan::reset_state [datafusion]
Posted by "nuno-faria (via GitHub)" <gi...@apache.org>.
nuno-faria commented on code in PR #17028:
URL: https://github.com/apache/datafusion/pull/17028#discussion_r2251238645
##########
datafusion/sqllogictest/test_files/cte.slt:
##########
@@ -996,6 +996,29 @@ physical_plan
08)----------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
09)------------WorkTableExec: name=numbers
+# Test for issue #16998: SortExec shares DynamicFilterPhysicalExpr across multiple executions
+# This should return 5 rows but currently returns only 2 due to the bug
Review Comment:
I think the test comment can now be updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add ExecutionPlan::reset_state [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #17028:
URL: https://github.com/apache/datafusion/pull/17028#discussion_r2252643695
##########
datafusion/physical-plan/src/sorts/sort.rs:
##########
@@ -1116,10 +1127,46 @@ impl ExecutionPlan for SortExec {
self: Arc<Self>,
children: Vec<Arc<dyn ExecutionPlan>>,
) -> Result<Arc<dyn ExecutionPlan>> {
- let mut new_sort = SortExec::new(self.expr.clone(), Arc::clone(&children[0]))
- .with_fetch(self.fetch)
- .with_preserve_partitioning(self.preserve_partitioning);
- new_sort.filter = self.filter.clone();
+ let mut new_sort = self.cloned();
+ assert!(
+ children.len() == 1,
+ "SortExec should have exactly one child"
+ );
+ new_sort.input = Arc::clone(&children[0]);
+ // Recompute the properties based on the new input since they may have changed.
+ let (cache, sort_prefix) = Self::compute_properties(
+ &new_sort.input,
+ new_sort.expr.clone(),
+ new_sort.preserve_partitioning,
+ )
+ .expect(concat!(
+ "Safety: we had already been calling `compute_properties(...).unwrap()` in `new()` ",
+ "and it seems to be okay",
+ "\n",
+ "We assumed that doing the same thing here directly instead ",
+ "of calling `new()` (as we did before this commit) is also okay but it's possible that ",
+ "implementations have drifted and this is no longer safe even if `new()` still works, ",
+ "for example if `new()` now does something different than just calling `compute_properties(...).unwrap()`",
+ "\n",
+ "This is clearly a bug, please report it!"
Review Comment:
@alamb this already had the possibility to panic in it because it called `SortExec::new()`:
https://github.com/apache/datafusion/blob/eb2b8c07b2f564fbd9046c67de41ac4ebfe7ae84/datafusion/physical-plan/src/sorts/sort.rs#L865-L872
https://github.com/apache/datafusion/blob/eb2b8c07b2f564fbd9046c67de41ac4ebfe7ae84/datafusion/physical-plan/src/sorts/sort.rs#L1115-L1119
Returning a `Result` would be a major breaking change to the `ExecutionPlan::with_new_children` API which I don't think we should do in this PR.
I do think `ExecutionPlan::with_new_children` returning a Result would be a good thing. In general I think trait methods should err on the side of returning a result in case some implementation needs to. If none of them do I'd expect compilation to make it pretty much a non issue for performance. But maybe let's do that as it's own PR if we really want to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add ExecutionPlan::reset_state [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #17028:
URL: https://github.com/apache/datafusion/pull/17028#issuecomment-3161606264
> Are we concerned that `reset_state` will disable dynamic the dynamic filter optimization? I wasn't able to cook up an example query, I don't do much recursive queries myself, but I imagine that unless the optimizer runs again after `reset_state` to re-connect dynamic filters to a data source they can't possibly work. Right?
I suggest we file a ticket to look into this in the future for anyone who might be interested in that usecase, and not worry about it in this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add ExecutionPlan::reset_state [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #17028:
URL: https://github.com/apache/datafusion/pull/17028#discussion_r2252643695
##########
datafusion/physical-plan/src/sorts/sort.rs:
##########
@@ -1116,10 +1127,46 @@ impl ExecutionPlan for SortExec {
self: Arc<Self>,
children: Vec<Arc<dyn ExecutionPlan>>,
) -> Result<Arc<dyn ExecutionPlan>> {
- let mut new_sort = SortExec::new(self.expr.clone(), Arc::clone(&children[0]))
- .with_fetch(self.fetch)
- .with_preserve_partitioning(self.preserve_partitioning);
- new_sort.filter = self.filter.clone();
+ let mut new_sort = self.cloned();
+ assert!(
+ children.len() == 1,
+ "SortExec should have exactly one child"
+ );
+ new_sort.input = Arc::clone(&children[0]);
+ // Recompute the properties based on the new input since they may have changed.
+ let (cache, sort_prefix) = Self::compute_properties(
+ &new_sort.input,
+ new_sort.expr.clone(),
+ new_sort.preserve_partitioning,
+ )
+ .expect(concat!(
+ "Safety: we had already been calling `compute_properties(...).unwrap()` in `new()` ",
+ "and it seems to be okay",
+ "\n",
+ "We assumed that doing the same thing here directly instead ",
+ "of calling `new()` (as we did before this commit) is also okay but it's possible that ",
+ "implementations have drifted and this is no longer safe even if `new()` still works, ",
+ "for example if `new()` now does something different than just calling `compute_properties(...).unwrap()`",
+ "\n",
+ "This is clearly a bug, please report it!"
Review Comment:
@alamb this already had the possibility to panic in it because it called `SortExec::new()`:
https://github.com/apache/datafusion/blob/eb2b8c07b2f564fbd9046c67de41ac4ebfe7ae84/datafusion/physical-plan/src/sorts/sort.rs#L865-L872
https://github.com/apache/datafusion/blob/eb2b8c07b2f564fbd9046c67de41ac4ebfe7ae84/datafusion/physical-plan/src/sorts/sort.rs#L1115-L1119
Returning a `Result` would be a major breaking change to the `ExecutionPlan::with_new_children` API which I don't think we should do in this PR.
I do think `ExecutionPlan::with_new_children` returning a Result would be a good thing. In general I think trait methods should err on the side of returning a result in case some implementation needs to. If none of them do I'd expect compilation to make it pretty much a non issue for performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add ExecutionPlan::reset_state [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #17028:
URL: https://github.com/apache/datafusion/pull/17028#discussion_r2254847146
##########
datafusion/physical-plan/src/sorts/sort.rs:
##########
@@ -1116,10 +1127,46 @@ impl ExecutionPlan for SortExec {
self: Arc<Self>,
children: Vec<Arc<dyn ExecutionPlan>>,
) -> Result<Arc<dyn ExecutionPlan>> {
- let mut new_sort = SortExec::new(self.expr.clone(), Arc::clone(&children[0]))
- .with_fetch(self.fetch)
- .with_preserve_partitioning(self.preserve_partitioning);
- new_sort.filter = self.filter.clone();
+ let mut new_sort = self.cloned();
+ assert!(
+ children.len() == 1,
+ "SortExec should have exactly one child"
+ );
+ new_sort.input = Arc::clone(&children[0]);
+ // Recompute the properties based on the new input since they may have changed.
+ let (cache, sort_prefix) = Self::compute_properties(
+ &new_sort.input,
+ new_sort.expr.clone(),
+ new_sort.preserve_partitioning,
+ )
Review Comment:
The logic for this change is:
1. We need to do similar work as `with_new_children` (i.e. clone `SortExec`) but each method has slightly different requirements (`with_new_children` needs to reset `cache` while `reset_state` needs to reset `filter`.
2. To solve this I created the new `SortExec::cloned` which does neither of those two things and moved the resetting of `cache` into `with_new_children` and the resetting of `filter` into `reset_state`.
In other words, it doesn't make sense to put `Self::compute_properties(...)` in `reset_state`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add ExecutionPlan::reset_state [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #17028:
URL: https://github.com/apache/datafusion/pull/17028#issuecomment-3165772813
> done! #17060
>
> should we wait for anything else before merging this?
Nope, I think we are good -- let's merge it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add ExecutionPlan::reset_state [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #17028:
URL: https://github.com/apache/datafusion/pull/17028#discussion_r2251377563
##########
datafusion/sqllogictest/test_files/cte.slt:
##########
@@ -996,6 +996,29 @@ physical_plan
08)----------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
09)------------WorkTableExec: name=numbers
+# Test for issue #16998: SortExec shares DynamicFilterPhysicalExpr across multiple executions
+# This should return 5 rows but currently returns only 2 due to the bug
Review Comment:
```suggestion
# Test for issue #16998: SortExec shares DynamicFilterPhysicalExpr across multiple executions
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org