You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "mdashti (via GitHub)" <gi...@apache.org> on 2026/02/20 02:02:38 UTC
[I] Support filter pushdown through `SortMergeJoinExec` [datafusion]
mdashti opened a new issue, #20443:
URL: https://github.com/apache/datafusion/issues/20443
### Is your feature request related to a problem or challenge?
`SortMergeJoinExec` currently relies on the default `gather_filters_for_pushdown` implementation, which marks all parent filters as unsupported. This prevents `DynamicFilterPhysicalExpr` (e.g. from `SortExec`'s TopK) from being pushed through sort-merge joins to reach scan nodes below them.
`HashJoinExec` already implements proper filter routing via `FilterDescription::from_children` + `FilterPushdownPropagation::if_any`, which routes each parent filter to whichever child's schema contains the referenced columns. There's no reason `SortMergeJoinExec` can't do the same — both join types have clearly delineated left/right children with independent schemas.
As a workaround, we currently wrap `SortMergeJoinExec` in a thin passthrough node that overrides the two filter-pushdown methods. This works but adds an unnecessary layer of indirection.
### Describe the solution you'd like
Implement `gather_filters_for_pushdown` and `handle_child_pushdown_result` on `SortMergeJoinExec`, following the same pattern as `HashJoinExec`:
```rust
// gather_filters_for_pushdown
fn gather_filters_for_pushdown(
&self,
_phase: FilterPushdownPhase,
parent_filters: Vec<Arc<dyn PhysicalExpr>>,
_config: &ConfigOptions,
) -> Result<FilterDescription> {
if self.join_type != JoinType::Inner {
return Ok(FilterDescription::all_unsupported(
&parent_filters,
&self.children(),
));
}
FilterDescription::from_children(parent_filters, &self.children())
}
// handle_child_pushdown_result
fn handle_child_pushdown_result(
&self,
_phase: FilterPushdownPhase,
child_pushdown_result: ChildPushdownResult,
_config: &ConfigOptions,
) -> Result<FilterPushdownPropagation<Arc<dyn ExecutionPlan>>> {
if self.join_type != JoinType::Inner {
return Ok(FilterPushdownPropagation::all_unsupported(child_pushdown_result));
}
Ok(FilterPushdownPropagation::if_any(child_pushdown_result))
}
```
This uses `from_children` for column-based routing (a filter on column `x` is routed only to the child whose schema contains `x`) and `if_any` for propagation (matching `HashJoinExec`'s behavior, since a filter can only apply to one child).
### Describe alternatives you've considered
Wrapping `SortMergeJoinExec` in a custom `ExecutionPlan` that overrides `gather_filters_for_pushdown` and `handle_child_pushdown_result`. This is what we're currently doing — it works, but adds unnecessary plan complexity. Ideally this should be built into `SortMergeJoinExec` itself.
### Additional context
The same gap may apply to other join types (`NestedLoopJoinExec`, `SymmetricHashJoinExec`, `CrossJoinExec`) that also use the default no-op filter pushdown. `SortMergeJoinExec` is the most impactful since it's used for sorted-input joins where TopK dynamic filters are most beneficial.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support filter pushdown through `SortMergeJoinExec` [datafusion]
Posted by "petern48 (via GitHub)" <gi...@apache.org>.
petern48 commented on issue #20443:
URL: https://github.com/apache/datafusion/issues/20443#issuecomment-3938028946
@mdashti No worries at all, I hadn't started yet anyway. Happy to leave it for you to tackle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support filter pushdown through `SortMergeJoinExec` [datafusion]
Posted by "stuhood (via GitHub)" <gi...@apache.org>.
stuhood commented on issue #20443:
URL: https://github.com/apache/datafusion/issues/20443#issuecomment-4805010015
https://github.com/apache/datafusion/pull/21267#issuecomment-4805005398 : sorry for the noise!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support filter pushdown through `SortMergeJoinExec` [datafusion]
Posted by "stuhood (via GitHub)" <gi...@apache.org>.
stuhood commented on issue #20443:
URL: https://github.com/apache/datafusion/issues/20443#issuecomment-4401502699
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support filter pushdown through `SortMergeJoinExec` [datafusion]
Posted by "petern48 (via GitHub)" <gi...@apache.org>.
petern48 commented on issue #20443:
URL: https://github.com/apache/datafusion/issues/20443#issuecomment-3935598399
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support filter pushdown through `SortMergeJoinExec` [datafusion]
Posted by "mdashti (via GitHub)" <gi...@apache.org>.
mdashti commented on issue #20443:
URL: https://github.com/apache/datafusion/issues/20443#issuecomment-3936312074
@petern48 Thanks for taking this. I didn't see your message, and didn't know the process to claim a ticket. I've opened a PR: https://github.com/apache/datafusion/pull/20455. Please see if you can build on top of it and add anything needed. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org