You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "yashrb24 (via GitHub)" <gi...@apache.org> on 2026/03/30 10:22:34 UTC
[PR] fix: FilterPushdown incorrectly remaps filters through ProjectionExec with duplicate column names [datafusion]
yashrb24 opened a new pull request, #21247:
URL: https://github.com/apache/datafusion/pull/21247
## Which issue does this PR close?
- Closes #21246
## Rationale for this change
`ProjectionExec::gather_filters_for_pushdown` silently rewrites filter predicates to the wrong source column when the output schema contains duplicate column names — a structure that arises above joins where both sides share a column name. Two functions use name-only schema lookups (`column_with_name` and `index_of`) that always return the first match, which is incorrect when duplicate names exist:
1. `collect_reverse_alias` — HashMap key collision causes the second duplicate to overwrite the first.
2. `FilterRemapper::try_remap` — `index_of` silently rewrites column indices from non-first duplicates to position 0.
This code path is not exercised through normal SQL because the logical optimizer's `PushDownFilter` resolves qualified column references and pushes filters below projections before the physical plan is created. However, it affects any direct construction of physical plans (custom planners, external systems, the DataFrame API with manual projections).
## What changes are included in this PR?
1. **`collect_reverse_alias`**: Use `enumerate()` index instead of `column_with_name()`. Projection expressions are positionally aligned with the output schema, so `idx` is the correct output column index.
2. **`gather_filters_for_pushdown`**: Replace `FilterRemapper::try_remap` (which uses `index_of`) with direct validation against the alias map's exact `(name, index)` keys. The `PhysicalColumnRewriter` already does an exact-key lookup, so `try_remap` was both redundant and wrong for this case.
## Are these changes tested?
Yes. A regression test is added that constructs the exact physical plan structure triggering the bug (FilterExec → ProjectionExec with duplicate column names → HashJoinExec), runs the FilterPushdown optimizer, and verifies the optimized plan returns correct results (3 rows instead of the previous 0).
## Are there any user-facing changes?
No API changes. Fixes incorrect query results for physical plans with duplicate column names in projections.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: FilterPushdown incorrectly remaps filters through ProjectionExec with duplicate column names [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #21247:
URL: https://github.com/apache/datafusion/pull/21247#discussion_r3597759115
##########
datafusion/core/tests/physical_optimizer/filter_pushdown.rs:
##########
@@ -3374,3 +3374,142 @@ fn test_filter_pushdown_through_sort_with_projection() {
"
);
}
+
+/// Reproduces a bug where FilterPushdown through ProjectionExec with duplicate
Review Comment:
Is this bug possible to reproduce with SQL? Can we write a sqllogictest (slt) test for it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: FilterPushdown incorrectly remaps filters through ProjectionExec with duplicate column names [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #21247: fix: FilterPushdown incorrectly remaps filters through ProjectionExec with duplicate column names
URL: https://github.com/apache/datafusion/pull/21247
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: FilterPushdown incorrectly remaps filters through ProjectionExec with duplicate column names [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21247:
URL: https://github.com/apache/datafusion/pull/21247#issuecomment-4964751386
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org