You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "pepijnve (via GitHub)" <gi...@apache.org> on 2026/09/16 21:14:06 UTC
[PR] Avoid duplicating keep-in-place expressions [datafusion]
pepijnve opened a new pull request, #25388:
URL: https://github.com/apache/datafusion/pull/25388
## Which issue does this PR close?
- Closes #25329.
## Rationale for this change
## What changes are included in this PR?
- Don't allow movement of keep-in-place expressions in filter pushdown
- Avoid duplication of keep-in-place expressions in during leaf expression extraction
## What is the testing strategy for this PR?
To be completed
## Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Avoid duplicating keep-in-place expressions [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on PR #25388:
URL: https://github.com/apache/datafusion/pull/25388#issuecomment-5705673812
There are still some test failures which is why this is still marked draft.
For this particular case, I'm wondering if the older version is actually an improvement. The arithmetic isn't super expensive, but we are evaluating it twice for all rows and we're not really saving any work in the scan.
The number of expressions being marked as 'keep in place' is probably a bit too broad at the moment though. I'm concerned that this change may prevent too many filter pushdown opportunities.
```
Expression: optimized_plan
────────────────────────────────────────────────────────────────────────────────
-old snapshot
+new results
────────────┬───────────────────────────────────────────────────────────────────
1 │-Projection: b * Int32(3) AS a, test.c
2 │- Projection: test.a * Int32(2) + test.c AS b, test.c
3 │- TableScan: test, full_filters=[(test.a * Int32(2) + test.c) * Int32(3) = Int64(1)]
1 │+Filter: a = Int64(1)
2 │+ Projection: b * Int32(3) AS a, test.c
3 │+ Projection: test.a * Int32(2) + test.c AS b, test.c
4 │+ TableScan: test
────────────┴───────────────────────────────────────────────────────────────────
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Avoid duplicating keep-in-place expressions [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on PR #25388:
URL: https://github.com/apache/datafusion/pull/25388#issuecomment-5704654290
@adriangb if you have a moment, I could use your input on this one. I got myself into an infinite in `push_extraction_pairs` when the double projection with the deferred extractions is created. This had me wondering if I put the 'don't duplicate' guard in the wrong place in this optimiser. Since you seem to be the original author of this code, you might have some ideas on how to achieve the same effect in a more appropriate way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org