You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Braedon-Wooding-Displayr (via GitHub)" <gi...@apache.org> on 2026/10/18 10:19:24 UTC
[PR] perf: deduplicate found expressions with an IndexSet [datafusion]
Braedon-Wooding-Displayr opened a new pull request, #26474:
URL: https://github.com/apache/datafusion/pull/26474
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
- Closes #26473
## Rationale for this change
`find_exprs_in_exprs` and `find_exprs_in_expr` deduplicate with `Vec::contains`, which is quadratic in the number of distinct matches. On 20,000 distinct aggregates `find_aggregate_exprs` takes 932 ms; with this change it takes 2.8 ms (release build, see the issue for the full table).
## What changes are included in this PR?
Both functions now collect into a single index set.
## Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: deduplicate found expressions with an IndexSet [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #26474:
URL: https://github.com/apache/datafusion/pull/26474#discussion_r4357388086
##########
datafusion/expr/src/utils.rs:
##########
@@ -880,11 +876,19 @@ fn find_exprs_in_expr<F>(expr: &Expr, test_fn: &F) -> Vec<Expr>
where
F: Fn(&Expr) -> bool,
{
- let mut exprs = vec![];
+ let mut found = IndexSet::new();
+ collect_exprs_in_expr(expr, test_fn, &mut found);
+ found.into_iter().collect()
Review Comment:
nit: this can just delegate to `find_exprs_in_exprs`, which leaves one place that builds the set:
```diff
- let mut found = IndexSet::new();
- collect_exprs_in_expr(expr, test_fn, &mut found);
- found.into_iter().collect()
+ find_exprs_in_exprs(std::iter::once(expr), test_fn)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: deduplicate found expressions with an IndexSet [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #26474:
URL: https://github.com/apache/datafusion/pull/26474#issuecomment-6248894667
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/26474?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 82.89%. Comparing base ([`3470bd0`](https://app.codecov.io/gh/apache/datafusion/commit/3470bd0fbc016df5280493bb41c2e7ced3a4e626?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`9d76a7d`](https://app.codecov.io/gh/apache/datafusion/commit/9d76a7d230dc06d2bd03fcff716350a83ad0ce8a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #26474 +/- ##
==========================================
- Coverage 82.90% 82.89% -0.01%
==========================================
Files 1154 1154
Lines 457654 457680 +26
Branches 457654 457680 +26
==========================================
+ Hits 379398 379412 +14
- Misses 55252 55261 +9
- Partials 23004 23007 +3
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/26474?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: deduplicate found expressions with an IndexSet [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #26474:
URL: https://github.com/apache/datafusion/pull/26474#issuecomment-6382876619
This PR appears to be waiting on the author to respond to reviewer feedback so marking as draft. Please mark ready to review when it is ready for another look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: deduplicate found expressions with an IndexSet [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #26474:
URL: https://github.com/apache/datafusion/pull/26474#issuecomment-6406419874
Thanks @Braedon-Wooding-Displayr !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: deduplicate found expressions with an IndexSet [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 merged PR #26474:
URL: https://github.com/apache/datafusion/pull/26474
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: deduplicate found expressions with an IndexSet [datafusion]
Posted by "Braedon-Wooding-Displayr (via GitHub)" <gi...@apache.org>.
Braedon-Wooding-Displayr commented on PR #26474:
URL: https://github.com/apache/datafusion/pull/26474#issuecomment-6405129746
Sorry! Australian timezones mean that it sits over weekends sometimes!
I've just fixed and rebase + pushed, should be ready for merge now @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org