You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "neilconway (via GitHub)" <gi...@apache.org> on 2026/02/25 03:00:45 UTC
[PR] perf: Use Hashbrown for array_distinct [datafusion]
neilconway opened a new pull request, #20538:
URL: https://github.com/apache/datafusion/pull/20538
## Which issue does this PR close?
N/A
## Rationale for this change
#20364 recently optimized `array_distinct` to use batched row conversion. As part of that PR, `std::HashSet` was used. This PR just replaces `std::HashSet` with `hashbrown::HashSet`, which measurably improves performance.
## What changes are included in this PR?
## Are these changes tested?
Yes.
## Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Use Hashbrown for array_distinct [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on PR #20538:
URL: https://github.com/apache/datafusion/pull/20538#issuecomment-3956425171
Benchmarks:
```
┌────────────────────┬──────────────┬───────────┬─────────────┐
│ Benchmark │ std::HashSet │ hashbrown │ Improvement │
├────────────────────┼──────────────┼───────────┼─────────────┤
│ high_duplicate/10 │ 145.00 µs │ 68.94 µs │ -52.4% │
├────────────────────┼──────────────┼───────────┼─────────────┤
│ high_duplicate/50 │ 719.46 µs │ 350.68 µs │ -51.3% │
├────────────────────┼──────────────┼───────────┼─────────────┤
│ high_duplicate/100 │ 1.404 ms │ 674.83 µs │ -52.0% │
├────────────────────┼──────────────┼───────────┼─────────────┤
│ low_duplicate/10 │ 180.33 µs │ 107.87 µs │ -40.0% │
├────────────────────┼──────────────┼───────────┼─────────────┤
│ low_duplicate/50 │ 849.26 µs │ 530.89 µs │ -37.6% │
├────────────────────┼──────────────┼───────────┼─────────────┤
│ low_duplicate/100 │ 1.706 ms │ 987.72 µs │ -42.0% │
└────────────────────┴──────────────┴───────────┴─────────────┘
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Use Hashbrown for array_distinct [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on PR #20538:
URL: https://github.com/apache/datafusion/pull/20538#issuecomment-3961843714
> Thanks @neilconway it is amazing how 1 line can change so much
Indeed; the choice of hash function in `std` seems a bit silly, at least for our use-case. #19869 might be worth considering...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Use Hashbrown for array_distinct [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich merged PR #20538:
URL: https://github.com/apache/datafusion/pull/20538
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org