You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Dandandan (via GitHub)" <gi...@apache.org> on 2026/03/16 10:54:30 UTC
[PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Dandandan opened a new pull request, #20958:
URL: https://github.com/apache/datafusion/pull/20958
## Summary
- Replace `ahash` with `foldhash` in `datafusion-common` hashing (`with_hashes`/`create_hashes`)
- Use `SeedableRandomState` for rehash paths: fold existing hash into hasher's initial state, eliminating the separate `combine_hashes` step
- Add `hash_write` method to `HashValue` trait for writing values into an existing hasher
- Use `valid_indices()` iterator for null paths instead of per-element `is_null()` checks
## Benchmark results (int64, 8192 rows, Apple M1)
| Benchmark | Before (ahash) | After (foldhash) | Improvement |
|---|---|---|---|
| single array, no nulls | 5.65 µs | 3.30 µs | **-42%** |
| multiple arrays, no nulls | 22.15 µs | 11.19 µs | **-49%** |
| single array, nulls | 11.94 µs | 9.47 µs | **-21%** |
| multiple arrays, nulls | 36.92 µs | 29.80 µs | **-19%** |
String view improvements (utf8_view, 8192 rows):
| Benchmark | Improvement |
|---|---|
| single, no nulls | **-13%** |
| multiple, no nulls | **-28%** |
| small strings, single | **-55%** |
| small strings, multiple | **-60%** |
## Test plan
- [x] All 36 `hash_utils` unit tests pass
- [ ] Run full CI suite
- [ ] Verify downstream crates still compile (they use `ahash::RandomState` directly — this PR only changes `datafusion-common`, further migration needed)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066823187
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4066805380)
`Linux bench-c4066805380-294-vb6vs 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing foldhash-hashing (b5a0933a53541f161cb336fdb0ffc01291cefc61) to 9c3c01a (merge-base) [diff](https://github.com/apache/datafusion/compare/9c3c01aa54fbf24bb0779e04d47a1492983ed4d7..b5a0933a53541f161cb336fdb0ffc01291cefc61) using: tpcds
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066953218
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4067244202
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4067242298)
`Linux bench-c4067242298-310-q76m4 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing foldhash-hashing (df6f5a0605ec157880abd10a1d7a7ce3898e3212) to 8609288 (merge-base) [diff](https://github.com/apache/datafusion/compare/8609288ae0595e715f6ab4fc6a5a56edf1b05eeb..df6f5a0605ec157880abd10a1d7a7ce3898e3212) using: clickbench_partitioned
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066944128
🤖 Criterion benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4066830279)
<details><summary>Details</summary>
<p>
```
group foldhash-hashing main
----- ---------------- ----
dense_union: multiple, no nulls 1.00 238.5±0.28µs ? ?/sec 1.09 259.4±0.50µs ? ?/sec
dense_union: single, no nulls 1.00 79.8±0.12µs ? ?/sec 1.09 87.2±0.13µs ? ?/sec
dictionary_utf8_int32: multiple, no nulls 1.11 16.5±0.01µs ? ?/sec 1.00 14.9±0.02µs ? ?/sec
dictionary_utf8_int32: multiple, nulls 1.00 32.7±0.06µs ? ?/sec 1.00 32.7±0.01µs ? ?/sec
dictionary_utf8_int32: single, no nulls 1.08 5.1±0.80µs ? ?/sec 1.00 4.7±0.03µs ? ?/sec
dictionary_utf8_int32: single, nulls 1.00 11.0±0.01µs ? ?/sec 1.02 11.2±0.01µs ? ?/sec
int64: multiple, no nulls 1.00 16.9±0.02µs ? ?/sec 1.97 33.4±0.03µs ? ?/sec
int64: multiple, nulls 1.00 26.7±0.01µs ? ?/sec 1.43 38.1±0.12µs ? ?/sec
int64: single, no nulls 1.00 4.6±0.01µs ? ?/sec 2.60 11.9±0.02µs ? ?/sec
int64: single, nulls 1.00 8.7±0.01µs ? ?/sec 1.35 11.8±0.05µs ? ?/sec
large_utf8: multiple, no nulls 1.00 109.8±0.40µs ? ?/sec 1.41 154.4±3.65µs ? ?/sec
large_utf8: multiple, nulls 1.00 136.2±0.79µs ? ?/sec 1.49 203.5±0.57µs ? ?/sec
large_utf8: single, no nulls 1.00 26.9±0.33µs ? ?/sec 1.44 38.7±0.08µs ? ?/sec
large_utf8: single, nulls 1.00 45.7±1.07µs ? ?/sec 1.31 59.6±1.03µs ? ?/sec
list_array: multiple, no nulls 1.00 146.4±0.21µs ? ?/sec 1.76 258.0±1.04µs ? ?/sec
list_array: multiple, nulls 1.00 176.0±0.26µs ? ?/sec 1.60 281.2±0.24µs ? ?/sec
list_array: single, no nulls 1.00 48.8±0.19µs ? ?/sec 1.78 86.8±0.22µs ? ?/sec
list_array: single, nulls 1.00 59.0±0.14µs ? ?/sec 1.59 94.0±0.13µs ? ?/sec
list_array_sliced: 1/10 of 81920 rows 1.00 48.9±0.20µs ? ?/sec 1.73 84.4±0.18µs ? ?/sec
list_array_sliced: 1/2 of 16384 rows 1.00 49.2±0.13µs ? ?/sec 1.72 84.9±0.32µs ? ?/sec
list_array_sliced: 1/5 of 40960 rows 1.00 49.3±0.02µs ? ?/sec 1.73 85.4±0.68µs ? ?/sec
map_array: multiple, no nulls 1.00 240.8±0.14µs ? ?/sec 1.74 418.5±0.59µs ? ?/sec
map_array: multiple, nulls 1.00 246.9±1.22µs ? ?/sec 1.73 426.8±0.89µs ? ?/sec
map_array: single, no nulls 1.00 80.9±0.12µs ? ?/sec 1.73 140.1±0.32µs ? ?/sec
map_array: single, nulls 1.00 82.7±0.10µs ? ?/sec 1.73 142.7±0.27µs ? ?/sec
map_array_sliced: 1/10 of 81920 rows 1.00 80.9±0.20µs ? ?/sec 1.72 139.5±0.21µs ? ?/sec
map_array_sliced: 1/2 of 16384 rows 1.00 80.4±0.05µs ? ?/sec 1.73 139.2±0.08µs ? ?/sec
map_array_sliced: 1/5 of 40960 rows 1.00 81.3±0.13µs ? ?/sec 1.72 139.9±0.68µs ? ?/sec
run_array_int32: multiple, no nulls 1.00 5.0±0.01µs ? ?/sec 1.16 5.8±0.01µs ? ?/sec
run_array_int32: multiple, nulls 1.00 6.7±0.01µs ? ?/sec 1.05 7.0±0.01µs ? ?/sec
run_array_int32: single, no nulls 1.00 1816.7±3.47ns ? ?/sec 1.14 2.1±0.00µs ? ?/sec
run_array_int32: single, nulls 1.00 2.4±0.01µs ? ?/sec 1.04 2.4±0.00µs ? ?/sec
sparse_union: multiple, no nulls 1.00 260.1±0.78µs ? ?/sec 1.08 282.1±0.45µs ? ?/sec
sparse_union: single, no nulls 1.00 87.0±0.22µs ? ?/sec 1.09 94.5±0.31µs ? ?/sec
sparse_union_sliced: 1/10 of 81920 rows 1.00 87.1±0.22µs ? ?/sec 1.09 95.0±0.97µs ? ?/sec
sparse_union_sliced: 1/2 of 16384 rows 1.00 87.1±0.14µs ? ?/sec 1.09 94.8±0.21µs ? ?/sec
sparse_union_sliced: 1/5 of 40960 rows 1.00 87.3±0.26µs ? ?/sec 1.09 94.8±0.30µs ? ?/sec
struct_array: multiple, no nulls 1.00 161.8±1.03µs ? ?/sec 1.69 272.9±0.62µs ? ?/sec
struct_array: multiple, nulls 1.00 169.8±0.60µs ? ?/sec 1.67 282.9±0.36µs ? ?/sec
struct_array: single, no nulls 1.00 54.1±0.36µs ? ?/sec 1.69 91.6±0.33µs ? ?/sec
struct_array: single, nulls 1.00 57.4±0.08µs ? ?/sec 1.66 95.5±0.31µs ? ?/sec
utf8: multiple, no nulls 1.00 103.4±0.48µs ? ?/sec 1.50 155.2±1.09µs ? ?/sec
utf8: multiple, nulls 1.00 142.2±0.49µs ? ?/sec 1.51 214.8±2.92µs ? ?/sec
utf8: single, no nulls 1.00 26.6±0.12µs ? ?/sec 1.48 39.4±0.18µs ? ?/sec
utf8: single, nulls 1.00 45.7±0.73µs ? ?/sec 1.33 61.0±0.74µs ? ?/sec
utf8_view (small): multiple, no nulls 1.00 19.0±0.01µs ? ?/sec 1.93 36.7±0.11µs ? ?/sec
utf8_view (small): multiple, nulls 1.00 29.2±0.02µs ? ?/sec 1.55 45.2±0.06µs ? ?/sec
utf8_view (small): single, no nulls 1.00 5.1±0.00µs ? ?/sec 2.27 11.5±0.08µs ? ?/sec
utf8_view (small): single, nulls 1.00 9.7±0.02µs ? ?/sec 1.45 14.1±0.01µs ? ?/sec
utf8_view: multiple, no nulls 1.00 107.4±0.37µs ? ?/sec 1.29 138.9±0.38µs ? ?/sec
utf8_view: multiple, nulls 1.00 120.6±0.44µs ? ?/sec 1.24 149.0±0.34µs ? ?/sec
utf8_view: single, no nulls 1.00 33.1±0.08µs ? ?/sec 1.13 37.5±0.04µs ? ?/sec
utf8_view: single, nulls 1.00 38.3±0.19µs ? ?/sec 1.17 44.7±1.27µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 528.3s |
| Peak memory | 3.5 GiB |
| Avg memory | 3.3 GiB |
| CPU user | 663.4s |
| CPU sys | 1.4s |
| Disk read | 0 B |
| Disk write | 555.8 MiB |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 523.1s |
| Peak memory | 3.4 GiB |
| Avg memory | 3.4 GiB |
| CPU user | 659.3s |
| CPU sys | 0.7s |
| Disk read | 0 B |
| Disk write | 13.8 MiB |
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066979961
Benchmark for [this request](https://github.com/apache/datafusion/pull/20958#issuecomment-4066953218) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
error[E0308]: mismatched types
--> datafusion/physical-plan/src/windows/bounded_window_agg_exec.rs:711:32
|
711 | create_hashes(columns, &self.random_state, &mut batch_hashes)?;
| ------------- ^^^^^^^^^^^^^^^^^^ expected `&FixedState`, found `&RandomState`
| |
| arguments to this function are incorrect
|
= note: expected reference `&foldhash::fast::FixedState`
found reference `&ahash::RandomState`
note: function defined here
--> datafusion/common/src/hash_utils.rs:1094:8
|
1094 | pub fn create_hashes<'a, I, T>(
| ^^^^^^^^^^^^^
For more information about this error, try `rustc --explain E0308`.
error: could not compile `datafusion-physical-plan` (lib) due to 10 previous errors
warning: build failed, waiting for other jobs to finish...
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4067123884
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4067468776
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4067452276)
`Linux bench-c4067452276-312-746ld 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing foldhash-hashing (e084d7fd5fdb06d3b3f0aca8655687a9df57da95) to 8609288 (merge-base) [diff](https://github.com/apache/datafusion/compare/8609288ae0595e715f6ab4fc6a5a56edf1b05eeb..e084d7fd5fdb06d3b3f0aca8655687a9df57da95) using: clickbench_extended
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4067587376
run benchmark clickbench_extended
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4067605580
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4067587376)
`Linux bench-c4067587376-313-twj2f 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing foldhash-hashing (0c011ff93ea88a34d48caef35fce2e966b43756a) to 8609288 (merge-base) [diff](https://github.com/apache/datafusion/compare/8609288ae0595e715f6ab4fc6a5a56edf1b05eeb..0c011ff93ea88a34d48caef35fce2e966b43756a) using: clickbench_extended
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066833334
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4066829844)
`Linux bench-c4066829844-299-prj6d 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing foldhash-hashing (83168bbc2f9d35a0e5082891c4afbfe6cb75bfbb) to 8609288 (merge-base) [diff](https://github.com/apache/datafusion/compare/8609288ae0595e715f6ab4fc6a5a56edf1b05eeb..83168bbc2f9d35a0e5082891c4afbfe6cb75bfbb) using: tpch
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066967928
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4066953218)
`Linux bench-c4066953218-302-vmnj2 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing foldhash-hashing (7fe428296c61e972616f09d516a063fa5e716cc3) to 8609288 (merge-base) [diff](https://github.com/apache/datafusion/compare/8609288ae0595e715f6ab4fc6a5a56edf1b05eeb..7fe428296c61e972616f09d516a063fa5e716cc3) using: tpcds
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4067133872
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4067123884)
`Linux bench-c4067123884-307-xgdzs 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing foldhash-hashing (c78a48406aec42bc01820343c88ccee37e54eabe) to 8609288 (merge-base) [diff](https://github.com/apache/datafusion/compare/8609288ae0595e715f6ab4fc6a5a56edf1b05eeb..c78a48406aec42bc01820343c88ccee37e54eabe) using: clickbench_partitioned
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4070144656
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4070030473)
<details><summary>Details</summary>
<p>
```
Comparing HEAD and foldhash-hashing
--------------------
Benchmark tpcds_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ foldhash-hashing ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 50.96 / 51.64 ±0.62 / 52.70 ms │ 46.90 / 48.29 ±1.14 / 49.65 ms │ +1.07x faster │
│ QQuery 2 │ 155.54 / 156.32 ±0.77 / 157.78 ms │ 152.55 / 153.60 ±1.37 / 156.22 ms │ no change │
│ QQuery 3 │ 119.28 / 119.57 ±0.20 / 119.85 ms │ 116.06 / 116.86 ±0.77 / 118.28 ms │ no change │
│ QQuery 4 │ 1401.17 / 1423.55 ±18.67 / 1451.52 ms │ 1394.35 / 1419.95 ±22.93 / 1455.35 ms │ no change │
│ QQuery 5 │ 185.41 / 189.02 ±2.58 / 192.99 ms │ 182.06 / 183.23 ±1.14 / 184.97 ms │ no change │
│ QQuery 6 │ 1007.41 / 1043.91 ±30.18 / 1082.24 ms │ 1013.24 / 1045.21 ±34.14 / 1098.16 ms │ no change │
│ QQuery 7 │ 360.78 / 363.04 ±2.25 / 367.20 ms │ 355.45 / 357.74 ±2.18 / 361.75 ms │ no change │
│ QQuery 8 │ 125.22 / 125.76 ±0.45 / 126.33 ms │ 119.94 / 120.54 ±0.60 / 121.59 ms │ no change │
│ QQuery 9 │ 117.52 / 120.91 ±2.78 / 124.43 ms │ 111.18 / 116.43 ±2.66 / 118.60 ms │ no change │
│ QQuery 10 │ 116.44 / 128.05 ±19.74 / 167.48 ms │ 111.08 / 112.46 ±1.17 / 114.62 ms │ +1.14x faster │
│ QQuery 11 │ 998.34 / 1013.93 ±14.77 / 1041.40 ms │ 979.00 / 982.62 ±2.96 / 986.37 ms │ no change │
│ QQuery 12 │ 48.06 / 49.74 ±1.49 / 51.65 ms │ 47.41 / 49.44 ±1.85 / 52.85 ms │ no change │
│ QQuery 13 │ 406.36 / 413.15 ±4.82 / 419.87 ms │ 407.96 / 410.11 ±1.82 / 412.77 ms │ no change │
│ QQuery 14 │ 1109.13 / 1122.07 ±12.11 / 1144.10 ms │ 1067.62 / 1075.93 ±9.01 / 1092.25 ms │ no change │
│ QQuery 15 │ 18.53 / 19.56 ±0.92 / 21.05 ms │ 18.40 / 19.82 ±1.13 / 21.33 ms │ no change │
│ QQuery 16 │ 47.63 / 48.54 ±0.80 / 49.57 ms │ 44.42 / 44.79 ±0.34 / 45.41 ms │ +1.08x faster │
│ QQuery 17 │ 251.91 / 253.96 ±1.92 / 256.88 ms │ 245.17 / 246.58 ±1.31 / 248.78 ms │ no change │
│ QQuery 18 │ 135.43 / 135.85 ±0.47 / 136.76 ms │ 131.23 / 132.59 ±0.95 / 133.90 ms │ no change │
│ QQuery 19 │ 163.16 / 165.81 ±1.75 / 168.41 ms │ 158.56 / 159.41 ±0.77 / 160.86 ms │ no change │
│ QQuery 20 │ 16.20 / 17.22 ±0.69 / 18.11 ms │ 16.04 / 16.59 ±0.46 / 17.29 ms │ no change │
│ QQuery 21 │ 25.25 / 25.59 ±0.33 / 26.03 ms │ 23.16 / 23.79 ±0.42 / 24.30 ms │ +1.08x faster │
│ QQuery 22 │ 516.38 / 519.53 ±3.75 / 526.89 ms │ 492.40 / 496.52 ±2.72 / 500.62 ms │ no change │
│ QQuery 23 │ 962.54 / 974.67 ±7.76 / 983.09 ms │ 931.86 / 941.13 ±6.55 / 951.29 ms │ no change │
│ QQuery 24 │ 442.59 / 445.55 ±2.96 / 449.98 ms │ 429.77 / 433.16 ±3.07 / 437.62 ms │ no change │
│ QQuery 25 │ 369.96 / 372.88 ±1.66 / 374.81 ms │ 359.19 / 362.31 ±1.80 / 364.05 ms │ no change │
│ QQuery 26 │ 89.14 / 90.61 ±1.48 / 92.74 ms │ 84.37 / 87.07 ±2.29 / 90.84 ms │ no change │
│ QQuery 27 │ 357.77 / 358.68 ±0.80 / 359.91 ms │ 352.56 / 355.41 ±2.89 / 359.37 ms │ no change │
│ QQuery 28 │ 154.49 / 156.43 ±1.36 / 158.03 ms │ 153.55 / 154.93 ±1.47 / 157.60 ms │ no change │
│ QQuery 29 │ 313.18 / 316.11 ±2.37 / 319.74 ms │ 301.96 / 306.02 ±2.58 / 310.05 ms │ no change │
│ QQuery 30 │ 49.60 / 52.95 ±2.22 / 55.24 ms │ 48.02 / 50.34 ±1.50 / 51.96 ms │ no change │
│ QQuery 31 │ 188.34 / 189.67 ±1.00 / 191.00 ms │ 176.50 / 178.32 ±1.19 / 180.20 ms │ +1.06x faster │
│ QQuery 32 │ 63.09 / 64.21 ±0.85 / 65.19 ms │ 60.15 / 60.84 ±0.84 / 62.49 ms │ +1.06x faster │
│ QQuery 33 │ 153.38 / 154.60 ±1.35 / 157.01 ms │ 147.63 / 149.45 ±1.49 / 151.63 ms │ no change │
│ QQuery 34 │ 113.28 / 114.65 ±1.85 / 118.26 ms │ 106.99 / 110.88 ±2.58 / 114.64 ms │ no change │
│ QQuery 35 │ 120.81 / 122.02 ±0.79 / 122.99 ms │ 113.30 / 114.25 ±0.79 / 115.41 ms │ +1.07x faster │
│ QQuery 36 │ 218.15 / 227.41 ±4.74 / 231.31 ms │ 215.96 / 222.80 ±4.17 / 228.55 ms │ no change │
│ QQuery 37 │ 188.95 / 190.04 ±1.12 / 191.99 ms │ 177.66 / 182.49 ±2.58 / 185.37 ms │ no change │
│ QQuery 38 │ 97.37 / 98.76 ±1.64 / 101.32 ms │ 89.39 / 92.66 ±2.03 / 95.02 ms │ +1.07x faster │
│ QQuery 39 │ 143.43 / 145.17 ±1.93 / 148.64 ms │ 136.91 / 138.81 ±1.79 / 141.83 ms │ no change │
│ QQuery 40 │ 120.81 / 125.70 ±4.75 / 134.21 ms │ 115.28 / 120.54 ±5.54 / 130.38 ms │ no change │
│ QQuery 41 │ 18.00 / 19.74 ±1.30 / 21.61 ms │ 18.60 / 19.56 ±0.96 / 21.21 ms │ no change │
│ QQuery 42 │ 112.14 / 114.43 ±1.60 / 116.41 ms │ 109.60 / 111.66 ±1.19 / 112.81 ms │ no change │
│ QQuery 43 │ 88.95 / 90.22 ±0.77 / 91.27 ms │ 85.12 / 86.34 ±0.77 / 87.41 ms │ no change │
│ QQuery 44 │ 18.99 / 19.60 ±0.62 / 20.66 ms │ 17.39 / 18.12 ±0.59 / 19.17 ms │ +1.08x faster │
│ QQuery 45 │ 57.48 / 58.58 ±0.79 / 59.55 ms │ 56.84 / 58.39 ±1.21 / 59.49 ms │ no change │
│ QQuery 46 │ 237.25 / 242.80 ±3.80 / 247.99 ms │ 239.28 / 241.32 ±2.07 / 245.07 ms │ no change │
│ QQuery 47 │ 774.92 / 779.81 ±6.71 / 792.89 ms │ 730.90 / 745.83 ±9.28 / 758.85 ms │ no change │
│ QQuery 48 │ 294.28 / 298.26 ±2.10 / 300.32 ms │ 286.60 / 293.89 ±4.81 / 299.53 ms │ no change │
│ QQuery 49 │ 267.78 / 270.53 ±2.89 / 276.09 ms │ 264.75 / 266.31 ±1.30 / 268.56 ms │ no change │
│ QQuery 50 │ 245.85 / 253.36 ±4.71 / 258.92 ms │ 235.47 / 239.67 ±2.71 / 242.66 ms │ +1.06x faster │
│ QQuery 51 │ 190.13 / 195.28 ±2.81 / 197.96 ms │ 186.43 / 189.68 ±2.14 / 191.83 ms │ no change │
│ QQuery 52 │ 113.49 / 114.85 ±1.02 / 116.08 ms │ 109.75 / 111.85 ±1.41 / 113.69 ms │ no change │
│ QQuery 53 │ 110.43 / 111.51 ±1.16 / 113.73 ms │ 106.45 / 107.03 ±0.50 / 107.87 ms │ no change │
│ QQuery 54 │ 155.98 / 158.74 ±2.27 / 162.27 ms │ 153.99 / 155.49 ±1.25 / 157.28 ms │ no change │
│ QQuery 55 │ 110.80 / 111.34 ±0.57 / 112.22 ms │ 109.31 / 110.06 ±0.51 / 110.59 ms │ no change │
│ QQuery 56 │ 151.76 / 154.47 ±2.13 / 157.48 ms │ 150.23 / 150.93 ±0.67 / 151.83 ms │ no change │
│ QQuery 57 │ 194.61 / 196.71 ±2.08 / 200.46 ms │ 185.46 / 187.87 ±1.44 / 189.38 ms │ no change │
│ QQuery 58 │ 315.10 / 324.69 ±7.79 / 337.81 ms │ 297.85 / 306.85 ±6.09 / 316.19 ms │ +1.06x faster │
│ QQuery 59 │ 209.96 / 212.89 ±1.90 / 215.17 ms │ 206.02 / 209.43 ±5.26 / 219.88 ms │ no change │
│ QQuery 60 │ 157.09 / 158.21 ±0.96 / 159.82 ms │ 150.09 / 152.29 ±1.42 / 154.53 ms │ no change │
│ QQuery 61 │ 180.52 / 182.79 ±1.21 / 183.90 ms │ 176.77 / 178.64 ±0.96 / 179.40 ms │ no change │
│ QQuery 62 │ 914.12 / 940.42 ±19.48 / 961.06 ms │ 914.25 / 953.08 ±27.60 / 1000.43 ms │ no change │
│ QQuery 63 │ 110.66 / 113.20 ±2.35 / 116.87 ms │ 106.81 / 108.87 ±1.83 / 111.58 ms │ no change │
│ QQuery 64 │ 729.70 / 733.51 ±2.65 / 737.82 ms │ 720.33 / 722.40 ±1.38 / 724.26 ms │ no change │
│ QQuery 65 │ 264.16 / 268.74 ±4.95 / 277.89 ms │ 256.14 / 260.35 ±2.29 / 262.87 ms │ no change │
│ QQuery 66 │ 258.24 / 263.76 ±4.60 / 272.15 ms │ 254.55 / 267.25 ±8.46 / 276.73 ms │ no change │
│ QQuery 67 │ 349.34 / 357.94 ±6.38 / 368.42 ms │ 315.03 / 324.51 ±5.27 / 329.63 ms │ +1.10x faster │
│ QQuery 68 │ 287.29 / 293.37 ±3.52 / 297.40 ms │ 286.72 / 290.49 ±3.46 / 296.62 ms │ no change │
│ QQuery 69 │ 116.71 / 117.83 ±1.09 / 119.42 ms │ 108.11 / 109.00 ±0.93 / 110.73 ms │ +1.08x faster │
│ QQuery 70 │ 356.38 / 372.68 ±11.95 / 393.37 ms │ 343.24 / 350.87 ±8.51 / 366.85 ms │ +1.06x faster │
│ QQuery 71 │ 140.54 / 143.61 ±2.58 / 147.79 ms │ 137.07 / 138.88 ±1.18 / 140.65 ms │ no change │
│ QQuery 72 │ 719.95 / 745.98 ±16.61 / 772.29 ms │ 720.55 / 734.97 ±13.14 / 755.73 ms │ no change │
│ QQuery 73 │ 108.39 / 110.71 ±1.90 / 113.68 ms │ 103.78 / 105.24 ±1.02 / 106.60 ms │ no change │
│ QQuery 74 │ 636.33 / 642.40 ±3.77 / 645.60 ms │ 604.01 / 617.45 ±9.74 / 626.17 ms │ no change │
│ QQuery 75 │ 297.23 / 299.30 ±1.65 / 301.42 ms │ 287.97 / 289.68 ±1.26 / 291.58 ms │ no change │
│ QQuery 76 │ 138.76 / 139.79 ±0.53 / 140.23 ms │ 137.43 / 139.78 ±1.97 / 143.11 ms │ no change │
│ QQuery 77 │ 206.61 / 209.48 ±1.84 / 211.75 ms │ 198.76 / 200.45 ±1.47 / 202.20 ms │ no change │
│ QQuery 78 │ 369.93 / 374.31 ±4.57 / 382.58 ms │ 358.06 / 361.99 ±4.08 / 367.79 ms │ no change │
│ QQuery 79 │ 245.00 / 247.46 ±2.33 / 251.53 ms │ 236.41 / 239.68 ±2.79 / 243.51 ms │ no change │
│ QQuery 80 │ 348.52 / 349.98 ±1.39 / 352.39 ms │ 336.56 / 340.16 ±2.32 / 343.69 ms │ no change │
│ QQuery 81 │ 33.49 / 34.40 ±0.91 / 35.95 ms │ 31.73 / 32.82 ±1.01 / 34.19 ms │ no change │
│ QQuery 82 │ 207.51 / 212.61 ±3.33 / 217.37 ms │ 201.70 / 203.88 ±1.83 / 206.21 ms │ no change │
│ QQuery 83 │ 49.42 / 51.32 ±1.00 / 52.17 ms │ 47.25 / 48.52 ±0.95 / 49.83 ms │ +1.06x faster │
│ QQuery 84 │ 53.04 / 53.99 ±0.69 / 55.00 ms │ 51.44 / 52.23 ±0.63 / 53.19 ms │ no change │
│ QQuery 85 │ 157.58 / 159.92 ±2.18 / 163.48 ms │ 153.34 / 155.69 ±1.95 / 159.25 ms │ no change │
│ QQuery 86 │ 42.15 / 43.48 ±0.68 / 44.07 ms │ 41.45 / 42.41 ±0.88 / 43.45 ms │ no change │
│ QQuery 87 │ 96.46 / 99.77 ±2.18 / 102.97 ms │ 90.65 / 94.99 ±3.27 / 100.23 ms │ no change │
│ QQuery 88 │ 118.26 / 119.80 ±1.18 / 121.76 ms │ 110.19 / 112.36 ±1.17 / 113.50 ms │ +1.07x faster │
│ QQuery 89 │ 126.89 / 128.42 ±1.75 / 131.75 ms │ 120.80 / 122.30 ±1.50 / 125.12 ms │ no change │
│ QQuery 90 │ 30.23 / 30.98 ±0.45 / 31.51 ms │ 27.90 / 28.73 ±0.61 / 29.25 ms │ +1.08x faster │
│ QQuery 91 │ 69.17 / 70.16 ±0.84 / 71.39 ms │ 64.50 / 66.25 ±0.90 / 67.09 ms │ +1.06x faster │
│ QQuery 92 │ 63.67 / 64.76 ±1.17 / 66.89 ms │ 61.12 / 62.55 ±1.12 / 64.10 ms │ no change │
│ QQuery 93 │ 199.17 / 203.05 ±2.20 / 205.58 ms │ 194.04 / 195.43 ±2.15 / 199.71 ms │ no change │
│ QQuery 94 │ 68.41 / 69.14 ±0.50 / 69.87 ms │ 65.76 / 66.29 ±0.36 / 66.80 ms │ no change │
│ QQuery 95 │ 147.49 / 150.66 ±2.98 / 155.31 ms │ 138.91 / 141.42 ±1.74 / 143.55 ms │ +1.07x faster │
│ QQuery 96 │ 81.37 / 82.29 ±1.04 / 84.12 ms │ 76.20 / 76.93 ±0.42 / 77.36 ms │ +1.07x faster │
│ QQuery 97 │ 141.19 / 141.85 ±0.38 / 142.22 ms │ 131.83 / 134.34 ±1.64 / 136.32 ms │ +1.06x faster │
│ QQuery 98 │ 156.20 / 158.73 ±2.18 / 162.12 ms │ 156.41 / 158.14 ±1.36 / 160.44 ms │ no change │
│ QQuery 99 │ 10799.67 / 10840.16 ±29.63 / 10882.86 ms │ 10858.77 / 10932.19 ±39.45 / 10969.92 ms │ no change │
└───────────┴──────────────────────────────────────────┴──────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 35215.56ms │
│ Total Time (foldhash-hashing) │ 34617.66ms │
│ Average Time (HEAD) │ 355.71ms │
│ Average Time (foldhash-hashing) │ 349.67ms │
│ Queries Faster │ 21 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 78 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 176.4s |
| Peak memory | 5.5 GiB |
| Avg memory | 4.6 GiB |
| CPU user | 278.9s |
| CPU sys | 22.1s |
| Disk read | 0 B |
| Disk write | 637.0 MiB |
**tpcds — branch**
| Metric | Value |
|--------|-------|
| Wall time | 173.4s |
| Peak memory | 5.2 GiB |
| Avg memory | 4.4 GiB |
| CPU user | 274.1s |
| CPU sys | 21.0s |
| Disk read | 0 B |
| Disk write | 784.0 KiB |
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066838997
Benchmark for [this request](https://github.com/apache/datafusion/pull/20958#issuecomment-4066829844) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
error[E0308]: mismatched types
--> datafusion/physical-expr/src/expressions/in_list.rs:200:34
|
200 | with_hashes([&in_array], &state, |hashes| -> Result<()> {
| ----------- ^^^^^^ expected `&FixedState`, found `&RandomState`
| |
| arguments to this function are incorrect
|
= note: expected reference `&foldhash::fast::FixedState`
found reference `&ahash::RandomState`
note: function defined here
--> datafusion/common/src/hash_utils.rs:97:8
|
97 | pub fn with_hashes<I, T, F, R>(
| ^^^^^^^^^^^
For more information about this error, try `rustc --explain E0308`.
error: could not compile `datafusion-physical-expr` (lib) due to 2 previous errors
warning: build failed, waiting for other jobs to finish...
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066979531
Benchmark for [this request](https://github.com/apache/datafusion/pull/20958#issuecomment-4066953218) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
error[E0308]: mismatched types
--> datafusion/physical-plan/src/windows/bounded_window_agg_exec.rs:711:32
|
711 | create_hashes(columns, &self.random_state, &mut batch_hashes)?;
| ------------- ^^^^^^^^^^^^^^^^^^ expected `&FixedState`, found `&RandomState`
| |
| arguments to this function are incorrect
|
= note: expected reference `&foldhash::fast::FixedState`
found reference `&ahash::RandomState`
note: function defined here
--> datafusion/common/src/hash_utils.rs:1094:8
|
1094 | pub fn create_hashes<'a, I, T>(
| ^^^^^^^^^^^^^
For more information about this error, try `rustc --explain E0308`.
error: could not compile `datafusion-physical-plan` (lib) due to 10 previous errors
warning: build failed, waiting for other jobs to finish...
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4067452276
run benchmark clickbench_extended
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#discussion_r2942405906
##########
datafusion/physical-plan/src/aggregates/mod.rs:
##########
@@ -87,8 +87,16 @@ pub fn topk_types_supported(key_type: &DataType, value_type: &DataType) -> bool
}
/// Hard-coded seed for aggregations to ensure hash values differ from `RepartitionExec`, avoiding collisions.
-const AGGREGATION_HASH_SEED: ahash::RandomState =
- ahash::RandomState::with_seeds('A' as u64, 'G' as u64, 'G' as u64, 'R' as u64);
+const AGGREGATION_HASH_SEED: datafusion_common::hash_utils::RandomState =
+ datafusion_common::hash_utils::RandomState::with_seed(
+ ('A' as u64)
+ .wrapping_mul(6364136223846793005)
Review Comment:
I made it just a random one now.
IMO we should also review the use of those fixed seeds in the code - I think some of them can be safely replaced by `RandomState::default()` without any issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#discussion_r2941040377
##########
datafusion/functions-aggregate/src/hyperloglog.rs:
##########
@@ -61,12 +61,7 @@ where
/// shared across cluster, this SEED will have to be consistent across all
/// parties otherwise we might have corruption. So ideally for later this seed
/// shall be part of the serialized form (or stay unchanged across versions).
-const SEED: RandomState = RandomState::with_seeds(
- 0x885f6cab121d01a3_u64,
- 0x71e4379f2976ad8f_u64,
- 0xbf30173dd28a8816_u64,
- 0x0eaea5d736d733a4_u64,
-);
+const SEED: foldhash::quality::FixedState = foldhash::quality::FixedState::with_seed(0);
Review Comment:
There are I think many things that break with new versions just HLL for DataFusion upgrades (protobuf changes being the major one).
HLL is one of them, but if we're going to change the hash function not something we can avoid.
One could use a versioning strategy (when deploying a new version of DataFusion).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#discussion_r2941275826
##########
datafusion/core/tests/sql/unparser.rs:
##########
@@ -368,8 +357,9 @@ async fn collect_results(ctx: &SessionContext, original: &str) -> TestCaseResult
}
};
- // Sort if needed for comparison
- if !is_sorted {
+ // Always sort for deterministic comparison — even "sorted" results can have
+ // tied rows in different order between original and unparsed SQL.
Review Comment:
The reason it now fails is
`RandomState::default()` of foldhash giving a random seed on each call, instead of returning the same random one that is generated once and reused after that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066979614
Benchmark for [this request](https://github.com/apache/datafusion/pull/20958#issuecomment-4066953218) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
error[E0308]: mismatched types
--> datafusion/physical-plan/src/windows/bounded_window_agg_exec.rs:711:32
|
711 | create_hashes(columns, &self.random_state, &mut batch_hashes)?;
| ------------- ^^^^^^^^^^^^^^^^^^ expected `&FixedState`, found `&RandomState`
| |
| arguments to this function are incorrect
|
= note: expected reference `&foldhash::fast::FixedState`
found reference `&ahash::RandomState`
note: function defined here
--> datafusion/common/src/hash_utils.rs:1094:8
|
1094 | pub fn create_hashes<'a, I, T>(
| ^^^^^^^^^^^^^
For more information about this error, try `rustc --explain E0308`.
error: could not compile `datafusion-physical-plan` (lib) due to 10 previous errors
warning: build failed, waiting for other jobs to finish...
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066805989
run benchmark with_hashes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#discussion_r2942958440
##########
datafusion/sqllogictest/test_files/aggregate.slt:
##########
@@ -1776,7 +1776,7 @@ SELECT approx_distinct(null)
query II
SELECT approx_distinct(c9) AS a, approx_distinct(c9) AS b FROM aggregate_test_100
----
-100 100
+99 99
Review Comment:
it is also consistent with the results from the query above which shows a slight difference
It is weird that approx_distinct in this query is 99 but it is 100 in the query right above it 🤔
##########
datafusion/proto/proto/datafusion.proto:
##########
@@ -1041,9 +1041,9 @@ message PhysicalExtensionExprNode {
message PhysicalHashExprNode {
repeated PhysicalExprNode on_columns = 1;
uint64 seed0 = 2;
- uint64 seed1 = 3;
- uint64 seed2 = 4;
- uint64 seed3 = 5;
+ // uint64 seed1 = 3;
Review Comment:
do we need this still? Shall we just remove them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066839107
Benchmark for [this request](https://github.com/apache/datafusion/pull/20958#issuecomment-4066829844) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
error[E0308]: mismatched types
--> datafusion/physical-expr/src/expressions/in_list.rs:200:34
|
200 | with_hashes([&in_array], &state, |hashes| -> Result<()> {
| ----------- ^^^^^^ expected `&FixedState`, found `&RandomState`
| |
| arguments to this function are incorrect
|
= note: expected reference `&foldhash::fast::FixedState`
found reference `&ahash::RandomState`
note: function defined here
--> datafusion/common/src/hash_utils.rs:97:8
|
97 | pub fn with_hashes<I, T, F, R>(
| ^^^^^^^^^^^
For more information about this error, try `rustc --explain E0308`.
error: could not compile `datafusion-physical-expr` (lib) due to 2 previous errors
warning: build failed, waiting for other jobs to finish...
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066833785
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4066829844)
`Linux bench-c4066829844-297-cqgq7 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing foldhash-hashing (83168bbc2f9d35a0e5082891c4afbfe6cb75bfbb) to 8609288 (merge-base) [diff](https://github.com/apache/datafusion/compare/8609288ae0595e715f6ab4fc6a5a56edf1b05eeb..83168bbc2f9d35a0e5082891c4afbfe6cb75bfbb) using: clickbench_partitioned
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4072451323
Thanks @alamb @adriangb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#discussion_r2940056327
##########
datafusion/sqllogictest/test_files/aggregate.slt:
##########
@@ -1776,7 +1776,7 @@ SELECT approx_distinct(null)
query II
SELECT approx_distinct(c9) AS a, approx_distinct(c9) AS b FROM aggregate_test_100
----
-100 100
+99 99
Review Comment:
This is within error bounds
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4067242298
run benchmark clickbench_partitioned
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4067236589
run clickbench_extended
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4070043922
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4070030473)
`Linux bench-c4070030473-337-2jcs5 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing foldhash-hashing (7d6860599bdcb705b20ef30e0770cfb5bf106b9d) to b61aee7 (merge-base) [diff](https://github.com/apache/datafusion/compare/b61aee77c849be3fb5391311870db41519ab6612..7d6860599bdcb705b20ef30e0770cfb5bf106b9d) using: tpcds
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066843441
Benchmark for [this request](https://github.com/apache/datafusion/pull/20958#issuecomment-4066829844) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
error[E0308]: mismatched types
--> datafusion/physical-expr/src/expressions/in_list.rs:200:34
|
200 | with_hashes([&in_array], &state, |hashes| -> Result<()> {
| ----------- ^^^^^^ expected `&FixedState`, found `&RandomState`
| |
| arguments to this function are incorrect
|
= note: expected reference `&foldhash::fast::FixedState`
found reference `&ahash::RandomState`
note: function defined here
--> datafusion/common/src/hash_utils.rs:97:8
|
97 | pub fn with_hashes<I, T, F, R>(
| ^^^^^^^^^^^
For more information about this error, try `rustc --explain E0308`.
error: could not compile `datafusion-physical-expr` (lib) due to 2 previous errors
warning: build failed, waiting for other jobs to finish...
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066819035
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4066805380)
`Linux bench-c4066805380-295-628pd 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing foldhash-hashing (b5a0933a53541f161cb336fdb0ffc01291cefc61) to 9c3c01a (merge-base) [diff](https://github.com/apache/datafusion/compare/9c3c01aa54fbf24bb0779e04d47a1492983ed4d7..b5a0933a53541f161cb336fdb0ffc01291cefc61) using: tpch
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4067205635
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4067123884)
<details><summary>Details</summary>
<p>
```
Comparing HEAD and foldhash-hashing
--------------------
Benchmark tpch_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ foldhash-hashing ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 1 │ 46.83 / 47.44 ±0.74 / 48.87 ms │ 46.03 / 46.64 ±0.70 / 47.94 ms │ no change │
│ QQuery 2 │ 24.86 / 25.29 ±0.41 / 26.04 ms │ 25.56 / 25.67 ±0.06 / 25.73 ms │ no change │
│ QQuery 3 │ 33.93 / 34.25 ±0.25 / 34.67 ms │ 33.82 / 34.68 ±0.91 / 36.36 ms │ no change │
│ QQuery 4 │ 21.92 / 22.47 ±0.48 / 23.05 ms │ 21.78 / 22.55 ±0.43 / 23.10 ms │ no change │
│ QQuery 5 │ 50.83 / 52.41 ±1.34 / 54.76 ms │ 50.55 / 52.25 ±1.31 / 53.84 ms │ no change │
│ QQuery 6 │ 17.59 / 17.99 ±0.40 / 18.55 ms │ 17.62 / 17.85 ±0.16 / 18.01 ms │ no change │
│ QQuery 7 │ 57.19 / 59.29 ±1.82 / 62.45 ms │ 57.43 / 59.06 ±1.67 / 61.36 ms │ no change │
│ QQuery 8 │ 51.59 / 52.01 ±0.43 / 52.82 ms │ 51.64 / 52.44 ±0.90 / 54.07 ms │ no change │
│ QQuery 9 │ 56.54 / 57.94 ±1.38 / 59.87 ms │ 56.95 / 57.50 ±0.50 / 58.35 ms │ no change │
│ QQuery 10 │ 74.15 / 75.05 ±1.06 / 77.11 ms │ 74.05 / 75.07 ±1.28 / 77.52 ms │ no change │
│ QQuery 11 │ 17.16 / 18.05 ±0.83 / 19.24 ms │ 17.27 / 18.39 ±1.74 / 21.85 ms │ no change │
│ QQuery 12 │ 28.88 / 29.35 ±0.40 / 29.98 ms │ 28.41 / 29.01 ±0.46 / 29.81 ms │ no change │
│ QQuery 13 │ 39.23 / 40.46 ±0.79 / 41.69 ms │ 40.69 / 41.19 ±0.48 / 41.85 ms │ no change │
│ QQuery 14 │ 29.61 / 29.80 ±0.10 / 29.89 ms │ 29.37 / 29.87 ±0.53 / 30.87 ms │ no change │
│ QQuery 15 │ 36.06 / 36.58 ±0.34 / 37.06 ms │ 36.43 / 37.00 ±0.45 / 37.80 ms │ no change │
│ QQuery 16 │ 18.10 / 18.44 ±0.20 / 18.71 ms │ 17.84 / 18.39 ±0.40 / 18.96 ms │ no change │
│ QQuery 17 │ 76.36 / 77.40 ±0.79 / 78.62 ms │ 73.15 / 74.32 ±0.73 / 75.38 ms │ no change │
│ QQuery 18 │ 82.66 / 83.71 ±0.82 / 84.53 ms │ 79.54 / 80.91 ±0.86 / 82.03 ms │ no change │
│ QQuery 19 │ 38.88 / 39.67 ±0.64 / 40.75 ms │ 38.93 / 39.53 ±0.34 / 39.86 ms │ no change │
│ QQuery 20 │ 44.00 / 44.56 ±0.44 / 45.12 ms │ 43.03 / 43.25 ±0.17 / 43.47 ms │ no change │
│ QQuery 21 │ 69.92 / 70.99 ±0.87 / 71.95 ms │ 71.12 / 71.85 ±0.97 / 73.67 ms │ no change │
│ QQuery 22 │ 20.11 / 20.30 ±0.19 / 20.66 ms │ 20.07 / 20.44 ±0.39 / 21.17 ms │ no change │
└───────────┴────────────────────────────────┴────────────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (HEAD) │ 953.47ms │
│ Total Time (foldhash-hashing) │ 947.86ms │
│ Average Time (HEAD) │ 43.34ms │
│ Average Time (foldhash-hashing) │ 43.08ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 22 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴──────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpch — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 5.1s |
| Peak memory | 4.0 GiB |
| Avg memory | 3.5 GiB |
| CPU user | 34.0s |
| CPU sys | 3.2s |
| Disk read | 0 B |
| Disk write | 148.0 KiB |
**tpch — branch**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 4.1 GiB |
| Avg memory | 3.5 GiB |
| CPU user | 33.6s |
| CPU sys | 3.1s |
| Disk read | 0 B |
| Disk write | 56.0 KiB |
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4067345967
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4067242298)
<details><summary>Details</summary>
<p>
```
Comparing HEAD and foldhash-hashing
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ foldhash-hashing ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.58 / 4.86 ±6.39 / 17.64 ms │ 1.63 / 4.95 ±6.48 / 17.91 ms │ no change │
│ QQuery 1 │ 14.98 / 15.30 ±0.22 / 15.51 ms │ 14.58 / 15.03 ±0.26 / 15.40 ms │ no change │
│ QQuery 2 │ 56.51 / 57.14 ±0.51 / 57.84 ms │ 57.12 / 58.20 ±1.33 / 60.79 ms │ no change │
│ QQuery 3 │ 50.10 / 52.40 ±1.42 / 54.34 ms │ 49.07 / 51.73 ±1.57 / 53.85 ms │ no change │
│ QQuery 4 │ 309.29 / 313.72 ±3.79 / 320.05 ms │ 298.41 / 305.77 ±8.91 / 321.54 ms │ no change │
│ QQuery 5 │ 351.82 / 357.61 ±3.58 / 362.66 ms │ 349.33 / 358.72 ±6.41 / 369.11 ms │ no change │
│ QQuery 6 │ 5.23 / 5.94 ±0.44 / 6.52 ms │ 5.70 / 6.67 ±0.91 / 7.83 ms │ 1.12x slower │
│ QQuery 7 │ 17.42 / 19.73 ±4.22 / 28.16 ms │ 17.34 / 18.11 ±0.93 / 19.91 ms │ +1.09x faster │
│ QQuery 8 │ 432.77 / 445.59 ±10.00 / 460.50 ms │ 430.52 / 440.10 ±6.03 / 445.28 ms │ no change │
│ QQuery 9 │ 683.66 / 697.93 ±7.33 / 704.44 ms │ 665.06 / 671.90 ±5.84 / 678.96 ms │ no change │
│ QQuery 10 │ 94.33 / 97.16 ±4.55 / 106.18 ms │ 93.47 / 96.29 ±2.62 / 100.33 ms │ no change │
│ QQuery 11 │ 109.70 / 111.65 ±1.49 / 113.97 ms │ 104.99 / 107.40 ±1.53 / 108.79 ms │ no change │
│ QQuery 12 │ 351.65 / 353.41 ±1.64 / 356.48 ms │ 350.21 / 357.46 ±6.87 / 368.92 ms │ no change │
│ QQuery 13 │ 456.24 / 469.90 ±9.11 / 482.26 ms │ 459.10 / 472.88 ±12.49 / 489.88 ms │ no change │
│ QQuery 14 │ 362.17 / 366.27 ±4.54 / 374.22 ms │ 358.71 / 361.80 ±4.51 / 370.53 ms │ no change │
│ QQuery 15 │ 372.00 / 383.07 ±8.75 / 398.55 ms │ 363.70 / 372.27 ±6.46 / 381.43 ms │ no change │
│ QQuery 16 │ 759.74 / 782.07 ±18.37 / 815.04 ms │ 741.25 / 762.78 ±26.49 / 814.14 ms │ no change │
│ QQuery 17 │ 751.27 / 773.21 ±28.42 / 828.75 ms │ 715.98 / 727.88 ±7.36 / 737.86 ms │ +1.06x faster │
│ QQuery 18 │ 1492.93 / 1565.76 ±50.97 / 1637.93 ms │ 1374.08 / 1388.07 ±8.37 / 1398.82 ms │ +1.13x faster │
│ QQuery 19 │ 36.87 / 39.30 ±1.74 / 42.17 ms │ 35.18 / 37.49 ±3.22 / 43.83 ms │ no change │
│ QQuery 20 │ 716.04 / 734.36 ±19.47 / 759.18 ms │ 708.36 / 711.96 ±2.88 / 715.40 ms │ no change │
│ QQuery 21 │ 763.51 / 769.47 ±5.05 / 776.54 ms │ 756.28 / 758.99 ±1.44 / 760.39 ms │ no change │
│ QQuery 22 │ 1138.85 / 1147.61 ±6.98 / 1160.21 ms │ 1128.27 / 1135.73 ±5.19 / 1144.18 ms │ no change │
│ QQuery 23 │ 3228.20 / 3252.43 ±19.98 / 3281.11 ms │ 3191.06 / 3208.00 ±15.89 / 3228.38 ms │ no change │
│ QQuery 24 │ 101.33 / 105.05 ±1.96 / 107.06 ms │ 101.88 / 106.61 ±3.48 / 111.09 ms │ no change │
│ QQuery 25 │ 143.72 / 145.44 ±2.09 / 149.54 ms │ 143.15 / 145.18 ±1.29 / 147.15 ms │ no change │
│ QQuery 26 │ 101.21 / 105.09 ±2.06 / 107.00 ms │ 104.38 / 105.40 ±0.89 / 106.91 ms │ no change │
│ QQuery 27 │ 853.90 / 861.36 ±4.37 / 866.23 ms │ 844.40 / 851.28 ±4.21 / 855.55 ms │ no change │
│ QQuery 28 │ 7794.13 / 7820.30 ±26.04 / 7868.98 ms │ 7749.32 / 7773.36 ±23.92 / 7815.53 ms │ no change │
│ QQuery 29 │ 56.26 / 62.14 ±8.35 / 78.56 ms │ 56.85 / 62.65 ±3.28 / 65.64 ms │ no change │
│ QQuery 30 │ 379.07 / 386.70 ±10.52 / 406.47 ms │ 372.85 / 379.36 ±6.05 / 390.22 ms │ no change │
│ QQuery 31 │ 377.96 / 386.54 ±5.00 / 392.62 ms │ 388.51 / 397.90 ±7.74 / 411.51 ms │ no change │
│ QQuery 32 │ 1099.48 / 1121.48 ±23.00 / 1165.84 ms │ 1132.91 / 1254.59 ±98.67 / 1383.36 ms │ 1.12x slower │
│ QQuery 33 │ 1529.77 / 1553.39 ±13.34 / 1563.96 ms │ 1491.41 / 1507.64 ±21.15 / 1549.32 ms │ no change │
│ QQuery 34 │ 1506.33 / 1523.58 ±8.71 / 1529.39 ms │ 1464.18 / 1498.24 ±21.35 / 1526.56 ms │ no change │
│ QQuery 35 │ 402.96 / 415.82 ±9.56 / 430.02 ms │ 389.36 / 399.12 ±5.37 / 403.62 ms │ no change │
│ QQuery 36 │ 120.59 / 126.62 ±3.43 / 129.98 ms │ 112.57 / 123.59 ±5.76 / 127.70 ms │ no change │
│ QQuery 37 │ 50.09 / 51.38 ±0.99 / 52.73 ms │ 51.03 / 52.25 ±0.99 / 53.19 ms │ no change │
│ QQuery 38 │ 75.13 / 76.33 ±1.28 / 78.07 ms │ 74.95 / 76.41 ±1.03 / 77.70 ms │ no change │
│ QQuery 39 │ 221.82 / 229.72 ±6.15 / 236.78 ms │ 214.83 / 219.91 ±5.60 / 228.28 ms │ no change │
│ QQuery 40 │ 27.31 / 27.90 ±0.46 / 28.66 ms │ 25.74 / 27.86 ±1.83 / 30.88 ms │ no change │
│ QQuery 41 │ 21.67 / 22.97 ±1.97 / 26.83 ms │ 21.74 / 22.51 ±0.42 / 23.01 ms │ no change │
│ QQuery 42 │ 21.22 / 21.80 ±0.39 / 22.18 ms │ 20.80 / 21.39 ±0.39 / 21.91 ms │ no change │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 27859.47ms │
│ Total Time (foldhash-hashing) │ 27455.44ms │
│ Average Time (HEAD) │ 647.89ms │
│ Average Time (foldhash-hashing) │ 638.50ms │
│ Queries Faster │ 3 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 38 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 140.8s |
| Peak memory | 48.3 GiB |
| Avg memory | 38.0 GiB |
| CPU user | 1318.1s |
| CPU sys | 91.5s |
| Disk read | 0 B |
| Disk write | 5.8 GiB |
**clickbench_partitioned — branch**
| Metric | Value |
|--------|-------|
| Wall time | 138.4s |
| Peak memory | 40.1 GiB |
| Avg memory | 33.0 GiB |
| CPU user | 1293.8s |
| CPU sys | 97.9s |
| Disk read | 0 B |
| Disk write | 116.0 KiB |
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4067568231
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4067452276)
<details><summary>Details</summary>
<p>
```
Comparing HEAD and foldhash-hashing
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ foldhash-hashing ┃ Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 837.21 / 852.18 ±14.24 / 878.58 ms │ 804.40 / 817.29 ±9.43 / 828.18 ms │ no change │
│ QQuery 1 │ 227.69 / 229.14 ±0.97 / 230.59 ms │ 205.52 / 206.46 ±0.73 / 207.34 ms │ +1.11x faster │
│ QQuery 2 │ 539.82 / 543.04 ±2.23 / 546.24 ms │ 494.41 / 497.67 ±2.48 / 501.78 ms │ +1.09x faster │
│ QQuery 3 │ 332.66 / 334.40 ±0.95 / 335.52 ms │ 316.22 / 317.22 ±1.01 / 319.03 ms │ +1.05x faster │
│ QQuery 4 │ 679.60 / 706.39 ±15.88 / 725.58 ms │ 661.89 / 683.66 ±13.14 / 701.34 ms │ no change │
│ QQuery 5 │ 9792.53 / 10044.39 ±206.48 / 10393.16 ms │ 9849.85 / 10093.84 ±234.40 / 10430.73 ms │ no change │
│ QQuery 6 │ 981.39 / 997.12 ±15.86 / 1027.36 ms │ 979.91 / 988.26 ±5.64 / 995.95 ms │ no change │
│ QQuery 7 │ 790.42 / 799.78 ±6.67 / 810.25 ms │ 787.77 / 845.31 ±54.15 / 916.42 ms │ 1.06x slower │
└──────────┴──────────────────────────────────────────┴──────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 14506.44ms │
│ Total Time (foldhash-hashing) │ 14449.70ms │
│ Average Time (HEAD) │ 1813.30ms │
│ Average Time (foldhash-hashing) │ 1806.21ms │
│ Queries Faster │ 3 │
│ Queries Slower │ 1 │
│ Queries with No Change │ 4 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_extended — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 73.2s |
| Peak memory | 32.5 GiB |
| Avg memory | 27.1 GiB |
| CPU user | 725.3s |
| CPU sys | 22.8s |
| Disk read | 0 B |
| Disk write | 21.8 MiB |
**clickbench_extended — branch**
| Metric | Value |
|--------|-------|
| Wall time | 72.7s |
| Peak memory | 30.7 GiB |
| Avg memory | 25.2 GiB |
| CPU user | 715.9s |
| CPU sys | 27.7s |
| Disk read | 0 B |
| Disk write | 724.0 KiB |
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#discussion_r2942405906
##########
datafusion/physical-plan/src/aggregates/mod.rs:
##########
@@ -87,8 +87,16 @@ pub fn topk_types_supported(key_type: &DataType, value_type: &DataType) -> bool
}
/// Hard-coded seed for aggregations to ensure hash values differ from `RepartitionExec`, avoiding collisions.
-const AGGREGATION_HASH_SEED: ahash::RandomState =
- ahash::RandomState::with_seeds('A' as u64, 'G' as u64, 'G' as u64, 'R' as u64);
+const AGGREGATION_HASH_SEED: datafusion_common::hash_utils::RandomState =
+ datafusion_common::hash_utils::RandomState::with_seed(
+ ('A' as u64)
+ .wrapping_mul(6364136223846793005)
Review Comment:
I made it just a random one now.
IMO we should also review the use of those fixed seeds in the code - I think many of them can be safely replaced by `RandomState::default()` without any issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066829844
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4067134104
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4067123884)
`Linux bench-c4067123884-309-qfqwh 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing foldhash-hashing (c78a48406aec42bc01820343c88ccee37e54eabe) to 8609288 (merge-base) [diff](https://github.com/apache/datafusion/compare/8609288ae0595e715f6ab4fc6a5a56edf1b05eeb..c78a48406aec42bc01820343c88ccee37e54eabe) using: tpch
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4067238363
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4067123884)
<details><summary>Details</summary>
<p>
```
Comparing HEAD and foldhash-hashing
--------------------
Benchmark tpcds_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ foldhash-hashing ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 50.08 / 51.04 ±0.81 / 52.51 ms │ 49.83 / 50.78 ±0.63 / 51.50 ms │ no change │
│ QQuery 2 │ 154.78 / 155.61 ±0.65 / 156.65 ms │ 148.82 / 150.07 ±0.77 / 151.23 ms │ no change │
│ QQuery 3 │ 121.06 / 121.70 ±0.64 / 122.60 ms │ 116.75 / 117.59 ±0.57 / 118.24 ms │ no change │
│ QQuery 4 │ 1437.09 / 1455.74 ±10.69 / 1468.67 ms │ 1396.18 / 1418.56 ±13.44 / 1430.06 ms │ no change │
│ QQuery 5 │ 187.11 / 189.08 ±1.33 / 190.95 ms │ 182.93 / 185.45 ±2.48 / 190.02 ms │ no change │
│ QQuery 6 │ 1035.79 / 1068.00 ±20.05 / 1098.88 ms │ 1004.22 / 1056.89 ±35.04 / 1102.66 ms │ no change │
│ QQuery 7 │ 361.93 / 365.74 ±3.24 / 370.63 ms │ 358.96 / 361.75 ±1.79 / 364.36 ms │ no change │
│ QQuery 8 │ 125.63 / 126.79 ±0.99 / 128.39 ms │ 121.31 / 122.40 ±1.01 / 124.10 ms │ no change │
│ QQuery 9 │ 120.86 / 122.81 ±1.15 / 123.89 ms │ 118.27 / 127.54 ±10.29 / 140.98 ms │ no change │
│ QQuery 10 │ 115.77 / 118.26 ±1.48 / 119.82 ms │ 113.26 / 114.16 ±0.68 / 114.77 ms │ no change │
│ QQuery 11 │ 1016.41 / 1031.94 ±13.81 / 1050.47 ms │ 997.90 / 1004.95 ±5.95 / 1012.78 ms │ no change │
│ QQuery 12 │ 48.17 / 49.90 ±1.48 / 52.32 ms │ 47.55 / 49.69 ±2.20 / 53.58 ms │ no change │
│ QQuery 13 │ 411.43 / 414.40 ±1.63 / 416.02 ms │ 410.27 / 411.93 ±1.82 / 415.23 ms │ no change │
│ QQuery 14 │ 1114.60 / 1119.27 ±4.00 / 1124.46 ms │ 1083.05 / 1085.16 ±2.48 / 1089.76 ms │ no change │
│ QQuery 15 │ 18.43 / 19.87 ±1.37 / 21.63 ms │ 19.02 / 19.94 ±1.15 / 22.03 ms │ no change │
│ QQuery 16 │ 47.28 / 47.68 ±0.33 / 48.27 ms │ 45.05 / 46.24 ±0.99 / 47.75 ms │ no change │
│ QQuery 17 │ 256.17 / 258.17 ±1.05 / 259.05 ms │ 246.30 / 248.90 ±1.62 / 250.44 ms │ no change │
│ QQuery 18 │ 134.65 / 136.95 ±1.42 / 138.87 ms │ 135.54 / 136.47 ±0.51 / 137.08 ms │ no change │
│ QQuery 19 │ 169.09 / 169.82 ±0.56 / 170.65 ms │ 161.59 / 163.69 ±1.74 / 166.66 ms │ no change │
│ QQuery 20 │ 16.98 / 17.62 ±0.50 / 18.37 ms │ 16.44 / 16.63 ±0.25 / 17.11 ms │ +1.06x faster │
│ QQuery 21 │ 25.24 / 25.97 ±0.53 / 26.68 ms │ 23.74 / 24.28 ±0.47 / 25.04 ms │ +1.07x faster │
│ QQuery 22 │ 525.09 / 530.74 ±3.73 / 535.04 ms │ 492.42 / 498.50 ±4.87 / 504.40 ms │ +1.06x faster │
│ QQuery 23 │ 976.83 / 989.67 ±8.73 / 1002.08 ms │ 950.99 / 955.19 ±4.46 / 963.23 ms │ no change │
│ QQuery 24 │ 446.26 / 449.18 ±2.82 / 453.64 ms │ 431.34 / 434.60 ±2.22 / 437.74 ms │ no change │
│ QQuery 25 │ 368.81 / 373.72 ±3.37 / 378.99 ms │ 358.65 / 362.00 ±2.19 / 365.20 ms │ no change │
│ QQuery 26 │ 88.09 / 89.11 ±1.68 / 92.47 ms │ 85.25 / 87.53 ±1.93 / 91.11 ms │ no change │
│ QQuery 27 │ 359.09 / 362.64 ±2.29 / 365.14 ms │ 354.18 / 355.99 ±2.14 / 360.07 ms │ no change │
│ QQuery 28 │ 154.74 / 156.26 ±0.94 / 157.44 ms │ 155.29 / 157.13 ±1.21 / 158.23 ms │ no change │
│ QQuery 29 │ 313.56 / 316.92 ±3.03 / 322.47 ms │ 301.59 / 306.03 ±2.34 / 308.48 ms │ no change │
│ QQuery 30 │ 50.12 / 52.16 ±1.76 / 55.13 ms │ 47.31 / 50.07 ±1.64 / 52.44 ms │ no change │
│ QQuery 31 │ 187.44 / 190.93 ±1.93 / 193.31 ms │ 178.22 / 180.69 ±1.60 / 183.13 ms │ +1.06x faster │
│ QQuery 32 │ 63.19 / 64.24 ±0.87 / 65.79 ms │ 63.55 / 64.49 ±0.61 / 65.33 ms │ no change │
│ QQuery 33 │ 153.32 / 154.00 ±0.61 / 155.00 ms │ 150.03 / 151.59 ±1.24 / 153.45 ms │ no change │
│ QQuery 34 │ 111.80 / 114.08 ±1.20 / 115.11 ms │ 110.07 / 110.85 ±0.81 / 112.36 ms │ no change │
│ QQuery 35 │ 120.20 / 121.50 ±0.95 / 123.12 ms │ 112.17 / 114.64 ±1.29 / 115.92 ms │ +1.06x faster │
│ QQuery 36 │ 226.55 / 228.66 ±2.06 / 231.70 ms │ 222.03 / 228.15 ±4.39 / 234.04 ms │ no change │
│ QQuery 37 │ 187.03 / 190.96 ±3.54 / 195.89 ms │ 177.47 / 180.95 ±2.56 / 184.36 ms │ +1.06x faster │
│ QQuery 38 │ 95.26 / 98.34 ±2.51 / 102.29 ms │ 95.17 / 98.02 ±1.81 / 100.81 ms │ no change │
│ QQuery 39 │ 141.96 / 144.87 ±1.51 / 146.11 ms │ 137.84 / 140.56 ±1.42 / 141.81 ms │ no change │
│ QQuery 40 │ 119.26 / 124.81 ±7.99 / 140.32 ms │ 118.20 / 122.61 ±3.60 / 128.98 ms │ no change │
│ QQuery 41 │ 18.31 / 18.82 ±0.58 / 19.93 ms │ 18.75 / 19.53 ±0.62 / 20.43 ms │ no change │
│ QQuery 42 │ 113.25 / 114.78 ±1.32 / 116.78 ms │ 111.46 / 111.67 ±0.13 / 111.87 ms │ no change │
│ QQuery 43 │ 89.12 / 90.56 ±1.06 / 91.76 ms │ 84.55 / 86.23 ±1.14 / 88.01 ms │ no change │
│ QQuery 44 │ 18.28 / 18.71 ±0.45 / 19.57 ms │ 17.72 / 18.16 ±0.34 / 18.65 ms │ no change │
│ QQuery 45 │ 58.78 / 59.12 ±0.34 / 59.58 ms │ 56.80 / 58.15 ±1.11 / 60.16 ms │ no change │
│ QQuery 46 │ 240.22 / 245.75 ±3.23 / 250.03 ms │ 240.14 / 242.54 ±1.27 / 243.60 ms │ no change │
│ QQuery 47 │ 771.26 / 781.87 ±5.45 / 785.66 ms │ 749.98 / 757.56 ±4.04 / 761.08 ms │ no change │
│ QQuery 48 │ 293.10 / 297.36 ±2.84 / 300.73 ms │ 295.66 / 299.84 ±3.23 / 303.59 ms │ no change │
│ QQuery 49 │ 267.52 / 269.32 ±2.02 / 272.80 ms │ 264.04 / 266.90 ±2.17 / 270.62 ms │ no change │
│ QQuery 50 │ 247.75 / 250.37 ±2.18 / 253.76 ms │ 237.82 / 241.33 ±4.24 / 248.98 ms │ no change │
│ QQuery 51 │ 194.84 / 198.89 ±3.93 / 204.82 ms │ 188.36 / 193.44 ±4.43 / 200.67 ms │ no change │
│ QQuery 52 │ 113.06 / 114.91 ±1.05 / 116.32 ms │ 110.55 / 111.80 ±1.23 / 113.75 ms │ no change │
│ QQuery 53 │ 109.47 / 111.38 ±1.73 / 114.17 ms │ 107.45 / 109.29 ±1.92 / 112.86 ms │ no change │
│ QQuery 54 │ 157.25 / 159.15 ±2.12 / 163.13 ms │ 151.54 / 153.06 ±0.93 / 154.33 ms │ no change │
│ QQuery 55 │ 112.61 / 113.78 ±0.97 / 115.58 ms │ 111.81 / 112.69 ±0.55 / 113.34 ms │ no change │
│ QQuery 56 │ 153.33 / 155.69 ±1.71 / 157.68 ms │ 151.02 / 152.08 ±0.71 / 153.06 ms │ no change │
│ QQuery 57 │ 192.87 / 196.42 ±2.19 / 199.31 ms │ 185.38 / 187.71 ±2.76 / 191.96 ms │ no change │
│ QQuery 58 │ 310.86 / 320.35 ±7.33 / 329.18 ms │ 297.21 / 306.35 ±6.41 / 314.07 ms │ no change │
│ QQuery 59 │ 210.94 / 212.70 ±1.59 / 214.76 ms │ 204.09 / 206.61 ±2.56 / 211.15 ms │ no change │
│ QQuery 60 │ 156.36 / 158.67 ±1.35 / 160.49 ms │ 152.32 / 154.53 ±1.34 / 156.04 ms │ no change │
│ QQuery 61 │ 181.39 / 183.47 ±1.42 / 185.22 ms │ 177.41 / 181.13 ±2.12 / 183.48 ms │ no change │
│ QQuery 62 │ 914.56 / 971.75 ±58.05 / 1083.50 ms │ 907.81 / 982.79 ±96.97 / 1172.30 ms │ no change │
│ QQuery 63 │ 109.95 / 111.49 ±1.30 / 113.22 ms │ 108.96 / 110.50 ±1.35 / 112.30 ms │ no change │
│ QQuery 64 │ 735.23 / 738.76 ±2.91 / 742.35 ms │ 722.96 / 727.39 ±2.77 / 730.35 ms │ no change │
│ QQuery 65 │ 271.12 / 275.28 ±3.67 / 280.76 ms │ 271.46 / 272.65 ±1.50 / 275.55 ms │ no change │
│ QQuery 66 │ 243.73 / 267.36 ±16.24 / 290.91 ms │ 257.76 / 268.48 ±12.91 / 289.16 ms │ no change │
│ QQuery 67 │ 345.15 / 352.70 ±4.47 / 358.29 ms │ 325.02 / 331.33 ±3.86 / 335.67 ms │ +1.06x faster │
│ QQuery 68 │ 289.72 / 297.83 ±4.35 / 301.79 ms │ 289.65 / 295.02 ±4.05 / 300.17 ms │ no change │
│ QQuery 69 │ 116.42 / 117.13 ±0.46 / 117.87 ms │ 108.44 / 109.67 ±0.77 / 110.67 ms │ +1.07x faster │
│ QQuery 70 │ 355.61 / 366.18 ±14.69 / 393.96 ms │ 347.15 / 355.73 ±13.44 / 382.44 ms │ no change │
│ QQuery 71 │ 141.37 / 143.44 ±2.21 / 147.30 ms │ 141.04 / 142.78 ±1.84 / 146.17 ms │ no change │
│ QQuery 72 │ 731.34 / 745.10 ±11.60 / 760.90 ms │ 726.18 / 737.48 ±10.00 / 751.75 ms │ no change │
│ QQuery 73 │ 108.40 / 110.45 ±1.30 / 112.37 ms │ 106.54 / 109.17 ±2.41 / 113.47 ms │ no change │
│ QQuery 74 │ 640.95 / 649.05 ±5.51 / 656.61 ms │ 635.83 / 642.69 ±4.22 / 647.70 ms │ no change │
│ QQuery 75 │ 297.99 / 300.51 ±2.07 / 303.43 ms │ 289.06 / 292.99 ±2.06 / 295.10 ms │ no change │
│ QQuery 76 │ 139.06 / 140.02 ±1.49 / 142.97 ms │ 139.19 / 140.95 ±1.20 / 142.88 ms │ no change │
│ QQuery 77 │ 206.68 / 207.80 ±0.86 / 209.19 ms │ 203.42 / 204.93 ±1.71 / 207.42 ms │ no change │
│ QQuery 78 │ 368.45 / 375.42 ±4.19 / 381.31 ms │ 364.67 / 367.00 ±1.92 / 370.43 ms │ no change │
│ QQuery 79 │ 242.95 / 248.14 ±2.69 / 250.45 ms │ 239.86 / 244.66 ±2.81 / 247.87 ms │ no change │
│ QQuery 80 │ 349.22 / 350.93 ±1.06 / 352.12 ms │ 341.52 / 345.75 ±3.19 / 350.12 ms │ no change │
│ QQuery 81 │ 33.17 / 33.95 ±0.67 / 34.77 ms │ 31.60 / 33.13 ±0.97 / 34.53 ms │ no change │
│ QQuery 82 │ 209.71 / 211.77 ±1.57 / 213.92 ms │ 204.38 / 206.09 ±1.12 / 207.80 ms │ no change │
│ QQuery 83 │ 48.80 / 50.01 ±0.77 / 50.93 ms │ 48.40 / 49.43 ±1.07 / 51.27 ms │ no change │
│ QQuery 84 │ 52.33 / 53.41 ±0.68 / 54.31 ms │ 52.24 / 52.83 ±0.36 / 53.28 ms │ no change │
│ QQuery 85 │ 156.03 / 157.44 ±1.41 / 159.61 ms │ 154.17 / 156.18 ±1.85 / 158.94 ms │ no change │
│ QQuery 86 │ 43.12 / 44.49 ±1.92 / 48.24 ms │ 41.76 / 43.01 ±1.16 / 45.13 ms │ no change │
│ QQuery 87 │ 95.01 / 99.61 ±3.36 / 103.43 ms │ 93.08 / 96.24 ±3.31 / 101.32 ms │ no change │
│ QQuery 88 │ 119.05 / 120.35 ±1.31 / 122.72 ms │ 113.45 / 114.60 ±0.91 / 116.23 ms │ no change │
│ QQuery 89 │ 127.23 / 128.78 ±0.99 / 130.20 ms │ 121.77 / 123.93 ±1.29 / 125.20 ms │ no change │
│ QQuery 90 │ 30.12 / 30.57 ±0.46 / 31.40 ms │ 28.91 / 29.42 ±0.32 / 29.77 ms │ no change │
│ QQuery 91 │ 67.16 / 68.36 ±0.79 / 69.47 ms │ 68.19 / 69.23 ±0.66 / 70.11 ms │ no change │
│ QQuery 92 │ 62.89 / 63.97 ±0.91 / 65.32 ms │ 63.54 / 64.76 ±0.70 / 65.49 ms │ no change │
│ QQuery 93 │ 201.48 / 203.86 ±2.49 / 208.19 ms │ 195.06 / 197.07 ±1.19 / 198.29 ms │ no change │
│ QQuery 94 │ 68.38 / 69.26 ±0.68 / 70.43 ms │ 65.69 / 66.69 ±0.68 / 67.47 ms │ no change │
│ QQuery 95 │ 144.07 / 147.60 ±2.78 / 151.48 ms │ 141.72 / 142.41 ±0.80 / 143.75 ms │ no change │
│ QQuery 96 │ 77.73 / 81.50 ±2.41 / 84.47 ms │ 73.38 / 76.36 ±1.72 / 78.51 ms │ +1.07x faster │
│ QQuery 97 │ 136.26 / 140.14 ±2.05 / 142.29 ms │ 135.18 / 137.96 ±2.33 / 141.78 ms │ no change │
│ QQuery 98 │ 156.49 / 159.43 ±2.60 / 162.95 ms │ 156.99 / 158.03 ±0.90 / 159.07 ms │ no change │
│ QQuery 99 │ 10777.14 / 10814.58 ±24.29 / 10853.81 ms │ 10807.55 / 10867.58 ±31.04 / 10897.17 ms │ no change │
└───────────┴──────────────────────────────────────────┴──────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 35339.61ms │
│ Total Time (foldhash-hashing) │ 34850.19ms │
│ Average Time (HEAD) │ 356.97ms │
│ Average Time (foldhash-hashing) │ 352.02ms │
│ Queries Faster │ 9 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 90 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 177.0s |
| Peak memory | 5.6 GiB |
| Avg memory | 4.4 GiB |
| CPU user | 280.7s |
| CPU sys | 21.9s |
| Disk read | 0 B |
| Disk write | 640.4 MiB |
**tpcds — branch**
| Metric | Value |
|--------|-------|
| Wall time | 174.5s |
| Peak memory | 5.2 GiB |
| Avg memory | 4.4 GiB |
| CPU user | 275.3s |
| CPU sys | 21.3s |
| Disk read | 0 B |
| Disk write | 792.0 KiB |
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#discussion_r2942998999
##########
datafusion/sqllogictest/test_files/aggregate.slt:
##########
@@ -1776,7 +1776,7 @@ SELECT approx_distinct(null)
query II
SELECT approx_distinct(c9) AS a, approx_distinct(c9) AS b FROM aggregate_test_100
----
-100 100
+99 99
Review Comment:
Hmm perhaps because it is executed slightly differently as a single column group by vs multi column? The second value above is also 99 (but also has 100 distinct values).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066819873
🤖 Criterion benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4066805989)
`Linux bench-c4066805989-296-p568z 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing foldhash-hashing (b5a0933a53541f161cb336fdb0ffc01291cefc61) to 9c3c01a (merge-base) [diff](https://github.com/apache/datafusion/compare/9c3c01aa54fbf24bb0779e04d47a1492983ed4d7..b5a0933a53541f161cb336fdb0ffc01291cefc61)
BENCH_NAME=with_hashes
BENCH_COMMAND=cargo bench --features=parquet --bench with_hashes
BENCH_FILTER=
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan merged PR #20958:
URL: https://github.com/apache/datafusion/pull/20958
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#discussion_r2941190385
##########
datafusion/physical-plan/src/joins/symmetric_hash_join.rs:
##########
@@ -239,7 +239,7 @@ impl SymmetricHashJoinExec {
build_join_schema(&left_schema, &right_schema, join_type);
// Initialize the random state for the join operation:
- let random_state = RandomState::with_seeds(0, 0, 0, 0);
+ let random_state = RandomState::with_seed(0);
Review Comment:
Not sure, I think might be good for a follow-up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4067139209
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4067123884)
`Linux bench-c4067123884-308-dshjt 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing foldhash-hashing (c78a48406aec42bc01820343c88ccee37e54eabe) to 8609288 (merge-base) [diff](https://github.com/apache/datafusion/compare/8609288ae0595e715f6ab4fc6a5a56edf1b05eeb..c78a48406aec42bc01820343c88ccee37e54eabe) using: tpcds
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4067238142
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4067123884)
<details><summary>Details</summary>
<p>
```
Comparing HEAD and foldhash-hashing
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ foldhash-hashing ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.64 / 4.98 ±6.49 / 17.96 ms │ 1.66 / 4.97 ±6.43 / 17.82 ms │ no change │
│ QQuery 1 │ 15.39 / 15.58 ±0.17 / 15.85 ms │ 16.04 / 16.55 ±0.30 / 16.88 ms │ 1.06x slower │
│ QQuery 2 │ 56.62 / 56.93 ±0.41 / 57.71 ms │ 58.05 / 58.36 ±0.24 / 58.68 ms │ no change │
│ QQuery 3 │ 50.18 / 51.77 ±1.17 / 52.99 ms │ 47.42 / 50.21 ±2.08 / 53.71 ms │ no change │
│ QQuery 4 │ 311.90 / 325.57 ±10.59 / 339.62 ms │ 304.26 / 310.54 ±4.25 / 315.08 ms │ no change │
│ QQuery 5 │ 364.25 / 367.53 ±3.05 / 371.89 ms │ 354.99 / 359.14 ±2.42 / 362.00 ms │ no change │
│ QQuery 6 │ 5.23 / 5.64 ±0.36 / 6.08 ms │ 5.31 / 6.37 ±0.99 / 8.03 ms │ 1.13x slower │
│ QQuery 7 │ 17.42 / 17.83 ±0.30 / 18.23 ms │ 18.19 / 19.08 ±1.55 / 22.17 ms │ 1.07x slower │
│ QQuery 8 │ 452.93 / 465.53 ±13.13 / 487.52 ms │ 441.60 / 449.30 ±4.47 / 454.66 ms │ no change │
│ QQuery 9 │ 702.23 / 709.90 ±4.53 / 714.95 ms │ 680.64 / 694.78 ±8.10 / 701.95 ms │ no change │
│ QQuery 10 │ 93.49 / 96.05 ±1.56 / 98.01 ms │ 96.13 / 99.95 ±5.75 / 111.33 ms │ no change │
│ QQuery 11 │ 107.71 / 109.16 ±0.83 / 110.29 ms │ 109.45 / 110.37 ±0.90 / 112.00 ms │ no change │
│ QQuery 12 │ 351.96 / 361.12 ±5.47 / 368.90 ms │ 357.59 / 359.74 ±2.53 / 363.63 ms │ no change │
│ QQuery 13 │ 462.57 / 480.00 ±15.36 / 503.17 ms │ 463.09 / 478.04 ±11.98 / 497.44 ms │ no change │
│ QQuery 14 │ 356.01 / 365.45 ±6.35 / 375.47 ms │ 361.96 / 371.82 ±8.66 / 387.20 ms │ no change │
│ QQuery 15 │ 377.90 / 393.13 ±14.53 / 416.91 ms │ 356.31 / 381.89 ±23.33 / 425.86 ms │ no change │
│ QQuery 16 │ 779.93 / 797.03 ±23.20 / 842.44 ms │ 733.93 / 751.43 ±23.23 / 796.57 ms │ +1.06x faster │
│ QQuery 17 │ 759.63 / 771.79 ±8.96 / 787.09 ms │ 728.55 / 738.08 ±5.72 / 746.42 ms │ no change │
│ QQuery 18 │ 1492.40 / 1522.90 ±29.34 / 1578.62 ms │ 1460.06 / 1524.40 ±45.48 / 1581.51 ms │ no change │
│ QQuery 19 │ 35.57 / 46.30 ±20.49 / 87.26 ms │ 36.50 / 39.17 ±1.93 / 42.49 ms │ +1.18x faster │
│ QQuery 20 │ 720.39 / 738.06 ±16.18 / 760.54 ms │ 715.98 / 727.36 ±13.52 / 748.66 ms │ no change │
│ QQuery 21 │ 759.61 / 764.62 ±4.86 / 772.60 ms │ 758.70 / 763.27 ±3.63 / 768.92 ms │ no change │
│ QQuery 22 │ 1136.43 / 1144.63 ±5.53 / 1153.02 ms │ 1136.17 / 1138.89 ±1.82 / 1141.10 ms │ no change │
│ QQuery 23 │ 3213.50 / 3236.35 ±19.03 / 3270.96 ms │ 3237.70 / 3261.02 ±16.65 / 3283.78 ms │ no change │
│ QQuery 24 │ 105.00 / 106.77 ±1.91 / 110.36 ms │ 105.46 / 108.46 ±2.33 / 112.66 ms │ no change │
│ QQuery 25 │ 141.88 / 145.32 ±2.85 / 149.54 ms │ 143.60 / 147.28 ±2.88 / 152.20 ms │ no change │
│ QQuery 26 │ 102.85 / 105.31 ±1.39 / 106.87 ms │ 103.72 / 106.54 ±1.61 / 108.35 ms │ no change │
│ QQuery 27 │ 851.30 / 857.01 ±8.61 / 874.06 ms │ 850.45 / 858.50 ±7.05 / 869.16 ms │ no change │
│ QQuery 28 │ 7722.99 / 7797.67 ±47.50 / 7851.07 ms │ 7727.76 / 7800.38 ±47.39 / 7864.21 ms │ no change │
│ QQuery 29 │ 56.39 / 97.63 ±74.84 / 247.21 ms │ 57.39 / 62.52 ±3.76 / 66.92 ms │ +1.56x faster │
│ QQuery 30 │ 382.51 / 388.57 ±6.45 / 400.64 ms │ 380.00 / 388.06 ±5.40 / 395.00 ms │ no change │
│ QQuery 31 │ 375.10 / 396.49 ±13.62 / 416.55 ms │ 388.45 / 397.53 ±10.83 / 417.62 ms │ no change │
│ QQuery 32 │ 1207.41 / 1326.52 ±82.62 / 1466.03 ms │ 1266.73 / 1323.71 ±44.55 / 1394.05 ms │ no change │
│ QQuery 33 │ 1477.90 / 1542.95 ±52.57 / 1631.05 ms │ 1504.79 / 1540.58 ±52.15 / 1639.98 ms │ no change │
│ QQuery 34 │ 1511.66 / 1530.12 ±22.56 / 1565.71 ms │ 1503.31 / 1511.99 ±6.80 / 1522.11 ms │ no change │
│ QQuery 35 │ 422.49 / 424.09 ±1.80 / 427.00 ms │ 399.28 / 407.41 ±4.68 / 413.72 ms │ no change │
│ QQuery 36 │ 121.00 / 126.06 ±3.40 / 131.67 ms │ 115.10 / 122.16 ±4.35 / 128.85 ms │ no change │
│ QQuery 37 │ 50.93 / 51.96 ±0.83 / 53.00 ms │ 50.57 / 53.60 ±1.91 / 56.14 ms │ no change │
│ QQuery 38 │ 76.72 / 78.58 ±1.17 / 79.90 ms │ 76.16 / 78.67 ±1.97 / 81.74 ms │ no change │
│ QQuery 39 │ 226.78 / 232.18 ±4.32 / 239.20 ms │ 214.68 / 222.46 ±6.56 / 233.04 ms │ no change │
│ QQuery 40 │ 25.79 / 28.32 ±1.78 / 30.33 ms │ 25.77 / 28.68 ±1.76 / 30.95 ms │ no change │
│ QQuery 41 │ 21.56 / 21.98 ±0.35 / 22.55 ms │ 21.62 / 23.22 ±0.97 / 24.50 ms │ 1.06x slower │
│ QQuery 42 │ 21.81 / 22.77 ±0.51 / 23.24 ms │ 20.79 / 21.82 ±0.82 / 23.20 ms │ no change │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 28128.14ms │
│ Total Time (foldhash-hashing) │ 27918.30ms │
│ Average Time (HEAD) │ 654.14ms │
│ Average Time (foldhash-hashing) │ 649.26ms │
│ Queries Faster │ 3 │
│ Queries Slower │ 4 │
│ Queries with No Change │ 36 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 141.7s |
| Peak memory | 40.3 GiB |
| Avg memory | 28.8 GiB |
| CPU user | 1317.4s |
| CPU sys | 104.0s |
| Disk read | 0 B |
| Disk write | 5.2 GiB |
**clickbench_partitioned — branch**
| Metric | Value |
|--------|-------|
| Wall time | 140.6s |
| Peak memory | 37.2 GiB |
| Avg memory | 28.1 GiB |
| CPU user | 1308.1s |
| CPU sys | 102.0s |
| Disk read | 0 B |
| Disk write | 756.0 KiB |
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4067714593
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4067587376)
<details><summary>Details</summary>
<p>
```
Comparing HEAD and foldhash-hashing
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ foldhash-hashing ┃ Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 858.97 / 888.06 ±37.67 / 958.53 ms │ 834.16 / 883.07 ±34.60 / 912.55 ms │ no change │
│ QQuery 1 │ 227.74 / 228.63 ±0.76 / 229.71 ms │ 205.48 / 207.44 ±1.72 / 210.35 ms │ +1.10x faster │
│ QQuery 2 │ 541.63 / 543.65 ±1.11 / 544.82 ms │ 499.61 / 502.19 ±1.76 / 504.94 ms │ +1.08x faster │
│ QQuery 3 │ 334.85 / 336.04 ±1.16 / 337.89 ms │ 316.43 / 317.67 ±1.02 / 319.48 ms │ +1.06x faster │
│ QQuery 4 │ 685.99 / 701.63 ±13.70 / 726.71 ms │ 692.46 / 704.06 ±13.23 / 724.37 ms │ no change │
│ QQuery 5 │ 9902.41 / 10285.01 ±255.92 / 10701.69 ms │ 9724.30 / 9894.96 ±128.18 / 10104.47 ms │ no change │
│ QQuery 6 │ 992.41 / 1009.01 ±18.31 / 1043.42 ms │ 999.09 / 1012.97 ±8.13 / 1021.97 ms │ no change │
│ QQuery 7 │ 795.02 / 806.15 ±7.80 / 818.08 ms │ 788.41 / 812.15 ±13.87 / 825.17 ms │ no change │
└──────────┴──────────────────────────────────────────┴─────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 14798.17ms │
│ Total Time (foldhash-hashing) │ 14334.51ms │
│ Average Time (HEAD) │ 1849.77ms │
│ Average Time (foldhash-hashing) │ 1791.81ms │
│ Queries Faster │ 3 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 5 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_extended — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 74.6s |
| Peak memory | 32.6 GiB |
| Avg memory | 28.0 GiB |
| CPU user | 737.3s |
| CPU sys | 24.0s |
| Disk read | 0 B |
| Disk write | 22.0 MiB |
**clickbench_extended — branch**
| Metric | Value |
|--------|-------|
| Wall time | 72.3s |
| Peak memory | 31.7 GiB |
| Avg memory | 27.2 GiB |
| CPU user | 714.7s |
| CPU sys | 23.9s |
| Disk read | 0 B |
| Disk write | 76.0 KiB |
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#discussion_r2941048984
##########
datafusion/physical-plan/src/joins/hash_join/partitioned_hash_eval.rs:
##########
@@ -47,8 +47,16 @@ pub struct SeededRandomState {
impl SeededRandomState {
/// Create a new SeededRandomState with the given seeds.
pub const fn with_seeds(k0: u64, k1: u64, k2: u64, k3: u64) -> Self {
+ // Combine 4 seeds into one for foldhash's single-seed API
Review Comment:
Probably would make sense. Probably the usage of `with_seeds` is low enough that we can just change the signature to use one seed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066820566
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4066805380)
`Linux bench-c4066805380-293-szvfg 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing foldhash-hashing (b5a0933a53541f161cb336fdb0ffc01291cefc61) to 9c3c01a (merge-base) [diff](https://github.com/apache/datafusion/compare/9c3c01aa54fbf24bb0779e04d47a1492983ed4d7..b5a0933a53541f161cb336fdb0ffc01291cefc61) using: clickbench_partitioned
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066830831
Benchmark for [this request](https://github.com/apache/datafusion/pull/20958#issuecomment-4066805380) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
| ^^^^^^^^^^^^^
error[E0308]: mismatched types
--> datafusion/physical-expr-common/src/binary_view_map.rs:259:33
|
259 | create_hashes([values], &self.random_state, batch_hashes)
| ------------- ^^^^^^^^^^^^^^^^^^ expected `&FixedState`, found `&RandomState`
| |
| arguments to this function are incorrect
|
= note: expected reference `&foldhash::fast::FixedState`
found reference `&ahash::RandomState`
note: function defined here
--> datafusion/common/src/hash_utils.rs:1094:8
|
1094 | pub fn create_hashes<'a, I, T>(
| ^^^^^^^^^^^^^
For more information about this error, try `rustc --explain E0308`.
error: could not compile `datafusion-physical-expr-common` (lib) due to 2 previous errors
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066831901
Benchmark for [this request](https://github.com/apache/datafusion/pull/20958#issuecomment-4066805380) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
| ^^^^^^^^^^^^^
error[E0308]: mismatched types
--> datafusion/physical-expr-common/src/binary_view_map.rs:259:33
|
259 | create_hashes([values], &self.random_state, batch_hashes)
| ------------- ^^^^^^^^^^^^^^^^^^ expected `&FixedState`, found `&RandomState`
| |
| arguments to this function are incorrect
|
= note: expected reference `&foldhash::fast::FixedState`
found reference `&ahash::RandomState`
note: function defined here
--> datafusion/common/src/hash_utils.rs:1094:8
|
1094 | pub fn create_hashes<'a, I, T>(
| ^^^^^^^^^^^^^
For more information about this error, try `rustc --explain E0308`.
error: could not compile `datafusion-physical-expr-common` (lib) due to 2 previous errors
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066844454
🤖 Criterion benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4066830279)
`Linux bench-c4066830279-300-wg5bp 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing foldhash-hashing (83168bbc2f9d35a0e5082891c4afbfe6cb75bfbb) to 8609288 (merge-base) [diff](https://github.com/apache/datafusion/compare/8609288ae0595e715f6ab4fc6a5a56edf1b05eeb..83168bbc2f9d35a0e5082891c4afbfe6cb75bfbb)
BENCH_NAME=with_hashes
BENCH_COMMAND=cargo bench --features=parquet --bench with_hashes
BENCH_FILTER=
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066830279
run benchmark with_hashes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4070148879
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4070030473)
<details><summary>Details</summary>
<p>
```
Comparing HEAD and foldhash-hashing
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ foldhash-hashing ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.69 / 5.05 ±6.51 / 18.06 ms │ 1.94 / 5.34 ±6.65 / 18.63 ms │ 1.06x slower │
│ QQuery 1 │ 15.38 / 15.82 ±0.35 / 16.35 ms │ 16.40 / 16.61 ±0.12 / 16.75 ms │ 1.05x slower │
│ QQuery 2 │ 57.84 / 58.61 ±0.44 / 59.07 ms │ 58.55 / 59.13 ±0.61 / 60.30 ms │ no change │
│ QQuery 3 │ 49.54 / 51.87 ±1.25 / 53.18 ms │ 53.34 / 55.22 ±1.02 / 56.16 ms │ 1.06x slower │
│ QQuery 4 │ 301.53 / 314.15 ±10.92 / 333.76 ms │ 327.81 / 342.08 ±8.97 / 351.43 ms │ 1.09x slower │
│ QQuery 5 │ 353.25 / 363.63 ±7.37 / 371.64 ms │ 366.59 / 387.96 ±15.02 / 407.91 ms │ 1.07x slower │
│ QQuery 6 │ 5.36 / 7.34 ±2.02 / 10.82 ms │ 5.77 / 6.31 ±0.32 / 6.77 ms │ +1.16x faster │
│ QQuery 7 │ 17.42 / 18.04 ±0.44 / 18.79 ms │ 18.58 / 19.82 ±1.16 / 21.40 ms │ 1.10x slower │
│ QQuery 8 │ 457.11 / 473.44 ±15.96 / 503.15 ms │ 471.78 / 496.83 ±18.20 / 518.49 ms │ no change │
│ QQuery 9 │ 675.38 / 701.27 ±17.49 / 726.03 ms │ 653.57 / 670.00 ±10.10 / 681.36 ms │ no change │
│ QQuery 10 │ 95.16 / 97.89 ±2.59 / 102.50 ms │ 93.90 / 96.50 ±1.58 / 98.82 ms │ no change │
│ QQuery 11 │ 107.79 / 109.66 ±1.68 / 111.99 ms │ 106.48 / 107.53 ±1.01 / 109.37 ms │ no change │
│ QQuery 12 │ 351.54 / 356.81 ±5.48 / 365.95 ms │ 343.61 / 360.02 ±10.77 / 372.08 ms │ no change │
│ QQuery 13 │ 465.21 / 475.81 ±8.96 / 491.73 ms │ 459.09 / 484.64 ±18.92 / 507.53 ms │ no change │
│ QQuery 14 │ 361.77 / 365.85 ±4.81 / 374.39 ms │ 347.32 / 361.84 ±8.43 / 371.70 ms │ no change │
│ QQuery 15 │ 377.23 / 405.62 ±22.47 / 438.20 ms │ 370.47 / 391.46 ±26.17 / 442.94 ms │ no change │
│ QQuery 16 │ 773.75 / 814.12 ±32.77 / 860.63 ms │ 740.56 / 776.18 ±40.94 / 853.25 ms │ no change │
│ QQuery 17 │ 755.84 / 783.57 ±16.42 / 800.29 ms │ 726.58 / 736.47 ±5.88 / 743.15 ms │ +1.06x faster │
│ QQuery 18 │ 1462.37 / 1528.24 ±55.26 / 1600.91 ms │ 1431.52 / 1519.66 ±60.46 / 1603.26 ms │ no change │
│ QQuery 19 │ 38.56 / 59.23 ±27.83 / 113.79 ms │ 37.98 / 39.69 ±1.47 / 42.08 ms │ +1.49x faster │
│ QQuery 20 │ 714.56 / 740.58 ±24.62 / 777.93 ms │ 716.02 / 731.57 ±16.35 / 762.94 ms │ no change │
│ QQuery 21 │ 755.23 / 763.21 ±5.94 / 772.22 ms │ 758.79 / 775.24 ±10.27 / 791.21 ms │ no change │
│ QQuery 22 │ 1127.16 / 1134.66 ±7.28 / 1147.05 ms │ 1132.46 / 1141.66 ±7.06 / 1150.52 ms │ no change │
│ QQuery 23 │ 3223.69 / 3261.72 ±24.92 / 3298.24 ms │ 3184.50 / 3210.01 ±28.80 / 3263.95 ms │ no change │
│ QQuery 24 │ 102.90 / 106.20 ±3.14 / 112.12 ms │ 103.03 / 107.26 ±4.50 / 115.88 ms │ no change │
│ QQuery 25 │ 141.21 / 143.92 ±2.22 / 147.62 ms │ 144.89 / 146.60 ±1.01 / 147.69 ms │ no change │
│ QQuery 26 │ 103.64 / 104.74 ±0.87 / 105.93 ms │ 106.34 / 108.48 ±1.67 / 110.93 ms │ no change │
│ QQuery 27 │ 844.76 / 860.11 ±11.86 / 874.68 ms │ 843.26 / 850.66 ±6.01 / 861.37 ms │ no change │
│ QQuery 28 │ 7764.55 / 7792.94 ±25.49 / 7839.33 ms │ 7720.47 / 7789.58 ±41.54 / 7843.91 ms │ no change │
│ QQuery 29 │ 56.72 / 60.44 ±5.93 / 72.16 ms │ 56.79 / 59.83 ±2.93 / 65.12 ms │ no change │
│ QQuery 30 │ 382.70 / 388.87 ±5.35 / 397.65 ms │ 364.71 / 375.46 ±9.03 / 387.48 ms │ no change │
│ QQuery 31 │ 373.77 / 403.78 ±18.34 / 424.40 ms │ 373.46 / 381.24 ±8.10 / 393.35 ms │ +1.06x faster │
│ QQuery 32 │ 1094.76 / 1142.90 ±33.39 / 1197.97 ms │ 1053.40 / 1070.45 ±18.56 / 1102.33 ms │ +1.07x faster │
│ QQuery 33 │ 1486.25 / 1531.45 ±33.09 / 1584.92 ms │ 1462.21 / 1515.87 ±31.49 / 1544.30 ms │ no change │
│ QQuery 34 │ 1527.34 / 1573.30 ±31.28 / 1610.52 ms │ 1451.54 / 1488.85 ±38.86 / 1561.96 ms │ +1.06x faster │
│ QQuery 35 │ 454.03 / 473.02 ±17.19 / 494.34 ms │ 385.89 / 403.72 ±18.19 / 438.24 ms │ +1.17x faster │
│ QQuery 36 │ 119.61 / 129.11 ±6.16 / 135.64 ms │ 124.03 / 127.52 ±1.94 / 129.59 ms │ no change │
│ QQuery 37 │ 52.29 / 55.79 ±2.15 / 58.00 ms │ 49.85 / 50.94 ±0.63 / 51.57 ms │ +1.10x faster │
│ QQuery 38 │ 76.88 / 79.58 ±1.44 / 80.99 ms │ 74.64 / 77.47 ±2.23 / 80.23 ms │ no change │
│ QQuery 39 │ 240.64 / 246.35 ±5.99 / 257.39 ms │ 217.18 / 224.36 ±10.00 / 243.87 ms │ +1.10x faster │
│ QQuery 40 │ 25.49 / 28.76 ±2.13 / 31.64 ms │ 23.71 / 26.56 ±1.98 / 29.10 ms │ +1.08x faster │
│ QQuery 41 │ 23.03 / 25.11 ±2.12 / 28.83 ms │ 21.67 / 23.12 ±1.60 / 26.23 ms │ +1.09x faster │
│ QQuery 42 │ 21.99 / 23.14 ±0.60 / 23.71 ms │ 20.09 / 21.06 ±0.60 / 21.90 ms │ +1.10x faster │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 28075.69ms │
│ Total Time (foldhash-hashing) │ 27640.76ms │
│ Average Time (HEAD) │ 652.92ms │
│ Average Time (foldhash-hashing) │ 642.81ms │
│ Queries Faster │ 12 │
│ Queries Slower │ 6 │
│ Queries with No Change │ 25 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 141.7s |
| Peak memory | 41.7 GiB |
| Avg memory | 31.9 GiB |
| CPU user | 1326.4s |
| CPU sys | 92.7s |
| Disk read | 0 B |
| Disk write | 4.2 GiB |
**clickbench_partitioned — branch**
| Metric | Value |
|--------|-------|
| Wall time | 139.4s |
| Peak memory | 43.0 GiB |
| Avg memory | 30.4 GiB |
| CPU user | 1307.4s |
| CPU sys | 93.5s |
| Disk read | 0 B |
| Disk write | 760.0 KiB |
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4070030473
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4070043884
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4070030473)
`Linux bench-c4070030473-336-scmmg 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing foldhash-hashing (7d6860599bdcb705b20ef30e0770cfb5bf106b9d) to b61aee7 (merge-base) [diff](https://github.com/apache/datafusion/compare/b61aee77c849be3fb5391311870db41519ab6612..7d6860599bdcb705b20ef30e0770cfb5bf106b9d) using: clickbench_partitioned
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4070040885
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4070030473)
`Linux bench-c4070030473-338-zmcdm 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing foldhash-hashing (7d6860599bdcb705b20ef30e0770cfb5bf106b9d) to b61aee7 (merge-base) [diff](https://github.com/apache/datafusion/compare/b61aee77c849be3fb5391311870db41519ab6612..7d6860599bdcb705b20ef30e0770cfb5bf106b9d) using: tpch
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4070108048
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4070030473)
<details><summary>Details</summary>
<p>
```
Comparing HEAD and foldhash-hashing
--------------------
Benchmark tpch_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ foldhash-hashing ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 1 │ 46.55 / 47.70 ±1.17 / 49.61 ms │ 46.44 / 47.03 ±0.68 / 48.35 ms │ no change │
│ QQuery 2 │ 25.33 / 26.07 ±0.49 / 26.81 ms │ 26.26 / 26.70 ±0.45 / 27.43 ms │ no change │
│ QQuery 3 │ 34.22 / 34.53 ±0.47 / 35.45 ms │ 34.45 / 34.81 ±0.19 / 35.00 ms │ no change │
│ QQuery 4 │ 21.87 / 22.99 ±1.07 / 24.54 ms │ 21.83 / 22.81 ±0.79 / 23.80 ms │ no change │
│ QQuery 5 │ 51.76 / 52.23 ±0.62 / 53.42 ms │ 51.52 / 53.76 ±2.12 / 57.63 ms │ no change │
│ QQuery 6 │ 17.65 / 18.23 ±0.56 / 19.02 ms │ 18.09 / 18.31 ±0.15 / 18.51 ms │ no change │
│ QQuery 7 │ 57.04 / 58.49 ±1.23 / 60.01 ms │ 57.21 / 58.82 ±0.99 / 60.28 ms │ no change │
│ QQuery 8 │ 52.38 / 52.91 ±0.49 / 53.61 ms │ 52.22 / 52.68 ±0.39 / 53.24 ms │ no change │
│ QQuery 9 │ 57.15 / 58.91 ±1.63 / 61.32 ms │ 57.33 / 59.35 ±1.58 / 61.94 ms │ no change │
│ QQuery 10 │ 72.92 / 75.26 ±2.23 / 78.24 ms │ 73.68 / 74.14 ±0.47 / 74.95 ms │ no change │
│ QQuery 11 │ 17.29 / 17.70 ±0.23 / 17.99 ms │ 17.93 / 18.15 ±0.12 / 18.32 ms │ no change │
│ QQuery 12 │ 28.84 / 29.32 ±0.54 / 30.37 ms │ 29.04 / 29.48 ±0.37 / 30.14 ms │ no change │
│ QQuery 13 │ 41.22 / 42.06 ±0.84 / 43.09 ms │ 39.75 / 40.71 ±1.13 / 42.91 ms │ no change │
│ QQuery 14 │ 29.72 / 29.96 ±0.26 / 30.43 ms │ 29.82 / 30.13 ±0.47 / 31.07 ms │ no change │
│ QQuery 15 │ 36.78 / 37.72 ±1.24 / 40.08 ms │ 37.73 / 39.10 ±1.78 / 42.58 ms │ no change │
│ QQuery 16 │ 18.46 / 18.92 ±0.34 / 19.35 ms │ 18.14 / 18.84 ±0.69 / 19.93 ms │ no change │
│ QQuery 17 │ 76.96 / 78.02 ±1.06 / 79.81 ms │ 75.72 / 76.96 ±0.83 / 78.12 ms │ no change │
│ QQuery 18 │ 81.95 / 84.42 ±2.56 / 88.45 ms │ 80.76 / 83.93 ±2.64 / 88.73 ms │ no change │
│ QQuery 19 │ 38.94 / 39.34 ±0.33 / 39.93 ms │ 39.89 / 40.48 ±0.94 / 42.34 ms │ no change │
│ QQuery 20 │ 44.95 / 45.96 ±0.83 / 47.21 ms │ 44.86 / 45.75 ±0.66 / 46.51 ms │ no change │
│ QQuery 21 │ 70.11 / 73.62 ±1.89 / 75.36 ms │ 71.42 / 72.44 ±1.06 / 74.00 ms │ no change │
│ QQuery 22 │ 20.48 / 21.12 ±0.38 / 21.62 ms │ 20.68 / 20.99 ±0.24 / 21.42 ms │ no change │
└───────────┴────────────────────────────────┴────────────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (HEAD) │ 965.47ms │
│ Total Time (foldhash-hashing) │ 965.37ms │
│ Average Time (HEAD) │ 43.88ms │
│ Average Time (foldhash-hashing) │ 43.88ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 22 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴──────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpch — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 5.1s |
| Peak memory | 4.0 GiB |
| Avg memory | 3.5 GiB |
| CPU user | 34.1s |
| CPU sys | 3.3s |
| Disk read | 0 B |
| Disk write | 148.0 KiB |
**tpch — branch**
| Metric | Value |
|--------|-------|
| Wall time | 5.1s |
| Peak memory | 4.0 GiB |
| Avg memory | 3.5 GiB |
| CPU user | 33.8s |
| CPU sys | 3.3s |
| Disk read | 0 B |
| Disk write | 56.0 KiB |
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066830666
Benchmark for [this request](https://github.com/apache/datafusion/pull/20958#issuecomment-4066805380) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
| ^^^^^^^^^^^^^
error[E0308]: mismatched types
--> datafusion/physical-expr-common/src/binary_view_map.rs:259:33
|
259 | create_hashes([values], &self.random_state, batch_hashes)
| ------------- ^^^^^^^^^^^^^^^^^^ expected `&FixedState`, found `&RandomState`
| |
| arguments to this function are incorrect
|
= note: expected reference `&foldhash::fast::FixedState`
found reference `&ahash::RandomState`
note: function defined here
--> datafusion/common/src/hash_utils.rs:1094:8
|
1094 | pub fn create_hashes<'a, I, T>(
| ^^^^^^^^^^^^^
For more information about this error, try `rustc --explain E0308`.
error: could not compile `datafusion-physical-expr-common` (lib) due to 2 previous errors
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066833218
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4066829844)
`Linux bench-c4066829844-298-znlk7 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing foldhash-hashing (83168bbc2f9d35a0e5082891c4afbfe6cb75bfbb) to 8609288 (merge-base) [diff](https://github.com/apache/datafusion/compare/8609288ae0595e715f6ab4fc6a5a56edf1b05eeb..83168bbc2f9d35a0e5082891c4afbfe6cb75bfbb) using: tpcds
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066805380
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066932449
🤖 Criterion benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4066805989)
<details><summary>Details</summary>
<p>
```
group foldhash-hashing main
----- ---------------- ----
dense_union: multiple, no nulls 1.00 237.8±0.32µs ? ?/sec 1.09 258.6±0.38µs ? ?/sec
dense_union: single, no nulls 1.00 79.5±0.09µs ? ?/sec 1.09 86.5±0.12µs ? ?/sec
dictionary_utf8_int32: multiple, no nulls 1.20 17.8±1.04µs ? ?/sec 1.00 14.9±0.11µs ? ?/sec
dictionary_utf8_int32: multiple, nulls 1.00 32.7±0.12µs ? ?/sec 1.00 32.7±0.02µs ? ?/sec
dictionary_utf8_int32: single, no nulls 1.00 4.6±0.00µs ? ?/sec 1.02 4.7±0.00µs ? ?/sec
dictionary_utf8_int32: single, nulls 1.00 11.0±0.01µs ? ?/sec 1.02 11.2±0.02µs ? ?/sec
int64: multiple, no nulls 1.00 16.9±0.01µs ? ?/sec 1.98 33.6±0.14µs ? ?/sec
int64: multiple, nulls 1.00 26.7±0.01µs ? ?/sec 1.43 38.1±0.03µs ? ?/sec
int64: single, no nulls 1.00 4.6±0.00µs ? ?/sec 2.61 11.9±0.02µs ? ?/sec
int64: single, nulls 1.00 8.7±0.01µs ? ?/sec 1.34 11.7±0.01µs ? ?/sec
large_utf8: multiple, no nulls 1.00 109.9±0.36µs ? ?/sec 1.39 153.1±0.65µs ? ?/sec
large_utf8: multiple, nulls 1.00 135.7±0.43µs ? ?/sec 1.50 203.1±0.76µs ? ?/sec
large_utf8: single, no nulls 1.00 26.9±0.35µs ? ?/sec 1.44 38.8±0.13µs ? ?/sec
large_utf8: single, nulls 1.00 44.7±0.89µs ? ?/sec 1.33 59.4±0.33µs ? ?/sec
list_array: multiple, no nulls 1.00 148.0±0.11µs ? ?/sec 1.74 257.9±2.72µs ? ?/sec
list_array: multiple, nulls 1.00 175.6±0.29µs ? ?/sec 1.60 280.9±1.06µs ? ?/sec
list_array: single, no nulls 1.00 49.4±0.11µs ? ?/sec 1.71 84.4±0.13µs ? ?/sec
list_array: single, nulls 1.00 58.7±0.07µs ? ?/sec 1.60 93.6±0.06µs ? ?/sec
list_array_sliced: 1/10 of 81920 rows 1.00 49.0±0.03µs ? ?/sec 1.72 84.2±0.15µs ? ?/sec
list_array_sliced: 1/2 of 16384 rows 1.00 49.5±0.21µs ? ?/sec 1.71 84.6±0.09µs ? ?/sec
list_array_sliced: 1/5 of 40960 rows 1.00 49.5±0.10µs ? ?/sec 1.73 85.5±0.96µs ? ?/sec
map_array: multiple, no nulls 1.00 241.3±0.34µs ? ?/sec 1.73 418.4±0.22µs ? ?/sec
map_array: multiple, nulls 1.00 247.6±0.15µs ? ?/sec 1.72 427.0±0.74µs ? ?/sec
map_array: single, no nulls 1.00 80.9±0.05µs ? ?/sec 1.72 139.3±0.14µs ? ?/sec
map_array: single, nulls 1.00 82.7±0.04µs ? ?/sec 1.74 143.4±0.10µs ? ?/sec
map_array_sliced: 1/10 of 81920 rows 1.00 80.8±0.05µs ? ?/sec 1.73 140.1±0.07µs ? ?/sec
map_array_sliced: 1/2 of 16384 rows 1.00 80.7±0.05µs ? ?/sec 1.72 139.2±0.09µs ? ?/sec
map_array_sliced: 1/5 of 40960 rows 1.00 80.5±0.05µs ? ?/sec 1.74 139.7±0.10µs ? ?/sec
run_array_int32: multiple, no nulls 1.00 4.9±0.01µs ? ?/sec 1.16 5.7±0.01µs ? ?/sec
run_array_int32: multiple, nulls 1.00 6.7±0.01µs ? ?/sec 1.04 7.0±0.01µs ? ?/sec
run_array_int32: single, no nulls 1.00 1815.3±4.21ns ? ?/sec 1.15 2.1±0.00µs ? ?/sec
run_array_int32: single, nulls 1.00 2.3±0.01µs ? ?/sec 1.04 2.4±0.00µs ? ?/sec
sparse_union: multiple, no nulls 1.00 260.1±0.55µs ? ?/sec 1.08 282.0±0.94µs ? ?/sec
sparse_union: single, no nulls 1.00 87.0±0.27µs ? ?/sec 1.08 94.3±0.20µs ? ?/sec
sparse_union_sliced: 1/10 of 81920 rows 1.00 87.1±0.32µs ? ?/sec 1.09 94.9±0.21µs ? ?/sec
sparse_union_sliced: 1/2 of 16384 rows 1.00 87.1±0.16µs ? ?/sec 1.09 94.8±0.49µs ? ?/sec
sparse_union_sliced: 1/5 of 40960 rows 1.00 87.0±0.23µs ? ?/sec 1.09 94.5±0.28µs ? ?/sec
struct_array: multiple, no nulls 1.00 159.5±0.26µs ? ?/sec 1.72 273.6±1.10µs ? ?/sec
struct_array: multiple, nulls 1.00 171.2±0.99µs ? ?/sec 1.65 282.7±0.32µs ? ?/sec
struct_array: single, no nulls 1.00 54.3±0.26µs ? ?/sec 1.69 91.9±0.13µs ? ?/sec
struct_array: single, nulls 1.00 57.0±0.26µs ? ?/sec 1.67 95.1±0.16µs ? ?/sec
utf8: multiple, no nulls 1.00 102.7±0.28µs ? ?/sec 1.51 155.2±1.71µs ? ?/sec
utf8: multiple, nulls 1.00 143.9±1.87µs ? ?/sec 1.46 210.2±0.69µs ? ?/sec
utf8: single, no nulls 1.00 27.3±0.16µs ? ?/sec 1.44 39.3±0.15µs ? ?/sec
utf8: single, nulls 1.00 46.2±0.76µs ? ?/sec 1.31 60.3±0.25µs ? ?/sec
utf8_view (small): multiple, no nulls 1.00 19.3±0.13µs ? ?/sec 1.88 36.4±0.01µs ? ?/sec
utf8_view (small): multiple, nulls 1.00 29.2±0.07µs ? ?/sec 1.55 45.2±0.02µs ? ?/sec
utf8_view (small): single, no nulls 1.00 5.1±0.01µs ? ?/sec 2.16 11.0±0.00µs ? ?/sec
utf8_view (small): single, nulls 1.00 9.7±0.01µs ? ?/sec 1.45 14.1±0.01µs ? ?/sec
utf8_view: multiple, no nulls 1.00 108.0±0.81µs ? ?/sec 1.29 139.0±0.34µs ? ?/sec
utf8_view: multiple, nulls 1.00 120.3±0.31µs ? ?/sec 1.24 149.1±0.25µs ? ?/sec
utf8_view: single, no nulls 1.00 33.1±0.06µs ? ?/sec 1.13 37.4±0.04µs ? ?/sec
utf8_view: single, nulls 1.00 39.2±0.70µs ? ?/sec 1.12 43.8±0.05µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 526.8s |
| Peak memory | 3.4 GiB |
| Avg memory | 3.4 GiB |
| CPU user | 662.2s |
| CPU sys | 1.3s |
| Disk read | 0 B |
| Disk write | 523.1 MiB |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 523.9s |
| Peak memory | 3.4 GiB |
| Avg memory | 3.4 GiB |
| CPU user | 660.3s |
| CPU sys | 0.7s |
| Disk read | 0 B |
| Disk write | 13.1 MiB |
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066967841
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4066953218)
`Linux bench-c4066953218-301-q2n6z 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing foldhash-hashing (7fe428296c61e972616f09d516a063fa5e716cc3) to 8609288 (merge-base) [diff](https://github.com/apache/datafusion/compare/8609288ae0595e715f6ab4fc6a5a56edf1b05eeb..7fe428296c61e972616f09d516a063fa5e716cc3) using: clickbench_partitioned
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#issuecomment-4066968384
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20958#issuecomment-4066953218)
`Linux bench-c4066953218-303-v8gkt 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing foldhash-hashing (7fe428296c61e972616f09d516a063fa5e716cc3) to 8609288 (merge-base) [diff](https://github.com/apache/datafusion/compare/8609288ae0595e715f6ab4fc6a5a56edf1b05eeb..7fe428296c61e972616f09d516a063fa5e716cc3) using: tpch
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Replace ahash with foldhash for faster hashing in datafusion-common [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20958:
URL: https://github.com/apache/datafusion/pull/20958#discussion_r2940913813
##########
datafusion/physical-plan/src/joins/hash_join/partitioned_hash_eval.rs:
##########
@@ -47,8 +47,16 @@ pub struct SeededRandomState {
impl SeededRandomState {
/// Create a new SeededRandomState with the given seeds.
pub const fn with_seeds(k0: u64, k1: u64, k2: u64, k3: u64) -> Self {
+ // Combine 4 seeds into one for foldhash's single-seed API
Review Comment:
Is the goal here to avoid changing the public signature? IIRC this is only public because it gets used across modules / crates -> maybe we can just change the signature? If that's the case it would be good to add to the docstring "this is only meant for internal use, we may break the method, etc.".
##########
datafusion/physical-plan/src/aggregates/mod.rs:
##########
@@ -87,8 +87,16 @@ pub fn topk_types_supported(key_type: &DataType, value_type: &DataType) -> bool
}
/// Hard-coded seed for aggregations to ensure hash values differ from `RepartitionExec`, avoiding collisions.
-const AGGREGATION_HASH_SEED: ahash::RandomState =
- ahash::RandomState::with_seeds('A' as u64, 'G' as u64, 'G' as u64, 'R' as u64);
+const AGGREGATION_HASH_SEED: datafusion_common::hash_utils::RandomState =
+ datafusion_common::hash_utils::RandomState::with_seed(
+ ('A' as u64)
+ .wrapping_mul(6364136223846793005)
Review Comment:
Could you add a comment here explaining to us mere mortals where this number comes from?
##########
datafusion/physical-plan/src/joins/symmetric_hash_join.rs:
##########
@@ -239,7 +239,7 @@ impl SymmetricHashJoinExec {
build_join_schema(&left_schema, &right_schema, join_type);
// Initialize the random state for the join operation:
- let random_state = RandomState::with_seeds(0, 0, 0, 0);
+ let random_state = RandomState::with_seed(0);
Review Comment:
I thought we abstracted out the 2 random states we use (`0,0,0,0` and `H,A,S,H`) somewhere - should we re-use those?
##########
datafusion/core/tests/sql/unparser.rs:
##########
@@ -368,8 +357,9 @@ async fn collect_results(ctx: &SessionContext, original: &str) -> TestCaseResult
}
};
- // Sort if needed for comparison
- if !is_sorted {
+ // Always sort for deterministic comparison — even "sorted" results can have
+ // tied rows in different order between original and unparsed SQL.
Review Comment:
Ah makes sense 👍🏻
##########
datafusion/functions-aggregate/src/hyperloglog.rs:
##########
@@ -61,12 +61,7 @@ where
/// shared across cluster, this SEED will have to be consistent across all
/// parties otherwise we might have corruption. So ideally for later this seed
/// shall be part of the serialized form (or stay unchanged across versions).
-const SEED: RandomState = RandomState::with_seeds(
- 0x885f6cab121d01a3_u64,
- 0x71e4379f2976ad8f_u64,
- 0xbf30173dd28a8816_u64,
- 0x0eaea5d736d733a4_u64,
-);
+const SEED: foldhash::quality::FixedState = foldhash::quality::FixedState::with_seed(0);
Review Comment:
Is this going to be problematic? It says in the comment "ideally [...] this seed shall [...] stay unchanged across versions..."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org