You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "tohuya6 (via GitHub)" <gi...@apache.org> on 2026/09/27 16:52:34 UTC
[PR] fix: keep the running hash for nested dictionaries in Spark xxhash64 [datafusion]
tohuya6 opened a new pull request, #25813:
URL: https://github.com/apache/datafusion/pull/25813
## Which issue does this PR close?
- Closes #25799.
## Rationale for this change
Spark `xxhash64` returns different results for the same data depending on whether a value inside a struct or list is dictionary-encoded. The dictionary fast path restarted from the default seed, so earlier arguments and earlier list elements had no effect.
## What changes are included in this PR?
Ports the fix from apache/datafusion-comet#5757:
- The dictionary fast path is used only when every row has the same running hash.
- The fast path starts from that running hash instead of 42.
## What is the testing strategy for this PR?
Two unit tests ported from Comet, both fail on main:
- `test_dictionary_element_in_list_matches_decoded`: a dictionary inside a list hashes the same as its decoded values.
- `test_dictionary_with_nonuniform_seeds_matches_decoded`: rows with different running hashes hash the same as the decoded values.
The existing Spark hash unit tests and sqllogictests pass. Both queries from the issue now return matching results for dictionary and plain values.
## Are there any user-facing changes?
`xxhash64` now matches Spark for dictionary-encoded values inside structs and lists. No API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: keep the running hash for nested dictionaries in Spark xxhash64 [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway merged PR #25813:
URL: https://github.com/apache/datafusion/pull/25813
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: keep the running hash for nested dictionaries in Spark xxhash64 [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25813:
URL: https://github.com/apache/datafusion/pull/25813#issuecomment-5858041653
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25813?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 82.51%. Comparing base ([`f029b94`](https://app.codecov.io/gh/apache/datafusion/commit/f029b944c8d2f8e70f0c213f87160c12ddad6a09?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`60f7473`](https://app.codecov.io/gh/apache/datafusion/commit/60f74737278e45bf790186fcde51ee05fb4f956b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 2 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #25813 +/- ##
=======================================
Coverage 82.51% 82.51%
=======================================
Files 1141 1141
Lines 439780 439843 +63
Branches 439780 439843 +63
=======================================
+ Hits 362888 362953 +65
+ Misses 54950 54944 -6
- Partials 21942 21946 +4
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25813?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: keep the running hash for nested dictionaries in Spark xxhash64 [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on PR #25813:
URL: https://github.com/apache/datafusion/pull/25813#issuecomment-5969542830
Thanks @tohuya6 !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org