You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "neilconway (via GitHub)" <gi...@apache.org> on 2026/09/27 13:47:41 UTC
[I] Spark `xxhash64` ignores the running hash for dictionary-encoded values inside structs and lists [datafusion]
neilconway opened a new issue, #25799:
URL: https://github.com/apache/datafusion/issues/25799
### Describe the bug
Dictionary-encoded values nested inside a struct or list change xxhash64's result, so the same data hashes differently depending on its encoding. When the first field of a struct, or any list element, is dictionary-encoded, the hash accumulated so far is thrown away. Earlier arguments, and earlier list elements, then have no effect on the result.
### To Reproduce
```
-- Dictionary-encoded field inside a struct: the first argument is dropped
SELECT xxhash64(7, named_struct('d', arrow_cast('a', 'Dictionary(Int32, Utf8)'))) AS dict_7,
xxhash64(8, named_struct('d', arrow_cast('a', 'Dictionary(Int32, Utf8)'))) AS dict_8,
xxhash64(7, named_struct('d', 'a')) AS plain_7,
xxhash64(8, named_struct('d', 'a')) AS plain_8;
-- Dictionary-encoded list elements: only the last element counts
SELECT xxhash64(make_array(arrow_cast('a', 'Dictionary(Int32, Utf8)'),
arrow_cast('b', 'Dictionary(Int32, Utf8)'))) AS dict_list,
xxhash64(make_array('a', 'b')) AS plain_list,
xxhash64(make_array('b')) AS plain_last_only;
$ cargo run --bin datafusion-cli -- --spark -f repro.sql
+----------------------+----------------------+---------------------+---------------------+
| dict_7 | dict_8 | plain_7 | plain_8 |
+----------------------+----------------------+---------------------+---------------------+
| -8582455328737087284 | -8582455328737087284 | 2701516948731497016 | -694621655434011405 |
+----------------------+----------------------+---------------------+---------------------+
+----------------------+----------------------+----------------------+
| dict_list | plain_list | plain_last_only |
+----------------------+----------------------+----------------------+
| -6391946315847899181 | -7357372313250070212 | -6391946315847899181 |
+----------------------+----------------------+----------------------+
```
### Expected behavior
Dictionary encoding should not change the hash: dict_7/dict_8 should equal plain_7/plain_8, and dict_list should equal plain_list.
### Additional context
Bug fixed in Comet already: apache/datafusion-comet#5757
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Spark `xxhash64` ignores the running hash for dictionary-encoded values inside structs and lists [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway closed issue #25799: Spark `xxhash64` ignores the running hash for dictionary-encoded values inside structs and lists
URL: https://github.com/apache/datafusion/issues/25799
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Spark `xxhash64` ignores the running hash for dictionary-encoded values inside structs and lists [datafusion]
Posted by "tohuya6 (via GitHub)" <gi...@apache.org>.
tohuya6 commented on issue #25799:
URL: https://github.com/apache/datafusion/issues/25799#issuecomment-5856733949
hi @neilconway, I have a patch ready with the fix from comet 5757, ok if I open a PR for review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Spark `xxhash64` ignores the running hash for dictionary-encoded values inside structs and lists [datafusion]
Posted by "tohuya6 (via GitHub)" <gi...@apache.org>.
tohuya6 commented on issue #25799:
URL: https://github.com/apache/datafusion/issues/25799#issuecomment-5954582857
Hi @neilconway, let me know if I replicated the patch well!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Spark `xxhash64` ignores the running hash for dictionary-encoded values inside structs and lists [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on issue #25799:
URL: https://github.com/apache/datafusion/issues/25799#issuecomment-5857696364
> hi [@neilconway](https://github.com/neilconway), I have a patch ready with the fix from comet 5757, ok if I open a PR for review?
Sure @tohuya6 !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org