You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "neilconway (via GitHub)" <gi...@apache.org> on 2026/09/27 13:47:41 UTC

[I] Spark `xxhash64` ignores the running hash for dictionary-encoded values inside structs and lists [datafusion]

neilconway opened a new issue, #25799:
URL: https://github.com/apache/datafusion/issues/25799

   ### Describe the bug
   
   Dictionary-encoded values nested inside a struct or list change xxhash64's result, so the same data hashes differently depending on its encoding. When the first field of a struct, or any list element, is dictionary-encoded, the hash accumulated so far is thrown away. Earlier arguments, and earlier list elements, then have no effect on the result.
   
   
   ### To Reproduce
   
   ```
   -- Dictionary-encoded field inside a struct: the first argument is dropped
   SELECT xxhash64(7, named_struct('d', arrow_cast('a', 'Dictionary(Int32, Utf8)'))) AS dict_7,
          xxhash64(8, named_struct('d', arrow_cast('a', 'Dictionary(Int32, Utf8)'))) AS dict_8,
          xxhash64(7, named_struct('d', 'a')) AS plain_7,
          xxhash64(8, named_struct('d', 'a')) AS plain_8;
   
   -- Dictionary-encoded list elements: only the last element counts
   SELECT xxhash64(make_array(arrow_cast('a', 'Dictionary(Int32, Utf8)'),
                              arrow_cast('b', 'Dictionary(Int32, Utf8)'))) AS dict_list,
          xxhash64(make_array('a', 'b')) AS plain_list,
          xxhash64(make_array('b')) AS plain_last_only;
   
   $ cargo run --bin datafusion-cli -- --spark -f repro.sql
   
   +----------------------+----------------------+---------------------+---------------------+
   | dict_7               | dict_8               | plain_7             | plain_8             |
   +----------------------+----------------------+---------------------+---------------------+
   | -8582455328737087284 | -8582455328737087284 | 2701516948731497016 | -694621655434011405 |
   +----------------------+----------------------+---------------------+---------------------+
   
   +----------------------+----------------------+----------------------+
   | dict_list            | plain_list           | plain_last_only      |
   +----------------------+----------------------+----------------------+
   | -6391946315847899181 | -7357372313250070212 | -6391946315847899181 |
   +----------------------+----------------------+----------------------+
   ```
   
   ### Expected behavior
   
   Dictionary encoding should not change the hash: dict_7/dict_8 should equal plain_7/plain_8, and dict_list should equal plain_list.
   
   
   ### Additional context
   
   Bug fixed in Comet already: apache/datafusion-comet#5757


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Spark `xxhash64` ignores the running hash for dictionary-encoded values inside structs and lists [datafusion]

Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway closed issue #25799: Spark `xxhash64` ignores the running hash for dictionary-encoded values inside structs and lists
URL: https://github.com/apache/datafusion/issues/25799


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Spark `xxhash64` ignores the running hash for dictionary-encoded values inside structs and lists [datafusion]

Posted by "tohuya6 (via GitHub)" <gi...@apache.org>.
tohuya6 commented on issue #25799:
URL: https://github.com/apache/datafusion/issues/25799#issuecomment-5856733949

   hi @neilconway, I have a patch ready with the fix from comet 5757, ok if I open a PR for review?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Spark `xxhash64` ignores the running hash for dictionary-encoded values inside structs and lists [datafusion]

Posted by "tohuya6 (via GitHub)" <gi...@apache.org>.
tohuya6 commented on issue #25799:
URL: https://github.com/apache/datafusion/issues/25799#issuecomment-5954582857

   Hi @neilconway, let me know if I replicated the patch well!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Spark `xxhash64` ignores the running hash for dictionary-encoded values inside structs and lists [datafusion]

Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on issue #25799:
URL: https://github.com/apache/datafusion/issues/25799#issuecomment-5857696364

   > hi [@neilconway](https://github.com/neilconway), I have a patch ready with the fix from comet 5757, ok if I open a PR for review?
   
   Sure @tohuya6 !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org