You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Huon Wilson (JIRA)" <ji...@apache.org> on 2019/03/08 03:06:00 UTC
[jira] [Created] (SPARK-27099) Expose xxHash64 as a flexible 64-bit
column hash like `hash`
Huon Wilson created SPARK-27099:
-----------------------------------
Summary: Expose xxHash64 as a flexible 64-bit column hash like `hash`
Key: SPARK-27099
URL: https://issues.apache.org/jira/browse/SPARK-27099
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 2.4.0, 2.3.3
Reporter: Huon Wilson
I’m working on something that requires deterministic randomness, i.e. a row gets the same “random” value no matter the order of the DataFrame. A seeded hash seems to be the perfect way to do this, but the existing hashes have various limitations:
- hash: 32-bit output (only 4 billion possibilities will result in a lot of collisions for many tables: the birthday paradox implies >50% chance of at least one for tables larger than 77000 rows, and likely ~1.6 billion collisions in a table of size 4 billion)
- sha1/sha2/md5: single binary column input, string output
It seems there’s already support for a 64-bit hash function that can work with an arbitrary number of arbitrary-typed columns (XxHash64), which could be exposed as xxHash64 or xxhash64 (or similar).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org