[jira] [Created] (SPARK-45900) Update hash functionalities from xxHash64 to XXH3

["Nathan Holland (Jira)" <ji...@apache.org>]

Nathan Holland created SPARK-45900:
--------------------------------------

             Summary: Update hash functionalities from xxHash64 to XXH3
                 Key: SPARK-45900
                 URL: https://issues.apache.org/jira/browse/SPARK-45900
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.5.0
            Reporter: Nathan Holland


I often work in projects that require deterministic randomness, especially when creating surrogate keys. For small volumes of data xxhash64 works well however this functionality doesn't scale well - with a 64-bit hash code, the chance of collision is one in a million when you hash just six million items increasing sharply due to the birthday paradox.

Currently there are a few ways to handle this
 - hash: 32-bit output (>50% chance of collision at least one for tables larger than 77000 rows, and likely ~1.6 billion collisions in a table of size 4 billion)
 - xxhash64: 64-bit output (>50% chance of collision at least one for tables larger than 5 billion rows)
 - shaXXX/md5: single binary column input, string output, quite computationally expensive.

I'd suggest adding the newest algorithm in the xxhash64 family, XXH3. The XXH3 family is a modern 64-bit and 128-bit hash function family that provides improved strength and performance across the board.

I'd imagine this would be a new function named xxhash3 and take 64 bit, and 128bit bit lengths. For usability I believe the bit length should default to 128bits to provide the best experience to reduce accidental collisions and leave users to set the bit length to 64 as an override if they need to for additional performance or interop reasons. (given the benchmarks, this would likely be quite rare)

References:
 * [Documentation|https://xxhash.com/]
 * xxHash64 Ticket
 * [Existing xxHash64 logic|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/XXH64.java]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org