[jira] [Created] (SPARK-40660) Switch XORShiftRandom to distribute elements

["Yuming Wang (Jira)" <ji...@apache.org>]

Yuming Wang created SPARK-40660:
-----------------------------------

             Summary: Switch XORShiftRandom to distribute elements
                 Key: SPARK-40660
                 URL: https://issues.apache.org/jira/browse/SPARK-40660
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.4.0
            Reporter: Yuming Wang



{code:scala}
import java.util.Random
import org.apache.spark.util.random.XORShiftRandom
import scala.util.hashing

def distribution(count: Int, partition: Int) = {
  println((1 to count).map(partitionId => new Random(partitionId).nextInt(partition))
    .groupBy(f => f)
    .map(_._2.size).mkString(". "))

  println((1 to count).map(partitionId => new Random(hashing.byteswap32(partitionId)).nextInt(partition))
    .groupBy(f => f)
    .map(_._2.size).mkString(". "))

  println((1 to count).map(partitionId => new XORShiftRandom(partitionId).nextInt(partition))
    .groupBy(f => f)
    .map(_._2.size).mkString(". "))
}

distribution(200, 4)
{code}


{noformat}
200
50. 60. 46. 44
55. 48. 43. 54
{noformat}




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org