You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Peter Toth (Jira)" <ji...@apache.org> on 2023/09/19 13:49:00 UTC
[jira] [Updated] (SPARK-45216) Fix non-deterministic seeded Dataset APIs
[ https://issues.apache.org/jira/browse/SPARK-45216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Toth updated SPARK-45216:
-------------------------------
Description:
If we run the following example the result is the expected equal 2 columns:
{noformat}
val c = rand()
df.select(c, c)
+--------------------------+--------------------------+
|rand(-4522010140232537566)|rand(-4522010140232537566)|
+--------------------------+--------------------------+
| 0.4520819282997137| 0.4520819282997137|
+--------------------------+--------------------------+
{noformat}
But if we run use other similar APIs their result is incorrect:
{noformat}
val r1 = random()
val r2 = uuid()
val r3 = shuffle(col("x"))
val x = df.select(r1, r1, r2, r2, r3, r3)
+------------------+------------------+--------------------+--------------------+----------+----------+
| rand()| rand()| uuid()| uuid()|shuffle(x)|shuffle(x)|
+------------------+------------------+--------------------+--------------------+----------+----------+
|0.7407604956381952|0.7957319451135009|e55bc4b0-74e6-4b0...|a587163b-d06b-4bb...| [1, 2, 3]| [2, 1, 3]|
+------------------+------------------+--------------------+--------------------+----------+----------+
{noformat}
> Fix non-deterministic seeded Dataset APIs
> -----------------------------------------
>
> Key: SPARK-45216
> URL: https://issues.apache.org/jira/browse/SPARK-45216
> Project: Spark
> Issue Type: Bug
> Components: Connect, SQL
> Affects Versions: 4.0.0
> Reporter: Peter Toth
> Priority: Major
>
> If we run the following example the result is the expected equal 2 columns:
> {noformat}
> val c = rand()
> df.select(c, c)
> +--------------------------+--------------------------+
> |rand(-4522010140232537566)|rand(-4522010140232537566)|
> +--------------------------+--------------------------+
> | 0.4520819282997137| 0.4520819282997137|
> +--------------------------+--------------------------+
> {noformat}
>
> But if we run use other similar APIs their result is incorrect:
> {noformat}
> val r1 = random()
> val r2 = uuid()
> val r3 = shuffle(col("x"))
> val x = df.select(r1, r1, r2, r2, r3, r3)
> +------------------+------------------+--------------------+--------------------+----------+----------+
> | rand()| rand()| uuid()| uuid()|shuffle(x)|shuffle(x)|
> +------------------+------------------+--------------------+--------------------+----------+----------+
> |0.7407604956381952|0.7957319451135009|e55bc4b0-74e6-4b0...|a587163b-d06b-4bb...| [1, 2, 3]| [2, 1, 3]|
> +------------------+------------------+--------------------+--------------------+----------+----------+
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org