[jira] [Commented] (SPARK-23599) The UUID() expression is too non-deterministic

["Stu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-23599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17466961#comment-17466961 ] 

Stu commented on SPARK-23599:
-----------------------------

We have encountered this problem with Spark 3.1.2, resulting in duplicate values in a situation where a spark executor died. As suggested in the description, this error was hard to track down and difficult to replicate. 

> The UUID() expression is too non-deterministic
> ----------------------------------------------
>
>                 Key: SPARK-23599
>                 URL: https://issues.apache.org/jira/browse/SPARK-23599
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.3.0
>            Reporter: Herman van Hövell
>            Assignee: L. C. Hsieh
>            Priority: Critical
>             Fix For: 2.3.1, 2.4.0
>
>
> The current {{Uuid()}} expression uses {{java.util.UUID.randomUUID}} for UUID generation. There are a couple of major problems with this:
> - It is non-deterministic across task retries. This breaks Spark's processing model, and this will to very hard to trace bugs, like non-deterministic shuffles, duplicates and missing rows.
> - It uses a single secure random for UUID generation. This uses a single JVM wide lock, and this can lead to lock contention and other performance problems.
> We should move to something that is deterministic between retries. This can be done by using seeded PRNGs for which we set the seed during planning. It is important here to use a PRNG that provides enough entropy for creating a proper UUID.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org