[jira] [Assigned] (SPARK-32671) Race condition in MapOutputTracker.getStatistics

["Apache Spark (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-32671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-32671:
------------------------------------

    Assignee:     (was: Apache Spark)

> Race condition in MapOutputTracker.getStatistics
> ------------------------------------------------
>
>                 Key: SPARK-32671
>                 URL: https://issues.apache.org/jira/browse/SPARK-32671
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 3.0.0, 3.0.1
>            Reporter: Andy Grove
>            Priority: Major
>
> MapOutputTracker.getStatistics builds an array of partition sizes for a shuffle id and in some cases uses multiple threads running in parallel to update this array. This code is not thread-safe and the output is non-deterministic when there are multiple MapStatus entries for the same partition.
> We have unit tests such as the skewed join tests in AdaptiveQueryExecSuite that depend on the output being deterministic, and intermittent failures in these tests led me to track this bug down.
> The issue is trivial to fix by using an AtomicLong when building the array of partition sizes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org