You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Andy Grove (Jira)" <ji...@apache.org> on 2020/08/20 17:05:00 UTC

[jira] [Created] (SPARK-32671) Race condition in MapOutputTracker.getStatistics

Andy Grove created SPARK-32671:
----------------------------------

             Summary: Race condition in MapOutputTracker.getStatistics
                 Key: SPARK-32671
                 URL: https://issues.apache.org/jira/browse/SPARK-32671
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 3.0.0, 3.0.1
            Reporter: Andy Grove


MapOutputTracker.getStatistics builds an array of partition sizes for a shuffle id and in some cases uses multiple threads running in parallel to update this array. This code is not thread-safe and the output is non-deterministic when there are multiple MapStatus entries for the same partition.

We have unit tests such as the skewed join tests in AdaptiveQueryExecSuite that depend on the output being deterministic, and intermittent failures in these tests led me to track this bug down.

The issue is trivial to fix by using an AtomicLong when building the array of partition sizes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org