You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2015/03/31 00:05:53 UTC

[jira] [Created] (SPARK-6614) OutputCommitCoordinator should clear authorized committers only after authorized committer fails, not after any failure

Josh Rosen created SPARK-6614:
---------------------------------

             Summary: OutputCommitCoordinator should clear authorized committers only after authorized committer fails, not after any failure
                 Key: SPARK-6614
                 URL: https://issues.apache.org/jira/browse/SPARK-6614
             Project: Spark
          Issue Type: Bug
    Affects Versions: 1.3.0, 1.2.1
            Reporter: Josh Rosen
            Assignee: Josh Rosen


In OutputCommitCoordinator, there is some logic to clear the authorized committer's lock on committing in case it fails.  However, it looks like the current code also clears this lock if _other_ tasks fail, which is an obvious bug: https://github.com/apache/spark/blob/df3550084c9975f999ed370dd9f7c495181a68ba/core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoordinator.scala#L118.  In theory, it's possible that this could allow a new committer to start, run to completion, and commit output before the authorized committer finished, but it's unlikely that this race occurs often in practice due to the complex combination of failure and timing conditions that would be required to expose it.  Still, we should fix this issue.

This was discovered by [~adav] while reading the OutputCommitCoordinator code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org