You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2015/04/01 01:22:53 UTC
[jira] [Resolved] (SPARK-6614) OutputCommitCoordinator should clear
authorized committers only after authorized committer fails, not after any
failure
[ https://issues.apache.org/jira/browse/SPARK-6614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Rosen resolved SPARK-6614.
-------------------------------
Resolution: Fixed
Fix Version/s: 1.4.0
1.3.1
> OutputCommitCoordinator should clear authorized committers only after authorized committer fails, not after any failure
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-6614
> URL: https://issues.apache.org/jira/browse/SPARK-6614
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
> Affects Versions: 1.3.0, 1.3.1, 1.4.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
> Fix For: 1.3.1, 1.4.0
>
>
> In OutputCommitCoordinator, there is some logic to clear the authorized committer's lock on committing in case it fails. However, it looks like the current code also clears this lock if _other_ tasks fail, which is an obvious bug: https://github.com/apache/spark/blob/df3550084c9975f999ed370dd9f7c495181a68ba/core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoordinator.scala#L118. In theory, it's possible that this could allow a new committer to start, run to completion, and commit output before the authorized committer finished, but it's unlikely that this race occurs often in practice due to the complex combination of failure and timing conditions that would be required to expose it. Still, we should fix this issue.
> This was discovered by [~adav] while reading the OutputCommitCoordinator code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org