You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sa...@apache.org on 2015/05/09 02:51:26 UTC
spark git commit: [SPARK-7451] [YARN] Preemption of executors is
counted as failure causing Spark job to fail
Repository: spark
Updated Branches:
refs/heads/master 84bf931f3 -> b6c797b08
[SPARK-7451] [YARN] Preemption of executors is counted as failure causing Spark job to fail
Added a check to handle container exit status for the preemption scenario, log an INFO message in such cases and move on.
andrewor14
Author: Ashwin Shankar <as...@netflix.com>
Closes #5993 from ashwinshankar77/SPARK-7451 and squashes the following commits:
90900cf [Ashwin Shankar] Fix log info message
cf8b6cf [Ashwin Shankar] Stop counting preemption of executors as failure
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b6c797b0
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b6c797b0
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b6c797b0
Branch: refs/heads/master
Commit: b6c797b08cbd08d7aab59ad0106af0f5f41ef186
Parents: 84bf931
Author: Ashwin Shankar <as...@netflix.com>
Authored: Fri May 8 17:51:00 2015 -0700
Committer: Sandy Ryza <sa...@cloudera.com>
Committed: Fri May 8 17:51:00 2015 -0700
----------------------------------------------------------------------
.../main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/b6c797b0/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
----------------------------------------------------------------------
diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
index b8f42da..88d68d5 100644
--- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
+++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
@@ -373,7 +373,9 @@ private[yarn] class YarnAllocator(
// Hadoop 2.2.X added a ContainerExitStatus we should switch to use
// there are some exit status' we shouldn't necessarily count against us, but for
// now I think its ok as none of the containers are expected to exit
- if (completedContainer.getExitStatus == -103) { // vmem limit exceeded
+ if (completedContainer.getExitStatus == ContainerExitStatus.PREEMPTED) {
+ logInfo("Container preempted: " + containerId)
+ } else if (completedContainer.getExitStatus == -103) { // vmem limit exceeded
logWarning(memLimitExceededLogMessage(
completedContainer.getDiagnostics,
VMEM_EXCEEDED_PATTERN))
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org