You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by tg...@apache.org on 2015/01/09 15:10:12 UTC
spark git commit: [SPARK-5169][YARN]fetch the correct max attempts
Repository: spark
Updated Branches:
refs/heads/master 167a5ab0b -> f3da4bd72
[SPARK-5169][YARN]fetch the correct max attempts
Soryy for fetching the wrong max attempts in this commit https://github.com/apache/spark/commit/8fdd48959c93b9cf809f03549e2ae6c4687d1fcd.
We need to fix it now.
tgravescs
If we set an spark.yarn.maxAppAttempts which is larger than `yarn.resourcemanager.am.max-attempts` in yarn side, it will be overrided as described here:
>The maximum number of application attempts. It's a global setting for all application masters. Each application master can specify its individual maximum number of application attempts via the API, but the individual number cannot be more than the global upper bound. If it is, the resourcemanager will override it. The default number is set to 2, to allow at least one retry for AM.
http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
Author: WangTaoTheTonic <ba...@aliyun.com>
Closes #3942 from WangTaoTheTonic/HOTFIX and squashes the following commits:
9ac16ce [WangTaoTheTonic] fetch the correct max attempts
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f3da4bd7
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f3da4bd7
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f3da4bd7
Branch: refs/heads/master
Commit: f3da4bd7289d493014ad3c5176ada60794dfcfe0
Parents: 167a5ab
Author: WangTaoTheTonic <ba...@aliyun.com>
Authored: Fri Jan 9 08:10:09 2015 -0600
Committer: Thomas Graves <tg...@apache.org>
Committed: Fri Jan 9 08:10:09 2015 -0600
----------------------------------------------------------------------
.../org/apache/spark/deploy/yarn/YarnRMClient.scala | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/f3da4bd7/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnRMClient.scala
----------------------------------------------------------------------
diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnRMClient.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnRMClient.scala
index e183efc..b45e599 100644
--- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnRMClient.scala
+++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnRMClient.scala
@@ -121,9 +121,15 @@ private[spark] class YarnRMClient(args: ApplicationMasterArguments) extends Logg
/** Returns the maximum number of attempts to register the AM. */
def getMaxRegAttempts(sparkConf: SparkConf, yarnConf: YarnConfiguration): Int = {
- sparkConf.getOption("spark.yarn.maxAppAttempts").map(_.toInt).getOrElse(
- yarnConf.getInt(
- YarnConfiguration.RM_AM_MAX_ATTEMPTS, YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS))
+ val sparkMaxAttempts = sparkConf.getOption("spark.yarn.maxAppAttempts").map(_.toInt)
+ val yarnMaxAttempts = yarnConf.getInt(
+ YarnConfiguration.RM_AM_MAX_ATTEMPTS, YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
+ val retval: Int = sparkMaxAttempts match {
+ case Some(x) => if (x <= yarnMaxAttempts) x else yarnMaxAttempts
+ case None => yarnMaxAttempts
+ }
+
+ retval
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org