You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by tg...@apache.org on 2014/07/08 20:55:50 UTC
git commit: SPARK-2400 : fix spark.yarn.max.executor.failures
explaination
Repository: spark
Updated Branches:
refs/heads/master c8a2313cd -> b520b6453
SPARK-2400 : fix spark.yarn.max.executor.failures explaination
According to
```scala
private val maxNumExecutorFailures = sparkConf.getInt("spark.yarn.max.executor.failures",
sparkConf.getInt("spark.yarn.max.worker.failures", math.max(args.numExecutors * 2, 3)))
```
default value should be numExecutors * 2, with minimum of 3, and it's same to the config
`spark.yarn.max.worker.failures`
Author: CrazyJvm <cr...@gmail.com>
Closes #1282 from CrazyJvm/yarn-doc and squashes the following commits:
1a5f25b [CrazyJvm] remove deprecated config
c438aec [CrazyJvm] fix style
86effa6 [CrazyJvm] change expression
211f130 [CrazyJvm] fix html tag
2900d23 [CrazyJvm] fix style
a4b2e27 [CrazyJvm] fix configuration spark.yarn.max.executor.failures
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b520b645
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b520b645
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b520b645
Branch: refs/heads/master
Commit: b520b6453ed76926108e0bdd56114d16e1d86850
Parents: c8a2313
Author: CrazyJvm <cr...@gmail.com>
Authored: Tue Jul 8 13:55:42 2014 -0500
Committer: Thomas Graves <tg...@apache.org>
Committed: Tue Jul 8 13:55:42 2014 -0500
----------------------------------------------------------------------
docs/running-on-yarn.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/b520b645/docs/running-on-yarn.md
----------------------------------------------------------------------
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md
index 5d8d603..0362f5a 100644
--- a/docs/running-on-yarn.md
+++ b/docs/running-on-yarn.md
@@ -55,7 +55,7 @@ Most of the configs are the same for Spark on YARN as for other deployment modes
</tr>
<tr>
<td><code>spark.yarn.max.executor.failures</code></td>
- <td>2*numExecutors</td>
+ <td>numExecutors * 2, with minimum of 3</td>
<td>
The maximum number of executor failures before failing the application.
</td>