You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by twinkle-sachdeva <gi...@git.apache.org> on 2015/04/10 10:23:42 UTC
[GitHub] spark pull request: SPARK-6735:[YARN] Adding properties to disable...
GitHub user twinkle-sachdeva opened a pull request:
https://github.com/apache/spark/pull/5449
SPARK-6735:[YARN] Adding properties to disable maximum number of executor failure's check or to make it relative to duration
For long running applications, user might want to disable this property or to make it relative to a duration window, so that some older failure does not cause Application to abort in long run.
Added properties 1) spark.yarn.max.executor.failures.disable: to disable maximum executor failure check 2) spark.yarn.max.executor.failures.relative: to make the maximum executor failure to be relative 3)spark.yarn.max.executor.failures.relative.window: specify relative window duration in sec , default being 600 sec
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/twinkle-sachdeva/spark SPARK-6735-MASTER
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5449.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5449
----
commit 5ab5a0784256ac953dcfe229087f911e5eb4c127
Author: twinkle.sachdeva <tw...@guavus.com>
Date: 2015-04-07T11:02:25Z
SPARK-6735:Adding properties 1) spark.yarn.max.executor.failures.disable: to disable maximum executor failure check 2) spark.yarn.max.executor.failures.relative to make the maximum executor failure to be relative 3)spark.yarn.max.executor.failures.relative.window : specify relative window duration in sec , default being 600 sec
commit 4750a012bcdc2cb55770f6e75c52cc1e0d6041fb
Author: twinkle.sachdeva <tw...@guavus.com>
Date: 2015-04-10T08:25:40Z
SPARK-6735:populating executorFailureTimeStamps only when required
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6735:[YARN] Adding properties to disable...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5449#discussion_r28139245
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -94,6 +98,14 @@ private[yarn] class YarnAllocator(
// Additional memory overhead.
protected val memoryOverhead: Int = sparkConf.getInt("spark.yarn.executor.memoryOverhead",
math.max((MEMORY_OVERHEAD_FACTOR * executorMemory).toInt, MEMORY_OVERHEAD_MIN))
+
+ // Make the maximum executor failure check to be relative with respect to duration
+ private val relativeMaxExecutorFailureCheck =
--- End diff --
Isn't this more easily solved by considering max failures to be per batch interval or something? why define another different window of time in another property?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6735:[YARN] Adding properties to disable...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5449#discussion_r28139207
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -119,7 +131,27 @@ private[yarn] class YarnAllocator(
def getNumExecutorsRunning: Int = numExecutorsRunning
- def getNumExecutorsFailed: Int = numExecutorsFailed
+ def getNumExecutorsFailed: Int = {
+ if(relativeMaxExecutorFailureCheck){
+ getRelevantNumExecutorsFailed
+ } else {
+ numExecutorsFailed.intValue
+ }
+ }
+
+ /**
+ * Returns the the relative number of executor failures within the specifid window duration.
+ */
+
+ def getRelevantNumExecutorsFailed : Int = {
+ var currentTime = System.currentTimeMillis / 1000
--- End diff --
Many style problems around here, like the disconnected javadoc, missing spaces around conditions, `var` vs `val`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6735:[YARN] Adding properties to disable...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/5449#discussion_r28159254
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
@@ -59,6 +59,10 @@ private[spark] class ApplicationMaster(
private val maxNumExecutorFailures = sparkConf.getInt("spark.yarn.max.executor.failures",
sparkConf.getInt("spark.yarn.max.worker.failures", math.max(args.numExecutors * 2, 3)))
+ // Disable the maximum executor failure check
+ private val disableMaxExecutorFailureCheck =
+ sparkConf.getBoolean("spark.yarn.max.executor.failures.disable", false)
--- End diff --
I agree. We could make special value to means disabled -> like 0 or -1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6735:[YARN] Adding properties to disable...
Posted by twinkle-sachdeva <gi...@git.apache.org>.
Github user twinkle-sachdeva commented on a diff in the pull request:
https://github.com/apache/spark/pull/5449#discussion_r28215620
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -94,6 +98,14 @@ private[yarn] class YarnAllocator(
// Additional memory overhead.
protected val memoryOverhead: Int = sparkConf.getInt("spark.yarn.executor.memoryOverhead",
math.max((MEMORY_OVERHEAD_FACTOR * executorMemory).toInt, MEMORY_OVERHEAD_MIN))
+
+ // Make the maximum executor failure check to be relative with respect to duration
+ private val relativeMaxExecutorFailureCheck =
--- End diff --
Hi,
I am not sure if just batch window will do, as the D stream window needs to be some multiple of it. Also, in our use case a long running spark application, there will be no concept of batch window as such.
Thanks,
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6735:[YARN] Adding properties to disable...
Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/5449#discussion_r28217902
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -119,7 +131,27 @@ private[yarn] class YarnAllocator(
def getNumExecutorsRunning: Int = numExecutorsRunning
- def getNumExecutorsFailed: Int = numExecutorsFailed
+ def getNumExecutorsFailed: Int = {
+ if(relativeMaxExecutorFailureCheck){
+ getRelevantNumExecutorsFailed
+ } else {
+ numExecutorsFailed.intValue
+ }
+ }
+
+ /**
+ * Returns the the relative number of executor failures within the specifid window duration.
+ */
+
+ def getRelevantNumExecutorsFailed : Int = {
+ var currentTime = System.currentTimeMillis / 1000
--- End diff --
Here's Spark's style guide: https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6735:[YARN] Adding properties to disable...
Posted by twinkle-sachdeva <gi...@git.apache.org>.
Github user twinkle-sachdeva commented on a diff in the pull request:
https://github.com/apache/spark/pull/5449#discussion_r28224409
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -94,6 +98,14 @@ private[yarn] class YarnAllocator(
// Additional memory overhead.
protected val memoryOverhead: Int = sparkConf.getInt("spark.yarn.executor.memoryOverhead",
math.max((MEMORY_OVERHEAD_FACTOR * executorMemory).toInt, MEMORY_OVERHEAD_MIN))
+
+ // Make the maximum executor failure check to be relative with respect to duration
+ private val relativeMaxExecutorFailureCheck =
--- End diff --
Hi @sryza ,
spark.yarn.max.executor.failuresPerMinute will make that to be per minute kind of failures, which means hardcoding the value of window.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6735:[YARN] Adding properties to disable...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/5449#issuecomment-136894778
@twinkle-sachdeva can you close this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6735:[YARN] Adding properties to disable...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5449#issuecomment-91476884
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6735:[YARN] Adding properties to disable...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5449#discussion_r28139153
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
@@ -59,6 +59,10 @@ private[spark] class ApplicationMaster(
private val maxNumExecutorFailures = sparkConf.getInt("spark.yarn.max.executor.failures",
sparkConf.getInt("spark.yarn.max.worker.failures", math.max(args.numExecutors * 2, 3)))
+ // Disable the maximum executor failure check
+ private val disableMaxExecutorFailureCheck =
+ sparkConf.getBoolean("spark.yarn.max.executor.failures.disable", false)
--- End diff --
There's a cost and weight to making a flag for everything, and I think this doesn't add value. Just set max to a high value to "disable" it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6735:[YARN] Adding properties to disable...
Posted by twinkle-sachdeva <gi...@git.apache.org>.
Github user twinkle-sachdeva commented on the pull request:
https://github.com/apache/spark/pull/5449#issuecomment-96515946
Hi @srowen ,
Please review the changes.
Thanks,
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6735:[YARN] Adding properties to disable...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5449#discussion_r28233526
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -94,6 +98,14 @@ private[yarn] class YarnAllocator(
// Additional memory overhead.
protected val memoryOverhead: Int = sparkConf.getInt("spark.yarn.executor.memoryOverhead",
math.max((MEMORY_OVERHEAD_FACTOR * executorMemory).toInt, MEMORY_OVERHEAD_MIN))
+
+ // Make the maximum executor failure check to be relative with respect to duration
+ private val relativeMaxExecutorFailureCheck =
--- End diff --
Ah right, I had in my head this was streaming-specific for some reason. For an app that may run forever, it seems like any maximum number of failures is insufficient, and you'd want to disable this entirely. How about that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6735:[YARN] Adding properties to disable...
Posted by twinkle-sachdeva <gi...@git.apache.org>.
Github user twinkle-sachdeva commented on a diff in the pull request:
https://github.com/apache/spark/pull/5449#discussion_r28394987
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -94,6 +98,14 @@ private[yarn] class YarnAllocator(
// Additional memory overhead.
protected val memoryOverhead: Int = sparkConf.getInt("spark.yarn.executor.memoryOverhead",
math.max((MEMORY_OVERHEAD_FACTOR * executorMemory).toInt, MEMORY_OVERHEAD_MIN))
+
+ // Make the maximum executor failure check to be relative with respect to duration
+ private val relativeMaxExecutorFailureCheck =
--- End diff --
Sounds reasonable.
Added the property as spark.yarn.max.executor.failuresPerMinute
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6735:[YARN] Adding properties to disable...
Posted by twinkle-sachdeva <gi...@git.apache.org>.
Github user twinkle-sachdeva commented on a diff in the pull request:
https://github.com/apache/spark/pull/5449#discussion_r28215570
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -119,7 +131,27 @@ private[yarn] class YarnAllocator(
def getNumExecutorsRunning: Int = numExecutorsRunning
- def getNumExecutorsFailed: Int = numExecutorsFailed
+ def getNumExecutorsFailed: Int = {
+ if(relativeMaxExecutorFailureCheck){
+ getRelevantNumExecutorsFailed
+ } else {
+ numExecutorsFailed.intValue
+ }
+ }
+
+ /**
+ * Returns the the relative number of executor failures within the specifid window duration.
+ */
+
+ def getRelevantNumExecutorsFailed : Int = {
+ var currentTime = System.currentTimeMillis / 1000
--- End diff --
Hi @srowen ,
I will fix it up, I just have a curiosity that while build process, it do shares some of the styling issues, is there any other extensive list also?
Thanks,
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6735:[YARN] Adding properties to disable...
Posted by twinkle-sachdeva <gi...@git.apache.org>.
Github user twinkle-sachdeva commented on a diff in the pull request:
https://github.com/apache/spark/pull/5449#discussion_r28215509
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
@@ -59,6 +59,10 @@ private[spark] class ApplicationMaster(
private val maxNumExecutorFailures = sparkConf.getInt("spark.yarn.max.executor.failures",
sparkConf.getInt("spark.yarn.max.worker.failures", math.max(args.numExecutors * 2, 3)))
+ // Disable the maximum executor failure check
+ private val disableMaxExecutorFailureCheck =
+ sparkConf.getBoolean("spark.yarn.max.executor.failures.disable", false)
--- End diff --
Looks good, will treat maxNumExecutorFailures = -1 as disable state for this check.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6735:[YARN] Adding properties to disable...
Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/5449#discussion_r28218090
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -94,6 +98,14 @@ private[yarn] class YarnAllocator(
// Additional memory overhead.
protected val memoryOverhead: Int = sparkConf.getInt("spark.yarn.executor.memoryOverhead",
math.max((MEMORY_OVERHEAD_FACTOR * executorMemory).toInt, MEMORY_OVERHEAD_MIN))
+
+ // Make the maximum executor failure check to be relative with respect to duration
+ private val relativeMaxExecutorFailureCheck =
--- End diff --
@srowen if you're suggesting tying it to the dstream window I think that could be confusing. It doesn't seem obvious to me that these should be proportional.
Another thing is that it seems confusing that, if I want to set a max failures per time interval, I need to set three different properties. It's also worth considering just adding a `spark.yarn.max.executor.failuresPerMinute` property.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6735:[YARN] Adding properties to disable...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5449#issuecomment-96767452
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6735:[YARN] Adding properties to disable...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/5449
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6735:[YARN] Adding properties to disable...
Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/5449#discussion_r28278175
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -94,6 +98,14 @@ private[yarn] class YarnAllocator(
// Additional memory overhead.
protected val memoryOverhead: Int = sparkConf.getInt("spark.yarn.executor.memoryOverhead",
math.max((MEMORY_OVERHEAD_FACTOR * executorMemory).toInt, MEMORY_OVERHEAD_MIN))
+
+ // Make the maximum executor failure check to be relative with respect to duration
+ private val relativeMaxExecutorFailureCheck =
--- End diff --
My point was that the size of the window might not need to vary. Are there examples that come to mind of scenarios where the ideal window size is widely different?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6735:[YARN] Adding properties to disable...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/5449#issuecomment-91482854
1. If the window logic was teased out, it could be testable
1. FWIW, in slider we track the node failures too, to try and see if recurrent failures are due to unreliable nodes. Its very hard to get the logic right here (on a small/busy cluster you may always get the same node). Again, long-lived apps need to window node unreliability.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org