You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by davies <gi...@git.apache.org> on 2014/12/01 22:07:30 UTC
[GitHub] spark pull request: [SPARK-4609] counting failures on host instead...
GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/3541
[SPARK-4609] counting failures on host instead of executor
Executor will restarted with a new id, so counting number of failures based on executor id will not work in this cases, this patch change it to host.
After this change, one downside may be that the task can not be launched on other executors on the same host if it failed on one executor of them. All the executors on the same host share many resources, there is not a good reason that a task can succeed in other executors on the same host. So this will not be a problem.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davies/spark fix_bad_slave
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3541.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3541
----
commit 592dd4f7f8e4aef8e6124fdbc4097b94ef1690c7
Author: Davies Liu <da...@databricks.com>
Date: 2014-12-01T20:21:46Z
counting failures on host instead of executor
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65301702
@davies when you can have multiple executors per host or executor restarted on host on failure, then this can manifest ... please refer to the comments that @kayousterhout referenced to above on why it was added.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-72128864
[Test build #26340 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26340/consoleFull) for PR 3541 at commit [`482804b`](https://github.com/apache/spark/commit/482804b9511567aaa02e20890361136b28812ac3).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/3541#discussion_r21121570
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -274,12 +274,12 @@ private[spark] class TaskSetManager(
* Is this re-execution of a failed task on an executor it already failed in before
* EXECUTOR_TASK_BLACKLIST_TIMEOUT has elapsed ?
*/
- private def executorIsBlacklisted(execId: String, taskId: Int): Boolean = {
- if (failedExecutors.contains(taskId)) {
- val failed = failedExecutors.get(taskId).get
+ private def executorIsBlacklisted(host: String, taskId: Int): Boolean = {
+ if (failedHosts.contains(taskId)) {
+ val hosts = failedHosts.get(taskId).get
- return failed.contains(execId) &&
- clock.getTime() - failed.get(execId).get < EXECUTOR_TASK_BLACKLIST_TIMEOUT
+ return hosts.contains(host) &&
+ clock.getTime() - hosts.get(host).get < EXECUTOR_TASK_BLACKLIST_TIMEOUT
--- End diff --
I think we can not change the name of config variable. How about just call it `TASK_BLACKLIST_TIMEOUT` ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-72128871
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26340/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by davies <gi...@git.apache.org>.
Github user davies closed the pull request at:
https://github.com/apache/spark/pull/3541
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] counting failures on host instead...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65137353
@davies can you change the title of the PR and the JIRA to be more specific+consistent and say "Blacklist hosts rather than executors when tasks fail" (the description is good -- just good to have a more specific title for when people are looking at commits or searching for a jira later).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65156189
[Test build #538 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/538/consoleFull) for PR 3541 at commit [`5ff8227`](https://github.com/apache/spark/commit/5ff82273a92dc83d59f28a6d19fc5f9f50098015).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` trait ConnectionFactory extends Serializable `
* `class MatrixFactorizationModel(`
* `class CompressedSerializer(FramedSerializer):`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65148816
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24003/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65141381
I made a JIRA for enabling this for the next version of Spark: https://issues.apache.org/jira/browse/SPARK-4681
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65141229
@pwendell @aarondav what about the naming though -- what do we usually do for consistency in this kind of scenario? The old config name is executor blacklist timeout, which is technically not correct after this patch (it's host blacklist timeout).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] counting failures on host instead...
Posted by aarondav <gi...@git.apache.org>.
Github user aarondav commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65137793
I didn't realize we had this behavior, actually. Just out of curiosity on the feature itself, is deadlock when a task is blacklisted on every remaining node avoided?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65312874
@mridulm In the case of one executor restarted on failure, the executor id will be changed, put the previous executor-id in blacklist does make sense. It only help when task failed on the particular status of a executor (such as OOM, too many open files). In this cases, host based blacklist also could help.
One situation that executor-id based blacklist will better than host based is that you have only one host and several executors on it. In this case, you will see some delay of scheduling once task failed. Also, executor-id based blacklist may improve data locality if reschedule the failed task into same host, the performance gain should be small, because the percentage of failed tasks should be small.
It's more common that user will have multiple hosts than multiple executors on single host, and host based blacklist can help in most of failure cases, it should be turned on by default. It will be good to have host based blacklist together with executor-id based, but it will complicated the code a lot, do the benefits in above worth it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65147017
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24000/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65140988
@aarondav Good catch! I just realized that this blacklist is not `enabled` by default.
Should we increase it to 10 seconds or 1 minutes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65145903
[Test build #538 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/538/consoleFull) for PR 3541 at commit [`5ff8227`](https://github.com/apache/spark/commit/5ff82273a92dc83d59f28a6d19fc5f9f50098015).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] counting failures on host instead...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3541#discussion_r21120179
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -79,8 +79,8 @@ private[spark] class TaskSetManager(
val copiesRunning = new Array[Int](numTasks)
val successful = new Array[Boolean](numTasks)
private val numFailures = new Array[Int](numTasks)
- // key is taskId, value is a Map of executor id to when it failed
- private val failedExecutors = new HashMap[Int, HashMap[String, Long]]()
+ // key is taskId, value is a Map of host to when it failed
+ private val failedHosts = new HashMap[Int, HashMap[String, Long]]()
--- End diff --
can you change this to failedHostsForTask? I think that would make it more clear that the indices here are task ids (realize this isn't directly related to your change -- but would improve readability)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65163725
[Test build #24006 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24006/consoleFull) for PR 3541 at commit [`6daec2d`](https://github.com/apache/spark/commit/6daec2da02de9d60bc56391bf8d9388fbcf19a92).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65138690
@aarondav no -- the idea with this was that eventually the blacklist timeout will expire. When @mridulm originally added this feature, the motivation described was: "The reason I had to initially look into the problem was due to repeated task failures on an executor which was going to die anyway (was in process of cleanup I believe when tasks getting assigned to it or some such) : which killed our tasksets and so jobs." For that case, deadlock isn't so much an issue -- although you're right that it could happen in other cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-66424149
On Thu, Dec 4, 2014 at 2:57 AM, Davies Liu <no...@github.com> wrote:
> @davies <https://github.com/davies> I am not sure I completely understood
> your comment.
> Sorry for that, maybe I didnot explain it clearly.
>
> As detailed above, there are multiple reasons why a task can fail - and
> quite a lot of them are non-fatal from 'rescheduling the task on same host'
> point of view : in particular race in spark between reporting executor
> going down, shutdown hooks running and task schedules due to locality
> preference.
> So we need per-executor blacklist - note that this is just a temporary -
> to either allow the executor to recover (in case task failures are due to
> transient reasons), or allow task to get scheduled elsewhere in meantime
> (if schedule locality constraints can be satisfied).
>
> Agreed that the executor based blacklist worked for you, and I think the
> host based blacklist will also work for you (there is a little regression
> about locality).
>
It is not a small regression - if you have 4 - 8 executors on a host (as is
common here) : this change will blacklist all of them instead of
blacklisting a single executor.
This is fairly severe regression : which is why I said I am -1 on modifying
existing behavior unless new functionality allows for existing feature to
continue to work as currently expected to.
The thing to understand is executor blacklist is not subsumed by host
blacklist other than in a very crude model.
> A different set of criterion would apply when we want to do host level
> blacklist - when we have determined that the node is unusable, and so task
> fails on all executors in the node : due to NODE_LOCAL locality level, we
> would keep trying other executors on the same node in case executor
> blacklist kicks in; so in case the node is temporarily unusable, executor
> black list might not help.
>
> So we need host based blacklist.
>
Yes, the reasons why we need host blacklist are valid and separate from why
we need executor blacklist.
They might overlap in some degenerate cases (since obviously host level
issues do impact executors too) : executor blacklist is more fine grained -
while host level issues are more coarser in comparison.
While executor blacklist might alleviate lack of host blacklist to some
extent (as exists currently), it is suboptimal to do so : so need for host
blacklist is justified.
> The timeout based temporary executor blacklist we currently have is
> still a stop gap solution which solves immediate problems observed at that
> time : without which spark was becoming unusable in large enough
> multi-tennet clusters.
>
> Agreed.
>
> If we want to it to a host level and do a principled solution - then we
> need a lot of other pieces to be put into place (since currently we only
> take task scheduling into account; which is insufficient).
> Top of my head - remove it from rdd replication, de-allocate executors
> already on the node, moving existing rdd blocks away from the executors on
> the node, blacklisting the node from further allocation requests (yarn,
> mesos), and so on. I am sure @kayousterhout
> <https://github.com/kayousterhout> might have other thoughts on this.
>
> Agreed. Figure out the failure domain is a hard thing in distributed
> environment, I'm doubt that who can contribute a principled solution to
> retry the failed tasks in the best position in near term (such as
> reschedule it in same executor, different executor on same host, different
> host, different rack).
>
> I think the host based blacklist is the simplest solution and work well in
> most failure cases.
>
> Unfortunately, I do not have the bandwidth to engage on this; so I am
> hoping the right thing gets done. Whatever it is, I am -1 on removing
> executor level blacklist - that is something we heavily depend on to get
> our jobs to work. A better solution while not regressing on this
> functionality is most welcome !
>
> Really appreciate your comments here, to have a better solution. Could you
> raise a detailed cases that the host based blacklist will break you job?
> Maybe there are some cases I did not figure out in your situation, please
> correct me.
>
The primary reason for executor blacklist, as @kayousterhout
<https://github.com/kayousterhout> also referred to, were initially quite
simple :
Task gets submitted to same executor repeatedly due to locality constraint
- but keeps failing on the executor since the executor might be in
inconsistent state (like in middle of shutdown, etc).This very quickly
caused applications to fail.
There are other cases where similar pattern is exhibited. Usually they are
transient in nature - and so why the executor blacklist was modelled as a
temporary timebound blacklist - that too for only a specific taskset (other
tasksets can run on the executor, and did not impact other tasks currently
running on the executor).
This was a conscious decision not just to simplify the implementation, but
also because that was the immediate space of problems we were targetting.
A host blacklist might have to consider other aspects - I dont know the
scope of the problem being approached.
1) We could make a simplifying assumption and assume the host issues are
transient in nature - like with executor and proceed with that assumption.
This might be valid to make in case failed executors are cleaning up their
state and the host cane recover and be healthy soon (like disk space
reclaimed on executor shutdown, etc).
2) We might want to model more serious issues - like what MR seems to do.
I have not looked at their implementation, but iirc once blacklisted a host
is not considered for a specific AM again.
@tgravescs <https://github.com/tgravescs> might be able to comment more.
3) Since spark manages more state - do we want to do anything about that :
list replicated blocks, etc.
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/3541#issuecomment-65493883>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65278813
@mridulm Thanks for the feedback. Could you provide some details about configurations (such as the number of executors on a host and the number of hosts)? And in which case the host level blacklist is worse than executor level?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by suyanNone <gi...@git.apache.org>.
Github user suyanNone commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-66863621
To be frankly, I am fresh in spark.
If have local fileSystem accessed errors, should we identify and mark as host-failed? may wrapAs ReadFileException or WriteFileException or sth
and I not sure is work for distinguish from exception in spark...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65148808
[Test build #24003 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24003/consoleFull) for PR 3541 at commit [`37d8fe0`](https://github.com/apache/spark/commit/37d8fe0b6920824c774a48a0583be43c98a6703c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65309912
Note: I am ignoring deterministic failure reasons here (which will fail on any host and usually points to bug in user or spark codebase).
Task failure could be due to a variety of transient reasons - which could be directly related to the task in question, indirectly related to it, or even completely unrelated to it.
For example:
- What other tasks are running on the executor and how it interacts with the failed task.
- What data is currently cached on the executor and the impact it has on resource utilization (rdd, broadcast, buffers, gc, etc).
- What the current state of the executor is (in process of shutdown, but not yet informed the driver about it).
... among others.
Also note that when you have resource constraints enforced (particularly in yarn - where memory limits are aggressively enforced) - one or more of the above can interact with that to cause further non deterministic failures : which is why we have more hacks like memory overheads to help alleviate (though not eliminate) them.
Since we have limits on number of times an application can have executors failures, number of times a task can fail before the application is failed, etc - we need executor level blacklist.
Note, this does not mean we do not need host level blacklist ! I can definitely see value in that if the issues above are host level - as pointed out, lack of hdd space, bad memory or cpu, thermal issues, etc.
Ideally, as I mentioned in the past, we need a better way to identify and blacklist executors/hosts/racks.
What we currently have is a stop gap hack - and upgrading that from executor level to host level does not solve problems (it causes regressions actually in our workloads - since we are not missing a replica completely for dfs data and the other replicas might not be in our allocated hosts/executors).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65152580
[Test build #24006 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24006/consoleFull) for PR 3541 at commit [`6daec2d`](https://github.com/apache/spark/commit/6daec2da02de9d60bc56391bf8d9388fbcf19a92).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] counting failures on host instead...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3541#discussion_r21119992
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -274,12 +274,12 @@ private[spark] class TaskSetManager(
* Is this re-execution of a failed task on an executor it already failed in before
* EXECUTOR_TASK_BLACKLIST_TIMEOUT has elapsed ?
*/
- private def executorIsBlacklisted(execId: String, taskId: Int): Boolean = {
- if (failedExecutors.contains(taskId)) {
- val failed = failedExecutors.get(taskId).get
+ private def executorIsBlacklisted(host: String, taskId: Int): Boolean = {
--- End diff --
can you change this to hostIsBlacklisted?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65147008
[Test build #24000 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24000/consoleFull) for PR 3541 at commit [`592dd4f`](https://github.com/apache/spark/commit/592dd4f7f8e4aef8e6124fdbc4097b94ef1690c7).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65141318
@kayousterhout for naming I think the best thing is to advertise a new name in the docs, and if someone has set the old name we just accept that also.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65140880
Hey this is a good idea. Also another thing - we should turn this feature on by default in the master branch and for 1.3+ IMO. I don't see any reason why a user wouldn't wont this behavior.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] counting failures on host instead...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3541#discussion_r21120040
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -274,12 +274,12 @@ private[spark] class TaskSetManager(
* Is this re-execution of a failed task on an executor it already failed in before
* EXECUTOR_TASK_BLACKLIST_TIMEOUT has elapsed ?
*/
- private def executorIsBlacklisted(execId: String, taskId: Int): Boolean = {
- if (failedExecutors.contains(taskId)) {
- val failed = failedExecutors.get(taskId).get
+ private def executorIsBlacklisted(host: String, taskId: Int): Boolean = {
+ if (failedHosts.contains(taskId)) {
+ val hosts = failedHosts.get(taskId).get
- return failed.contains(execId) &&
- clock.getTime() - failed.get(execId).get < EXECUTOR_TASK_BLACKLIST_TIMEOUT
+ return hosts.contains(host) &&
+ clock.getTime() - hosts.get(host).get < EXECUTOR_TASK_BLACKLIST_TIMEOUT
--- End diff --
hm one thing that's ugly here is we should ideally change this to HOST_TASK_BLACKLIST_TIMEOUT -- but then the name of the config variable is inconsistent...any ideas for what makes most sense here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65140196
Yeah, @mridulm made this "secret" based on an assumption that it was a stopgap solution:
"Regarding timeout variable name : I was considering a very specific variable name to allow for a better/future approach to handling this issue - and at that time allow us to retire this variable without potential variable name conflicts (spark.scheduler.blacklistTimeout implies a more general black-list handling, which this is not unfortunately); IMO, this is a stop gap solution until we add support for a better black list approach which handles both executors and blocks.
But until we have that, this will atleast unblock us - thankfully, this is not something which a lot of users are hitting (but is fairly common in our case unfortunately).
Given this, should we expose this in documentation ?"
Seems like we should document this now if more folks are running into it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] counting failures on host instead...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65135475
[Test build #24000 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24000/consoleFull) for PR 3541 at commit [`592dd4f`](https://github.com/apache/spark/commit/592dd4f7f8e4aef8e6124fdbc4097b94ef1690c7).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65163730
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24006/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by aarondav <gi...@git.apache.org>.
Github user aarondav commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65139319
Ah, so this appears to be a secret feature anyway, as you have to set this undocumented timeout property to a nonzero value to enable it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65138728
@aarondav The blacklist have timeout in it, so eventually the tasks could be launched on all hosts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65140169
[Test build #24003 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24003/consoleFull) for PR 3541 at commit [`37d8fe0`](https://github.com/apache/spark/commit/37d8fe0b6920824c774a48a0583be43c98a6703c).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-90656662
Close this now, will re-open if needed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-66396977
@suyanNone Yes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by suyanNone <gi...@git.apache.org>.
Github user suyanNone commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-66396760
@davies IIUC, current executor-backlist or host-backlist is all for one TaskSetManager. There is a chance that tow or more taskSetManager run on the same host or executor... right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65272551
Thx @kayousterhout for the ping !
We are fairly aggressively using blacklisting executors - not hosts.
The assumption that a task failed on an executor in a host will imply it will fail on all executors on the host is incorrect - particularly in multi-tennet environment : where number of executors per host can be fairly high : which is exactly the reason why I added this hack to begin with.
Instead of modifying the existing behavior of executor level blacklist, it would be better to add to it - add a way to configure host level blacklisting in addition to existing functionality.
I am -1 on changing the existing behavior.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65144979
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24004/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65303921
@mridulm I think maybe what @davies was asking (which I'm also wondering about) was: why would a task fail on executor A but not on executor B, if they're both on the same host? Many of the reasons I can think of for something failing are host-specific (e.g., bad disk) and not executor-specific.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] counting failures on host instead...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-65136878
@mridulm -- this PR changes the blacklisting behavior to blacklist a host rather than just an executor. Since you wrote the original blacklist code, I just wanted to check that this seems reasonable to you?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3541#issuecomment-72119497
[Test build #26340 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26340/consoleFull) for PR 3541 at commit [`482804b`](https://github.com/apache/spark/commit/482804b9511567aaa02e20890361136b28812ac3).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org