You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by mwws <gi...@git.apache.org> on 2015/09/15 08:40:59 UTC
[GitHub] spark pull request: Spark 8426
GitHub user mwws opened a pull request:
https://github.com/apache/spark/pull/8760
Spark 8426
1. add a global BlacklistTracker which is able to share blacklisted executors information across TaskSetManager. Different BlacklistStrategy could be applied to determine which executors should be put in blacklist. And it's convenient to add new strategy later according to current design.
2. remove duplicate "failedExecutors" list in TaskSetManager and make BlacklistTracker being the only interface for blacklist mechanism in codebase which is logically cleaner.
3. for backward compatibility, implement DefaultStrategy to keep the same semantics as current by default
4. apply BlacklistTracker in YARN to avoid allocating new executors on blacklisted node, the blacklisted node information will be passed to YARN ResourceManager. it affects in dynamic allocation mode.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mwws/spark SPARK-8426
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8760.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8760
----
commit 8519cb5ca4bbe86885fd7cc09fc9aeaf6fab1436
Author: unknown <ma...@maowei-mobl.ccr.corp.intel.com>
Date: 2015-07-28T05:19:29Z
change package name to test.org.apache.spark.sql.hive.execution
commit 7c91bb55b9424657409c7e5131cf9b791b9e4a66
Author: unknown <ma...@maowei-mobl.ccr.corp.intel.com>
Date: 2015-09-02T07:54:31Z
Merge remote-tracking branch 'apache-spark/master'
commit 9a25d71025a411e1fad4190e914465218a1a2d66
Author: mwws <we...@intel.com>
Date: 2015-09-10T05:29:44Z
add executor blacklist feature
1. Add blackListTracker and blackListStrategy class
2. remove deplicated failedExecutor in TaskSetManager
3. enhance YARN allocator to pass node blacklist info to ResourceManager
commit f648b4c185042ee9d38e11b7f213fe68868c8d72
Author: mwws <we...@intel.com>
Date: 2015-09-11T05:25:31Z
some enhancement
1. Add taskId to FailureInfo to be able to contain enough informantion
for backward compatibility
2. replace Seq with Set to speedup contains method and remove duplicate
element by natual
3. remove executors from failureExecutors when
(1)any task succeed on the executor
(2)the executor is removed by ExecutorAllocateClient
3. Move IsBlacklist() method into BlacklistTracker to make logic cleaner
4. fit spark code style, eg: lineend, comments start, max length of line
commit 954886f18b3811958b1c2a588f00652d678cf871
Author: mwws <we...@intel.com>
Date: 2015-09-11T07:47:40Z
fix compile error
fix compile error
commit 78feafbe23729f2102effec9abc90d6a865ba211
Author: mwws <we...@intel.com>
Date: 2015-09-14T04:29:08Z
farther change according to comment
1. rename some varilables and methods
2. correct FailureExecutors logic in CoarseGrainedSchedulerBackend
commit c84f362b4002eddde22f43c421434f413b04a5ee
Author: mwws <we...@intel.com>
Date: 2015-09-14T04:36:53Z
some change to fit scala style
fit scala style
commit 90ee814686e3bf7906fa447b7594aa6b3643de80
Author: mwws <we...@intel.com>
Date: 2015-09-14T05:22:00Z
revert incorrect package name change
revert incorrect package name change
commit 413b403ccc577d062b6eae10d995077631d78e97
Author: mwws <we...@intel.com>
Date: 2015-09-15T05:06:01Z
change strategy to keep semanetic as original
The original blacklist logic is taskId related, so I change interface
and add a default strategy to keep same semanetic as it.
commit 7c6eb632fb4132dc053683db617454ce7b335a31
Author: unknown <ma...@maowei-mobl.ccr.corp.intel.com>
Date: 2015-07-28T05:19:29Z
change package name to test.org.apache.spark.sql.hive.execution
commit b306caecf6d5112c44029ee016654119d3c5185d
Author: mwws <we...@intel.com>
Date: 2015-09-10T05:29:44Z
add executor blacklist feature
1. Add blackListTracker and blackListStrategy class
2. remove deplicated failedExecutor in TaskSetManager
3. enhance YARN allocator to pass node blacklist info to ResourceManager
commit 395773f1a91d13c4e21bd19bcc56e5a02c9c2784
Author: mwws <we...@intel.com>
Date: 2015-09-11T05:25:31Z
some enhancement
1. Add taskId to FailureInfo to be able to contain enough informantion
for backward compatibility
2. replace Seq with Set to speedup contains method and remove duplicate
element by natual
3. remove executors from failureExecutors when
(1)any task succeed on the executor
(2)the executor is removed by ExecutorAllocateClient
3. Move IsBlacklist() method into BlacklistTracker to make logic cleaner
4. fit spark code style, eg: lineend, comments start, max length of line
commit a7dfa20fec17de80755edde3b967f9345d8c2834
Author: mwws <we...@intel.com>
Date: 2015-09-11T07:47:40Z
fix compile error
fix compile error
commit 94d3a5d90e126d8c95580f2e7f3d5fff6dd7bc4c
Author: mwws <we...@intel.com>
Date: 2015-09-14T04:29:08Z
farther change according to comment
1. rename some varilables and methods
2. correct FailureExecutors logic in CoarseGrainedSchedulerBackend
commit 26fb5a5c41bbbdba793d8ea0c1d40cad299ffedb
Author: mwws <we...@intel.com>
Date: 2015-09-14T04:36:53Z
some change to fit scala style
fit scala style
commit f7ecc784788e7b3f0f75a975a781f1af7040bbc4
Author: mwws <we...@intel.com>
Date: 2015-09-14T05:22:00Z
revert incorrect package name change
revert incorrect package name change
commit f44d4e134f725dc42b3017bebe20f459b625df8c
Author: mwws <we...@intel.com>
Date: 2015-09-15T05:06:01Z
change strategy to keep semanetic as original
The original blacklist logic is taskId related, so I change interface
and add a default strategy to keep same semanetic as it.
commit c36ea5b44e9a3c4a91bb3625abc3863e2df2aa66
Author: mwws <we...@intel.com>
Date: 2015-09-15T06:03:46Z
Merge branch 'SPARK-8426' of https://github.com/mwws/spark into SPARK-8426
Conflicts:
core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
commit 24cdf30d6144d084b5c8ff23984e037bab18b4dd
Author: mwws <we...@intel.com>
Date: 2015-09-15T06:19:45Z
fix compile error
fix compile error casued by rebase
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153266133
**[Test build #44892 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44892/consoleFull)** for PR 8760 at commit [`d03cc04`](https://github.com/apache/spark/commit/d03cc04c34ecd12c222d9e325eca58dfa8d5a91e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r43798070
--- Diff: docs/configuration.md ---
@@ -1141,6 +1141,48 @@ Apart from these, the following properties are also available, and may be useful
</td>
</tr>
<tr>
+ <td><code>spark.scheduler.blacklist.enabled</code></td>
+ <td>true</td>
+ <td>
+ If set to ture, executor blacklist feature will be enabled to avoid allocte new task on bad executor. The logic to define bad executor is based on BlacklistStrategy which is also configurable.
+ </td>
+</tr>
+<tr>
+ <td><code>spark.scheduler.executorTaskBlacklistTime</code></td>
+ <td>0L</td>
+ <td>
+ The threshold to deside blacklist executor, if last failed time is older than current time subtract executorTaskBlacklistTime, then the executor could be removed from blacklist.
+ </td>
+</tr>
+<tr>
+ <td><code>spark.scheduler.blacklist.recoverPeriod</code></td>
+ <td>60L</td>
+ <td>
+ The period between runnning blacklist recover process.
+ </td>
+</tr>
+<tr>
+ <td><code>spark.scheduler.blacklist.strategy</code></td>
+ <td>default</td>
+ <td>
+ The strategy to determine executor blacklist and node blacklist. There are "default" strategy which is the srandard behavior before spark 1.6 and "threshold" strategy which is taskId unrelated and can learn experience from other taskSet to avoid allocating tasks on problematic executors.
--- End diff --
this definitely needs more thorough documentation ... probably more than should go into the description of a parameter. Maybe it deserves an additional section like http://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation ?
I will think about this a bit more ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-160884866
What would be more ideal is an implementation which allows for blacklisting single task (as is current), to multiple tasks for a taskset on a single executor (in case the taskset has issues on the executor - memory constraints perhaps) to blacklisting the node : time bound ofcourse, and for specific taskset and/or all taskset until timeout expires (directory cleanup catching up, etc).
Did I miss this in there somewhere ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40285734
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -58,15 +58,6 @@ private[spark] class TaskSetManager(
val conf = sched.sc.conf
- /*
- * Sometimes if an executor is dead or in an otherwise invalid state, the driver
- * does not realize right away leading to repeated task failures. If enabled,
- * this temporarily prevents a task from re-launching on an executor where
- * it just failed.
- */
- private val EXECUTOR_TASK_BLACKLIST_TIMEOUT =
- conf.getLong("spark.scheduler.executorTaskBlacklistTime", 0L)
--- End diff --
Good point, I will use this name instead of "spark.scheduler.blacklist.timeout"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-148658070
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153962783
Build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153962785
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45091/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-170446175
**[Test build #49099 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49099/consoleFull)** for PR 8760 at commit [`d94bdd7`](https://github.com/apache/spark/commit/d94bdd7d16b77317bdfe5851e9ba297a229b4113).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149566499
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40217450
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -184,13 +188,21 @@ private[yarn] class YarnAllocator(
def requestTotalExecutorsWithPreferredLocalities(
requestedTotal: Int,
localityAwareTasks: Int,
- hostToLocalTaskCount: Map[String, Int]): Boolean = synchronized {
+ hostToLocalTaskCount: Map[String, Int],
+ nodeBlacklist: Set[String]): Boolean = synchronized {
--- End diff --
update the doc to include this param, and also mention in the description that we update the blacklist w/ the AM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-160890985
Since I am no longer very familiar with scheduler codebase, I am sort of giving up on my review halfway through - save the notes already added. Hopefully someone else more active can give a +1 !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-148208198
[Test build #43743 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43743/consoleFull) for PR 8760 at commit [`46ee0ff`](https://github.com/apache/spark/commit/46ee0ff4cb20e2ae18436cfacb5a4847e53d2a91).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-148207609
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42330850
--- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import org.apache.spark.SparkFunSuite
+import org.scalatest.BeforeAndAfter
+import org.apache.spark.SparkConf
+import org.apache.spark.TaskEndReason
+import org.apache.spark.Success
+import org.apache.spark.ExceptionFailure
+import org.apache.spark.LocalSparkContext
+import org.apache.spark.SparkContext
+import org.mockito.Mockito.{when,spy}
+import org.apache.spark.util.ManualClock
+
+class BlacklistTrackerSuite extends SparkFunSuite with BeforeAndAfter with LocalSparkContext {
+
+ val FAILURE: TaskEndReason = new ExceptionFailure(
+ "Fake",
+ "fake failure",
+ Array.empty[StackTraceElement],
+ "fake stack trace",
+ None,
+ None)
+
+ // Variable name can indicate basic information of taskInfo
+ // The format is "taskInfo_executorId_taskId_hostName"
+ val taskInfo_1_1_hostA = new TaskInfo(1L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_1_2_hostA = new TaskInfo(2L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_1_hostA = new TaskInfo(1L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_2_hostA = new TaskInfo(2L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_3_3_hostB = new TaskInfo(3L,1,1,0L,"3","hostB", TaskLocality.ANY, false)
--- End diff --
It seems that "sbt scalastyle" command skipped test folders, I have explicitly check with "sbt test:scalastyle" locally. It should be fine this time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149078383
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149564536
@mwws yes, looks like it was just a timeout running the test, just give it another run. I will also add you to the whitelist so you can trigger builds yourself. Btw, this is still needs to update the docs with the new config options.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-148658048
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42217021
--- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -0,0 +1,162 @@
+package org.apache.spark.scheduler
+
+import org.apache.spark.SparkFunSuite
+import org.scalatest.BeforeAndAfter
+import org.apache.spark.SparkConf
+import org.apache.spark.TaskEndReason
+import org.apache.spark.Success
+import org.apache.spark.ExceptionFailure
+import org.apache.spark.LocalSparkContext
+import org.apache.spark.SparkContext
+
+import org.mockito.Mockito.{when,spy}
+
+class BlacklistTrackerSuite extends SparkFunSuite with BeforeAndAfter with LocalSparkContext {
+
+ val FAILURE: TaskEndReason = new ExceptionFailure(
+ "Fake",
+ "fake failure",
+ Array.empty[StackTraceElement],
+ "fake stack trace",
+ None,
+ None)
+
+ // Variable name can indicate basic information of taskInfo
+ // The format is "taskInfo_executorId_taskId_hostName"
+ val taskInfo_1_1_hostA = new TaskInfo(1L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_1_2_hostA = new TaskInfo(2L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_1_hostA = new TaskInfo(1L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_2_hostA = new TaskInfo(2L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_3_3_hostB = new TaskInfo(3L,1,1,0L,"3","hostB", TaskLocality.ANY, false)
+
+ test ("expireExecutorsInBlacklist works ") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.recoverPeriod", "2")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.start()
+ Thread.sleep(100)
--- End diff --
Got it, thanks for the tips.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-154347311
**[Test build #45200 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45200/consoleFull)** for PR 8760 at commit [`33e3542`](https://github.com/apache/spark/commit/33e3542bb0b3c88c079dff4635a17c669636fd0d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `final class FailureStatus(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40226708
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
@@ -568,12 +568,12 @@ private[spark] class ApplicationMaster(
}
override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
- case RequestExecutors(requestedTotal, localityAwareTasks, hostToLocalTaskCount) =>
+ case RequestExecutors(requestedTotal, localityAwareTasks, hostToLocalTaskCount, nodeBalcklist) =>
--- End diff --
typo: `nodeBlacklist`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42008413
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.util.SystemClock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+trait BlacklistStrategy {
+ val expireTimeInMillisecond: Long
+
+ // Return executors in blacklist which are related to given TaskId
+ def getExecutorBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ taskId: Long): Set[String]
+
+ // Return all nodes in blacklist
+ def getNodeBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Set[String]
+
+ // Default implementation to remove failure executors from HashMap based on given time period.
+ def expireExecutorsInBlackList(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Unit = {
+ val now = new SystemClock().getTimeMillis()
+ executorIdToFailureStatus.retain((executorid, failureStatus) => {
+ (now - failureStatus.updatedTime) < expireTimeInMillisecond
+ })
+ }
+}
+
+/**
+ * This strategy is simply based on given threshold and is taskId unrelated. An executor will be
+ * in blacklist, if it failed more than "maxFailureTaskNumber" times. A node will be in blacklist,
+ * if there are more than "maxBlackExecutorNumber" executors on it in executor blacklist.
+ *
+ * In this case, provided taskId will be ignored. The benefit for taskId unrelated strategy is that
+ * different taskSets can learn experience from other taskSet to avoid allocating tasks on
+ * problematic executors.
+ */
+class SimpleStrategy(
+ maxFailureTaskNumber: Int,
+ maxBlackExecutorNumber: Int,
+ val expireTimeInMillisecond: Long
+ )extends BlacklistStrategy {
+
+ private def getSelectedExecutorMap(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]) = {
+ executorIdToFailureStatus.filter{
+ case (id, failureStatus) => failureStatus.totalNumFailures > maxFailureTaskNumber
+ }
+ }
+
+ // As this is a taskId unrelated strategy, the input taskId will be ignored
+ def getExecutorBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ taskId: Long): Set[String] = {
+ getSelectedExecutorMap(executorIdToFailureStatus).keys.toSet
+ }
+
+ def getNodeBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Set[String] = {
+ getSelectedExecutorMap(executorIdToFailureStatus)
+ .groupBy{case (id, failureStatus) => failureStatus.host}
+ .filter {case (host, executorIdToFailureStatus) =>
+ executorIdToFailureStatus.size > maxBlackExecutorNumber}
+ .keys.toSet
+ }
+}
+
+/**
+ * This strategy is applied as default to keep the same semantics as original. It's an taskId
+ * related strategy. If an executor failed running "task A", then we think this executor is
+ * blacked for "task A". And we think the executor is still healthy for other task. node blacklist
+ * is always empty.
+ *
+ * It was the standard behavior before spark 1.6
+ */
+class DefaultStrategy(val expireTimeInMillisecond: Long) extends BlacklistStrategy {
+ def getExecutorBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ taskId: Long): Set[String] = {
+ executorIdToFailureStatus.filter{
+ case (_, failureStatus) => failureStatus.numFailuresPerTask.keySet.contains(taskId)
+ }.keys.toSet
+ }
+
+ def getNodeBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Set[String] =
+ Set.empty[String]
+}
+
+/**
+ * Create BlacklistStrategy instance according to SparkConf
+ */
+object BlacklistStrategy {
+ def apply(sparkConf: SparkConf): BlacklistStrategy = {
+ val timeout = sparkConf.getLong("spark.scheduler.executorTaskBlacklistTime", 0L)
+ sparkConf.get("spark.scheduler.blacklist.strategy", "default") match {
+ case "default" =>
+ new DefaultStrategy(timeout)
+ case "threshold" =>
+ new SimpleStrategy(
+ sparkConf.getInt("spark.scheduler.blacklist.threshold.maxFailureTaskNumber", 3),
+ sparkConf.getInt("spark.scheduler.blacklist.threshold.maxBlackExecutorNumber", 3),
+ timeout)
+ case "strict" =>
+ // A special case of SimpleStrategy: Once task failed at executor,
+ // put the executor and its node into blacklist.
+ new SimpleStrategy(0, 0, timeout)
--- End diff --
I don't really see the point of adding "strict" -- do you think it will be a very common use case? Otherwise its just another thing to document and might confuse users a bit.
I'd also like to think of better names for the other strategies as well. It seems the key difference is whether or not they are task-specific, which isn't very clear from "default" and "threshold". I'm terrible at naming myself ... something like "singleTask" and "completeExecutor"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r53165346
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Clock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+private [scheduler] trait BlacklistStrategy {
+ /** Define a time interval to expire failure information of executors */
+ val expireTimeInMilliseconds: Long
+
+ /** Return executors in blacklist which are related to given stage and partition */
+ def getExecutorBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ atomTask: StageAndPartition,
+ clock: Clock): Set[String]
+
+ /** Return all nodes in blacklist */
+ def getNodeBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ clock: Clock): Set[String]
+
+ /** Return all nodes in blacklist for specified stage. By default it returns the same result as
+ * getNodeBlacklist. It could be override in strategy implementation.
+ */
+ def getNodeBlacklistForStage(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ stageId: Int,
+ clock: Clock): Set[String] = getNodeBlacklist(executorIdToFailureStatus, clock)
+
+ /**
+ * Choose which executors should be removed from blacklist. Return true if any executors are
+ * removed from the blacklist, false otherwise. The default implementation removes executors from
+ * the blacklist after [[expireTimeInMilliseconds]]
+ */
+ def expireExecutorsInBlackList(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ clock: Clock): Boolean = {
+ val now = clock.getTimeMillis()
+ val expiredKey = executorIdToFailureStatus.filter {
+ case (executorid, failureStatus) => {
+ (now - failureStatus.updatedTime) >= expireTimeInMilliseconds
+ }
+ }.keySet
+
+ if (expiredKey.isEmpty) {
+ false
+ } else {
+ executorIdToFailureStatus --= expiredKey
+ true
+ }
+ }
+}
+
+/**
+ * This strategy adds an executor to the blacklist for all tasks when the executor has too many
+ * task failures. An executor is placed in the blacklist when there are more than
+ * [[maxFailedTasks]] failed tasks. Furthermore, all executors in one node are put into the
+ * blacklist if there are more than [[maxBlacklistedExecutors]] blacklisted executors on one node.
+ * The benefit of this strategy is that different taskSets can learn experience from other taskSet
+ * to avoid allocating tasks on problematic executors.
+ */
+private[scheduler] class ExecutorAndNodeStrategy(
+ maxFailedTasks: Int,
+ maxBlacklistedExecutors: Int,
+ val expireTimeInMilliseconds: Long
+ ) extends BlacklistStrategy {
+
+ private def getExecutorBlacklistInfo(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]) = {
+ executorIdToFailureStatus.filter{
+ case (id, failureStatus) => failureStatus.totalNumFailures > maxFailedTasks
+ }
+ }
+
+ // As this is a task unrelated strategy, the input StageAndPartition info will be ignored
+ def getExecutorBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ atomTask: StageAndPartition,
+ clock: Clock): Set[String] = {
+ getExecutorBlacklistInfo(executorIdToFailureStatus).keys.toSet
+ }
+
+ def getNodeBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ clock: Clock): Set[String] = {
+ getExecutorBlacklistInfo(executorIdToFailureStatus)
+ .groupBy{case (id, failureStatus) => failureStatus.host}
+ .filter {case (host, executorIdToFailureStatus) =>
+ executorIdToFailureStatus.size > maxBlacklistedExecutors}
+ .keys.toSet
+ }
+}
+
+/**
+ * This strategy is applied as default to keep the same semantics as original. It's an task
+ * related strategy. If an executor failed running "task A", then we think this executor is
+ * blacked for "task A". And we think the executor is still healthy for other task. node blacklist
+ * is always empty.
+ *
+ * It was the standard behavior before spark 1.6
+ */
+private[scheduler] class SingleTaskStrategy(
+ val expireTimeInMilliseconds: Long) extends BlacklistStrategy {
+ def getExecutorBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ atomTask: StageAndPartition,
+ clock: Clock): Set[String] = {
+ executorIdToFailureStatus.filter{
+ case (_, failureStatus) => failureStatus.numFailuresPerTask.keySet.contains(atomTask) &&
+ clock.getTimeMillis() - failureStatus.updatedTime < expireTimeInMilliseconds
+ }.keys.toSet
+ }
+
+ def getNodeBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ clock: Clock): Set[String] = Set.empty[String]
+}
+
+/**
+ * Support getNodeBlacklistForStage. With this strategy, once executor failed running a task, we
+ * put all executors on the same node into blacklist, so all tasks on the same stage will not be
+ * allocated to that node.
+ */
+private[scheduler] class AdvancedSingleTaskStrategy(
+ expireTimeInMilliseconds: Long) extends SingleTaskStrategy(expireTimeInMilliseconds) {
+
+ override def getNodeBlacklistForStage(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ stageId: Int,
+ clock: Clock): Set[String] = {
+ executorIdToFailureStatus.filter{
+ case (_, failureStatus) =>
+ failureStatus.numFailuresPerTask.keySet.map(_.stageId).contains(stageId) &&
+ clock.getTimeMillis() - failureStatus.updatedTime < expireTimeInMilliseconds
+ }.values.map(_.host).toSet
+ }
+}
+
+/**
+ * Create BlacklistStrategy instance according to SparkConf
+ */
+private[scheduler] object BlacklistStrategy {
+ def apply(sparkConf: SparkConf): BlacklistStrategy = {
+ val timeout = sparkConf.getTimeAsMs("spark.scheduler.blacklist.timeout",
+ sparkConf.getLong("spark.scheduler.executorTaskBlacklistTime", 0L).toString() + "ms")
+ sparkConf.get("spark.scheduler.blacklist.strategy", "singleTask") match {
--- End diff --
As we talked offline, I am wondering if we can simplify the configuration by just enabling the experimental feature or not?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42056865
--- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -0,0 +1,162 @@
+package org.apache.spark.scheduler
--- End diff --
need to add the license header here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40168257
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.util.SystemClock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+trait BlacklistStrategy {
+ val expireTimeInMillisecond: Long
+
+ // TaskId is optional here because the BlacklistStrategy could be unrelated to TaskId
+ def getExecutorBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String]
+
+ def getNodeBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String]
+
+ // Default implementation to remove failure executors from HashMap based on given time period.
+ def expireExecutorsInBlackList(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]): Unit = {
+ val now = new SystemClock().getTimeMillis()
+ failedExecutorMap.retain((executorid, failureStatus) => {
+ (now - failureStatus.updatedTime) < expireTimeInMillisecond
+ })
+ }
+}
+
+/**
+ * The interface for taskId unrelated strategy. In this case, provided taskId will be ignored. The
+ * benefit for taskId unrelated strategy is different taskSets can learn experience from other
+ * taskSet to avoid allocating tasks to problematic executors.
+ */
+trait TaskUnrelatedBlacklistStrategy extends BlacklistStrategy with Logging{
+ def getExecutorBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String] = {
+ if (taskId.isDefined){
+ logWarning("TaskId unrelated strategy is applied, given taskId will be ignored")
+ }
+ getExecutorBlacklist(failedExecutorMap)
+ }
+
+ def getNodeBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String] = {
+ if (taskId.isDefined){
+ logWarning("TaskId unrelated strategy is applied, given taskId will be ignored")
+ }
+ getNodeBlacklist(failedExecutorMap)
+ }
+
+ // define new interface which remove taskId from parameter.
+ def getExecutorBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]): Set[String]
+ def getNodeBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]): Set[String]
+}
+
+/**
+ * This strategy is simply based on given threshold and is taskId unrelated. An executor will be
+ * in blacklist, if it failed more than "maxFailureTaskNumber" times. A node will be in blacklist,
+ * if there are more than "maxBlackExecutorNumber" executors on it in executor blacklist.
+ */
+class SimpleStrategy(
+ maxFailureTaskNumber: Int,
+ maxBlackExecutorNumber: Int,
+ val expireTimeInMillisecond: Long
+ )extends TaskUnrelatedBlacklistStrategy {
+
+ private def getSelectedExecutorMap(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]) = {
+ failedExecutorMap.filter{
+ case (id, failureStatus) => failureStatus.failureTimes > maxFailureTaskNumber
+ }
+ }
+
+ def getExecutorBlacklist(
+ failureExecutors: mutable.HashMap[String, FailureStatus]): Set[String] = {
+ getSelectedExecutorMap(failureExecutors).keys.toSet
+ }
+
+ def getNodeBlacklist(
+ failureExecutors: mutable.HashMap[String, FailureStatus]): Set[String] = {
--- End diff --
If the failureExecutors become large, it would be an issue. For the cache, do you mean maintain a local HashMap in this class as cache?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-152970172
**[Test build #44800 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44800/consoleFull)** for PR 8760 at commit [`6b7627f`](https://github.com/apache/spark/commit/6b7627f063ca7de930c8fb0802c7ae95a20fa226).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `final class FailureStatus(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149645257
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-170446190
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49099/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153298062
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44902/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149220589
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-152944169
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149640762
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40216505
--- Diff: core/src/test/scala/org/apache/spark/HeartbeatReceiverSuite.scala ---
@@ -261,7 +261,7 @@ private class FakeClusterManager(override val rpcEnv: RpcEnv) extends RpcEndpoin
def getExecutorIdsToKill: Set[String] = executorIdsToKill.toSet
override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
- case RequestExecutors(requestedTotal, _, _) =>
+ case RequestExecutors(requestedTotal, _, _,_) =>
--- End diff --
nit: space after the comma
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42045636
--- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -0,0 +1,162 @@
+package org.apache.spark.scheduler
+
+import org.apache.spark.SparkFunSuite
+import org.scalatest.BeforeAndAfter
+import org.apache.spark.SparkConf
+import org.apache.spark.TaskEndReason
+import org.apache.spark.Success
+import org.apache.spark.ExceptionFailure
+import org.apache.spark.LocalSparkContext
+import org.apache.spark.SparkContext
+
+import org.mockito.Mockito.{when,spy}
+
+class BlacklistTrackerSuite extends SparkFunSuite with BeforeAndAfter with LocalSparkContext {
+
+ val FAILURE: TaskEndReason = new ExceptionFailure(
+ "Fake",
+ "fake failure",
+ Array.empty[StackTraceElement],
+ "fake stack trace",
+ None,
+ None)
+
+ // Variable name can indicate basic information of taskInfo
+ // The format is "taskInfo_executorId_taskId_hostName"
+ val taskInfo_1_1_hostA = new TaskInfo(1L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_1_2_hostA = new TaskInfo(2L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_1_hostA = new TaskInfo(1L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_2_hostA = new TaskInfo(2L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_3_3_hostB = new TaskInfo(3L,1,1,0L,"3","hostB", TaskLocality.ANY, false)
+
+ test ("expireExecutorsInBlacklist works ") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.recoverPeriod", "2")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.start()
+ Thread.sleep(100)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1"))
+
+ // sleep 2 second to wait for expireExecutorsInBlackList be triggered
+ Thread.sleep(2000)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+
+ tracker.stop()
+ }
+
+ test("default strategy works as exceptation") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1","2"))
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set())
+
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("2"))
+ assert(tracker.nodeBlacklist() === Set())
+
+ tracker.updateFailureExecutors(taskInfo_3_3_hostB, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 3L) === Set("3"))
+ assert(tracker.nodeBlacklist() === Set())
+
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+ }
+
+ test ("strict strategy works as exception") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.strategy", "strict")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1"))
+ tracker.updateFailureExecutors(taskInfo_1_2_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set("1"))
--- End diff --
shouldn't this assert work even before you have the failure on the line above? Doesn't this strategy put executor 1 in the blacklist for all tasks?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42041212
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update
+ * executorIdToFailureStatus Map.
+ */
+private[spark] class BlacklistTracker(sparkConf: SparkConf) extends WithCache{
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val executorIdToFailureStatus: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val clock = new SystemClock()
+
+ private val enableBlacklistSpeculate = sparkConf.getBoolean(
+ "spark.scheduler.blacklist.speculate", false)
+
+ private val recoverPeriod = sparkConf.getLong(
+ "spark.scheduler.blacklist.recoverPeriod", 60L)
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList(executorIdToFailureStatus))
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, recoverPeriod, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ private def expireExecutorsInBlackList(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Unit = synchronized {
+ strategy.expireExecutorsInBlackList(executorIdToFailureStatus)
+
+ invalidAllCache()
+ }
+
+ def updateFailureExecutors(info: TaskInfo, reason: TaskEndReason) : Unit = synchronized {
+ reason match {
+ // If task succeeding, remove related record from executorIdToFailureStatus
+ case Success =>
+ removeFailureExecutorsForTaskId(info.executorId, info.taskId)
+
+ // If task failing, update latest failure time and failedTaskIds
+ case _ =>
+ val executorId = info.executorId
+ executorIdToFailureStatus.get(executorId) match {
+ case Some(failureStatus) =>
+ failureStatus.updatedTime = clock.getTimeMillis()
+ val failedTimes = failureStatus.numFailuresPerTask.getOrElse(info.taskId, 0) + 1
+ failureStatus.numFailuresPerTask.update(info.taskId, failedTimes)
+ case None =>
+ val failedTasks = mutable.HashMap(info.taskId -> 1)
+ val failureStatus = new FailureStatus(
+ clock.getTimeMillis(),
+ info.host,
+ failedTasks)
+ executorIdToFailureStatus.update(executorId, failureStatus)
+ }
+ invalidAllCache()
+ }
+ }
+
+ // remove the executorId from executorIdToFailureStatus
+ def removeFailureExecutors(executorId: String) : Unit = synchronized {
+ executorIdToFailureStatus.remove(executorId)
+ invalidAllCache()
+ }
+
+ // remove the failure record related to given taskId from executorIdToFailureStatus. If the
+ // number of records of given executorId becomes 0, remove the completed executorId.
+ def removeFailureExecutorsForTaskId(
+ executorId: String,
+ taskId: Long) : Unit = synchronized {
+ executorIdToFailureStatus.get(executorId).map(fs => {
+ fs.numFailuresPerTask.remove(taskId)
+ if(fs.numFailuresPerTask.isEmpty){
+ executorIdToFailureStatus.remove(executorId)
+ }
+ invalidAllCache()
+ })
+ }
+
+ def executorIsBlacklisted(
+ executorId: String,
+ sched: TaskSchedulerImpl,
+ taskId: Long) : Boolean = {
+
+ executorBlacklist(sched, taskId).contains(executorId)
+ }
+
+ // If the node is in blacklist, all executors allocated on that node will
+ // also be put into executor blacklist.
+ // By default it's turned off, user can enable it in sparkConf.
+ private def speculationFailedExecutor(
+ sched: TaskSchedulerImpl): Set[String] = {
--- End diff --
I know we discussed this previously, but I still don't really get the point of "speculation". I don't really see a strong use case, I feel like you'd always want it to be `true`. It seems that speculation is mostly covered by the "maxBlackExecutorNumber" conf. If you want to avoid blacklisting nodes, then you set that value very high; if you want to blacklist nodes as soon as you think its got back executors, you set it low. The distinction you get with speculation, is that you would differentiate two different parts of node blacklisting: (a) do you request new executors on those nodes? and (b) do you schedule tasks on _existing_ executors on those nodes?
Do we really think that there is an important use case for updating one without the other? Without a clear use case, I'm against it even just for the difficulty in clearly documenting to the end user. The name "speculation" is very confusing to me, but i dunno what else to call it.
Repeating myself again, but if you really want to leave the door open for some future strategy to distinguish the two, you could change the api:
(a) `BlacklistStrategy.getNodeBlacklist()` is used only in requests to the cluster manager when requesting new nodes. When scheduling tasks, the tracker only calls `strategy.getExecutorBlacklist()`
(b) When `SimpleStrategy` adds a blacklisted node, it will also update its set of blacklisted executors to go along with it (effectively turning on "speculation") -- but that is now an internal implementation of that particular strategy.
Then you could still have a different strategy without speculation if you really wanted it, but again I wouldn't add that till there was a use case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r46248240
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+import org.apache.spark.util.Clock
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update
+ * executorIdToFailureStatus Map.
+ */
+private[spark] class BlacklistTracker(
+ sparkConf: SparkConf,
+ clock: Clock = new SystemClock()) extends BlacklistCache{
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val executorIdToFailureStatus: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val recoverPeriod = sparkConf.getTimeAsSeconds(
+ "spark.scheduler.blacklist.recoverPeriod", "60s")
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList())
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, recoverPeriod, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ /** VisibleForTesting */
+ private[scheduler] def expireExecutorsInBlackList(): Unit = synchronized {
+ val updated = strategy.expireExecutorsInBlackList(executorIdToFailureStatus, clock)
+ if (updated) {
+ invalidateCache()
+ }
+ }
--- End diff --
@kayousterhout Would be good to have your thoughts here since I am fairly out of touch with scheduler now.
If I read it correctly, multiple methods in this class are directly called from scheduler as part of identifying, starting, updating tasks to be run from TaskSetManager/scheduler and they all use the same lock as updateFailedExecutors - which can actually run fairly non trivial computation as number of failures increase in the system. Is this a concern or is it alleviated somehow and I missed it ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153935641
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-148208308
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149568683
**[Test build #43980 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43980/consoleFull)** for PR 8760 at commit [`a2c8b40`](https://github.com/apache/spark/commit/a2c8b4021940606f17d122dc71ff444ca47c25a5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-142683197
Hi @mwws, I've been through in more detail, sorry if my comments have been a bit scattered as I worked through understanding this. I have one more high-level thought about the approach:
Its great that we can update yarn on the blacklisted nodes. However, do the strategies also need someway of *immediately* killing the blacklisted executors? Eg., say you request 5 nodes on a 100 node cluster, and in the early phase of your app you discover that one of the executors is failing repeatedly. The change proposed lets us go to scheduling tasks directly on the 4 good executors, but wouldn't you rather tell yarn you've got a bad executor, kill it, and request another one? This could be left to the strategy to choose, but seems like it should be possible and in one of the provided implementations. In that case it wouldn't really make sense to have an expiry time for the executors, since you just completely kill them, but I suppose you would still have an expiry time for the node blacklist. Then you also naturally escalate from assuming you've got one bad executor, and then if you get another bad executor on the same node, you'd blacklist the entire node.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-152970224
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44800/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-148191747
HI @mwws , just finished another round of review. I think the main things left are making sure we have the right decision for speculation (left a bigger inline comment), and also adding docs for this feature to http://spark.apache.org/docs/latest/configuration.html#scheduling by editing [`docs/configuration.md`](https://github.com/apache/spark/blob/master/docs/configuration.md). (It looks like "spark.scheduler.executorTaskBlacklistTime" isn't even documented currently, which it really should be.)
thanks for the update, looking good!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153938185
[Test build #45091 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45091/consoleFull) for PR 8760 at commit [`9a5ec8e`](https://github.com/apache/spark/commit/9a5ec8e51bbf5ae685f550f9af7eea2db3502fad).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-160814564
`spark.scheduler.executorTaskBlacklistTime` is renamed to `spark.scheduler.blacklist.timeout` in this PR to make blacklist related configuration following unified pattern spark.scheduler.blacklist.XXXX. By the way, the usage of this timeout configuration was missed in configuration doc before, I have added it back with new name.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-151616397
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-161100737
@kayousterhout this is pretty important for users running clusters with a larger clusters, eg. a few hundred nodes. We've seen cases where there is some weird misconfiguration on a node, which leads to all tasks failing on that node. The job will eventually succeed if you use the existing blacklist behavior, but with even one bad node you get so many task failure messages its very hard to make any sense of what is going on -- apps with thousands of tasks per stage and tens of stages generate tons of task failures. This leads to users complaining that things are broken, when its actually things are "ok" in the sense that they run successfully after completing, and also users being unable to find the "real" failures hidden in all the bogus msgs.
Cluster managers don't help here -- the problem is when there is a gap between the manager's notion of a "healthy" node and the particular spark application (eg., one particular library fails to load, or bad memory or something).
There is also an efficiency advantage to blacklisting the entire executor, rather than just letting tasks fail on your bad executors. In general the tasks will fail quickly and just get rescheduled (I've only actually seen one case where the tasks did not fail almost immediately), so maybe this is more minor, but seems like a nice thing to do right. But beyond just the initial task scheduling, this PR adds the ability for the cluster manager to avoid giving more executors on the bad node. Plus this opens the ability for spark to actively request a new executor -- eg., if you request 5 executors and one is bad, rather than running with 80% of your requested resources, you should just throw away that executor and get another one (if the cluster has capacity). though that is not directly included in this patch -- in earlier discussion @mwws thought it best to push off to follow up work.
I also like the idea of minimizing user configuration, but also recognize that we are working from a position of some ignorance on the failure modes people encounter and how they want fixes. Eg., I imagine that everyone would want to use the `ExecutorAndNode` strategy and not change any other configuration, but perhaps for some users they want the old behavior and more knobs. That's also why I'm reluctant to change the default behavior (but not drawing a hard line either). After providing some options we could try to collect some more feedback on what works.
You've got a good point about only blacklisting executors across jobs if the tasks succeed elsewhere. I was initially only thinking about a linear spark application -- in that case, if some tasks in a job fail everywhere, then the job will fail and the spark app is done in any case. But of course apps may try to recover from failed jobs, or even have completely independent jobs, in a "job server" style deployment.
Another point you made on another pr is that we have to ensure that we don't prevent any progress from being made if all executors get blacklisted and there is nowhere for any tasks to run. We should make sure we have some sensible behavior there, eg., fail the job.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40212790
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala ---
@@ -56,8 +56,12 @@ private[spark] abstract class YarnSchedulerBackend(
* This includes executors already pending or running.
*/
override def doRequestTotalExecutors(requestedTotal: Int): Boolean = {
+
+ val nodeBlacklist: Set[String] = scheduler.sc.blacklistTracker
+ .fold(Set.empty[String])(_.nodeBlacklist())
--- End diff --
I know this is normal scala code, but in spark we've decided to adopt a convention of *not* using `option.fold` -- instead we use `option.map(...).getOrElse(...)`, or maybe pattern matching depending on the case. its just a little too confusing for those new to scala.
(you use this in a few other places too, I think)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-167728914
@mridulm I have changed the interface a little bit and create `AdvancedSingleTaskStrategy` to support the use case you described above. With this new strategy, we enable node blacklist and scope of influence is limited in every stage. A unit test *AdvancedSingleTask strategy works* is added to show how it works.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r43799112
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.Clock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+private [scheduler] trait BlacklistStrategy {
+ /** Defined a time interval to expire failure information of executors */
+ val expireTimeInMilliseconds: Long
+
+ /** Return executors in blacklist which are related to given taskIndex */
+ def getExecutorBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ taskIndex: Int, clock: Clock): Set[String]
+
+ /** Return all nodes in blacklist */
+ def getNodeBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Set[String]
+
+ /**
+ * Default implementation to remove failure executors from HashMap based on given time period.
+ * The return value identity whether or not it updated anything
+ */
+ def expireExecutorsInBlackList(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus], clock: Clock): Boolean = {
+ val now = clock.getTimeMillis()
+ val expiredKey = executorIdToFailureStatus.filter {
+ case (executorid, failureStatus) => {
+ (now - failureStatus.updatedTime) >= expireTimeInMilliseconds
+ }
+ }.keySet
+
+ if (expiredKey.isEmpty) {
+ false
+ } else {
+ executorIdToFailureStatus --= expiredKey
+ true
+ }
+ }
+}
+
+/**
+ * This strategy is simply based on given threshold and is taskId unrelated. An executor will be
+ * in blacklist, if it failed more than "maxFailureTaskNumber" times. A node will be in blacklist,
+ * if there are more than "maxBlackExecutorNumber" executors on it in executor blacklist.
+ *
+ * In this case, provided taskId will be ignored. The benefit for taskId unrelated strategy is that
+ * different taskSets can learn experience from other taskSet to avoid allocating tasks on
+ * problematic executors.
+ */
+private[scheduler] class SimpleStrategy(
+ maxFailureTaskNumber: Int,
+ maxBlackExecutorNumber: Int,
+ val expireTimeInMilliseconds: Long
+ )extends BlacklistStrategy {
+
+ private def getSelectedExecutorMap(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]) = {
+ executorIdToFailureStatus.filter{
+ case (id, failureStatus) => failureStatus.totalNumFailures > maxFailureTaskNumber
+ }
+ }
+
+ // As this is a taskId unrelated strategy, the input taskId will be ignored
+ def getExecutorBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ taskIndex: Int, clock: Clock): Set[String] = {
+ getSelectedExecutorMap(executorIdToFailureStatus).keys.toSet
+ }
+
+ def getNodeBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Set[String] = {
+ getSelectedExecutorMap(executorIdToFailureStatus)
+ .groupBy{case (id, failureStatus) => failureStatus.host}
+ .filter {case (host, executorIdToFailureStatus) =>
+ executorIdToFailureStatus.size > maxBlackExecutorNumber}
+ .keys.toSet
+ }
+}
+
+/**
+ * This strategy is applied as default to keep the same semantics as original. It's an taskId
+ * related strategy. If an executor failed running "task A", then we think this executor is
+ * blacked for "task A". And we think the executor is still healthy for other task. node blacklist
+ * is always empty.
+ *
+ * It was the standard behavior before spark 1.6
+ */
+private[scheduler] class DefaultStrategy(
+ val expireTimeInMilliseconds: Long) extends BlacklistStrategy {
+ def getExecutorBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ taskIndex: Int, clock: Clock): Set[String] = {
+ executorIdToFailureStatus.filter{
+ case (_, failureStatus) => failureStatus.numFailuresPerTask.keySet.contains(taskIndex) &&
+ clock.getTimeMillis() - failureStatus.updatedTime < expireTimeInMilliseconds
+ }.keys.toSet
+ }
+
+ def getNodeBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Set[String] =
+ Set.empty[String]
+}
+
+/**
+ * Create BlacklistStrategy instance according to SparkConf
+ */
+private[scheduler] object BlacklistStrategy {
+ def apply(sparkConf: SparkConf): BlacklistStrategy = {
+ val timeout = sparkConf.getLong("spark.scheduler.executorTaskBlacklistTime", 0L)
--- End diff --
given my comments on the docs, change this to
```scala
sparkConf.getTimeAsSeconds("spark.scheduler.blacklist.timeout", sparkConf.getLong("spark.scheduler.executorTaskBlacklistTime", 0L) / 1000)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40215133
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.util.SystemClock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+trait BlacklistStrategy {
+ val expireTimeInMillisecond: Long
+
+ // TaskId is optional here because the BlacklistStrategy could be unrelated to TaskId
--- End diff --
ah, I see now. I was originally referring to both `getExecutorBlacklist` and `getNodeBlacklist` -- thanks for pointing out that `getNodeBlacklist` actually is used w/out a `taskId`, I had missed that.
I'd like to avoid over-engineering this api -- it'll be more clear to future readers & easier to test if we keep it minimal, so I am thinking that maybe we should just have two methods:
```scala
def getExecutorBlacklist(
failedExecutorMap: mutable.HashMap[String, FailureStatus],
taskId: Long): Set[String]
)
def getNodeBlacklist(failedExecutorMap: mutable.HashMap[String, FailureStatus]): Set[String]
```
that is, `getExecutorBlacklist` always takes a `taskId`, and `getNodeBlacklist` never does. Does that support the required use case? (Docs would also really help explain the purpose of these methods.) Feel free to push back if I'm still missing something.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r43798545
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -21,9 +21,10 @@ import java.util.Collections
import java.util.concurrent._
import java.util.regex.Pattern
+import scala.collection.JavaConverters._
import scala.collection.mutable
import scala.collection.mutable.{ArrayBuffer, HashMap, HashSet}
-import scala.collection.JavaConverters._
+import scala.collection.immutable.Set
--- End diff --
super nit: I think JavaConverters was actually correct in the old location (order is not just alphabetic -- full classes sort before packages). Also immutable should go before mutable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-151675882
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153252089
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-147312498
@andrewor14 , could you also help review and comment. Thanks in advance.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r43786312
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -83,8 +74,6 @@ private[spark] class TaskSetManager(
val copiesRunning = new Array[Int](numTasks)
val successful = new Array[Boolean](numTasks)
private val numFailures = new Array[Int](numTasks)
- // key is taskId, value is a Map of executor id to when it failed
--- End diff --
ah, I'm glad you pointed this out, b/c this is actually a really important difference. So I think this change is broken in the way it works across multiple task sets. In the original implementation, this distinction didn't matter so much because all of the bookkeeping was kept within one TaskSetManager. However, now that BlacklistTracker is global, it being used for multiple stages, where a task index refers to completely different things. Eg., you might have a failure in task 1 of stage 1, and another failure of task 1 in stage 2, but those are totally different tasks. so the default strategy is doing something different than before.
I think the general solution here is for all the `BlacklistXXX` classes to always refer to both a `stageId` and `taskId`, in the method arguments and also the bookkeeping. You should add a test case for this as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r43711057
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -83,8 +74,6 @@ private[spark] class TaskSetManager(
val copiesRunning = new Array[Int](numTasks)
val successful = new Array[Boolean](numTasks)
private val numFailures = new Array[Int](numTasks)
- // key is taskId, value is a Map of executor id to when it failed
--- End diff --
The key here is taskIndex actually, comment is misleading!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r43796970
--- Diff: docs/configuration.md ---
@@ -1141,6 +1141,48 @@ Apart from these, the following properties are also available, and may be useful
</td>
</tr>
<tr>
+ <td><code>spark.scheduler.blacklist.enabled</code></td>
+ <td>true</td>
+ <td>
+ If set to ture, executor blacklist feature will be enabled to avoid allocte new task on bad executor. The logic to define bad executor is based on BlacklistStrategy which is also configurable.
+ </td>
+</tr>
+<tr>
+ <td><code>spark.scheduler.executorTaskBlacklistTime</code></td>
+ <td>0L</td>
+ <td>
+ The threshold to deside blacklist executor, if last failed time is older than current time subtract executorTaskBlacklistTime, then the executor could be removed from blacklist.
--- End diff --
sorry I'm going to backtrack against what I said earlier. Now I think it probably is better if you go with your original name "spark.scheduler.blacklist.timeout", so that all the options are "spark.scheduler.blacklist.xxx". We should just still support "spark.scheduler.executorTaskBlacklistTime", even if its undocumented".
Also, lets change the conf to include a timeunit like the others, so the default would be "0s".
and I'd like to suggest rewording the actual text to: "If executor blacklisting is enabled, this controls how long an executor remains in the blacklist before it is returned to the pool of available executors."
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-185212009
@mwws can you update the code?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42199960
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update
+ * executorIdToFailureStatus Map.
+ */
+private[spark] class BlacklistTracker(sparkConf: SparkConf) extends WithCache{
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val executorIdToFailureStatus: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val clock = new SystemClock()
+
+ private val enableBlacklistSpeculate = sparkConf.getBoolean(
+ "spark.scheduler.blacklist.speculate", false)
+
+ private val recoverPeriod = sparkConf.getLong(
+ "spark.scheduler.blacklist.recoverPeriod", 60L)
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList(executorIdToFailureStatus))
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, recoverPeriod, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ private def expireExecutorsInBlackList(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Unit = synchronized {
+ strategy.expireExecutorsInBlackList(executorIdToFailureStatus)
+
+ invalidAllCache()
+ }
+
+ def updateFailureExecutors(info: TaskInfo, reason: TaskEndReason) : Unit = synchronized {
+ reason match {
+ // If task succeeding, remove related record from executorIdToFailureStatus
+ case Success =>
+ removeFailureExecutorsForTaskId(info.executorId, info.taskId)
+
+ // If task failing, update latest failure time and failedTaskIds
+ case _ =>
+ val executorId = info.executorId
+ executorIdToFailureStatus.get(executorId) match {
+ case Some(failureStatus) =>
+ failureStatus.updatedTime = clock.getTimeMillis()
+ val failedTimes = failureStatus.numFailuresPerTask.getOrElse(info.taskId, 0) + 1
+ failureStatus.numFailuresPerTask.update(info.taskId, failedTimes)
+ case None =>
+ val failedTasks = mutable.HashMap(info.taskId -> 1)
+ val failureStatus = new FailureStatus(
+ clock.getTimeMillis(),
+ info.host,
+ failedTasks)
+ executorIdToFailureStatus.update(executorId, failureStatus)
+ }
+ invalidAllCache()
+ }
+ }
+
+ // remove the executorId from executorIdToFailureStatus
+ def removeFailureExecutors(executorId: String) : Unit = synchronized {
+ executorIdToFailureStatus.remove(executorId)
+ invalidAllCache()
+ }
+
+ // remove the failure record related to given taskId from executorIdToFailureStatus. If the
+ // number of records of given executorId becomes 0, remove the completed executorId.
+ def removeFailureExecutorsForTaskId(
+ executorId: String,
+ taskId: Long) : Unit = synchronized {
+ executorIdToFailureStatus.get(executorId).map(fs => {
+ fs.numFailuresPerTask.remove(taskId)
+ if(fs.numFailuresPerTask.isEmpty){
+ executorIdToFailureStatus.remove(executorId)
+ }
+ invalidAllCache()
+ })
+ }
+
+ def executorIsBlacklisted(
+ executorId: String,
+ sched: TaskSchedulerImpl,
+ taskId: Long) : Boolean = {
+
+ executorBlacklist(sched, taskId).contains(executorId)
+ }
+
+ // If the node is in blacklist, all executors allocated on that node will
+ // also be put into executor blacklist.
+ // By default it's turned off, user can enable it in sparkConf.
+ private def speculationFailedExecutor(
+ sched: TaskSchedulerImpl): Set[String] = {
+ if (!enableBlacklistSpeculate) {
+ Set.empty[String]
+ } else {
+ nodeBlacklist().flatMap(sched.getExecutorsAliveOnHost(_)
+ .getOrElse(Set.empty[String])).toSet
+ }
+ }
+
+ def executorBlacklist(
+ sched: TaskSchedulerImpl,
+ taskId: Long): Set[String] = synchronized {
+ if (!isBlacklistExecutorCacheValid) {
+ reEvaluateExecutorBlacklistAndUpdateCache(sched, taskId)
+ } else {
+ getExecutorBlacklistFromCache(taskId).getOrElse {
+ reEvaluateExecutorBlacklistAndUpdateCache(sched, taskId)
+ }
+ }
+ }
+
+ private def reEvaluateExecutorBlacklistAndUpdateCache(
+ sched: TaskSchedulerImpl,
+ taskId: Long): Set[String] = {
+ val executors = speculationFailedExecutor(sched) ++
+ strategy.getExecutorBlacklist(executorIdToFailureStatus, taskId)
+ updateBlacklistExecutorCache(taskId, executors)
+ executors
+ }
+
+ // The actual implementation is delegated to strategy
+ def nodeBlacklist(): Set[String] = synchronized {
+ if (isBlacklistNodeCacheValid) {
+ getNodeBlacklistFromCache
+ } else {
+ val nodes = strategy.getNodeBlacklist(executorIdToFailureStatus)
+ updateBlacklistNodeCache(nodes)
+ nodes
+ }
+ }
+}
+
+// Hide cache details in this trait to make code clean and avoid operation mistake
+trait WithCache {
--- End diff --
Sure, will do
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-167741274
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48401/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-170444236
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149074911
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-151675843
**[Test build #44460 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44460/consoleFull)** for PR 8760 at commit [`a2c8b40`](https://github.com/apache/spark/commit/a2c8b4021940606f17d122dc71ff444ca47c25a5).
* This patch **fails from timeout after a configured wait of \`250m\`**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `final class FailureStatus(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149564553
Jenkins, add to whitelist
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42330786
--- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import org.apache.spark.SparkFunSuite
+import org.scalatest.BeforeAndAfter
+import org.apache.spark.SparkConf
+import org.apache.spark.TaskEndReason
+import org.apache.spark.Success
+import org.apache.spark.ExceptionFailure
+import org.apache.spark.LocalSparkContext
+import org.apache.spark.SparkContext
+import org.mockito.Mockito.{when,spy}
+import org.apache.spark.util.ManualClock
+
+class BlacklistTrackerSuite extends SparkFunSuite with BeforeAndAfter with LocalSparkContext {
+
+ val FAILURE: TaskEndReason = new ExceptionFailure(
+ "Fake",
+ "fake failure",
+ Array.empty[StackTraceElement],
+ "fake stack trace",
+ None,
+ None)
+
+ // Variable name can indicate basic information of taskInfo
+ // The format is "taskInfo_executorId_taskId_hostName"
+ val taskInfo_1_1_hostA = new TaskInfo(1L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_1_2_hostA = new TaskInfo(2L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_1_hostA = new TaskInfo(1L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_2_hostA = new TaskInfo(2L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_3_3_hostB = new TaskInfo(3L,1,1,0L,"3","hostB", TaskLocality.ANY, false)
+
+ val clock = new ManualClock(0)
+
+ test ("expireExecutorsInBlacklist works") {
+ // expire time is set to 6s
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.executorTaskBlacklistTime", "6000")
+
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf, clock)
+ // Executor 1 into blacklist at Time 00:00:00
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1"))
+
+ clock.setTime(2000)
+ tracker.expireExecutorsInBlackList()
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1"))
+ // Executor 1 failed again at Time 00::00:02
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+
+ clock.setTime(3000)
+ // Executor 2 failed at Time 00:00:03
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1", "2"))
+
+ clock.setTime(6000)
+ tracker.expireExecutorsInBlackList()
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1", "2"))
+
+ clock.setTime(8000)
+ tracker.expireExecutorsInBlackList()
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("2"))
+
+ clock.setTime(10000)
+ tracker.expireExecutorsInBlackList()
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+ }
+
+ test("default strategy works") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf, clock)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1","2"))
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set())
+
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("2"))
+ assert(tracker.nodeBlacklist() === Set())
+
+ tracker.updateFailureExecutors(taskInfo_3_3_hostB, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("2"))
+
+ tracker.updateFailureExecutors(taskInfo_3_3_hostB, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 3L) === Set("3"))
+ assert(tracker.nodeBlacklist() === Set())
+
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+ }
+
+ test ("threshold strategy works") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.strategy", "threshold")
+ .set("spark.scheduler.blacklist.threshold.maxFailureTaskNumber", "1")
+ .set("spark.scheduler.blacklist.threshold.maxBlackExecutorNumber", "0")
+
+ sc = new SparkContext(conf)
+ val scheduler = spy(new TaskSchedulerImpl(sc, 1))
+ when(scheduler.getExecutorsAliveOnHost("hostA")).thenReturn(Some(Set("1","2")))
+
+ val tracker = new BlacklistTracker(conf, clock)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+ tracker.updateFailureExecutors(taskInfo_1_2_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set("1", "2"))
+ assert(tracker.nodeBlacklist() === Set("hostA"))
--- End diff --
It's a meaningful test, I will add it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149564504
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-170446186
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153278694
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153251575
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40285521
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -184,13 +188,21 @@ private[yarn] class YarnAllocator(
def requestTotalExecutorsWithPreferredLocalities(
requestedTotal: Int,
localityAwareTasks: Int,
- hostToLocalTaskCount: Map[String, Int]): Boolean = synchronized {
+ hostToLocalTaskCount: Map[String, Int],
+ nodeBlacklist: Set[String]): Boolean = synchronized {
this.numLocalityAwareTasks = localityAwareTasks
this.hostToLocalTaskCounts = hostToLocalTaskCount
if (requestedTotal != targetNumExecutors) {
logInfo(s"Driver requested a total number of $requestedTotal executor(s).")
targetNumExecutors = requestedTotal
+
+ //Update blacklist infomation to YARN ResouceManager for this application,
+ //in order to avoid allocating new Container on the problematic nodes.
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-167741137
**[Test build #48401 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48401/consoleFull)** for PR 8760 at commit [`d94bdd7`](https://github.com/apache/spark/commit/d94bdd7d16b77317bdfe5851e9ba297a229b4113).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40298903
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.util.SystemClock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+trait BlacklistStrategy {
+ val expireTimeInMillisecond: Long
+
+ // TaskId is optional here because the BlacklistStrategy could be unrelated to TaskId
--- End diff --
Rethink about it, for "speculate" mode, we can also call taskId unrelated ```getNodeBlacklist```, it should be fine. And I have simplified the API as you suggested.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40222362
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update failedExecutorMap.
+ */
+class BlacklistTracker(sparkConf: SparkConf) {
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val failedExecutorMap: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
--- End diff --
can you rename this to `executorIdToFailureStatus` , and similarly rename all method params from `failureExecutors` to the same thing?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40285628
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -302,7 +278,8 @@ private[spark] class TaskSetManager(
speculatableTasks.retain(index => !successful(index)) // Remove finished tasks from set
def canRunOnHost(index: Int): Boolean =
- !hasAttemptOnHost(index, host) && !executorIsBlacklisted(execId, index)
+ !hasAttemptOnHost(index, host) &&
+ !blacklistTracker.fold(false)(_.executorIsBlacklisted(execId, sched, Some(index)))
--- End diff --
FIxed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40226351
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.util.SystemClock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+trait BlacklistStrategy {
+ val expireTimeInMillisecond: Long
+
+ // TaskId is optional here because the BlacklistStrategy could be unrelated to TaskId
+ def getExecutorBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String]
+
+ def getNodeBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String]
+
+ // Default implementation to remove failure executors from HashMap based on given time period.
+ def expireExecutorsInBlackList(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]): Unit = {
+ val now = new SystemClock().getTimeMillis()
+ failedExecutorMap.retain((executorid, failureStatus) => {
+ (now - failureStatus.updatedTime) < expireTimeInMillisecond
+ })
--- End diff --
oops, ignore (a) -- sorry I had overlooked the `synchronized`s in `BlackListTracker`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by markhamstra <gi...@git.apache.org>.
Github user markhamstra commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-161729752
I still like the idea of using speculative tasks to parole blacklisted executors and hosts, but that new role for speculative execution will complicate https://github.com/apache/spark/pull/8683.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149313562
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153252101
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-152943965
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40087258
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update failedExecutorMap.
+ */
+class BlacklistTracker(sparkConf: SparkConf) {
--- End diff --
this should be at least `private[spark]`, maybe `private[scheduler]` if possible. Same goes for all the other classes added
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-171180163
**[Test build #49294 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49294/consoleFull)** for PR 8760 at commit [`25a7c6f`](https://github.com/apache/spark/commit/25a7c6fc329687310f1ae88e56bcd72a5c259b61).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40216019
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update failedExecutorMap.
+ */
+class BlacklistTracker(sparkConf: SparkConf) {
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val failedExecutorMap: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val clock = new SystemClock()
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList(failedExecutorMap))
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, 60, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ private def expireExecutorsInBlackList(
+ failureExecutors: mutable.HashMap[String, FailureStatus]): Unit = synchronized {
+ strategy.expireExecutorsInBlackList(failureExecutors)
+ }
+
+ def updateFailureExecutors(info: TaskInfo, reason: TaskEndReason) : Unit = synchronized {
+ reason match {
+ // If task succeeding, remove related record from failedExecutorMap
+ case Success =>
+ removeFailureExecutors(info.executorId, Some(info.taskId))
+
+ // If task failing, update latest failure time and failedTaskIds
+ case _ =>
+ val executorId = info.executorId
+ failedExecutorMap.get(executorId) match {
+ case Some(failureStatus) =>
+ failureStatus.updatedTime = clock.getTimeMillis()
+ val failedTimes = failureStatus.failedTasks.getOrElse(info.taskId, 0) + 1
+ failureStatus.failedTasks.update(info.taskId, failedTimes)
+ case None =>
+ val failedTasks = mutable.HashMap(info.taskId -> 1)
+ val failureStatus = new FailureStatus(
+ clock.getTimeMillis(),
+ info.host,
+ failedTasks)
+ failedExecutorMap.update(executorId, failureStatus)
+ }
+ }
+ }
+
+ def removeFailureExecutors(
+ executorId: String,
+ taskId: Option[Long]) : Unit = synchronized {
+ taskId match {
+ // If taskId is provided, remove the taskId from failedTasks
+ case Some(id) =>
+ failedExecutorMap.get(executorId).map(fs => {
+ fs.updatedTime = clock.getTimeMillis()
+ fs.failedTasks.remove(id)
+ if(fs.failedTasks.isEmpty){
+ failedExecutorMap.remove(executorId)
+ }
+ })
+ // if taskId is not provided, remove the whole record related to given executorId
+ case None =>
+ failedExecutorMap.remove(executorId)
+ }
+ }
+
+ def executorIsBlacklisted(
+ executorId: String,
+ sched: TaskSchedulerImpl,
+ taskId: Option[Long]) : Boolean = {
+
+ executorBlacklist(sched, taskId).contains(executorId)
+ }
+
+ // The actual implementation is delegated to strategy
+ def executorBlacklist(
+ sched: TaskSchedulerImpl,
+ taskId: Option[Long]): Set[String] = synchronized {
+
+ // If the node is in blacklist, all executors allocated on that node will
+ // also be put into executor blacklist.
+ // By default it's turned off, user can enable it in sparkConf.
+ val speculationFailedExecutor: Set[String] =
+ if (sparkConf.getBoolean("spark.scheduler.blacklist.speculate", false)) {
+ Set.empty[String]
+ } else {
+ nodeBlacklist(taskId).flatMap(sched.getExecutorsAliveOnHost(_)
+ .getOrElse(Set.empty[String])).toSet
+ }
--- End diff --
nit: fix indentation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40164406
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update failedExecutorMap.
+ */
+class BlacklistTracker(sparkConf: SparkConf) {
--- End diff --
It need to be accessed in SparkContext which is not in scheduler package. I will add private[spark]
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40281631
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.util.SystemClock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+trait BlacklistStrategy {
+ val expireTimeInMillisecond: Long
+
+ // TaskId is optional here because the BlacklistStrategy could be unrelated to TaskId
--- End diff --
My original thinking was becasue (1) nodeBlacklist could depend on the result of executorBlacklist (2) executorBlacklist might be taskId related ==> then nodeBlacklist could also be taskId related. Your suggestion extinguish the possibility to make nodeBlacklist taskId related. And In "speculate" mode, when calling ```getExecutorBlacklist``` related to some taskId, we would like to ```getNodeBlacklist``` related to the same taskId.
Maybe we just keep both method with/without taskId to make the API more extendable and provide additional layer of trait to simplify the implementation?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40280578
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update failedExecutorMap.
+ */
+class BlacklistTracker(sparkConf: SparkConf) {
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val failedExecutorMap: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val clock = new SystemClock()
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList(failedExecutorMap))
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, 60, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ private def expireExecutorsInBlackList(
+ failureExecutors: mutable.HashMap[String, FailureStatus]): Unit = synchronized {
+ strategy.expireExecutorsInBlackList(failureExecutors)
+ }
+
+ def updateFailureExecutors(info: TaskInfo, reason: TaskEndReason) : Unit = synchronized {
+ reason match {
+ // If task succeeding, remove related record from failedExecutorMap
+ case Success =>
+ removeFailureExecutors(info.executorId, Some(info.taskId))
+
+ // If task failing, update latest failure time and failedTaskIds
+ case _ =>
+ val executorId = info.executorId
+ failedExecutorMap.get(executorId) match {
+ case Some(failureStatus) =>
+ failureStatus.updatedTime = clock.getTimeMillis()
+ val failedTimes = failureStatus.failedTasks.getOrElse(info.taskId, 0) + 1
+ failureStatus.failedTasks.update(info.taskId, failedTimes)
+ case None =>
+ val failedTasks = mutable.HashMap(info.taskId -> 1)
+ val failureStatus = new FailureStatus(
+ clock.getTimeMillis(),
+ info.host,
+ failedTasks)
+ failedExecutorMap.update(executorId, failureStatus)
+ }
+ }
+ }
+
+ def removeFailureExecutors(
+ executorId: String,
+ taskId: Option[Long]) : Unit = synchronized {
+ taskId match {
+ // If taskId is provided, remove the taskId from failedTasks
+ case Some(id) =>
+ failedExecutorMap.get(executorId).map(fs => {
+ fs.updatedTime = clock.getTimeMillis()
+ fs.failedTasks.remove(id)
+ if(fs.failedTasks.isEmpty){
+ failedExecutorMap.remove(executorId)
+ }
+ })
+ // if taskId is not provided, remove the whole record related to given executorId
+ case None =>
+ failedExecutorMap.remove(executorId)
+ }
+ }
+
+ def executorIsBlacklisted(
+ executorId: String,
+ sched: TaskSchedulerImpl,
+ taskId: Option[Long]) : Boolean = {
+
+ executorBlacklist(sched, taskId).contains(executorId)
+ }
+
+ // The actual implementation is delegated to strategy
+ def executorBlacklist(
+ sched: TaskSchedulerImpl,
+ taskId: Option[Long]): Set[String] = synchronized {
+
+ // If the node is in blacklist, all executors allocated on that node will
+ // also be put into executor blacklist.
+ // By default it's turned off, user can enable it in sparkConf.
+ val speculationFailedExecutor: Set[String] =
+ if (sparkConf.getBoolean("spark.scheduler.blacklist.speculate", false)) {
+ Set.empty[String]
+ } else {
+ nodeBlacklist(taskId).flatMap(sched.getExecutorsAliveOnHost(_)
+ .getOrElse(Set.empty[String])).toSet
+ }
--- End diff --
It's hard to delegate *speculate* part into strategy, because ```getNodeBlacklist``` can depend on (call) ```getExecutorBlacklist```, and it would be a circular reference if "getExecutorBlacklist" need the result of getNodeBlacklist. I would prefer to still keeping it here to make the "speculate" functionality independent with strategy. And in original semantics, there is no such functionality, so I make it configurable and setting to ```false``` by default.
For the sparkConf call, I plan to move it to outer layer and evaluate earlier to reduce the extra cost of this frequently used method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-142872717
Hi @squito , I do appreciate your detail review and comment, it helped a lot to improve the code quality. I have submit the latest change based on your comment and will add some unit test later.
About your comment about *immediately killing blacklisted executors*, to have a clear segregation of duty, I don't think BlacklistTracker should handled it. In my point of view, BlacklistTracker is duty to decide which executors and nodes are problematic and provide such information to TaskScheduler and ExecutorAllocator, ect. And the ExecutorAllocator(Yarn, Mesos, Standalnoe has different allocator) should actually kill or create executors, and the strategy about whether kill executor or not might be better to decide in allocator instead. Your suggestation could be doable with dynamic allocation which adjust the number of executors periodically, but I think it's another thing apart from what we focus here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149313997
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153271115
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44892/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153278773
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42044558
--- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -0,0 +1,162 @@
+package org.apache.spark.scheduler
+
+import org.apache.spark.SparkFunSuite
+import org.scalatest.BeforeAndAfter
+import org.apache.spark.SparkConf
+import org.apache.spark.TaskEndReason
+import org.apache.spark.Success
+import org.apache.spark.ExceptionFailure
+import org.apache.spark.LocalSparkContext
+import org.apache.spark.SparkContext
+
+import org.mockito.Mockito.{when,spy}
+
+class BlacklistTrackerSuite extends SparkFunSuite with BeforeAndAfter with LocalSparkContext {
+
+ val FAILURE: TaskEndReason = new ExceptionFailure(
+ "Fake",
+ "fake failure",
+ Array.empty[StackTraceElement],
+ "fake stack trace",
+ None,
+ None)
+
+ // Variable name can indicate basic information of taskInfo
+ // The format is "taskInfo_executorId_taskId_hostName"
+ val taskInfo_1_1_hostA = new TaskInfo(1L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_1_2_hostA = new TaskInfo(2L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_1_hostA = new TaskInfo(1L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_2_hostA = new TaskInfo(2L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_3_3_hostB = new TaskInfo(3L,1,1,0L,"3","hostB", TaskLocality.ANY, false)
+
+ test ("expireExecutorsInBlacklist works ") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.recoverPeriod", "2")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.start()
+ Thread.sleep(100)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1"))
+
+ // sleep 2 second to wait for expireExecutorsInBlackList be triggered
+ Thread.sleep(2000)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+
+ tracker.stop()
+ }
+
+ test("default strategy works as exceptation") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1","2"))
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set())
+
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("2"))
+ assert(tracker.nodeBlacklist() === Set())
+
+ tracker.updateFailureExecutors(taskInfo_3_3_hostB, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 3L) === Set("3"))
+ assert(tracker.nodeBlacklist() === Set())
+
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+ }
+
+ test ("strict strategy works as exception") {
--- End diff --
same here, just "strict strategy works"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40235676
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update failedExecutorMap.
+ */
+class BlacklistTracker(sparkConf: SparkConf) {
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val failedExecutorMap: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val clock = new SystemClock()
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList(failedExecutorMap))
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, 60, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ private def expireExecutorsInBlackList(
+ failureExecutors: mutable.HashMap[String, FailureStatus]): Unit = synchronized {
+ strategy.expireExecutorsInBlackList(failureExecutors)
+ }
+
+ def updateFailureExecutors(info: TaskInfo, reason: TaskEndReason) : Unit = synchronized {
+ reason match {
+ // If task succeeding, remove related record from failedExecutorMap
+ case Success =>
+ removeFailureExecutors(info.executorId, Some(info.taskId))
+
+ // If task failing, update latest failure time and failedTaskIds
+ case _ =>
+ val executorId = info.executorId
+ failedExecutorMap.get(executorId) match {
+ case Some(failureStatus) =>
+ failureStatus.updatedTime = clock.getTimeMillis()
+ val failedTimes = failureStatus.failedTasks.getOrElse(info.taskId, 0) + 1
+ failureStatus.failedTasks.update(info.taskId, failedTimes)
+ case None =>
+ val failedTasks = mutable.HashMap(info.taskId -> 1)
+ val failureStatus = new FailureStatus(
+ clock.getTimeMillis(),
+ info.host,
+ failedTasks)
+ failedExecutorMap.update(executorId, failureStatus)
+ }
+ }
+ }
+
+ def removeFailureExecutors(
+ executorId: String,
+ taskId: Option[Long]) : Unit = synchronized {
+ taskId match {
+ // If taskId is provided, remove the taskId from failedTasks
+ case Some(id) =>
+ failedExecutorMap.get(executorId).map(fs => {
+ fs.updatedTime = clock.getTimeMillis()
+ fs.failedTasks.remove(id)
+ if(fs.failedTasks.isEmpty){
+ failedExecutorMap.remove(executorId)
+ }
+ })
+ // if taskId is not provided, remove the whole record related to given executorId
+ case None =>
+ failedExecutorMap.remove(executorId)
+ }
+ }
+
+ def executorIsBlacklisted(
+ executorId: String,
+ sched: TaskSchedulerImpl,
+ taskId: Option[Long]) : Boolean = {
+
+ executorBlacklist(sched, taskId).contains(executorId)
+ }
+
+ // The actual implementation is delegated to strategy
+ def executorBlacklist(
+ sched: TaskSchedulerImpl,
+ taskId: Option[Long]): Set[String] = synchronized {
+
+ // If the node is in blacklist, all executors allocated on that node will
+ // also be put into executor blacklist.
+ // By default it's turned off, user can enable it in sparkConf.
+ val speculationFailedExecutor: Set[String] =
+ if (sparkConf.getBoolean("spark.scheduler.blacklist.speculate", false)) {
+ Set.empty[String]
+ } else {
+ nodeBlacklist(taskId).flatMap(sched.getExecutorsAliveOnHost(_)
+ .getOrElse(Set.empty[String])).toSet
+ }
--- End diff --
sorry one more thought: does it actually make more sense to more completely de-tangle these methods into different purposes? Maybe `getExecutorBlacklist` is the only thing called when actually scheduling tasks, and separately `getNodeBlacklist` is only used when talking to the cluster manager?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153264189
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40089514
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.util.SystemClock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+trait BlacklistStrategy {
+ val expireTimeInMillisecond: Long
+
+ // TaskId is optional here because the BlacklistStrategy could be unrelated to TaskId
+ def getExecutorBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String]
+
+ def getNodeBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String]
+
+ // Default implementation to remove failure executors from HashMap based on given time period.
+ def expireExecutorsInBlackList(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]): Unit = {
+ val now = new SystemClock().getTimeMillis()
+ failedExecutorMap.retain((executorid, failureStatus) => {
+ (now - failureStatus.updatedTime) < expireTimeInMillisecond
+ })
+ }
+}
+
+/**
+ * The interface for taskId unrelated strategy. In this case, provided taskId will be ignored. The
+ * benefit for taskId unrelated strategy is different taskSets can learn experience from other
+ * taskSet to avoid allocating tasks to problematic executors.
+ */
+trait TaskUnrelatedBlacklistStrategy extends BlacklistStrategy with Logging{
+ def getExecutorBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String] = {
+ if (taskId.isDefined){
+ logWarning("TaskId unrelated strategy is applied, given taskId will be ignored")
+ }
+ getExecutorBlacklist(failedExecutorMap)
+ }
+
+ def getNodeBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String] = {
+ if (taskId.isDefined){
+ logWarning("TaskId unrelated strategy is applied, given taskId will be ignored")
+ }
+ getNodeBlacklist(failedExecutorMap)
+ }
+
+ // define new interface which remove taskId from parameter.
+ def getExecutorBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]): Set[String]
+ def getNodeBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]): Set[String]
+}
+
+/**
+ * This strategy is simply based on given threshold and is taskId unrelated. An executor will be
+ * in blacklist, if it failed more than "maxFailureTaskNumber" times. A node will be in blacklist,
+ * if there are more than "maxBlackExecutorNumber" executors on it in executor blacklist.
+ */
+class SimpleStrategy(
+ maxFailureTaskNumber: Int,
+ maxBlackExecutorNumber: Int,
+ val expireTimeInMillisecond: Long
+ )extends TaskUnrelatedBlacklistStrategy {
+
+ private def getSelectedExecutorMap(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]) = {
+ failedExecutorMap.filter{
+ case (id, failureStatus) => failureStatus.failureTimes > maxFailureTaskNumber
+ }
+ }
+
+ def getExecutorBlacklist(
+ failureExecutors: mutable.HashMap[String, FailureStatus]): Set[String] = {
+ getSelectedExecutorMap(failureExecutors).keys.toSet
+ }
+
+ def getNodeBlacklist(
+ failureExecutors: mutable.HashMap[String, FailureStatus]): Set[String] = {
--- End diff --
these functions get called a *lot* (at least once per scheduled task). They will almost always return the same answer in successive calls, but they are doing non-trivial work in each call (several operations over large collections, eg. filtering, summing values, etc.). We should probably cache those values and just update when there are more failures (probably need to be careful on thread-safety).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42214583
--- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -0,0 +1,162 @@
+package org.apache.spark.scheduler
+
+import org.apache.spark.SparkFunSuite
+import org.scalatest.BeforeAndAfter
+import org.apache.spark.SparkConf
+import org.apache.spark.TaskEndReason
+import org.apache.spark.Success
+import org.apache.spark.ExceptionFailure
+import org.apache.spark.LocalSparkContext
+import org.apache.spark.SparkContext
+
+import org.mockito.Mockito.{when,spy}
+
+class BlacklistTrackerSuite extends SparkFunSuite with BeforeAndAfter with LocalSparkContext {
+
+ val FAILURE: TaskEndReason = new ExceptionFailure(
+ "Fake",
+ "fake failure",
+ Array.empty[StackTraceElement],
+ "fake stack trace",
+ None,
+ None)
+
+ // Variable name can indicate basic information of taskInfo
+ // The format is "taskInfo_executorId_taskId_hostName"
+ val taskInfo_1_1_hostA = new TaskInfo(1L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_1_2_hostA = new TaskInfo(2L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_1_hostA = new TaskInfo(1L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_2_hostA = new TaskInfo(2L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_3_3_hostB = new TaskInfo(3L,1,1,0L,"3","hostB", TaskLocality.ANY, false)
+
+ test ("expireExecutorsInBlacklist works ") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.recoverPeriod", "2")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.start()
+ Thread.sleep(100)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1"))
+
+ // sleep 2 second to wait for expireExecutorsInBlackList be triggered
+ Thread.sleep(2000)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+
+ tracker.stop()
+ }
+
+ test("default strategy works as exceptation") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1","2"))
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set())
+
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("2"))
+ assert(tracker.nodeBlacklist() === Set())
+
+ tracker.updateFailureExecutors(taskInfo_3_3_hostB, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 3L) === Set("3"))
+ assert(tracker.nodeBlacklist() === Set())
+
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+ }
+
+ test ("strict strategy works as exception") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.strategy", "strict")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1"))
+ tracker.updateFailureExecutors(taskInfo_1_2_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set("1"))
--- End diff --
As the strict strategy/threshold strategy is taskID unrelated, executor 1 is in the blacklist for all tasks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153262860
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-154328641
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-151615731
I have no idea what is going on here. All the tests you touched complete very quickly for me locally. Maybe this is just really bad luck, I'll give it another spin ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-148660224
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43838/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42199540
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.util.SystemClock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+trait BlacklistStrategy {
+ val expireTimeInMillisecond: Long
+
+ // Return executors in blacklist which are related to given TaskId
+ def getExecutorBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ taskId: Long): Set[String]
+
+ // Return all nodes in blacklist
+ def getNodeBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Set[String]
+
+ // Default implementation to remove failure executors from HashMap based on given time period.
+ def expireExecutorsInBlackList(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Unit = {
+ val now = new SystemClock().getTimeMillis()
+ executorIdToFailureStatus.retain((executorid, failureStatus) => {
+ (now - failureStatus.updatedTime) < expireTimeInMillisecond
+ })
+ }
+}
+
+/**
+ * This strategy is simply based on given threshold and is taskId unrelated. An executor will be
+ * in blacklist, if it failed more than "maxFailureTaskNumber" times. A node will be in blacklist,
+ * if there are more than "maxBlackExecutorNumber" executors on it in executor blacklist.
+ *
+ * In this case, provided taskId will be ignored. The benefit for taskId unrelated strategy is that
+ * different taskSets can learn experience from other taskSet to avoid allocating tasks on
+ * problematic executors.
+ */
+class SimpleStrategy(
+ maxFailureTaskNumber: Int,
+ maxBlackExecutorNumber: Int,
+ val expireTimeInMillisecond: Long
+ )extends BlacklistStrategy {
+
+ private def getSelectedExecutorMap(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]) = {
+ executorIdToFailureStatus.filter{
+ case (id, failureStatus) => failureStatus.totalNumFailures > maxFailureTaskNumber
+ }
+ }
+
+ // As this is a taskId unrelated strategy, the input taskId will be ignored
+ def getExecutorBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ taskId: Long): Set[String] = {
+ getSelectedExecutorMap(executorIdToFailureStatus).keys.toSet
+ }
+
+ def getNodeBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Set[String] = {
+ getSelectedExecutorMap(executorIdToFailureStatus)
+ .groupBy{case (id, failureStatus) => failureStatus.host}
+ .filter {case (host, executorIdToFailureStatus) =>
+ executorIdToFailureStatus.size > maxBlackExecutorNumber}
+ .keys.toSet
+ }
+}
+
+/**
+ * This strategy is applied as default to keep the same semantics as original. It's an taskId
+ * related strategy. If an executor failed running "task A", then we think this executor is
+ * blacked for "task A". And we think the executor is still healthy for other task. node blacklist
+ * is always empty.
+ *
+ * It was the standard behavior before spark 1.6
+ */
+class DefaultStrategy(val expireTimeInMillisecond: Long) extends BlacklistStrategy {
+ def getExecutorBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ taskId: Long): Set[String] = {
+ executorIdToFailureStatus.filter{
+ case (_, failureStatus) => failureStatus.numFailuresPerTask.keySet.contains(taskId)
+ }.keys.toSet
+ }
+
+ def getNodeBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Set[String] =
+ Set.empty[String]
+}
+
+/**
+ * Create BlacklistStrategy instance according to SparkConf
+ */
+object BlacklistStrategy {
+ def apply(sparkConf: SparkConf): BlacklistStrategy = {
+ val timeout = sparkConf.getLong("spark.scheduler.executorTaskBlacklistTime", 0L)
+ sparkConf.get("spark.scheduler.blacklist.strategy", "default") match {
+ case "default" =>
+ new DefaultStrategy(timeout)
+ case "threshold" =>
+ new SimpleStrategy(
+ sparkConf.getInt("spark.scheduler.blacklist.threshold.maxFailureTaskNumber", 3),
+ sparkConf.getInt("spark.scheduler.blacklist.threshold.maxBlackExecutorNumber", 3),
+ timeout)
+ case "strict" =>
+ // A special case of SimpleStrategy: Once task failed at executor,
+ // put the executor and its node into blacklist.
+ new SimpleStrategy(0, 0, timeout)
--- End diff --
No strong reason to keep "strict" strategy, I will delete it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-148660221
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42208578
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update
+ * executorIdToFailureStatus Map.
+ */
+private[spark] class BlacklistTracker(sparkConf: SparkConf) extends WithCache{
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val executorIdToFailureStatus: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val clock = new SystemClock()
+
+ private val enableBlacklistSpeculate = sparkConf.getBoolean(
+ "spark.scheduler.blacklist.speculate", false)
+
+ private val recoverPeriod = sparkConf.getLong(
+ "spark.scheduler.blacklist.recoverPeriod", 60L)
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList(executorIdToFailureStatus))
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, recoverPeriod, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ private def expireExecutorsInBlackList(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Unit = synchronized {
+ strategy.expireExecutorsInBlackList(executorIdToFailureStatus)
+
+ invalidAllCache()
--- End diff --
good point, I will change the it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149563407
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149640677
**[Test build #43977 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43977/consoleFull)** for PR 8760 at commit [`a2c8b40`](https://github.com/apache/spark/commit/a2c8b4021940606f17d122dc71ff444ca47c25a5).
* This patch **fails from timeout after a configured wait of \`250m\`**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `final class FailureStatus(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r46248217
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+import org.apache.spark.util.Clock
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update
+ * executorIdToFailureStatus Map.
+ */
+private[spark] class BlacklistTracker(
+ sparkConf: SparkConf,
+ clock: Clock = new SystemClock()) extends BlacklistCache{
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val executorIdToFailureStatus: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val recoverPeriod = sparkConf.getTimeAsSeconds(
+ "spark.scheduler.blacklist.recoverPeriod", "60s")
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList())
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, recoverPeriod, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ /** VisibleForTesting */
+ private[scheduler] def expireExecutorsInBlackList(): Unit = synchronized {
+ val updated = strategy.expireExecutorsInBlackList(executorIdToFailureStatus, clock)
+ if (updated) {
+ invalidateCache()
+ }
+ }
+
+ def updateFailedExecutors(
+ stageId: Int,
+ partation: Int,
--- End diff --
sp
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40298959
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update failedExecutorMap.
+ */
+class BlacklistTracker(sparkConf: SparkConf) {
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val failedExecutorMap: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val clock = new SystemClock()
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList(failedExecutorMap))
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, 60, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ private def expireExecutorsInBlackList(
+ failureExecutors: mutable.HashMap[String, FailureStatus]): Unit = synchronized {
+ strategy.expireExecutorsInBlackList(failureExecutors)
+ }
+
+ def updateFailureExecutors(info: TaskInfo, reason: TaskEndReason) : Unit = synchronized {
+ reason match {
+ // If task succeeding, remove related record from failedExecutorMap
+ case Success =>
+ removeFailureExecutors(info.executorId, Some(info.taskId))
+
+ // If task failing, update latest failure time and failedTaskIds
+ case _ =>
+ val executorId = info.executorId
+ failedExecutorMap.get(executorId) match {
+ case Some(failureStatus) =>
+ failureStatus.updatedTime = clock.getTimeMillis()
+ val failedTimes = failureStatus.failedTasks.getOrElse(info.taskId, 0) + 1
+ failureStatus.failedTasks.update(info.taskId, failedTimes)
+ case None =>
+ val failedTasks = mutable.HashMap(info.taskId -> 1)
+ val failureStatus = new FailureStatus(
+ clock.getTimeMillis(),
+ info.host,
+ failedTasks)
+ failedExecutorMap.update(executorId, failureStatus)
+ }
+ }
+ }
+
+ def removeFailureExecutors(
+ executorId: String,
+ taskId: Option[Long]) : Unit = synchronized {
+ taskId match {
+ // If taskId is provided, remove the taskId from failedTasks
+ case Some(id) =>
+ failedExecutorMap.get(executorId).map(fs => {
+ fs.updatedTime = clock.getTimeMillis()
+ fs.failedTasks.remove(id)
+ if(fs.failedTasks.isEmpty){
+ failedExecutorMap.remove(executorId)
+ }
+ })
+ // if taskId is not provided, remove the whole record related to given executorId
+ case None =>
+ failedExecutorMap.remove(executorId)
+ }
+ }
+
+ def executorIsBlacklisted(
+ executorId: String,
+ sched: TaskSchedulerImpl,
+ taskId: Option[Long]) : Boolean = {
+
+ executorBlacklist(sched, taskId).contains(executorId)
+ }
+
+ // The actual implementation is delegated to strategy
+ def executorBlacklist(
+ sched: TaskSchedulerImpl,
+ taskId: Option[Long]): Set[String] = synchronized {
+
+ // If the node is in blacklist, all executors allocated on that node will
+ // also be put into executor blacklist.
+ // By default it's turned off, user can enable it in sparkConf.
+ val speculationFailedExecutor: Set[String] =
+ if (sparkConf.getBoolean("spark.scheduler.blacklist.speculate", false)) {
+ Set.empty[String]
+ } else {
+ nodeBlacklist(taskId).flatMap(sched.getExecutorsAliveOnHost(_)
+ .getOrElse(Set.empty[String])).toSet
+ }
+
+ speculationFailedExecutor ++ strategy.getExecutorBlacklist(failedExecutorMap, taskId)
+ }
+
+ // The actual implementation is delegated to strategy
+ def nodeBlacklist(taskId: Option[Long] = None): Set[String] = synchronized {
+ strategy.getNodeBlacklist(failedExecutorMap, taskId)
+ }
+}
+
+/**
+ * A class to record details of failure.
+ *
+ * @param initialTime the time when failure status be created
+ * @param host the node name which running executor on
+ * @param failedTasks all tasks failed on the executor (key is task id,
+ * value is failure time of this task)
+ */
+final class FailureStatus(
+ initialTime: Long,
+ val host: String,
+ val failedTasks: mutable.HashMap[Long, Int]) {
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153000900
about the new failure test in TaskSetManagerSuite, I take a deep look and has known how it failed. But I am surprised that the the original code could pass. In the failed test, task 0 failed on executor 1 and task 1 failed on executor 1.1, as I understand task 1 should still be able to allocate on exectuor 1, but in the test, it assert not. It must be something wrong, I will debug more later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-150693567
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-165304077
@squito Unless an executor or node is dying/has resource issues, in most cases, the failure is not repeatable across tasksets due to differing execution/resource characterstics. So blanket blacklist can be extremely suboptimal.
For example: taskset requiring disk usage (shuffle) vs taskset for executing collect() - the latter is not impacted by disk space being full/being reclaimed in background while former will keep failing.
An additional state might most probably handle the requirement - will be able to comment more when I actually see the design ofcourse :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-142299126
We should be sure to check the effect this has on scheduler throughput with some performance test. (I don't think there are any insurmountable issues, but its worth verifying.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153279192
**[Test build #44902 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44902/consoleFull)** for PR 8760 at commit [`b6b25e1`](https://github.com/apache/spark/commit/b6b25e1b035599e3568b331c70900f44d6976b92).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-154446468
Hi @mwws thanks for the updating. Did you see my comment about taskId vs taskIndex? That is really important and needs to be fixed and tested. Sorry I only just caught that issue.
If I can't come up with something better for the docs, then sure, we can leave that for follow up since the minimal docs are in place.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #8760: [Spark-8426] [scheduler] enhance blacklist mechani...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/8760
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40216306
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update failedExecutorMap.
+ */
+class BlacklistTracker(sparkConf: SparkConf) {
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val failedExecutorMap: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val clock = new SystemClock()
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList(failedExecutorMap))
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, 60, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ private def expireExecutorsInBlackList(
+ failureExecutors: mutable.HashMap[String, FailureStatus]): Unit = synchronized {
+ strategy.expireExecutorsInBlackList(failureExecutors)
+ }
+
+ def updateFailureExecutors(info: TaskInfo, reason: TaskEndReason) : Unit = synchronized {
+ reason match {
+ // If task succeeding, remove related record from failedExecutorMap
+ case Success =>
+ removeFailureExecutors(info.executorId, Some(info.taskId))
+
+ // If task failing, update latest failure time and failedTaskIds
+ case _ =>
+ val executorId = info.executorId
+ failedExecutorMap.get(executorId) match {
+ case Some(failureStatus) =>
+ failureStatus.updatedTime = clock.getTimeMillis()
+ val failedTimes = failureStatus.failedTasks.getOrElse(info.taskId, 0) + 1
+ failureStatus.failedTasks.update(info.taskId, failedTimes)
+ case None =>
+ val failedTasks = mutable.HashMap(info.taskId -> 1)
+ val failureStatus = new FailureStatus(
+ clock.getTimeMillis(),
+ info.host,
+ failedTasks)
+ failedExecutorMap.update(executorId, failureStatus)
+ }
+ }
+ }
+
+ def removeFailureExecutors(
+ executorId: String,
+ taskId: Option[Long]) : Unit = synchronized {
+ taskId match {
+ // If taskId is provided, remove the taskId from failedTasks
+ case Some(id) =>
+ failedExecutorMap.get(executorId).map(fs => {
+ fs.updatedTime = clock.getTimeMillis()
+ fs.failedTasks.remove(id)
+ if(fs.failedTasks.isEmpty){
+ failedExecutorMap.remove(executorId)
+ }
+ })
+ // if taskId is not provided, remove the whole record related to given executorId
+ case None =>
+ failedExecutorMap.remove(executorId)
+ }
+ }
+
+ def executorIsBlacklisted(
+ executorId: String,
+ sched: TaskSchedulerImpl,
+ taskId: Option[Long]) : Boolean = {
+
+ executorBlacklist(sched, taskId).contains(executorId)
+ }
+
+ // The actual implementation is delegated to strategy
+ def executorBlacklist(
+ sched: TaskSchedulerImpl,
+ taskId: Option[Long]): Set[String] = synchronized {
+
+ // If the node is in blacklist, all executors allocated on that node will
+ // also be put into executor blacklist.
+ // By default it's turned off, user can enable it in sparkConf.
+ val speculationFailedExecutor: Set[String] =
+ if (sparkConf.getBoolean("spark.scheduler.blacklist.speculate", false)) {
+ Set.empty[String]
+ } else {
+ nodeBlacklist(taskId).flatMap(sched.getExecutorsAliveOnHost(_)
+ .getOrElse(Set.empty[String])).toSet
+ }
+
+ speculationFailedExecutor ++ strategy.getExecutorBlacklist(failedExecutorMap, taskId)
+ }
+
+ // The actual implementation is delegated to strategy
+ def nodeBlacklist(taskId: Option[Long] = None): Set[String] = synchronized {
+ strategy.getNodeBlacklist(failedExecutorMap, taskId)
+ }
+}
+
+/**
+ * A class to record details of failure.
+ *
+ * @param initialTime the time when failure status be created
+ * @param host the node name which running executor on
+ * @param failedTasks all tasks failed on the executor (key is task id,
+ * value is failure time of this task)
+ */
+final class FailureStatus(
+ initialTime: Long,
+ val host: String,
+ val failedTasks: mutable.HashMap[Long, Int]) {
--- End diff --
thanks. Also I forgot to mention this -- but can you also update the doc correspondingly? I think I was especially confused by "value is failure time of this task", can you change to "value is the number of failures of this task"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r52685053
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+import org.apache.spark.util.Clock
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update
+ * executorIdToFailureStatus Map.
+ */
+private[spark] class BlacklistTracker(
+ sparkConf: SparkConf,
+ clock: Clock = new SystemClock()) extends BlacklistCache{
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val executorIdToFailureStatus: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val recoverPeriod = sparkConf.getTimeAsSeconds(
+ "spark.scheduler.blacklist.recoverPeriod", "60s")
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList())
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, recoverPeriod, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ /** VisibleForTesting */
+ private[scheduler] def expireExecutorsInBlackList(): Unit = synchronized {
+ val updated = strategy.expireExecutorsInBlackList(executorIdToFailureStatus, clock)
+ if (updated) {
+ invalidateCache()
+ }
+ }
--- End diff --
this is a good point. effort was made to avoid doing too much work with this lock, by caching the set of blacklisted nodes and executors. But maybe we can do a bit better.
The only reason we need to synchronize is b/c of the background thread that expires executors from the blacklist -- this is just called from a `TaskSetManger`, which in turn can only get called from threads with a lock on the `TaskScheduler`. So if instead of updating the cache in the background thread, we just have each of the methods check themselves if the blacklist needs to be updated, I think we could completely eliminate the need for the lock.
You'd still occasionally be pausing scheduling to run updateFailedExecutors, but even with 1000s of executors this seems pretty minor, and it is not running very often (60s by default). We could avoid the overhead of synchronization for scheduling every task, however.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40219031
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.util.SystemClock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+trait BlacklistStrategy {
+ val expireTimeInMillisecond: Long
+
+ // TaskId is optional here because the BlacklistStrategy could be unrelated to TaskId
+ def getExecutorBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String]
+
+ def getNodeBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String]
+
+ // Default implementation to remove failure executors from HashMap based on given time period.
+ def expireExecutorsInBlackList(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]): Unit = {
+ val now = new SystemClock().getTimeMillis()
+ failedExecutorMap.retain((executorid, failureStatus) => {
+ (now - failureStatus.updatedTime) < expireTimeInMillisecond
+ })
+ }
+}
+
+/**
+ * The interface for taskId unrelated strategy. In this case, provided taskId will be ignored. The
+ * benefit for taskId unrelated strategy is different taskSets can learn experience from other
+ * taskSet to avoid allocating tasks to problematic executors.
+ */
+trait TaskUnrelatedBlacklistStrategy extends BlacklistStrategy with Logging{
--- End diff --
given my comments above on the interface, I think this trait is not necessary. Its easy enough to have the strategy's just ignore `taskId` in `getExecutorBlacklist` w/out adding this extra level.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42260833
--- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import org.apache.spark.SparkFunSuite
+import org.scalatest.BeforeAndAfter
+import org.apache.spark.SparkConf
+import org.apache.spark.TaskEndReason
+import org.apache.spark.Success
+import org.apache.spark.ExceptionFailure
+import org.apache.spark.LocalSparkContext
+import org.apache.spark.SparkContext
+import org.mockito.Mockito.{when,spy}
+import org.apache.spark.util.ManualClock
+
+class BlacklistTrackerSuite extends SparkFunSuite with BeforeAndAfter with LocalSparkContext {
+
+ val FAILURE: TaskEndReason = new ExceptionFailure(
+ "Fake",
+ "fake failure",
+ Array.empty[StackTraceElement],
+ "fake stack trace",
+ None,
+ None)
+
+ // Variable name can indicate basic information of taskInfo
+ // The format is "taskInfo_executorId_taskId_hostName"
+ val taskInfo_1_1_hostA = new TaskInfo(1L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_1_2_hostA = new TaskInfo(2L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_1_hostA = new TaskInfo(1L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_2_hostA = new TaskInfo(2L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_3_3_hostB = new TaskInfo(3L,1,1,0L,"3","hostB", TaskLocality.ANY, false)
--- End diff --
nit: space after every comma. style checker is complaining about that now
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42217298
--- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -0,0 +1,162 @@
+package org.apache.spark.scheduler
+
+import org.apache.spark.SparkFunSuite
+import org.scalatest.BeforeAndAfter
+import org.apache.spark.SparkConf
+import org.apache.spark.TaskEndReason
+import org.apache.spark.Success
+import org.apache.spark.ExceptionFailure
+import org.apache.spark.LocalSparkContext
+import org.apache.spark.SparkContext
+
+import org.mockito.Mockito.{when,spy}
+
+class BlacklistTrackerSuite extends SparkFunSuite with BeforeAndAfter with LocalSparkContext {
+
+ val FAILURE: TaskEndReason = new ExceptionFailure(
+ "Fake",
+ "fake failure",
+ Array.empty[StackTraceElement],
+ "fake stack trace",
+ None,
+ None)
+
+ // Variable name can indicate basic information of taskInfo
+ // The format is "taskInfo_executorId_taskId_hostName"
+ val taskInfo_1_1_hostA = new TaskInfo(1L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_1_2_hostA = new TaskInfo(2L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_1_hostA = new TaskInfo(1L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_2_hostA = new TaskInfo(2L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_3_3_hostB = new TaskInfo(3L,1,1,0L,"3","hostB", TaskLocality.ANY, false)
+
+ test ("expireExecutorsInBlacklist works ") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.recoverPeriod", "2")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.start()
+ Thread.sleep(100)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1"))
+
+ // sleep 2 second to wait for expireExecutorsInBlackList be triggered
+ Thread.sleep(2000)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+
--- End diff --
sure, I will add it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-187547524
**[Test build #51737 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51737/consoleFull)** for PR 8760 at commit [`ecad5ff`](https://github.com/apache/spark/commit/ecad5ffe2bcc02f9b181a209172f22294e6af4ef).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149099247
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-148208305
[Test build #43743 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43743/console) for PR 8760 at commit [`46ee0ff`](https://github.com/apache/spark/commit/46ee0ff4cb20e2ae18436cfacb5a4847e53d2a91).
* This patch **fails RAT tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait BlacklistStrategy `
* `class SimpleStrategy(`
* `class DefaultStrategy(val expireTimeInMillisecond: Long) extends BlacklistStrategy `
* `trait WithCache `
* `final class FailureStatus(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r52682469
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -21,6 +21,7 @@ import java.util.Collections
import java.util.concurrent._
import java.util.regex.Pattern
+import scala.collection.immutable.Set
--- End diff --
this is very subjective, but I find it really confusing to have fully imported both immutable and mutable collections in the same file. (a) If you really want the immutable one, I'd just fully qualify it where you need to `collection.immutable.Set`. Or maybe (b) you could just use a mutable Set here for consistency? Or (c) if the code here doesn't care whether its mutable or immutable, you could take a `collection.Set`, and just fully qualify it where you use it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-170442019
@squito @kayousterhout @mridulm do you have any further comments on this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42059416
--- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -0,0 +1,162 @@
+package org.apache.spark.scheduler
+
+import org.apache.spark.SparkFunSuite
+import org.scalatest.BeforeAndAfter
+import org.apache.spark.SparkConf
+import org.apache.spark.TaskEndReason
+import org.apache.spark.Success
+import org.apache.spark.ExceptionFailure
+import org.apache.spark.LocalSparkContext
+import org.apache.spark.SparkContext
+
+import org.mockito.Mockito.{when,spy}
+
+class BlacklistTrackerSuite extends SparkFunSuite with BeforeAndAfter with LocalSparkContext {
+
+ val FAILURE: TaskEndReason = new ExceptionFailure(
+ "Fake",
+ "fake failure",
+ Array.empty[StackTraceElement],
+ "fake stack trace",
+ None,
+ None)
+
+ // Variable name can indicate basic information of taskInfo
+ // The format is "taskInfo_executorId_taskId_hostName"
+ val taskInfo_1_1_hostA = new TaskInfo(1L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_1_2_hostA = new TaskInfo(2L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_1_hostA = new TaskInfo(1L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_2_hostA = new TaskInfo(2L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_3_3_hostB = new TaskInfo(3L,1,1,0L,"3","hostB", TaskLocality.ANY, false)
+
+ test ("expireExecutorsInBlacklist works ") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.recoverPeriod", "2")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.start()
+ Thread.sleep(100)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1"))
+
+ // sleep 2 second to wait for expireExecutorsInBlackList be triggered
+ Thread.sleep(2000)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+
--- End diff --
Can you also check two more things:
(1) If you get another later failure, the executor stays in the blacklist
(2) if you have two executors in the blacklist, with different failure times, you correctly only remove one executor at first, and then remove both?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-151616363
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-171210806
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-175342880
@mwws can you rebase the code?
@kayousterhout any more comments on this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-165279831
@markhamstra I'm not entirely sure I understand the idea of de-blacklisting executors via speculation. The two seem rather independent, and I'm actually pretty interested in this feature even w/ speculation turned off. I don't really see how you'd leverage speculative tasks for this, it seems like it would complicate things a lot by tying them together, but maybe I just need the details spelled out more clearly?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42213571
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update
+ * executorIdToFailureStatus Map.
+ */
+private[spark] class BlacklistTracker(sparkConf: SparkConf) extends WithCache{
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val executorIdToFailureStatus: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val clock = new SystemClock()
+
+ private val enableBlacklistSpeculate = sparkConf.getBoolean(
+ "spark.scheduler.blacklist.speculate", false)
+
+ private val recoverPeriod = sparkConf.getLong(
+ "spark.scheduler.blacklist.recoverPeriod", 60L)
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList(executorIdToFailureStatus))
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, recoverPeriod, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ private def expireExecutorsInBlackList(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Unit = synchronized {
+ strategy.expireExecutorsInBlackList(executorIdToFailureStatus)
+
+ invalidAllCache()
+ }
+
+ def updateFailureExecutors(info: TaskInfo, reason: TaskEndReason) : Unit = synchronized {
+ reason match {
+ // If task succeeding, remove related record from executorIdToFailureStatus
+ case Success =>
+ removeFailureExecutorsForTaskId(info.executorId, info.taskId)
+
+ // If task failing, update latest failure time and failedTaskIds
+ case _ =>
+ val executorId = info.executorId
+ executorIdToFailureStatus.get(executorId) match {
+ case Some(failureStatus) =>
+ failureStatus.updatedTime = clock.getTimeMillis()
+ val failedTimes = failureStatus.numFailuresPerTask.getOrElse(info.taskId, 0) + 1
+ failureStatus.numFailuresPerTask.update(info.taskId, failedTimes)
+ case None =>
+ val failedTasks = mutable.HashMap(info.taskId -> 1)
+ val failureStatus = new FailureStatus(
+ clock.getTimeMillis(),
+ info.host,
+ failedTasks)
+ executorIdToFailureStatus.update(executorId, failureStatus)
+ }
+ invalidAllCache()
+ }
+ }
+
+ // remove the executorId from executorIdToFailureStatus
+ def removeFailureExecutors(executorId: String) : Unit = synchronized {
+ executorIdToFailureStatus.remove(executorId)
+ invalidAllCache()
+ }
+
+ // remove the failure record related to given taskId from executorIdToFailureStatus. If the
+ // number of records of given executorId becomes 0, remove the completed executorId.
+ def removeFailureExecutorsForTaskId(
+ executorId: String,
+ taskId: Long) : Unit = synchronized {
+ executorIdToFailureStatus.get(executorId).map(fs => {
+ fs.numFailuresPerTask.remove(taskId)
+ if(fs.numFailuresPerTask.isEmpty){
+ executorIdToFailureStatus.remove(executorId)
+ }
+ invalidAllCache()
+ })
+ }
+
+ def executorIsBlacklisted(
+ executorId: String,
+ sched: TaskSchedulerImpl,
+ taskId: Long) : Boolean = {
+
+ executorBlacklist(sched, taskId).contains(executorId)
+ }
+
+ // If the node is in blacklist, all executors allocated on that node will
+ // also be put into executor blacklist.
+ // By default it's turned off, user can enable it in sparkConf.
+ private def speculationFailedExecutor(
+ sched: TaskSchedulerImpl): Set[String] = {
--- End diff --
I am fine to make it always on and remove the configuration. It was there because I want to achieve the same logic as before. But I just noticed that according to current implementation, node blacklist will always be empty under "default" strategy and "speculation" is not useful. I will remove it. Thanks for the comment.
I
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by markhamstra <gi...@git.apache.org>.
Github user markhamstra commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-161133600
@kayousterhout Yeah, that's kind of interesting -- kind of like "arbeit macht frei", except it's not a lie. Probably shouldn't have gone there....
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42009817
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update
+ * executorIdToFailureStatus Map.
+ */
+private[spark] class BlacklistTracker(sparkConf: SparkConf) extends WithCache{
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val executorIdToFailureStatus: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val clock = new SystemClock()
+
+ private val enableBlacklistSpeculate = sparkConf.getBoolean(
+ "spark.scheduler.blacklist.speculate", false)
+
+ private val recoverPeriod = sparkConf.getLong(
+ "spark.scheduler.blacklist.recoverPeriod", 60L)
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList(executorIdToFailureStatus))
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, recoverPeriod, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ private def expireExecutorsInBlackList(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Unit = synchronized {
+ strategy.expireExecutorsInBlackList(executorIdToFailureStatus)
+
+ invalidAllCache()
+ }
+
+ def updateFailureExecutors(info: TaskInfo, reason: TaskEndReason) : Unit = synchronized {
+ reason match {
+ // If task succeeding, remove related record from executorIdToFailureStatus
+ case Success =>
+ removeFailureExecutorsForTaskId(info.executorId, info.taskId)
+
+ // If task failing, update latest failure time and failedTaskIds
+ case _ =>
+ val executorId = info.executorId
+ executorIdToFailureStatus.get(executorId) match {
+ case Some(failureStatus) =>
+ failureStatus.updatedTime = clock.getTimeMillis()
+ val failedTimes = failureStatus.numFailuresPerTask.getOrElse(info.taskId, 0) + 1
+ failureStatus.numFailuresPerTask.update(info.taskId, failedTimes)
+ case None =>
+ val failedTasks = mutable.HashMap(info.taskId -> 1)
+ val failureStatus = new FailureStatus(
+ clock.getTimeMillis(),
+ info.host,
+ failedTasks)
+ executorIdToFailureStatus.update(executorId, failureStatus)
+ }
+ invalidAllCache()
+ }
+ }
+
+ // remove the executorId from executorIdToFailureStatus
+ def removeFailureExecutors(executorId: String) : Unit = synchronized {
+ executorIdToFailureStatus.remove(executorId)
+ invalidAllCache()
+ }
+
+ // remove the failure record related to given taskId from executorIdToFailureStatus. If the
+ // number of records of given executorId becomes 0, remove the completed executorId.
+ def removeFailureExecutorsForTaskId(
+ executorId: String,
+ taskId: Long) : Unit = synchronized {
+ executorIdToFailureStatus.get(executorId).map(fs => {
+ fs.numFailuresPerTask.remove(taskId)
+ if(fs.numFailuresPerTask.isEmpty){
+ executorIdToFailureStatus.remove(executorId)
+ }
+ invalidAllCache()
+ })
+ }
+
+ def executorIsBlacklisted(
+ executorId: String,
+ sched: TaskSchedulerImpl,
+ taskId: Long) : Boolean = {
+
+ executorBlacklist(sched, taskId).contains(executorId)
+ }
+
+ // If the node is in blacklist, all executors allocated on that node will
+ // also be put into executor blacklist.
+ // By default it's turned off, user can enable it in sparkConf.
+ private def speculationFailedExecutor(
+ sched: TaskSchedulerImpl): Set[String] = {
+ if (!enableBlacklistSpeculate) {
+ Set.empty[String]
+ } else {
+ nodeBlacklist().flatMap(sched.getExecutorsAliveOnHost(_)
+ .getOrElse(Set.empty[String])).toSet
+ }
+ }
+
+ def executorBlacklist(
+ sched: TaskSchedulerImpl,
+ taskId: Long): Set[String] = synchronized {
+ if (!isBlacklistExecutorCacheValid) {
+ reEvaluateExecutorBlacklistAndUpdateCache(sched, taskId)
+ } else {
+ getExecutorBlacklistFromCache(taskId).getOrElse {
+ reEvaluateExecutorBlacklistAndUpdateCache(sched, taskId)
+ }
+ }
+ }
+
+ private def reEvaluateExecutorBlacklistAndUpdateCache(
+ sched: TaskSchedulerImpl,
+ taskId: Long): Set[String] = {
+ val executors = speculationFailedExecutor(sched) ++
+ strategy.getExecutorBlacklist(executorIdToFailureStatus, taskId)
+ updateBlacklistExecutorCache(taskId, executors)
+ executors
+ }
+
+ // The actual implementation is delegated to strategy
+ def nodeBlacklist(): Set[String] = synchronized {
+ if (isBlacklistNodeCacheValid) {
+ getNodeBlacklistFromCache
+ } else {
+ val nodes = strategy.getNodeBlacklist(executorIdToFailureStatus)
+ updateBlacklistNodeCache(nodes)
+ nodes
+ }
+ }
+}
+
+// Hide cache details in this trait to make code clean and avoid operation mistake
+trait WithCache {
--- End diff --
`private[scheduler]`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149074907
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42199742
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update
+ * executorIdToFailureStatus Map.
+ */
+private[spark] class BlacklistTracker(sparkConf: SparkConf) extends WithCache{
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val executorIdToFailureStatus: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val clock = new SystemClock()
+
+ private val enableBlacklistSpeculate = sparkConf.getBoolean(
+ "spark.scheduler.blacklist.speculate", false)
+
+ private val recoverPeriod = sparkConf.getLong(
+ "spark.scheduler.blacklist.recoverPeriod", 60L)
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList(executorIdToFailureStatus))
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, recoverPeriod, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ private def expireExecutorsInBlackList(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Unit = synchronized {
+ strategy.expireExecutorsInBlackList(executorIdToFailureStatus)
+
+ invalidAllCache()
+ }
+
+ def updateFailureExecutors(info: TaskInfo, reason: TaskEndReason) : Unit = synchronized {
+ reason match {
+ // If task succeeding, remove related record from executorIdToFailureStatus
+ case Success =>
+ removeFailureExecutorsForTaskId(info.executorId, info.taskId)
+
+ // If task failing, update latest failure time and failedTaskIds
+ case _ =>
+ val executorId = info.executorId
+ executorIdToFailureStatus.get(executorId) match {
+ case Some(failureStatus) =>
+ failureStatus.updatedTime = clock.getTimeMillis()
+ val failedTimes = failureStatus.numFailuresPerTask.getOrElse(info.taskId, 0) + 1
+ failureStatus.numFailuresPerTask.update(info.taskId, failedTimes)
+ case None =>
+ val failedTasks = mutable.HashMap(info.taskId -> 1)
+ val failureStatus = new FailureStatus(
+ clock.getTimeMillis(),
+ info.host,
+ failedTasks)
--- End diff --
Sure, I will follow the idiomatic way in Spark. The intention for splitting into different lines is just to make length check happy.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153935310
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42043964
--- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -0,0 +1,162 @@
+package org.apache.spark.scheduler
+
+import org.apache.spark.SparkFunSuite
+import org.scalatest.BeforeAndAfter
+import org.apache.spark.SparkConf
+import org.apache.spark.TaskEndReason
+import org.apache.spark.Success
+import org.apache.spark.ExceptionFailure
+import org.apache.spark.LocalSparkContext
+import org.apache.spark.SparkContext
+
+import org.mockito.Mockito.{when,spy}
+
+class BlacklistTrackerSuite extends SparkFunSuite with BeforeAndAfter with LocalSparkContext {
+
+ val FAILURE: TaskEndReason = new ExceptionFailure(
+ "Fake",
+ "fake failure",
+ Array.empty[StackTraceElement],
+ "fake stack trace",
+ None,
+ None)
+
+ // Variable name can indicate basic information of taskInfo
+ // The format is "taskInfo_executorId_taskId_hostName"
+ val taskInfo_1_1_hostA = new TaskInfo(1L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_1_2_hostA = new TaskInfo(2L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_1_hostA = new TaskInfo(1L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_2_hostA = new TaskInfo(2L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_3_3_hostB = new TaskInfo(3L,1,1,0L,"3","hostB", TaskLocality.ANY, false)
+
+ test ("expireExecutorsInBlacklist works ") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.recoverPeriod", "2")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.start()
+ Thread.sleep(100)
--- End diff --
We should avoid `Thread.sleep` for slow tests / flakiness by
(1) changing BlacklistTracker constructor to take an optional second arg, the `Clock`
(2) factor out the scheduled blacklist expiration into a method:
```scala
@VisibleForTesting
private[scheduler] def expireBlacklist(): Unit = expireExecutorsInBlackList(executorIdToFailureStatus)
```
(3) in the tests, use a `ManualClock` which you update directly and also call `expireBlacklist` directly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153252656
**[Test build #44884 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44884/consoleFull)** for PR 8760 at commit [`fc992b9`](https://github.com/apache/spark/commit/fc992b90daf3c4c9ac580d6d7929452febd80ae8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149078369
[Test build #43908 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43908/console) for PR 8760 at commit [`a2c8b40`](https://github.com/apache/spark/commit/a2c8b4021940606f17d122dc71ff444ca47c25a5).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `final class FailureStatus(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-187586804
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51737/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40217528
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -184,13 +188,21 @@ private[yarn] class YarnAllocator(
def requestTotalExecutorsWithPreferredLocalities(
requestedTotal: Int,
localityAwareTasks: Int,
- hostToLocalTaskCount: Map[String, Int]): Boolean = synchronized {
+ hostToLocalTaskCount: Map[String, Int],
+ nodeBlacklist: Set[String]): Boolean = synchronized {
this.numLocalityAwareTasks = localityAwareTasks
this.hostToLocalTaskCounts = hostToLocalTaskCount
if (requestedTotal != targetNumExecutors) {
logInfo(s"Driver requested a total number of $requestedTotal executor(s).")
targetNumExecutors = requestedTotal
+
+ //Update blacklist infomation to YARN ResouceManager for this application,
+ //in order to avoid allocating new Container on the problematic nodes.
--- End diff --
nit: space after "//" (style-checker will complain about this)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153962718
[Test build #45091 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45091/console) for PR 8760 at commit [`9a5ec8e`](https://github.com/apache/spark/commit/9a5ec8e51bbf5ae685f550f9af7eea2db3502fad).
* This patch **passes all tests**.
* This patch **does not merge cleanly**.
* This patch adds the following public classes _(experimental)_:
* `final class FailureStatus(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-154328628
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r43796097
--- Diff: docs/configuration.md ---
@@ -1141,6 +1141,48 @@ Apart from these, the following properties are also available, and may be useful
</td>
</tr>
<tr>
+ <td><code>spark.scheduler.blacklist.enabled</code></td>
+ <td>true</td>
+ <td>
+ If set to ture, executor blacklist feature will be enabled to avoid allocte new task on bad executor. The logic to define bad executor is based on BlacklistStrategy which is also configurable.
--- End diff --
A few typos here (ture, allocte), but I'd also like to suggest a rewording:
If set to "true", prevent Spark from scheduling tasks on executors that have been blacklisted due to too many task failures. The blacklisting algorithm can be further controlled by the other "spark.scheduler.blacklist" configuration options.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 8426
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-140295809
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-187586803
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153276955
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149302204
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43925/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-220069649
Hi everyone, sorry this has sat around for so long, but this is a super important feature, so I'd like to finish up the missing pieces and get this in (with credit to @mwws still, of course). Other than just fixing merge conflicts, it seems there are two things which need to be addressed: (1) addressing performance concerns with some profiling and (2) investigating a lock-free implementation (think this can be done but need to look more carefully). I'll also build on the tests introduced here https://github.com/apache/spark/pull/8559.
I'll open a WIP pr in the next day or two at least, but please let me know if I'm missing any major concern.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40279072
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala ---
@@ -56,8 +56,12 @@ private[spark] abstract class YarnSchedulerBackend(
* This includes executors already pending or running.
*/
override def doRequestTotalExecutors(requestedTotal: Int): Boolean = {
+
+ val nodeBlacklist: Set[String] = scheduler.sc.blacklistTracker
+ .fold(Set.empty[String])(_.nodeBlacklist())
--- End diff --
I will make change to follow spark idiomatic usage
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42215540
--- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -0,0 +1,162 @@
+package org.apache.spark.scheduler
+
+import org.apache.spark.SparkFunSuite
+import org.scalatest.BeforeAndAfter
+import org.apache.spark.SparkConf
+import org.apache.spark.TaskEndReason
+import org.apache.spark.Success
+import org.apache.spark.ExceptionFailure
+import org.apache.spark.LocalSparkContext
+import org.apache.spark.SparkContext
+
+import org.mockito.Mockito.{when,spy}
+
+class BlacklistTrackerSuite extends SparkFunSuite with BeforeAndAfter with LocalSparkContext {
+
+ val FAILURE: TaskEndReason = new ExceptionFailure(
+ "Fake",
+ "fake failure",
+ Array.empty[StackTraceElement],
+ "fake stack trace",
+ None,
+ None)
+
+ // Variable name can indicate basic information of taskInfo
+ // The format is "taskInfo_executorId_taskId_hostName"
+ val taskInfo_1_1_hostA = new TaskInfo(1L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_1_2_hostA = new TaskInfo(2L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_1_hostA = new TaskInfo(1L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_2_hostA = new TaskInfo(2L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_3_3_hostB = new TaskInfo(3L,1,1,0L,"3","hostB", TaskLocality.ANY, false)
+
+ test ("expireExecutorsInBlacklist works ") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.recoverPeriod", "2")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.start()
+ Thread.sleep(100)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1"))
+
+ // sleep 2 second to wait for expireExecutorsInBlackList be triggered
+ Thread.sleep(2000)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+
+ tracker.stop()
+ }
+
+ test("default strategy works as exceptation") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1","2"))
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set())
+
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("2"))
+ assert(tracker.nodeBlacklist() === Set())
+
+ tracker.updateFailureExecutors(taskInfo_3_3_hostB, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 3L) === Set("3"))
+ assert(tracker.nodeBlacklist() === Set())
+
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+ }
+
+ test ("strict strategy works as exception") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.strategy", "strict")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1"))
+ tracker.updateFailureExecutors(taskInfo_1_2_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set("1"))
+ assert(tracker.nodeBlacklist() === Set("hostA"))
+
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set("1", "2"))
+ assert(tracker.nodeBlacklist() === Set("hostA"))
+
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1","2"))
+
+ tracker.updateFailureExecutors(taskInfo_3_3_hostB, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 3L) === Set("1","2","3"))
+ assert(tracker.nodeBlacklist() === Set("hostA","hostB"))
+ }
+
+ test ("threshold strategy works as exception") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.strategy", "threshold")
+ .set("spark.scheduler.blacklist.threshold.maxFailureTaskNumber", "1")
+ .set("spark.scheduler.blacklist.threshold.maxBlackExecutorNumber", "0")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+ tracker.updateFailureExecutors(taskInfo_1_2_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set("1"))
+ assert(tracker.nodeBlacklist() === Set("hostA"))
+
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set("1"))
+ assert(tracker.nodeBlacklist() === Set("hostA"))
+
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
--- End diff --
Yes, that's the case. And I will add one more check as your suggestation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40285505
--- Diff: core/src/test/scala/org/apache/spark/HeartbeatReceiverSuite.scala ---
@@ -261,7 +261,7 @@ private class FakeClusterManager(override val rpcEnv: RpcEnv) extends RpcEndpoin
def getExecutorIdsToKill: Set[String] = executorIdsToKill.toSet
override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
- case RequestExecutors(requestedTotal, _, _) =>
+ case RequestExecutors(requestedTotal, _, _,_) =>
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40164318
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update failedExecutorMap.
+ */
+class BlacklistTracker(sparkConf: SparkConf) {
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val failedExecutorMap: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val clock = new SystemClock()
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList(failedExecutorMap))
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, 60, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ private def expireExecutorsInBlackList(
+ failureExecutors: mutable.HashMap[String, FailureStatus]): Unit = synchronized {
+ strategy.expireExecutorsInBlackList(failureExecutors)
+ }
+
+ def updateFailureExecutors(info: TaskInfo, reason: TaskEndReason) : Unit = synchronized {
+ reason match {
+ // If task succeeding, remove related record from failedExecutorMap
+ case Success =>
+ removeFailureExecutors(info.executorId, Some(info.taskId))
+
+ // If task failing, update latest failure time and failedTaskIds
+ case _ =>
+ val executorId = info.executorId
+ failedExecutorMap.get(executorId) match {
+ case Some(failureStatus) =>
+ failureStatus.updatedTime = clock.getTimeMillis()
+ val failedTimes = failureStatus.failedTasks.getOrElse(info.taskId, 0) + 1
+ failureStatus.failedTasks.update(info.taskId, failedTimes)
+ case None =>
+ val failedTasks = mutable.HashMap(info.taskId -> 1)
+ val failureStatus = new FailureStatus(
+ clock.getTimeMillis(),
+ info.host,
+ failedTasks)
+ failedExecutorMap.update(executorId, failureStatus)
+ }
+ }
+ }
+
+ def removeFailureExecutors(
+ executorId: String,
+ taskId: Option[Long]) : Unit = synchronized {
+ taskId match {
+ // If taskId is provided, remove the taskId from failedTasks
+ case Some(id) =>
+ failedExecutorMap.get(executorId).map(fs => {
+ fs.updatedTime = clock.getTimeMillis()
+ fs.failedTasks.remove(id)
+ if(fs.failedTasks.isEmpty){
+ failedExecutorMap.remove(executorId)
+ }
+ })
+ // if taskId is not provided, remove the whole record related to given executorId
+ case None =>
+ failedExecutorMap.remove(executorId)
+ }
+ }
+
+ def executorIsBlacklisted(
+ executorId: String,
+ sched: TaskSchedulerImpl,
+ taskId: Option[Long]) : Boolean = {
+
+ executorBlacklist(sched, taskId).contains(executorId)
+ }
+
+ // The actual implementation is delegated to strategy
+ def executorBlacklist(
+ sched: TaskSchedulerImpl,
+ taskId: Option[Long]): Set[String] = synchronized {
+
+ // If the node is in blacklist, all executors allocated on that node will
+ // also be put into executor blacklist.
+ // By default it's turned off, user can enable it in sparkConf.
+ val speculationFailedExecutor: Set[String] =
+ if (sparkConf.getBoolean("spark.scheduler.blacklist.speculate", false)) {
+ Set.empty[String]
+ } else {
+ nodeBlacklist(taskId).flatMap(sched.getExecutorsAliveOnHost(_)
+ .getOrElse(Set.empty[String])).toSet
+ }
+
+ speculationFailedExecutor ++ strategy.getExecutorBlacklist(failedExecutorMap, taskId)
+ }
+
+ // The actual implementation is delegated to strategy
+ def nodeBlacklist(taskId: Option[Long] = None): Set[String] = synchronized {
+ strategy.getNodeBlacklist(failedExecutorMap, taskId)
+ }
+}
+
+/**
+ * A class to record details of failure.
+ *
+ * @param initialTime the time when failure status be created
+ * @param host the node name which running executor on
+ * @param failedTasks all tasks failed on the executor (key is task id,
+ * value is failure time of this task)
+ */
+final class FailureStatus(
+ initialTime: Long,
+ val host: String,
+ val failedTasks: mutable.HashMap[Long, Int]) {
+
+ var updatedTime = initialTime
+ def failureTimes : Int = failedTasks.values.sum
--- End diff --
I will change it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153297941
**[Test build #44902 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44902/consoleFull)** for PR 8760 at commit [`b6b25e1`](https://github.com/apache/spark/commit/b6b25e1b035599e3568b331c70900f44d6976b92).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `final class FailureStatus(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r43970555
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.Clock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+private [scheduler] trait BlacklistStrategy {
+ /** Defined a time interval to expire failure information of executors */
+ val expireTimeInMilliseconds: Long
+
+ /** Return executors in blacklist which are related to given taskIndex */
+ def getExecutorBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ taskIndex: Int, clock: Clock): Set[String]
+
+ /** Return all nodes in blacklist */
+ def getNodeBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Set[String]
+
+ /**
+ * Default implementation to remove failure executors from HashMap based on given time period.
+ * The return value identity whether or not it updated anything
+ */
+ def expireExecutorsInBlackList(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus], clock: Clock): Boolean = {
+ val now = clock.getTimeMillis()
+ val expiredKey = executorIdToFailureStatus.filter {
+ case (executorid, failureStatus) => {
+ (now - failureStatus.updatedTime) >= expireTimeInMilliseconds
+ }
+ }.keySet
+
+ if (expiredKey.isEmpty) {
+ false
+ } else {
+ executorIdToFailureStatus --= expiredKey
+ true
+ }
+ }
+}
+
+/**
+ * This strategy is simply based on given threshold and is taskId unrelated. An executor will be
+ * in blacklist, if it failed more than "maxFailureTaskNumber" times. A node will be in blacklist,
+ * if there are more than "maxBlackExecutorNumber" executors on it in executor blacklist.
+ *
+ * In this case, provided taskId will be ignored. The benefit for taskId unrelated strategy is that
+ * different taskSets can learn experience from other taskSet to avoid allocating tasks on
+ * problematic executors.
+ */
+private[scheduler] class SimpleStrategy(
+ maxFailureTaskNumber: Int,
+ maxBlackExecutorNumber: Int,
+ val expireTimeInMilliseconds: Long
+ )extends BlacklistStrategy {
+
+ private def getSelectedExecutorMap(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]) = {
+ executorIdToFailureStatus.filter{
+ case (id, failureStatus) => failureStatus.totalNumFailures > maxFailureTaskNumber
+ }
+ }
+
+ // As this is a taskId unrelated strategy, the input taskId will be ignored
+ def getExecutorBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ taskIndex: Int, clock: Clock): Set[String] = {
+ getSelectedExecutorMap(executorIdToFailureStatus).keys.toSet
+ }
+
+ def getNodeBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Set[String] = {
+ getSelectedExecutorMap(executorIdToFailureStatus)
+ .groupBy{case (id, failureStatus) => failureStatus.host}
+ .filter {case (host, executorIdToFailureStatus) =>
+ executorIdToFailureStatus.size > maxBlackExecutorNumber}
+ .keys.toSet
+ }
+}
+
+/**
+ * This strategy is applied as default to keep the same semantics as original. It's an taskId
+ * related strategy. If an executor failed running "task A", then we think this executor is
+ * blacked for "task A". And we think the executor is still healthy for other task. node blacklist
+ * is always empty.
+ *
+ * It was the standard behavior before spark 1.6
+ */
+private[scheduler] class DefaultStrategy(
+ val expireTimeInMilliseconds: Long) extends BlacklistStrategy {
+ def getExecutorBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ taskIndex: Int, clock: Clock): Set[String] = {
+ executorIdToFailureStatus.filter{
+ case (_, failureStatus) => failureStatus.numFailuresPerTask.keySet.contains(taskIndex) &&
+ clock.getTimeMillis() - failureStatus.updatedTime < expireTimeInMilliseconds
+ }.keys.toSet
+ }
+
+ def getNodeBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Set[String] =
+ Set.empty[String]
+}
+
+/**
+ * Create BlacklistStrategy instance according to SparkConf
+ */
+private[scheduler] object BlacklistStrategy {
+ def apply(sparkConf: SparkConf): BlacklistStrategy = {
+ val timeout = sparkConf.getLong("spark.scheduler.executorTaskBlacklistTime", 0L)
--- End diff --
use *getTimeAsMs* instead and I think time unit could be automatically converted "s <--> ms <--> us " by this method, so it's not necessary to divide 1000 here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149373970
**[Test build #43935 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43935/consoleFull)** for PR 8760 at commit [`a2c8b40`](https://github.com/apache/spark/commit/a2c8b4021940606f17d122dc71ff444ca47c25a5).
* This patch **fails from timeout after a configured wait of \`250m\`**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `final class FailureStatus(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-171210609
**[Test build #49294 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49294/consoleFull)** for PR 8760 at commit [`25a7c6f`](https://github.com/apache/spark/commit/25a7c6fc329687310f1ae88e56bcd72a5c259b61).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r43797266
--- Diff: docs/configuration.md ---
@@ -1141,6 +1141,48 @@ Apart from these, the following properties are also available, and may be useful
</td>
</tr>
<tr>
+ <td><code>spark.scheduler.blacklist.enabled</code></td>
+ <td>true</td>
+ <td>
+ If set to ture, executor blacklist feature will be enabled to avoid allocte new task on bad executor. The logic to define bad executor is based on BlacklistStrategy which is also configurable.
+ </td>
+</tr>
+<tr>
+ <td><code>spark.scheduler.executorTaskBlacklistTime</code></td>
+ <td>0L</td>
+ <td>
+ The threshold to deside blacklist executor, if last failed time is older than current time subtract executorTaskBlacklistTime, then the executor could be removed from blacklist.
+ </td>
+</tr>
+<tr>
+ <td><code>spark.scheduler.blacklist.recoverPeriod</code></td>
+ <td>60L</td>
+ <td>
+ The period between runnning blacklist recover process.
--- End diff --
lets include a timeunit here too, so "60s", and change the text to "If executor blacklisting is enabled, this controls how often to check if executors can be returned to the pool of active executors."
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153253238
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44884/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149643268
**[Test build #43980 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43980/consoleFull)** for PR 8760 at commit [`a2c8b40`](https://github.com/apache/spark/commit/a2c8b4021940606f17d122dc71ff444ca47c25a5).
* This patch **fails from timeout after a configured wait of \`250m\`**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `final class FailureStatus(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149224169
**[Test build #43925 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43925/consoleFull)** for PR 8760 at commit [`a2c8b40`](https://github.com/apache/spark/commit/a2c8b4021940606f17d122dc71ff444ca47c25a5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-151675883
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44460/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149401264
I can't find obvious failure according to log. Is it caused by unstable Jenkins?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149645260
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43980/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r46249345
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+import org.apache.spark.util.Clock
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update
+ * executorIdToFailureStatus Map.
+ */
+private[spark] class BlacklistTracker(
+ sparkConf: SparkConf,
+ clock: Clock = new SystemClock()) extends BlacklistCache{
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val executorIdToFailureStatus: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val recoverPeriod = sparkConf.getTimeAsSeconds(
+ "spark.scheduler.blacklist.recoverPeriod", "60s")
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList())
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, recoverPeriod, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ /** VisibleForTesting */
+ private[scheduler] def expireExecutorsInBlackList(): Unit = synchronized {
+ val updated = strategy.expireExecutorsInBlackList(executorIdToFailureStatus, clock)
+ if (updated) {
+ invalidateCache()
+ }
+ }
+
+ def updateFailedExecutors(
+ stageId: Int,
+ partation: Int,
+ info: TaskInfo,
+ reason: TaskEndReason) : Unit = synchronized {
+
+ val atomTask = StageAndPartition(stageId, partation)
+ reason match {
+ // If the task succeeded, remove related record from executorIdToFailureStatus
+ case Success =>
+ removeFailedExecutorsForTaskId(info.executorId, stageId, partation)
+
+ // If the task failed, update latest failure time and failedTaskIds
+ case _ =>
+ val executorId = info.executorId
+ executorIdToFailureStatus.get(executorId) match {
+ case Some(failureStatus) =>
+ failureStatus.updatedTime = clock.getTimeMillis()
+ val failedTimes = failureStatus.numFailuresPerTask.getOrElse(atomTask, 0) + 1
+ failureStatus.numFailuresPerTask(atomTask) = failedTimes
+ case None =>
+ val failedTasks = mutable.HashMap(atomTask -> 1)
+ val failureStatus = new FailureStatus(
+ clock.getTimeMillis(), info.host, failedTasks)
+ executorIdToFailureStatus(executorId) = failureStatus
+ }
+ invalidateCache()
+ }
+ }
+
+ /** remove the executorId from executorIdToFailureStatus */
+ def removeFailedExecutors(executorId: String) : Unit = synchronized {
+ executorIdToFailureStatus.remove(executorId)
+ invalidateCache()
+ }
+
+ /**
+ * remove the failure record related to given taskId from executorIdToFailureStatus. If the
+ * number of records of given executorId becomes 0, remove the completed executorId.
+ */
+ def removeFailedExecutorsForTaskId(
+ executorId: String,
+ stageId: Int,
+ partation: Int) : Unit = synchronized {
+ val atomTask = StageAndPartition(stageId, partation)
+ executorIdToFailureStatus.get(executorId).map{ fs =>
+ fs.numFailuresPerTask.remove(atomTask)
+ if (fs.numFailuresPerTask.isEmpty) {
+ executorIdToFailureStatus.remove(executorId)
+ }
+ invalidateCache()
+ }
+ }
+
+ def isExecutorBlacklisted(
+ executorId: String,
+ sched: TaskSchedulerImpl,
+ stageId: Int,
+ partation: Int) : Boolean = {
+
+ executorBlacklist(sched, stageId, partation).contains(executorId)
+ }
+
+ // If the node is in blacklist, all executors allocated on that node will
+ // also be put into executor blacklist.
+ private def executorsOnBlacklistedNode(
+ sched: TaskSchedulerImpl): Set[String] = {
+ nodeBlacklist().flatMap(sched.getExecutorsAliveOnHost(_)
+ .getOrElse(Set.empty[String])).toSet
+ }
+
+ def executorBlacklist(
+ sched: TaskSchedulerImpl, stageId: Int, partation: Int): Set[String] = synchronized {
+ val atomTask = StageAndPartition(stageId, partation)
+ if (!isBlacklistExecutorCacheValid) {
+ reEvaluateExecutorBlacklistAndUpdateCache(sched, atomTask, clock)
+ } else {
+ getExecutorBlacklistFromCache(atomTask).getOrElse {
+ reEvaluateExecutorBlacklistAndUpdateCache(sched, atomTask, clock)
+ }
+ }
+ }
+
+ private def reEvaluateExecutorBlacklistAndUpdateCache(
+ sched: TaskSchedulerImpl,
+ atomTask: StageAndPartition,
+ clock: Clock): Set[String] = {
+ val executors = executorsOnBlacklistedNode(sched) ++
+ strategy.getExecutorBlacklist(executorIdToFailureStatus, atomTask, clock)
+ updateBlacklistExecutorCache(atomTask, executors)
+ executors
+ }
+
+ // The actual implementation is delegated to strategy
+ def nodeBlacklist(): Set[String] = synchronized {
+ if (isBlacklistNodeCacheValid) {
+ getNodeBlacklistFromCache
+ } else {
+ val nodes = strategy.getNodeBlacklist(executorIdToFailureStatus)
+ updateBlacklistNodeCache(nodes)
+ nodes
+ }
+ }
+}
+
+/**
+ * Hide cache details in this trait to make code clean and avoid operation mistake
+ */
+private[scheduler] trait BlacklistCache {
+
+ // local cache to minimize the the work when query blacklisted executor and node
+ private val blacklistExecutorCache = mutable.HashMap.empty[StageAndPartition, Set[String]]
+ private val blacklistNodeCache = mutable.Set.empty[String]
+
+ // The flag to mark if cache is valid, it will be set to false when executorIdToFailureStatus be
+ // updated and it will be set to true, when called executorBlacklist and nodeBlacklist.
+ private var _isBlacklistExecutorCacheValid : Boolean = false
+ private var _isBlacklistNodeCacheValid: Boolean = false
+
+ private val cacheLock = new Object()
+
+ protected def isBlacklistExecutorCacheValid : Boolean = _isBlacklistExecutorCacheValid
+ protected def isBlacklistNodeCacheValid: Boolean = _isBlacklistNodeCacheValid
+
+ protected def updateBlacklistExecutorCache(
+ atomTask: StageAndPartition,
+ blacklistExecutor: Set[String]): Unit = cacheLock.synchronized {
+ if (!_isBlacklistExecutorCacheValid) {
+ blacklistExecutorCache.clear()
+ }
+ blacklistExecutorCache.update(atomTask, blacklistExecutor)
+ _isBlacklistExecutorCacheValid = true
+ }
+
+ protected def updateBlacklistNodeCache(
+ blacklistNode: Set[String]): Unit = cacheLock.synchronized {
+ if (!_isBlacklistNodeCacheValid) {
+ blacklistNodeCache.clear()
+ }
+ blacklistNodeCache ++= blacklistNode
+ _isBlacklistNodeCacheValid = true
+ }
+
+ protected def invalidateCache(): Unit = cacheLock.synchronized {
+ _isBlacklistExecutorCacheValid = false
+ _isBlacklistNodeCacheValid = false
+ }
+
+ protected def getExecutorBlacklistFromCache(
+ atomTask: StageAndPartition): Option[Set[String]] = {
+ blacklistExecutorCache.get(atomTask)
+ }
+
+ protected def getNodeBlacklistFromCache: Set[String] = blacklistNodeCache.toSet
+}
+
+/**
+ * A class to record details of failure.
+ *
+ * @param initialTime the time when failure status be created
+ * @param host the node name which running executor on
+ * @param numFailuresPerTask all tasks failed on the executor (key is StageAndPartition, value
+ * is the number of failures of this task)
+ */
+private[scheduler] final class FailureStatus(
+ initialTime: Long,
+ val host: String,
+ val numFailuresPerTask: mutable.HashMap[StageAndPartition, Int]) {
+
+ var updatedTime = initialTime
--- End diff --
A potential issue with the design is the huge variance possible in how long two executors/nodes remain black-listed depending on how the failures get reported since time is always updated to last failure time.
Not sure what the implication of that would be though - would be interesting to monitor in subsequent releases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42044542
--- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -0,0 +1,162 @@
+package org.apache.spark.scheduler
+
+import org.apache.spark.SparkFunSuite
+import org.scalatest.BeforeAndAfter
+import org.apache.spark.SparkConf
+import org.apache.spark.TaskEndReason
+import org.apache.spark.Success
+import org.apache.spark.ExceptionFailure
+import org.apache.spark.LocalSparkContext
+import org.apache.spark.SparkContext
+
+import org.mockito.Mockito.{when,spy}
+
+class BlacklistTrackerSuite extends SparkFunSuite with BeforeAndAfter with LocalSparkContext {
+
+ val FAILURE: TaskEndReason = new ExceptionFailure(
+ "Fake",
+ "fake failure",
+ Array.empty[StackTraceElement],
+ "fake stack trace",
+ None,
+ None)
+
+ // Variable name can indicate basic information of taskInfo
+ // The format is "taskInfo_executorId_taskId_hostName"
+ val taskInfo_1_1_hostA = new TaskInfo(1L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_1_2_hostA = new TaskInfo(2L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_1_hostA = new TaskInfo(1L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_2_hostA = new TaskInfo(2L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_3_3_hostB = new TaskInfo(3L,1,1,0L,"3","hostB", TaskLocality.ANY, false)
+
+ test ("expireExecutorsInBlacklist works ") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.recoverPeriod", "2")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.start()
+ Thread.sleep(100)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1"))
+
+ // sleep 2 second to wait for expireExecutorsInBlackList be triggered
+ Thread.sleep(2000)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+
+ tracker.stop()
+ }
+
+ test("default strategy works as exceptation") {
--- End diff --
typo: expectation (I think that is what you mean)
but actually, might as well just rename to simply "default strategy works"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42004744
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.util.SystemClock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+trait BlacklistStrategy {
+ val expireTimeInMillisecond: Long
--- End diff --
can you rename to `expireTimeInMilliseconds` ('s' at the end) and add a doc?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149565623
**[Test build #43977 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43977/consoleFull)** for PR 8760 at commit [`a2c8b40`](https://github.com/apache/spark/commit/a2c8b4021940606f17d122dc71ff444ca47c25a5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-154347404
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-148207561
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-167741271
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42263885
--- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import org.apache.spark.SparkFunSuite
+import org.scalatest.BeforeAndAfter
+import org.apache.spark.SparkConf
+import org.apache.spark.TaskEndReason
+import org.apache.spark.Success
+import org.apache.spark.ExceptionFailure
+import org.apache.spark.LocalSparkContext
+import org.apache.spark.SparkContext
+import org.mockito.Mockito.{when,spy}
+import org.apache.spark.util.ManualClock
+
+class BlacklistTrackerSuite extends SparkFunSuite with BeforeAndAfter with LocalSparkContext {
+
+ val FAILURE: TaskEndReason = new ExceptionFailure(
+ "Fake",
+ "fake failure",
+ Array.empty[StackTraceElement],
+ "fake stack trace",
+ None,
+ None)
+
+ // Variable name can indicate basic information of taskInfo
+ // The format is "taskInfo_executorId_taskId_hostName"
+ val taskInfo_1_1_hostA = new TaskInfo(1L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_1_2_hostA = new TaskInfo(2L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_1_hostA = new TaskInfo(1L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_2_hostA = new TaskInfo(2L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_3_3_hostB = new TaskInfo(3L,1,1,0L,"3","hostB", TaskLocality.ANY, false)
+
+ val clock = new ManualClock(0)
+
+ test ("expireExecutorsInBlacklist works") {
+ // expire time is set to 6s
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.executorTaskBlacklistTime", "6000")
+
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf, clock)
+ // Executor 1 into blacklist at Time 00:00:00
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1"))
+
+ clock.setTime(2000)
+ tracker.expireExecutorsInBlackList()
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1"))
+ // Executor 1 failed again at Time 00::00:02
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+
+ clock.setTime(3000)
+ // Executor 2 failed at Time 00:00:03
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1", "2"))
+
+ clock.setTime(6000)
+ tracker.expireExecutorsInBlackList()
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1", "2"))
+
+ clock.setTime(8000)
+ tracker.expireExecutorsInBlackList()
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("2"))
+
+ clock.setTime(10000)
+ tracker.expireExecutorsInBlackList()
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+ }
+
+ test("default strategy works") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf, clock)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1","2"))
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set())
+
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("2"))
+ assert(tracker.nodeBlacklist() === Set())
+
+ tracker.updateFailureExecutors(taskInfo_3_3_hostB, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("2"))
+
+ tracker.updateFailureExecutors(taskInfo_3_3_hostB, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 3L) === Set("3"))
+ assert(tracker.nodeBlacklist() === Set())
+
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+ }
+
+ test ("threshold strategy works") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.strategy", "threshold")
+ .set("spark.scheduler.blacklist.threshold.maxFailureTaskNumber", "1")
+ .set("spark.scheduler.blacklist.threshold.maxBlackExecutorNumber", "0")
+
+ sc = new SparkContext(conf)
+ val scheduler = spy(new TaskSchedulerImpl(sc, 1))
+ when(scheduler.getExecutorsAliveOnHost("hostA")).thenReturn(Some(Set("1","2")))
+
+ val tracker = new BlacklistTracker(conf, clock)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+ tracker.updateFailureExecutors(taskInfo_1_2_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set("1", "2"))
+ assert(tracker.nodeBlacklist() === Set("hostA"))
--- End diff --
Sorry this is a pain, but I'd like you to make a few changes to increase test coverage a bit:
(a) add executor "3" to hostA
(b) change "maxBlackExecutorNumber" to 1
(c) after failing two tasks on executor 1, check that all tasks are blacklisted on executor 1, (regardless of task id):
```scala
tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
assert(tracker.executorBlacklist(scheduler, 1L) === Set())
tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
// executor 1 is now blacklisted from *all* tasks
(0 until 10).foreach { assert(tracker.executorBlacklist(scheduler, _) === Set("1")) }
assert(tracker.nodeBlacklist() === Set())
```
(d) then fail add more task failures, so that you blacklist executor 2, which will trigger the node blacklist so also blacklist executor 3.
(e) after removing executor 1 from the blacklist, then for all tasks, the blacklist is just executor 2 (executor 3 is also removed).
the point is to check that the separation of executor & node blacklists.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153253236
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-150732959
**[Test build #44259 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44259/consoleFull)** for PR 8760 at commit [`a2c8b40`](https://github.com/apache/spark/commit/a2c8b4021940606f17d122dc71ff444ca47c25a5).
* This patch **fails from timeout after a configured wait of \`250m\`**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `final class FailureStatus(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-152970222
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-151617756
**[Test build #44460 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44460/consoleFull)** for PR 8760 at commit [`a2c8b40`](https://github.com/apache/spark/commit/a2c8b4021940606f17d122dc71ff444ca47c25a5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-154329464
**[Test build #45200 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45200/consoleFull)** for PR 8760 at commit [`33e3542`](https://github.com/apache/spark/commit/33e3542bb0b3c88c079dff4635a17c669636fd0d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-165280794
@mridulm going back to one of your previous comments:
> What would be more ideal is an implementation which allows for blacklisting single task (as is current), to multiple tasks for a taskset on a single executor (in case the taskset has issues on the executor - memory constraints perhaps) to blacklisting the node : time bound ofcourse, and for specific taskset and/or all taskset until timeout expires (directory cleanup catching up, etc).
Did I miss this in there somewhere ?
this isn't specifically included, but I think it could be added as an alternate strategy without a huge change. If I understand correctly, you're basically proposing adding another blacklisted state, where an executor is completely blacklisted, but only for *one taskset*, and after its blacklisted for multiple tasksets then its promoted to being blacklisted for all tasksets (still subject to some timeout or something)? I don't think that would be too hard but also trying to find the balance between adding too many knobs -- is there a clear use case for it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40164305
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update failedExecutorMap.
+ */
+class BlacklistTracker(sparkConf: SparkConf) {
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val failedExecutorMap: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val clock = new SystemClock()
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList(failedExecutorMap))
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, 60, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ private def expireExecutorsInBlackList(
+ failureExecutors: mutable.HashMap[String, FailureStatus]): Unit = synchronized {
+ strategy.expireExecutorsInBlackList(failureExecutors)
+ }
+
+ def updateFailureExecutors(info: TaskInfo, reason: TaskEndReason) : Unit = synchronized {
+ reason match {
+ // If task succeeding, remove related record from failedExecutorMap
+ case Success =>
+ removeFailureExecutors(info.executorId, Some(info.taskId))
+
+ // If task failing, update latest failure time and failedTaskIds
+ case _ =>
+ val executorId = info.executorId
+ failedExecutorMap.get(executorId) match {
+ case Some(failureStatus) =>
+ failureStatus.updatedTime = clock.getTimeMillis()
+ val failedTimes = failureStatus.failedTasks.getOrElse(info.taskId, 0) + 1
+ failureStatus.failedTasks.update(info.taskId, failedTimes)
+ case None =>
+ val failedTasks = mutable.HashMap(info.taskId -> 1)
+ val failureStatus = new FailureStatus(
+ clock.getTimeMillis(),
+ info.host,
+ failedTasks)
+ failedExecutorMap.update(executorId, failureStatus)
+ }
+ }
+ }
+
+ def removeFailureExecutors(
+ executorId: String,
+ taskId: Option[Long]) : Unit = synchronized {
+ taskId match {
+ // If taskId is provided, remove the taskId from failedTasks
+ case Some(id) =>
+ failedExecutorMap.get(executorId).map(fs => {
+ fs.updatedTime = clock.getTimeMillis()
+ fs.failedTasks.remove(id)
+ if(fs.failedTasks.isEmpty){
+ failedExecutorMap.remove(executorId)
+ }
+ })
+ // if taskId is not provided, remove the whole record related to given executorId
+ case None =>
+ failedExecutorMap.remove(executorId)
+ }
+ }
+
+ def executorIsBlacklisted(
+ executorId: String,
+ sched: TaskSchedulerImpl,
+ taskId: Option[Long]) : Boolean = {
+
+ executorBlacklist(sched, taskId).contains(executorId)
+ }
+
+ // The actual implementation is delegated to strategy
+ def executorBlacklist(
+ sched: TaskSchedulerImpl,
+ taskId: Option[Long]): Set[String] = synchronized {
+
+ // If the node is in blacklist, all executors allocated on that node will
+ // also be put into executor blacklist.
+ // By default it's turned off, user can enable it in sparkConf.
+ val speculationFailedExecutor: Set[String] =
+ if (sparkConf.getBoolean("spark.scheduler.blacklist.speculate", false)) {
+ Set.empty[String]
+ } else {
+ nodeBlacklist(taskId).flatMap(sched.getExecutorsAliveOnHost(_)
+ .getOrElse(Set.empty[String])).toSet
+ }
+
+ speculationFailedExecutor ++ strategy.getExecutorBlacklist(failedExecutorMap, taskId)
+ }
+
+ // The actual implementation is delegated to strategy
+ def nodeBlacklist(taskId: Option[Long] = None): Set[String] = synchronized {
+ strategy.getNodeBlacklist(failedExecutorMap, taskId)
+ }
+}
+
+/**
+ * A class to record details of failure.
+ *
+ * @param initialTime the time when failure status be created
+ * @param host the node name which running executor on
+ * @param failedTasks all tasks failed on the executor (key is task id,
+ * value is failure time of this task)
+ */
+final class FailureStatus(
+ initialTime: Long,
+ val host: String,
+ val failedTasks: mutable.HashMap[Long, Int]) {
--- End diff --
Thanks for naming suggestion, it's always welcome:) I will change it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-152944782
@squito the timeout do related to my change. In "resubmitted" test case, the failure executor is in blacklist by accident and waiting for recovering process to make it back. I have fixed it and add a new test case to make sure blacklisting is off by default. Also doc is added to configuration.md.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42048978
--- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -0,0 +1,162 @@
+package org.apache.spark.scheduler
+
+import org.apache.spark.SparkFunSuite
+import org.scalatest.BeforeAndAfter
+import org.apache.spark.SparkConf
+import org.apache.spark.TaskEndReason
+import org.apache.spark.Success
+import org.apache.spark.ExceptionFailure
+import org.apache.spark.LocalSparkContext
+import org.apache.spark.SparkContext
+
+import org.mockito.Mockito.{when,spy}
+
+class BlacklistTrackerSuite extends SparkFunSuite with BeforeAndAfter with LocalSparkContext {
+
+ val FAILURE: TaskEndReason = new ExceptionFailure(
+ "Fake",
+ "fake failure",
+ Array.empty[StackTraceElement],
+ "fake stack trace",
+ None,
+ None)
+
+ // Variable name can indicate basic information of taskInfo
+ // The format is "taskInfo_executorId_taskId_hostName"
+ val taskInfo_1_1_hostA = new TaskInfo(1L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_1_2_hostA = new TaskInfo(2L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_1_hostA = new TaskInfo(1L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_2_hostA = new TaskInfo(2L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_3_3_hostB = new TaskInfo(3L,1,1,0L,"3","hostB", TaskLocality.ANY, false)
+
+ test ("expireExecutorsInBlacklist works ") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.recoverPeriod", "2")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.start()
+ Thread.sleep(100)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1"))
+
+ // sleep 2 second to wait for expireExecutorsInBlackList be triggered
+ Thread.sleep(2000)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+
+ tracker.stop()
+ }
+
+ test("default strategy works as exceptation") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1","2"))
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set())
+
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("2"))
+ assert(tracker.nodeBlacklist() === Set())
+
+ tracker.updateFailureExecutors(taskInfo_3_3_hostB, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 3L) === Set("3"))
+ assert(tracker.nodeBlacklist() === Set())
+
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+ }
+
+ test ("strict strategy works as exception") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.strategy", "strict")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1"))
+ tracker.updateFailureExecutors(taskInfo_1_2_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set("1"))
+ assert(tracker.nodeBlacklist() === Set("hostA"))
+
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set("1", "2"))
+ assert(tracker.nodeBlacklist() === Set("hostA"))
+
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1","2"))
+
+ tracker.updateFailureExecutors(taskInfo_3_3_hostB, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 3L) === Set("1","2","3"))
+ assert(tracker.nodeBlacklist() === Set("hostA","hostB"))
+ }
+
+ test ("threshold strategy works as exception") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.strategy", "threshold")
+ .set("spark.scheduler.blacklist.threshold.maxFailureTaskNumber", "1")
+ .set("spark.scheduler.blacklist.threshold.maxBlackExecutorNumber", "0")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+ tracker.updateFailureExecutors(taskInfo_1_2_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set("1"))
+ assert(tracker.nodeBlacklist() === Set("hostA"))
+
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set("1"))
+ assert(tracker.nodeBlacklist() === Set("hostA"))
+
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+ }
+
+ test ("test speculate executor feature under threshold strategy") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.strategy", "threshold")
+ .set("spark.scheduler.blacklist.threshold.maxFailureTaskNumber", "1")
+ .set("spark.scheduler.blacklist.threshold.maxBlackExecutorNumber", "0")
+ .set("spark.scheduler.blacklist.speculate","true")
+ sc = new SparkContext(conf)
+ val scheduler = spy(new TaskSchedulerImpl(sc, 1))
+ when(scheduler.getExecutorsAliveOnHost("hostA")).thenReturn(Some(Set("1","2")))
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+ tracker.updateFailureExecutors(taskInfo_1_2_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set("1", "2"))
+ assert(tracker.nodeBlacklist() === Set("hostA"))
+
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set("1", "2"))
+ assert(tracker.nodeBlacklist() === Set("hostA"))
+
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+ }
+
+ test ("unsupported strategy wors as exception") {
--- End diff --
nit: "unsupported strategy throws helpful exception"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42009023
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update
+ * executorIdToFailureStatus Map.
+ */
+private[spark] class BlacklistTracker(sparkConf: SparkConf) extends WithCache{
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val executorIdToFailureStatus: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val clock = new SystemClock()
+
+ private val enableBlacklistSpeculate = sparkConf.getBoolean(
+ "spark.scheduler.blacklist.speculate", false)
+
+ private val recoverPeriod = sparkConf.getLong(
+ "spark.scheduler.blacklist.recoverPeriod", 60L)
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList(executorIdToFailureStatus))
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, recoverPeriod, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ private def expireExecutorsInBlackList(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Unit = synchronized {
+ strategy.expireExecutorsInBlackList(executorIdToFailureStatus)
+
+ invalidAllCache()
+ }
+
+ def updateFailureExecutors(info: TaskInfo, reason: TaskEndReason) : Unit = synchronized {
+ reason match {
+ // If task succeeding, remove related record from executorIdToFailureStatus
+ case Success =>
+ removeFailureExecutorsForTaskId(info.executorId, info.taskId)
+
+ // If task failing, update latest failure time and failedTaskIds
+ case _ =>
+ val executorId = info.executorId
+ executorIdToFailureStatus.get(executorId) match {
+ case Some(failureStatus) =>
+ failureStatus.updatedTime = clock.getTimeMillis()
+ val failedTimes = failureStatus.numFailuresPerTask.getOrElse(info.taskId, 0) + 1
+ failureStatus.numFailuresPerTask.update(info.taskId, failedTimes)
+ case None =>
+ val failedTasks = mutable.HashMap(info.taskId -> 1)
+ val failureStatus = new FailureStatus(
+ clock.getTimeMillis(),
+ info.host,
+ failedTasks)
--- End diff --
super nit: when *calling* methods, each arg doesn't have to go on its own line. Thats only for when defining methods. you could put all the args on one line:
```scala
val failureStatus = new FailureStatus(
clock.getTimeMillis(), info.host, failedTasks)
```
this is subjective, though, sometimes I do like to put each arg. IMO I'd put them all on the same line, but no big deal
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40279471
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update failedExecutorMap.
+ */
+class BlacklistTracker(sparkConf: SparkConf) {
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val failedExecutorMap: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val clock = new SystemClock()
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList(failedExecutorMap))
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, 60, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ private def expireExecutorsInBlackList(
+ failureExecutors: mutable.HashMap[String, FailureStatus]): Unit = synchronized {
+ strategy.expireExecutorsInBlackList(failureExecutors)
+ }
+
+ def updateFailureExecutors(info: TaskInfo, reason: TaskEndReason) : Unit = synchronized {
+ reason match {
+ // If task succeeding, remove related record from failedExecutorMap
+ case Success =>
+ removeFailureExecutors(info.executorId, Some(info.taskId))
+
+ // If task failing, update latest failure time and failedTaskIds
+ case _ =>
+ val executorId = info.executorId
+ failedExecutorMap.get(executorId) match {
+ case Some(failureStatus) =>
+ failureStatus.updatedTime = clock.getTimeMillis()
+ val failedTimes = failureStatus.failedTasks.getOrElse(info.taskId, 0) + 1
+ failureStatus.failedTasks.update(info.taskId, failedTimes)
+ case None =>
+ val failedTasks = mutable.HashMap(info.taskId -> 1)
+ val failureStatus = new FailureStatus(
+ clock.getTimeMillis(),
+ info.host,
+ failedTasks)
+ failedExecutorMap.update(executorId, failureStatus)
+ }
+ }
+ }
+
+ def removeFailureExecutors(
+ executorId: String,
+ taskId: Option[Long]) : Unit = synchronized {
+ taskId match {
--- End diff --
Yes, I will divide it into two methods
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42043304
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.util.SystemClock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+trait BlacklistStrategy {
+ val expireTimeInMillisecond: Long
+
+ // Return executors in blacklist which are related to given TaskId
+ def getExecutorBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ taskId: Long): Set[String]
+
+ // Return all nodes in blacklist
+ def getNodeBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Set[String]
+
+ // Default implementation to remove failure executors from HashMap based on given time period.
+ def expireExecutorsInBlackList(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Unit = {
+ val now = new SystemClock().getTimeMillis()
--- End diff --
this should take a `Clock` for testing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-148659469
[Test build #43838 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43838/consoleFull) for PR 8760 at commit [`cecaa49`](https://github.com/apache/spark/commit/cecaa49124ea8dcdc78db6d93e5c7d8f743df653).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-142303197
HI @mwws thanks for working on this. I think the overall approach makes sense, and this will be a big help for users on large clusters. I left some comments on the code, though I haven't gone through very detailed yet, I tried to mostly look at a higher level. Once the code is a little closer you'll also need to add a lot more comments (esp scaladoc on the new interfaces & methods) and also a lot of tests. But this is looking promising.
btw, I would add "[wip]" (for "work in progress") to the PR title for now, at least until there are unit tests added.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40285516
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -184,13 +188,21 @@ private[yarn] class YarnAllocator(
def requestTotalExecutorsWithPreferredLocalities(
requestedTotal: Int,
localityAwareTasks: Int,
- hostToLocalTaskCount: Map[String, Int]): Boolean = synchronized {
+ hostToLocalTaskCount: Map[String, Int],
+ nodeBlacklist: Set[String]): Boolean = synchronized {
--- End diff --
Sure, I will
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-171210808
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49294/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40278499
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update failedExecutorMap.
+ */
+class BlacklistTracker(sparkConf: SparkConf) {
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val failedExecutorMap: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
--- End diff --
Sure, I will
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-167728825
**[Test build #48401 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48401/consoleFull)** for PR 8760 at commit [`d94bdd7`](https://github.com/apache/spark/commit/d94bdd7d16b77317bdfe5851e9ba297a229b4113).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153271107
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40213155
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -302,7 +278,8 @@ private[spark] class TaskSetManager(
speculatableTasks.retain(index => !successful(index)) // Remove finished tasks from set
def canRunOnHost(index: Int): Boolean =
- !hasAttemptOnHost(index, host) && !executorIsBlacklisted(execId, index)
+ !hasAttemptOnHost(index, host) &&
+ !blacklistTracker.fold(false)(_.executorIsBlacklisted(execId, sched, Some(index)))
--- End diff --
nit: indent two spaces
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r46247212
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.Clock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+private [scheduler] trait BlacklistStrategy {
+ /** Define a time interval to expire failure information of executors */
+ val expireTimeInMilliseconds: Long
+
+ /** Return executors in blacklist which are related to given stage and partition */
+ def getExecutorBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ atomTask: StageAndPartition,
+ clock: Clock): Set[String]
+
+ /** Return all nodes in blacklist */
+ def getNodeBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Set[String]
+
+ /**
+ * Choose which executors should be removed from blacklist. Return true if any executors are
+ * removed from the blacklist, false otherwise. The default implementation removes executors from
+ * the blacklist after [[expireTimeInMilliseconds]]
+ */
+ def expireExecutorsInBlackList(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ clock: Clock): Boolean = {
+ val now = clock.getTimeMillis()
+ val expiredKey = executorIdToFailureStatus.filter {
+ case (executorid, failureStatus) => {
+ (now - failureStatus.updatedTime) >= expireTimeInMilliseconds
+ }
+ }.keySet
+
+ if (expiredKey.isEmpty) {
+ false
+ } else {
+ executorIdToFailureStatus --= expiredKey
+ true
+ }
+ }
+}
+
+/**
+ * This strategy adds an executor to the blacklist for all tasks when the executor has too many
+ * task failures. An executor is placed in the blacklist when there are more than
+ * [[maxFailedTasks]] failed tasks. Furthermore, all executors in one node are put into the
+ * blacklist if there are more than [[maxBlacklistedExecutors]] blacklisted executors on one node.
+ * The benefit of this strategy is that different taskSets can learn experience from other taskSet
+ * to avoid allocating tasks on problematic executors.
+ */
+private[scheduler] class ExecutorAndNodeStrategy(
+ maxFailedTasks: Int,
+ maxBlacklistedExecutors: Int,
+ val expireTimeInMilliseconds: Long
+ ) extends BlacklistStrategy {
+
+ private def getSelectedExecutorMap(
--- End diff --
This needs to be renamed to something more appropriate - filterFailedExecutors or some such - so that it is clear what it does. Selected Executor does not really convey what it is supposed to do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40224979
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.util.SystemClock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+trait BlacklistStrategy {
+ val expireTimeInMillisecond: Long
+
+ // TaskId is optional here because the BlacklistStrategy could be unrelated to TaskId
+ def getExecutorBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String]
+
+ def getNodeBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String]
+
+ // Default implementation to remove failure executors from HashMap based on given time period.
+ def expireExecutorsInBlackList(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]): Unit = {
+ val now = new SystemClock().getTimeMillis()
+ failedExecutorMap.retain((executorid, failureStatus) => {
+ (now - failureStatus.updatedTime) < expireTimeInMillisecond
+ })
--- End diff --
(a) this is modifying `failedExecutorMap` in the blacklist-expire-timer thread, while everything else is happening in the scheduler main event loop, so we need to synchronize on all access to it.
(b) I think there is a change in behavior here. `failureStatus.updatedTime` is getting updated even on successful task completion. (`taskSetManager.handleSuccessfulTask` -> `tracker.updateFailureExecutors` -> `tracker.removeFailureExecutors `). So you'll never expire an executor if it has just one failure, but then a series of successes of other tasks. I think the right thing to do is probably to not update `failureStatus.updatedTime` on task success, and maybe rename it to `lastFailureTime`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40049131
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -21,9 +21,11 @@ import java.util.Collections
import java.util.concurrent._
import java.util.regex.Pattern
+import scala.collection.JavaConversions._
+import scala.collection.JavaConverters._
--- End diff --
OK, I will remove it and use asScala/asJava explicitly
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-160864884
@kayousterhout It's an interesting discussion on PR: #3541. As I understand, this PR can satisfy both use case.
For @davies , you might want to use `ExecutorAndNodeStrategy` by change `spark.scheduler.blacklist.strategy` configuration to enable host level blacklist. By default if 3 executors on the same host are blacklisted, we think the host should also be blacklisted. Also the threshold are configurable.
For @mridulm , I tried to keep the same behavior as before in this PR, the default "SingleTaskStrategy" is what you want. It maintain an executor level blacklist for every TaskSet, and the timeout mechanism is retained. (Although `spark.scheduler.executorTaskBlacklistTime` is renamed to `spark.scheduler.blacklist.timeout`, but the old configuration name is also supported to keep backward compatibility. So nothing need to be changed from your side.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r52681082
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+import org.apache.spark.util.Clock
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update
+ * executorIdToFailureStatus Map.
+ */
+private[spark] class BlacklistTracker(
+ sparkConf: SparkConf,
+ clock: Clock = new SystemClock()) extends BlacklistCache{
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val executorIdToFailureStatus: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val recoverPeriod = sparkConf.getTimeAsSeconds(
+ "spark.scheduler.blacklist.recoverPeriod", "60s")
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList())
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, recoverPeriod, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ /** VisibleForTesting */
+ private[scheduler] def expireExecutorsInBlackList(): Unit = synchronized {
+ val updated = strategy.expireExecutorsInBlackList(executorIdToFailureStatus, clock)
+ if (updated) {
+ invalidateCache()
+ }
+ }
+
+ def updateFailedExecutors(
+ stageId: Int,
+ partation: Int,
+ info: TaskInfo,
+ reason: TaskEndReason) : Unit = synchronized {
+
+ val atomTask = StageAndPartition(stageId, partation)
+ reason match {
+ // If the task succeeded, remove related record from executorIdToFailureStatus
+ case Success =>
+ removeFailedExecutorsForTaskId(info.executorId, stageId, partation)
+
+ // If the task failed, update latest failure time and failedTaskIds
+ case _ =>
+ val executorId = info.executorId
+ executorIdToFailureStatus.get(executorId) match {
+ case Some(failureStatus) =>
+ failureStatus.updatedTime = clock.getTimeMillis()
+ val failedTimes = failureStatus.numFailuresPerTask.getOrElse(atomTask, 0) + 1
+ failureStatus.numFailuresPerTask(atomTask) = failedTimes
+ case None =>
+ val failedTasks = mutable.HashMap(atomTask -> 1)
+ val failureStatus = new FailureStatus(
+ clock.getTimeMillis(), info.host, failedTasks)
+ executorIdToFailureStatus(executorId) = failureStatus
+ }
+ invalidateCache()
+ }
+ }
+
+ /** remove the executorId from executorIdToFailureStatus */
+ def removeFailedExecutors(executorId: String) : Unit = synchronized {
+ executorIdToFailureStatus.remove(executorId)
+ invalidateCache()
+ }
+
+ /**
+ * remove the failure record related to given taskId from executorIdToFailureStatus. If the
+ * number of records of given executorId becomes 0, remove the completed executorId.
+ */
+ def removeFailedExecutorsForTaskId(
+ executorId: String,
+ stageId: Int,
+ partation: Int) : Unit = synchronized {
+ val atomTask = StageAndPartition(stageId, partation)
+ executorIdToFailureStatus.get(executorId).map{ fs =>
+ fs.numFailuresPerTask.remove(atomTask)
+ if (fs.numFailuresPerTask.isEmpty) {
+ executorIdToFailureStatus.remove(executorId)
+ }
+ invalidateCache()
+ }
+ }
+
+ def isExecutorBlacklisted(
+ executorId: String,
+ sched: TaskSchedulerImpl,
+ stageId: Int,
+ partation: Int) : Boolean = {
+
+ executorBlacklist(sched, stageId, partation).contains(executorId)
+ }
+
+ // If the node is in blacklist, all executors allocated on that node will
+ // also be put into executor blacklist.
+ private def executorsOnBlacklistedNode(
+ sched: TaskSchedulerImpl): Set[String] = {
+ nodeBlacklist().flatMap(sched.getExecutorsAliveOnHost(_)
+ .getOrElse(Set.empty[String])).toSet
+ }
+
+ def executorBlacklist(
+ sched: TaskSchedulerImpl, stageId: Int, partation: Int): Set[String] = synchronized {
+ val atomTask = StageAndPartition(stageId, partation)
+ if (!isBlacklistExecutorCacheValid) {
+ reEvaluateExecutorBlacklistAndUpdateCache(sched, atomTask, clock)
+ } else {
+ getExecutorBlacklistFromCache(atomTask).getOrElse {
+ reEvaluateExecutorBlacklistAndUpdateCache(sched, atomTask, clock)
+ }
+ }
+ }
+
+ private def reEvaluateExecutorBlacklistAndUpdateCache(
+ sched: TaskSchedulerImpl,
+ atomTask: StageAndPartition,
+ clock: Clock): Set[String] = {
+ val executors = executorsOnBlacklistedNode(sched) ++
+ strategy.getExecutorBlacklist(executorIdToFailureStatus, atomTask, clock)
+ updateBlacklistExecutorCache(atomTask, executors)
+ executors
+ }
+
+ // The actual implementation is delegated to strategy
+ def nodeBlacklist(): Set[String] = synchronized {
+ if (isBlacklistNodeCacheValid) {
+ getNodeBlacklistFromCache
+ } else {
+ val nodes = strategy.getNodeBlacklist(executorIdToFailureStatus)
+ updateBlacklistNodeCache(nodes)
+ nodes
+ }
+ }
+}
+
+/**
+ * Hide cache details in this trait to make code clean and avoid operation mistake
+ */
+private[scheduler] trait BlacklistCache {
+
+ // local cache to minimize the the work when query blacklisted executor and node
+ private val blacklistExecutorCache = mutable.HashMap.empty[StageAndPartition, Set[String]]
+ private val blacklistNodeCache = mutable.Set.empty[String]
+
+ // The flag to mark if cache is valid, it will be set to false when executorIdToFailureStatus be
+ // updated and it will be set to true, when called executorBlacklist and nodeBlacklist.
+ private var _isBlacklistExecutorCacheValid : Boolean = false
+ private var _isBlacklistNodeCacheValid: Boolean = false
+
+ private val cacheLock = new Object()
+
+ protected def isBlacklistExecutorCacheValid : Boolean = _isBlacklistExecutorCacheValid
+ protected def isBlacklistNodeCacheValid: Boolean = _isBlacklistNodeCacheValid
+
+ protected def updateBlacklistExecutorCache(
+ atomTask: StageAndPartition,
+ blacklistExecutor: Set[String]): Unit = cacheLock.synchronized {
+ if (!_isBlacklistExecutorCacheValid) {
+ blacklistExecutorCache.clear()
+ }
+ blacklistExecutorCache.update(atomTask, blacklistExecutor)
+ _isBlacklistExecutorCacheValid = true
+ }
+
+ protected def updateBlacklistNodeCache(
+ blacklistNode: Set[String]): Unit = cacheLock.synchronized {
+ if (!_isBlacklistNodeCacheValid) {
+ blacklistNodeCache.clear()
+ }
+ blacklistNodeCache ++= blacklistNode
+ _isBlacklistNodeCacheValid = true
+ }
+
+ protected def invalidateCache(): Unit = cacheLock.synchronized {
+ _isBlacklistExecutorCacheValid = false
+ _isBlacklistNodeCacheValid = false
+ }
+
+ protected def getExecutorBlacklistFromCache(
+ atomTask: StageAndPartition): Option[Set[String]] = {
+ blacklistExecutorCache.get(atomTask)
+ }
+
+ protected def getNodeBlacklistFromCache: Set[String] = blacklistNodeCache.toSet
+}
+
+/**
+ * A class to record details of failure.
+ *
+ * @param initialTime the time when failure status be created
+ * @param host the node name which running executor on
+ * @param numFailuresPerTask all tasks failed on the executor (key is StageAndPartition, value
+ * is the number of failures of this task)
+ */
+private[scheduler] final class FailureStatus(
+ initialTime: Long,
+ val host: String,
+ val numFailuresPerTask: mutable.HashMap[StageAndPartition, Int]) {
+
+ var updatedTime = initialTime
--- End diff --
I don't see the problem here. We're tracking the last time the executor was known to be bad. If two executors have the same number & type of failures, but one executor has those failures much earlier, then we'll retry that executor sooner. Its a heuristic, sure, but I don't see any fundamental flaw here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153271089
**[Test build #44892 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44892/consoleFull)** for PR 8760 at commit [`d03cc04`](https://github.com/apache/spark/commit/d03cc04c34ecd12c222d9e325eca58dfa8d5a91e).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `final class FailureStatus(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153298058
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149314021
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-151615760
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-185965488
@kayousterhout @markhamstra one other thing to follow up on was the earlier idea of using speculation to test out whether blacklisted executors can be added back. While I think its a neat _idea_, I really do not think it needs to be included here. First of all, speculation has always been somewhat poorly tested and its taken a while to root out some race conditions -- I dont' think users would want speculation turned on without explicitly requesting it, and I think we can still do useful blacklisting without requiring users to turn it on. Second, I think it could lead to lots of user confusion from the errors generated by these speculative tasks. Third, it would add a lot of complexity, and it can always be added as an option later on.
I think the other concern is how this will affect scheduler throughput. I ran the spark-perf scheduler throughput tests (just locally), and didn't see any significant performance difference. However those tests aren't really stressing this part of the code, in particular there aren't failed executors, so its probably worth thinking of another test for this. The per-task overhead introduced with the lock-less approach will just be one call to `clock.getTimeMillis()`, which is pretty negligible, plus an occasional "blip" in the scheduling when the blacklists are updated. That would be non-zero but I also doubt it would be a problem, however some profiling is in order.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40090436
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.util.SystemClock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+trait BlacklistStrategy {
+ val expireTimeInMillisecond: Long
+
+ // TaskId is optional here because the BlacklistStrategy could be unrelated to TaskId
+ def getExecutorBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String]
+
+ def getNodeBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String]
+
+ // Default implementation to remove failure executors from HashMap based on given time period.
+ def expireExecutorsInBlackList(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]): Unit = {
+ val now = new SystemClock().getTimeMillis()
+ failedExecutorMap.retain((executorid, failureStatus) => {
+ (now - failureStatus.updatedTime) < expireTimeInMillisecond
+ })
+ }
+}
+
+/**
+ * The interface for taskId unrelated strategy. In this case, provided taskId will be ignored. The
+ * benefit for taskId unrelated strategy is different taskSets can learn experience from other
+ * taskSet to avoid allocating tasks to problematic executors.
+ */
+trait TaskUnrelatedBlacklistStrategy extends BlacklistStrategy with Logging{
+ def getExecutorBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String] = {
+ if (taskId.isDefined){
+ logWarning("TaskId unrelated strategy is applied, given taskId will be ignored")
+ }
+ getExecutorBlacklist(failedExecutorMap)
+ }
+
+ def getNodeBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String] = {
+ if (taskId.isDefined){
+ logWarning("TaskId unrelated strategy is applied, given taskId will be ignored")
+ }
+ getNodeBlacklist(failedExecutorMap)
+ }
+
+ // define new interface which remove taskId from parameter.
+ def getExecutorBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]): Set[String]
+ def getNodeBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]): Set[String]
+}
+
+/**
+ * This strategy is simply based on given threshold and is taskId unrelated. An executor will be
+ * in blacklist, if it failed more than "maxFailureTaskNumber" times. A node will be in blacklist,
+ * if there are more than "maxBlackExecutorNumber" executors on it in executor blacklist.
+ */
+class SimpleStrategy(
+ maxFailureTaskNumber: Int,
+ maxBlackExecutorNumber: Int,
+ val expireTimeInMillisecond: Long
+ )extends TaskUnrelatedBlacklistStrategy {
+
+ private def getSelectedExecutorMap(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]) = {
+ failedExecutorMap.filter{
+ case (id, failureStatus) => failureStatus.failureTimes > maxFailureTaskNumber
+ }
+ }
+
+ def getExecutorBlacklist(
+ failureExecutors: mutable.HashMap[String, FailureStatus]): Set[String] = {
+ getSelectedExecutorMap(failureExecutors).keys.toSet
+ }
+
+ def getNodeBlacklist(
+ failureExecutors: mutable.HashMap[String, FailureStatus]): Set[String] = {
+ getSelectedExecutorMap(failureExecutors)
+ .groupBy{case (id, failureStatus) => failureStatus.host}
+ .filter {case (host, failureExecutors) => failureExecutors.size > maxBlackExecutorNumber}
+ .keys.toSet
+ }
+}
+
+/**
+ * This strategy is applied as default to keep the same semantics as original.
--- End diff --
This comment is helpful for me reviewing now, but I think for future readers you should describe the actual behavior :) You can add that it was the standard behavior before spark 1.6
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40048736
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -21,9 +21,11 @@ import java.util.Collections
import java.util.concurrent._
import java.util.regex.Pattern
+import scala.collection.JavaConversions._
+import scala.collection.JavaConverters._
--- End diff --
use of `JavaConversions` is discouraged -- its very easy to get some implicit conversions happening when you had no intention of converting anything. stick w/ just `JavaConverters`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-161070513
Before fixing style issues on this change, I think it's worth considering whether this is the right approach for blacklisting. Based on discussion here and in #3541 and #10045, it seems like there are a few high-level goals here:
(1) Minimize the required user configuration (ideally this should just work without any configuration)
(2) If an executor/host isn't working well, stop using it for the particular job
(3) If an executor/host is failing for multiple jobs, stop using it across all jobs
(4) Eventually re-try the executor/host, since the failure may be transient
(5) Don't overcorrect for bad tasks / jobs (that fail regardless of whether they're run)
My concern with the approach in this PR is that it requires a lot of user configuration, and as has been discussed in the various PRs, the appropriate blacklist timeout is very workload-dependent and type-of-failure-dependent. I wonder if it would make sense to do something like an exponentially increasing timeout (where each consecutive failure triggers a longer timeout) to make this have a lower configuration overhead. I pinged @mateiz to see if he has any other ideas about how to do this gracefully.
The other issue is (5). One way to handle that is to only "count" a task failure if the task fails on the executor *and* succeeds elsewhere.
It would be great if we could make blacklisting work well out-of-the-box, so I think it's worth putting some thought into the right approach here. It would be useful to get others folks' feedback about whether these are the right goals and if there are better ideas for how to achieve them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149221691
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-151689989
Hi @mwws -- I took a closer look and figured out what is going on with the tests. By looking at the full logs from the unit tests (available under "Build Artifacts" here: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44460/) I could see that all the time is getting sunk into [`AccumulatorSuite:internal accumulators in fully resubmitted stages`](https://github.com/apache/spark/blob/master/core/src/test/scala/org/apache/spark/AccumulatorSuite.scala#L250). From trying it out locally, "internal accumulators in partially resubmitted stages" also seems to take really long. Can you take a closer look at those tests and figure out why they are taking so long? You should be able to debug locally. My guess is that somehow the blacklisting is getting turned on in these tests that fail executors though it shouldn't be. (Perhaps there should even be some check that blacklisting isn't turned on when in local mode ...)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40219253
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update failedExecutorMap.
+ */
+class BlacklistTracker(sparkConf: SparkConf) {
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val failedExecutorMap: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val clock = new SystemClock()
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList(failedExecutorMap))
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, 60, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ private def expireExecutorsInBlackList(
+ failureExecutors: mutable.HashMap[String, FailureStatus]): Unit = synchronized {
+ strategy.expireExecutorsInBlackList(failureExecutors)
+ }
+
+ def updateFailureExecutors(info: TaskInfo, reason: TaskEndReason) : Unit = synchronized {
+ reason match {
+ // If task succeeding, remove related record from failedExecutorMap
+ case Success =>
+ removeFailureExecutors(info.executorId, Some(info.taskId))
+
+ // If task failing, update latest failure time and failedTaskIds
+ case _ =>
+ val executorId = info.executorId
+ failedExecutorMap.get(executorId) match {
+ case Some(failureStatus) =>
+ failureStatus.updatedTime = clock.getTimeMillis()
+ val failedTimes = failureStatus.failedTasks.getOrElse(info.taskId, 0) + 1
+ failureStatus.failedTasks.update(info.taskId, failedTimes)
+ case None =>
+ val failedTasks = mutable.HashMap(info.taskId -> 1)
+ val failureStatus = new FailureStatus(
+ clock.getTimeMillis(),
+ info.host,
+ failedTasks)
+ failedExecutorMap.update(executorId, failureStatus)
+ }
+ }
+ }
+
+ def removeFailureExecutors(
+ executorId: String,
+ taskId: Option[Long]) : Unit = synchronized {
+ taskId match {
--- End diff --
similar comment here on `taskId` -- I think this is more naturally just two different methods, one with a `taskId`, and one without, since the callsite always knows if it has a taskId or not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42049028
--- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -0,0 +1,162 @@
+package org.apache.spark.scheduler
+
+import org.apache.spark.SparkFunSuite
+import org.scalatest.BeforeAndAfter
+import org.apache.spark.SparkConf
+import org.apache.spark.TaskEndReason
+import org.apache.spark.Success
+import org.apache.spark.ExceptionFailure
+import org.apache.spark.LocalSparkContext
+import org.apache.spark.SparkContext
+
+import org.mockito.Mockito.{when,spy}
+
+class BlacklistTrackerSuite extends SparkFunSuite with BeforeAndAfter with LocalSparkContext {
+
+ val FAILURE: TaskEndReason = new ExceptionFailure(
+ "Fake",
+ "fake failure",
+ Array.empty[StackTraceElement],
+ "fake stack trace",
+ None,
+ None)
+
+ // Variable name can indicate basic information of taskInfo
+ // The format is "taskInfo_executorId_taskId_hostName"
+ val taskInfo_1_1_hostA = new TaskInfo(1L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_1_2_hostA = new TaskInfo(2L,1,1,0L,"1","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_1_hostA = new TaskInfo(1L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_2_2_hostA = new TaskInfo(2L,1,1,0L,"2","hostA", TaskLocality.ANY, false)
+ val taskInfo_3_3_hostB = new TaskInfo(3L,1,1,0L,"3","hostB", TaskLocality.ANY, false)
+
+ test ("expireExecutorsInBlacklist works ") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.recoverPeriod", "2")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.start()
+ Thread.sleep(100)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1"))
+
+ // sleep 2 second to wait for expireExecutorsInBlackList be triggered
+ Thread.sleep(2000)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+
+ tracker.stop()
+ }
+
+ test("default strategy works as exceptation") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1","2"))
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set())
+
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("2"))
+ assert(tracker.nodeBlacklist() === Set())
+
+ tracker.updateFailureExecutors(taskInfo_3_3_hostB, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 3L) === Set("3"))
+ assert(tracker.nodeBlacklist() === Set())
+
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+ }
+
+ test ("strict strategy works as exception") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.strategy", "strict")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1"))
+ tracker.updateFailureExecutors(taskInfo_1_2_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set("1"))
+ assert(tracker.nodeBlacklist() === Set("hostA"))
+
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set("1", "2"))
+ assert(tracker.nodeBlacklist() === Set("hostA"))
+
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set("1","2"))
+
+ tracker.updateFailureExecutors(taskInfo_3_3_hostB, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 3L) === Set("1","2","3"))
+ assert(tracker.nodeBlacklist() === Set("hostA","hostB"))
+ }
+
+ test ("threshold strategy works as exception") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.strategy", "threshold")
+ .set("spark.scheduler.blacklist.threshold.maxFailureTaskNumber", "1")
+ .set("spark.scheduler.blacklist.threshold.maxBlackExecutorNumber", "0")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+ tracker.updateFailureExecutors(taskInfo_1_2_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set("1"))
+ assert(tracker.nodeBlacklist() === Set("hostA"))
+
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set("1"))
+ assert(tracker.nodeBlacklist() === Set("hostA"))
+
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+ }
+
+ test ("test speculate executor feature under threshold strategy") {
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.strategy", "threshold")
+ .set("spark.scheduler.blacklist.threshold.maxFailureTaskNumber", "1")
+ .set("spark.scheduler.blacklist.threshold.maxBlackExecutorNumber", "0")
+ .set("spark.scheduler.blacklist.speculate","true")
+ sc = new SparkContext(conf)
+ val scheduler = spy(new TaskSchedulerImpl(sc, 1))
+ when(scheduler.getExecutorsAliveOnHost("hostA")).thenReturn(Some(Set("1","2")))
+
+ val tracker = new BlacklistTracker(conf)
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+ tracker.updateFailureExecutors(taskInfo_1_2_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set("1", "2"))
+ assert(tracker.nodeBlacklist() === Set("hostA"))
+
+ tracker.updateFailureExecutors(taskInfo_2_1_hostA, FAILURE)
+ assert(tracker.executorBlacklist(scheduler, 2L) === Set("1", "2"))
+ assert(tracker.nodeBlacklist() === Set("hostA"))
+
+ tracker.updateFailureExecutors(taskInfo_1_1_hostA, Success)
+ assert(tracker.executorBlacklist(scheduler, 1L) === Set())
+ }
+
+ test ("unsupported strategy wors as exception") {
+ val ex = intercept[Exception]{
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.ui.enabled","false")
+ .set("spark.scheduler.blacklist.strategy", "FAKE")
+ sc = new SparkContext(conf)
+ val scheduler = new TaskSchedulerImpl(sc, 1)
+ }
+ assert(ex.getMessage === "No match blacklist strategy for: FAKE")
--- End diff --
nit: "No matching blacklist ..."
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42004518
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.util.SystemClock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+trait BlacklistStrategy {
--- End diff --
`private[scheduler]` for all classes in this file
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149315434
**[Test build #43935 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43935/consoleFull)** for PR 8760 at commit [`a2c8b40`](https://github.com/apache/spark/commit/a2c8b4021940606f17d122dc71ff444ca47c25a5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-150693842
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-147064097
@squito , I have add the unit test and fix several bug, could you help to review and help merge into Spark?
About the important case you described above, it make sense, the question would be who and when "relaying" bad executors information to allocator. YarnSchedulerBackend might be a good info sender, it can periodically check if *(totalRegisteredExecutors - blackliskExecutors.size < minExpectedExecuors)*. Let's open another JIRA to discuss more details.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-161165662
@kayousterhout I believe in most of case user just can just use default configuration or set `spark.scheduler.blacklist.strategy` to `executorAndNode`. I also like idea about minimizing user configuration, but at the same time, it would better to provide extendability and flexibility in case the "smart" system can not satisfy all user requirements.
About "running a speculative copy of a task", it might be doable but as I understand it add much complex to maintain correct status of TaskSet/Stage which the "speculative task" belows to. What if all other tasks has finished and the "speculative task" is still running? Should the speculative task be retried as normal tasks? What if "speculative task" succeeded but original task failed? Should next stage be aware of these "speculative tasks"? etc. And from system monitoring point of view, additional duplicated tasks are created which may confuse user.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-152945048
**[Test build #44800 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44800/consoleFull)** for PR 8760 at commit [`6b7627f`](https://github.com/apache/spark/commit/6b7627f063ca7de930c8fb0802c7ae95a20fa226).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149374003
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153253232
**[Test build #44884 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44884/consoleFull)** for PR 8760 at commit [`fc992b9`](https://github.com/apache/spark/commit/fc992b90daf3c4c9ac580d6d7929452febd80ae8).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `final class FailureStatus(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153264017
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42200088
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update
+ * executorIdToFailureStatus Map.
+ */
+private[spark] class BlacklistTracker(sparkConf: SparkConf) extends WithCache{
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val executorIdToFailureStatus: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val clock = new SystemClock()
+
+ private val enableBlacklistSpeculate = sparkConf.getBoolean(
+ "spark.scheduler.blacklist.speculate", false)
+
+ private val recoverPeriod = sparkConf.getLong(
+ "spark.scheduler.blacklist.recoverPeriod", 60L)
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList(executorIdToFailureStatus))
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, recoverPeriod, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ private def expireExecutorsInBlackList(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Unit = synchronized {
+ strategy.expireExecutorsInBlackList(executorIdToFailureStatus)
+
+ invalidAllCache()
+ }
+
+ def updateFailureExecutors(info: TaskInfo, reason: TaskEndReason) : Unit = synchronized {
+ reason match {
+ // If task succeeding, remove related record from executorIdToFailureStatus
+ case Success =>
+ removeFailureExecutorsForTaskId(info.executorId, info.taskId)
+
+ // If task failing, update latest failure time and failedTaskIds
+ case _ =>
+ val executorId = info.executorId
+ executorIdToFailureStatus.get(executorId) match {
+ case Some(failureStatus) =>
+ failureStatus.updatedTime = clock.getTimeMillis()
+ val failedTimes = failureStatus.numFailuresPerTask.getOrElse(info.taskId, 0) + 1
+ failureStatus.numFailuresPerTask.update(info.taskId, failedTimes)
+ case None =>
+ val failedTasks = mutable.HashMap(info.taskId -> 1)
+ val failureStatus = new FailureStatus(
+ clock.getTimeMillis(),
+ info.host,
+ failedTasks)
+ executorIdToFailureStatus.update(executorId, failureStatus)
+ }
+ invalidAllCache()
+ }
+ }
+
+ // remove the executorId from executorIdToFailureStatus
+ def removeFailureExecutors(executorId: String) : Unit = synchronized {
+ executorIdToFailureStatus.remove(executorId)
+ invalidAllCache()
+ }
+
+ // remove the failure record related to given taskId from executorIdToFailureStatus. If the
+ // number of records of given executorId becomes 0, remove the completed executorId.
+ def removeFailureExecutorsForTaskId(
+ executorId: String,
+ taskId: Long) : Unit = synchronized {
+ executorIdToFailureStatus.get(executorId).map(fs => {
+ fs.numFailuresPerTask.remove(taskId)
+ if(fs.numFailuresPerTask.isEmpty){
+ executorIdToFailureStatus.remove(executorId)
+ }
+ invalidAllCache()
+ })
+ }
+
+ def executorIsBlacklisted(
+ executorId: String,
+ sched: TaskSchedulerImpl,
+ taskId: Long) : Boolean = {
+
+ executorBlacklist(sched, taskId).contains(executorId)
+ }
+
+ // If the node is in blacklist, all executors allocated on that node will
+ // also be put into executor blacklist.
+ // By default it's turned off, user can enable it in sparkConf.
+ private def speculationFailedExecutor(
+ sched: TaskSchedulerImpl): Set[String] = {
+ if (!enableBlacklistSpeculate) {
+ Set.empty[String]
+ } else {
+ nodeBlacklist().flatMap(sched.getExecutorsAliveOnHost(_)
+ .getOrElse(Set.empty[String])).toSet
+ }
+ }
+
+ def executorBlacklist(
+ sched: TaskSchedulerImpl,
+ taskId: Long): Set[String] = synchronized {
+ if (!isBlacklistExecutorCacheValid) {
+ reEvaluateExecutorBlacklistAndUpdateCache(sched, taskId)
+ } else {
+ getExecutorBlacklistFromCache(taskId).getOrElse {
+ reEvaluateExecutorBlacklistAndUpdateCache(sched, taskId)
+ }
+ }
+ }
+
+ private def reEvaluateExecutorBlacklistAndUpdateCache(
+ sched: TaskSchedulerImpl,
+ taskId: Long): Set[String] = {
+ val executors = speculationFailedExecutor(sched) ++
+ strategy.getExecutorBlacklist(executorIdToFailureStatus, taskId)
+ updateBlacklistExecutorCache(taskId, executors)
+ executors
+ }
+
+ // The actual implementation is delegated to strategy
+ def nodeBlacklist(): Set[String] = synchronized {
+ if (isBlacklistNodeCacheValid) {
+ getNodeBlacklistFromCache
+ } else {
+ val nodes = strategy.getNodeBlacklist(executorIdToFailureStatus)
+ updateBlacklistNodeCache(nodes)
+ nodes
+ }
+ }
+}
+
+// Hide cache details in this trait to make code clean and avoid operation mistake
+trait WithCache {
--- End diff --
Naming is really the hardest problem in programming :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149078384
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43908/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-150695289
**[Test build #44259 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44259/consoleFull)** for PR 8760 at commit [`a2c8b40`](https://github.com/apache/spark/commit/a2c8b4021940606f17d122dc71ff444ca47c25a5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40279615
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
@@ -568,12 +568,12 @@ private[spark] class ApplicationMaster(
}
override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
- case RequestExecutors(requestedTotal, localityAwareTasks, hostToLocalTaskCount) =>
+ case RequestExecutors(requestedTotal, localityAwareTasks, hostToLocalTaskCount, nodeBalcklist) =>
--- End diff --
Shame. nice catch!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40048793
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.util.SystemClock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+trait BlacklistStrategy {
+ val expireTimeInMillisecond: Long
+
+ // TaskId is optional here because the BlacklistStrategy could be unrelated to TaskId
--- End diff --
you mean, implementations are free to ignore the taskID -- but the caller *must* provide the taskID, right? so it should not be an `Option[Long]` in the signature, it should just be a `Long` and the implementations can ignore.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-167052293
@squito @mridulm I am afraid an additional state can not help here, your use case requires "TaskSet related nodeBlacklist" which is not supported with current implementation. I am making change to support it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149302201
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149566458
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-153935218
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-170445768
**[Test build #49099 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49099/consoleFull)** for PR 8760 at commit [`d94bdd7`](https://github.com/apache/spark/commit/d94bdd7d16b77317bdfe5851e9ba297a229b4113).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149564532
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149221631
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-150732980
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44259/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-185602366
@chenghao-intel Sure, I will update soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42049087
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
@@ -568,12 +568,16 @@ private[spark] class ApplicationMaster(
}
override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
- case RequestExecutors(requestedTotal, localityAwareTasks, hostToLocalTaskCount) =>
+ case RequestExecutors(
+ requestedTotal,
+ localityAwareTasks,
+ hostToLocalTaskCount,
+ nodeBlacklist) =>
--- End diff --
same here for each arg on its own line -- no big deal, but IMO I'd put them on the same line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40088359
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update failedExecutorMap.
+ */
+class BlacklistTracker(sparkConf: SparkConf) {
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val failedExecutorMap: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val clock = new SystemClock()
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList(failedExecutorMap))
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, 60, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ private def expireExecutorsInBlackList(
+ failureExecutors: mutable.HashMap[String, FailureStatus]): Unit = synchronized {
+ strategy.expireExecutorsInBlackList(failureExecutors)
+ }
+
+ def updateFailureExecutors(info: TaskInfo, reason: TaskEndReason) : Unit = synchronized {
+ reason match {
+ // If task succeeding, remove related record from failedExecutorMap
+ case Success =>
+ removeFailureExecutors(info.executorId, Some(info.taskId))
+
+ // If task failing, update latest failure time and failedTaskIds
+ case _ =>
+ val executorId = info.executorId
+ failedExecutorMap.get(executorId) match {
+ case Some(failureStatus) =>
+ failureStatus.updatedTime = clock.getTimeMillis()
+ val failedTimes = failureStatus.failedTasks.getOrElse(info.taskId, 0) + 1
+ failureStatus.failedTasks.update(info.taskId, failedTimes)
+ case None =>
+ val failedTasks = mutable.HashMap(info.taskId -> 1)
+ val failureStatus = new FailureStatus(
+ clock.getTimeMillis(),
+ info.host,
+ failedTasks)
+ failedExecutorMap.update(executorId, failureStatus)
+ }
+ }
+ }
+
+ def removeFailureExecutors(
+ executorId: String,
+ taskId: Option[Long]) : Unit = synchronized {
+ taskId match {
+ // If taskId is provided, remove the taskId from failedTasks
+ case Some(id) =>
+ failedExecutorMap.get(executorId).map(fs => {
+ fs.updatedTime = clock.getTimeMillis()
+ fs.failedTasks.remove(id)
+ if(fs.failedTasks.isEmpty){
+ failedExecutorMap.remove(executorId)
+ }
+ })
+ // if taskId is not provided, remove the whole record related to given executorId
+ case None =>
+ failedExecutorMap.remove(executorId)
+ }
+ }
+
+ def executorIsBlacklisted(
+ executorId: String,
+ sched: TaskSchedulerImpl,
+ taskId: Option[Long]) : Boolean = {
+
+ executorBlacklist(sched, taskId).contains(executorId)
+ }
+
+ // The actual implementation is delegated to strategy
+ def executorBlacklist(
+ sched: TaskSchedulerImpl,
+ taskId: Option[Long]): Set[String] = synchronized {
+
+ // If the node is in blacklist, all executors allocated on that node will
+ // also be put into executor blacklist.
+ // By default it's turned off, user can enable it in sparkConf.
+ val speculationFailedExecutor: Set[String] =
+ if (sparkConf.getBoolean("spark.scheduler.blacklist.speculate", false)) {
+ Set.empty[String]
+ } else {
+ nodeBlacklist(taskId).flatMap(sched.getExecutorsAliveOnHost(_)
+ .getOrElse(Set.empty[String])).toSet
+ }
+
+ speculationFailedExecutor ++ strategy.getExecutorBlacklist(failedExecutorMap, taskId)
+ }
+
+ // The actual implementation is delegated to strategy
+ def nodeBlacklist(taskId: Option[Long] = None): Set[String] = synchronized {
+ strategy.getNodeBlacklist(failedExecutorMap, taskId)
+ }
+}
+
+/**
+ * A class to record details of failure.
+ *
+ * @param initialTime the time when failure status be created
+ * @param host the node name which running executor on
+ * @param failedTasks all tasks failed on the executor (key is task id,
+ * value is failure time of this task)
+ */
+final class FailureStatus(
+ initialTime: Long,
+ val host: String,
+ val failedTasks: mutable.HashMap[Long, Int]) {
--- End diff --
probably just a stupid mistake on my part, but when I first read this I thought it was the (clock) time at which the failure occurred. can we rename it to something like `numFailuresPerTask`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-148207378
Jenkins, ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40222054
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.util.SystemClock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+trait BlacklistStrategy {
+ val expireTimeInMillisecond: Long
+
+ // TaskId is optional here because the BlacklistStrategy could be unrelated to TaskId
+ def getExecutorBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String]
+
+ def getNodeBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String]
+
+ // Default implementation to remove failure executors from HashMap based on given time period.
+ def expireExecutorsInBlackList(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]): Unit = {
+ val now = new SystemClock().getTimeMillis()
+ failedExecutorMap.retain((executorid, failureStatus) => {
+ (now - failureStatus.updatedTime) < expireTimeInMillisecond
+ })
+ }
+}
+
+/**
+ * The interface for taskId unrelated strategy. In this case, provided taskId will be ignored. The
+ * benefit for taskId unrelated strategy is different taskSets can learn experience from other
+ * taskSet to avoid allocating tasks to problematic executors.
+ */
+trait TaskUnrelatedBlacklistStrategy extends BlacklistStrategy with Logging{
+ def getExecutorBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String] = {
+ if (taskId.isDefined){
+ logWarning("TaskId unrelated strategy is applied, given taskId will be ignored")
+ }
+ getExecutorBlacklist(failedExecutorMap)
+ }
+
+ def getNodeBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String] = {
+ if (taskId.isDefined){
+ logWarning("TaskId unrelated strategy is applied, given taskId will be ignored")
+ }
+ getNodeBlacklist(failedExecutorMap)
+ }
+
+ // define new interface which remove taskId from parameter.
+ def getExecutorBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]): Set[String]
+ def getNodeBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]): Set[String]
+}
+
+/**
+ * This strategy is simply based on given threshold and is taskId unrelated. An executor will be
+ * in blacklist, if it failed more than "maxFailureTaskNumber" times. A node will be in blacklist,
+ * if there are more than "maxBlackExecutorNumber" executors on it in executor blacklist.
+ */
+class SimpleStrategy(
+ maxFailureTaskNumber: Int,
+ maxBlackExecutorNumber: Int,
+ val expireTimeInMillisecond: Long
+ )extends TaskUnrelatedBlacklistStrategy {
+
+ private def getSelectedExecutorMap(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]) = {
+ failedExecutorMap.filter{
+ case (id, failureStatus) => failureStatus.failureTimes > maxFailureTaskNumber
+ }
+ }
+
+ def getExecutorBlacklist(
+ failureExecutors: mutable.HashMap[String, FailureStatus]): Set[String] = {
+ getSelectedExecutorMap(failureExecutors).keys.toSet
+ }
+
+ def getNodeBlacklist(
+ failureExecutors: mutable.HashMap[String, FailureStatus]): Set[String] = {
--- End diff --
ah, I was thinking before that the runtime would be proportional to the number of executors, but its really just dependent on the executors w/ failed tasks. Still I think it is probably good to minimize the work done here if we can and its not too hard. For instance, one easy thing is just avoiding querying the sparkConf inside this function ( currently there is `sparkConf.getBoolean("spark.scheduler.blacklist.speculate", false)`).
I see now that it is a little hard to do the caching, given the separation between `BlacklistTracker` and `BlacklistStrategy`, since `BlacklistStrategy` is responsible for updating `failureExecutors`, and there is also a time-expiration as well. I dont' see how a `isChanged` boolean is enough to manage this, but maybe I'm wrong -- happy to look at the code. I was thinking you would need a `failedExecutorVersion: Long`, which gets incremented by the tracker if its ever updated, but maybe there is something simpler.
I will think about this some more ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40088646
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update failedExecutorMap.
+ */
+class BlacklistTracker(sparkConf: SparkConf) {
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val failedExecutorMap: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val clock = new SystemClock()
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList(failedExecutorMap))
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, 60, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ private def expireExecutorsInBlackList(
+ failureExecutors: mutable.HashMap[String, FailureStatus]): Unit = synchronized {
+ strategy.expireExecutorsInBlackList(failureExecutors)
+ }
+
+ def updateFailureExecutors(info: TaskInfo, reason: TaskEndReason) : Unit = synchronized {
+ reason match {
+ // If task succeeding, remove related record from failedExecutorMap
+ case Success =>
+ removeFailureExecutors(info.executorId, Some(info.taskId))
+
+ // If task failing, update latest failure time and failedTaskIds
+ case _ =>
+ val executorId = info.executorId
+ failedExecutorMap.get(executorId) match {
+ case Some(failureStatus) =>
+ failureStatus.updatedTime = clock.getTimeMillis()
+ val failedTimes = failureStatus.failedTasks.getOrElse(info.taskId, 0) + 1
+ failureStatus.failedTasks.update(info.taskId, failedTimes)
+ case None =>
+ val failedTasks = mutable.HashMap(info.taskId -> 1)
+ val failureStatus = new FailureStatus(
+ clock.getTimeMillis(),
+ info.host,
+ failedTasks)
+ failedExecutorMap.update(executorId, failureStatus)
+ }
+ }
+ }
+
+ def removeFailureExecutors(
+ executorId: String,
+ taskId: Option[Long]) : Unit = synchronized {
+ taskId match {
+ // If taskId is provided, remove the taskId from failedTasks
+ case Some(id) =>
+ failedExecutorMap.get(executorId).map(fs => {
+ fs.updatedTime = clock.getTimeMillis()
+ fs.failedTasks.remove(id)
+ if(fs.failedTasks.isEmpty){
+ failedExecutorMap.remove(executorId)
+ }
+ })
+ // if taskId is not provided, remove the whole record related to given executorId
+ case None =>
+ failedExecutorMap.remove(executorId)
+ }
+ }
+
+ def executorIsBlacklisted(
+ executorId: String,
+ sched: TaskSchedulerImpl,
+ taskId: Option[Long]) : Boolean = {
+
+ executorBlacklist(sched, taskId).contains(executorId)
+ }
+
+ // The actual implementation is delegated to strategy
+ def executorBlacklist(
+ sched: TaskSchedulerImpl,
+ taskId: Option[Long]): Set[String] = synchronized {
+
+ // If the node is in blacklist, all executors allocated on that node will
+ // also be put into executor blacklist.
+ // By default it's turned off, user can enable it in sparkConf.
+ val speculationFailedExecutor: Set[String] =
+ if (sparkConf.getBoolean("spark.scheduler.blacklist.speculate", false)) {
+ Set.empty[String]
+ } else {
+ nodeBlacklist(taskId).flatMap(sched.getExecutorsAliveOnHost(_)
+ .getOrElse(Set.empty[String])).toSet
+ }
+
+ speculationFailedExecutor ++ strategy.getExecutorBlacklist(failedExecutorMap, taskId)
+ }
+
+ // The actual implementation is delegated to strategy
+ def nodeBlacklist(taskId: Option[Long] = None): Set[String] = synchronized {
+ strategy.getNodeBlacklist(failedExecutorMap, taskId)
+ }
+}
+
+/**
+ * A class to record details of failure.
+ *
+ * @param initialTime the time when failure status be created
+ * @param host the node name which running executor on
+ * @param failedTasks all tasks failed on the executor (key is task id,
+ * value is failure time of this task)
+ */
+final class FailureStatus(
+ initialTime: Long,
+ val host: String,
+ val failedTasks: mutable.HashMap[Long, Int]) {
+
+ var updatedTime = initialTime
+ def failureTimes : Int = failedTasks.values.sum
--- End diff --
and then rename this to `totalNumFailures` or something like that
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149640764
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43977/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42004674
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.util.SystemClock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+trait BlacklistStrategy {
+ val expireTimeInMillisecond: Long
+
+ // Return executors in blacklist which are related to given TaskId
+ def getExecutorBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus],
+ taskId: Long): Set[String]
+
+ // Return all nodes in blacklist
+ def getNodeBlacklist(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Set[String]
+
+ // Default implementation to remove failure executors from HashMap based on given time period.
--- End diff --
can you change all of these to doc comments?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-143099031
I don't think I was totally clear about the suggestion for immediately killing executors. I definitely did not mean that `BlacklistTracker` should be making direct calls to yarn. But I do think it needs some way of "relaying" that info on to appropriate allocators. It could "suggest" that the allocator kill the executor (and request a new one), and the allocator is free to do what it likes with that suggestion (eg., it could ignore completely). Meanwhile, the tracker would blacklist the executor, and if eventually the allocator does decide to remove it, then it can stop tracking that executor.
I think this is an important case even without dynamic allocation. If you just request a static 5 executors, you're really want 5 good executors, not 5 executors with some mix of good and bad ones.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40285964
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -58,15 +58,6 @@ private[spark] class TaskSetManager(
val conf = sched.sc.conf
- /*
- * Sometimes if an executor is dead or in an otherwise invalid state, the driver
- * does not realize right away leading to repeated task failures. If enabled,
- * this temporarily prevents a task from re-launching on an executor where
- * it just failed.
- */
- private val EXECUTOR_TASK_BLACKLIST_TIMEOUT =
- conf.getLong("spark.scheduler.executorTaskBlacklistTime", 0L)
--- End diff --
Also I will make the default value the same as here "0L"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40231288
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update failedExecutorMap.
+ */
+class BlacklistTracker(sparkConf: SparkConf) {
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val failedExecutorMap: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val clock = new SystemClock()
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList(failedExecutorMap))
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, 60, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ private def expireExecutorsInBlackList(
+ failureExecutors: mutable.HashMap[String, FailureStatus]): Unit = synchronized {
+ strategy.expireExecutorsInBlackList(failureExecutors)
+ }
+
+ def updateFailureExecutors(info: TaskInfo, reason: TaskEndReason) : Unit = synchronized {
+ reason match {
+ // If task succeeding, remove related record from failedExecutorMap
+ case Success =>
+ removeFailureExecutors(info.executorId, Some(info.taskId))
+
+ // If task failing, update latest failure time and failedTaskIds
+ case _ =>
+ val executorId = info.executorId
+ failedExecutorMap.get(executorId) match {
+ case Some(failureStatus) =>
+ failureStatus.updatedTime = clock.getTimeMillis()
+ val failedTimes = failureStatus.failedTasks.getOrElse(info.taskId, 0) + 1
+ failureStatus.failedTasks.update(info.taskId, failedTimes)
+ case None =>
+ val failedTasks = mutable.HashMap(info.taskId -> 1)
+ val failureStatus = new FailureStatus(
+ clock.getTimeMillis(),
+ info.host,
+ failedTasks)
+ failedExecutorMap.update(executorId, failureStatus)
+ }
+ }
+ }
+
+ def removeFailureExecutors(
+ executorId: String,
+ taskId: Option[Long]) : Unit = synchronized {
+ taskId match {
+ // If taskId is provided, remove the taskId from failedTasks
+ case Some(id) =>
+ failedExecutorMap.get(executorId).map(fs => {
+ fs.updatedTime = clock.getTimeMillis()
+ fs.failedTasks.remove(id)
+ if(fs.failedTasks.isEmpty){
+ failedExecutorMap.remove(executorId)
+ }
+ })
+ // if taskId is not provided, remove the whole record related to given executorId
+ case None =>
+ failedExecutorMap.remove(executorId)
+ }
+ }
+
+ def executorIsBlacklisted(
+ executorId: String,
+ sched: TaskSchedulerImpl,
+ taskId: Option[Long]) : Boolean = {
+
+ executorBlacklist(sched, taskId).contains(executorId)
+ }
+
+ // The actual implementation is delegated to strategy
+ def executorBlacklist(
+ sched: TaskSchedulerImpl,
+ taskId: Option[Long]): Set[String] = synchronized {
+
+ // If the node is in blacklist, all executors allocated on that node will
+ // also be put into executor blacklist.
+ // By default it's turned off, user can enable it in sparkConf.
+ val speculationFailedExecutor: Set[String] =
+ if (sparkConf.getBoolean("spark.scheduler.blacklist.speculate", false)) {
+ Set.empty[String]
+ } else {
+ nodeBlacklist(taskId).flatMap(sched.getExecutorsAliveOnHost(_)
+ .getOrElse(Set.empty[String])).toSet
+ }
--- End diff --
actually looking at this in more detail now -- I don't really see the point of this conf option. To me it seems more natural to just always have the executorBlacklist include the nodeBlacklist. I imagine that you would always want to exclude all executors on a blacklisted node. If you really want to make this configurable, it could be left to the strategy to figure out. (That is, the function just calls `strategy.getExecutorBlacklist`, and most strategies would choose to include all blacklisted nodes in this as well.)
But unless there is a good use case, maybe we should just remove the option completely?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40225407
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -58,15 +58,6 @@ private[spark] class TaskSetManager(
val conf = sched.sc.conf
- /*
- * Sometimes if an executor is dead or in an otherwise invalid state, the driver
- * does not realize right away leading to repeated task failures. If enabled,
- * this temporarily prevents a task from re-launching on an executor where
- * it just failed.
- */
- private val EXECUTOR_TASK_BLACKLIST_TIMEOUT =
- conf.getLong("spark.scheduler.executorTaskBlacklistTime", 0L)
--- End diff --
we need to be sure to still support this conf name. In fact I think you can just stick w/ this instead of switching to "spark.scheduler.blacklist.timeout", right? perhaps that conf should only be for strategies which do per-task blacklisting, but I feel like it could be for all cases.
(In any case, I think we defer these exact naming decisions till we're closer on the implementation, just mentioning it now while I'm noticing it.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r42009885
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkConf
+import org.apache.spark.Success
+import org.apache.spark.TaskEndReason
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.Utils
+
+
+/**
+ * BlacklistTracker is design to track problematic executors and node on application level.
+ * It is shared by all TaskSet, so that once a new TaskSet coming, it could be benefit from
+ * previous experience of other TaskSet.
+ *
+ * Once task finished, the callback method in TaskSetManager should update
+ * executorIdToFailureStatus Map.
+ */
+private[spark] class BlacklistTracker(sparkConf: SparkConf) extends WithCache{
+ // maintain a ExecutorId --> FailureStatus HashMap
+ private val executorIdToFailureStatus: mutable.HashMap[String, FailureStatus] = mutable.HashMap()
+
+ // Apply Strategy pattern here to change different blacklist detection logic
+ private val strategy = BlacklistStrategy(sparkConf)
+
+ // A daemon thread to expire blacklist executor periodically
+ private val scheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor(
+ "spark-scheduler-blacklist-expire-timer")
+
+ private val clock = new SystemClock()
+
+ private val enableBlacklistSpeculate = sparkConf.getBoolean(
+ "spark.scheduler.blacklist.speculate", false)
+
+ private val recoverPeriod = sparkConf.getLong(
+ "spark.scheduler.blacklist.recoverPeriod", 60L)
+
+ def start(): Unit = {
+ val scheduleTask = new Runnable() {
+ override def run(): Unit = {
+ Utils.logUncaughtExceptions(expireExecutorsInBlackList(executorIdToFailureStatus))
+ }
+ }
+ scheduler.scheduleAtFixedRate(scheduleTask, 0L, recoverPeriod, TimeUnit.SECONDS)
+ }
+
+ def stop(): Unit = {
+ scheduler.shutdown()
+ scheduler.awaitTermination(10, TimeUnit.SECONDS)
+ }
+
+ // The actual implementation is delegated to strategy
+ private def expireExecutorsInBlackList(
+ executorIdToFailureStatus: mutable.HashMap[String, FailureStatus]): Unit = synchronized {
+ strategy.expireExecutorsInBlackList(executorIdToFailureStatus)
+
+ invalidAllCache()
+ }
+
+ def updateFailureExecutors(info: TaskInfo, reason: TaskEndReason) : Unit = synchronized {
+ reason match {
+ // If task succeeding, remove related record from executorIdToFailureStatus
+ case Success =>
+ removeFailureExecutorsForTaskId(info.executorId, info.taskId)
+
+ // If task failing, update latest failure time and failedTaskIds
+ case _ =>
+ val executorId = info.executorId
+ executorIdToFailureStatus.get(executorId) match {
+ case Some(failureStatus) =>
+ failureStatus.updatedTime = clock.getTimeMillis()
+ val failedTimes = failureStatus.numFailuresPerTask.getOrElse(info.taskId, 0) + 1
+ failureStatus.numFailuresPerTask.update(info.taskId, failedTimes)
+ case None =>
+ val failedTasks = mutable.HashMap(info.taskId -> 1)
+ val failureStatus = new FailureStatus(
+ clock.getTimeMillis(),
+ info.host,
+ failedTasks)
+ executorIdToFailureStatus.update(executorId, failureStatus)
+ }
+ invalidAllCache()
+ }
+ }
+
+ // remove the executorId from executorIdToFailureStatus
+ def removeFailureExecutors(executorId: String) : Unit = synchronized {
+ executorIdToFailureStatus.remove(executorId)
+ invalidAllCache()
+ }
+
+ // remove the failure record related to given taskId from executorIdToFailureStatus. If the
+ // number of records of given executorId becomes 0, remove the completed executorId.
+ def removeFailureExecutorsForTaskId(
+ executorId: String,
+ taskId: Long) : Unit = synchronized {
+ executorIdToFailureStatus.get(executorId).map(fs => {
+ fs.numFailuresPerTask.remove(taskId)
+ if(fs.numFailuresPerTask.isEmpty){
+ executorIdToFailureStatus.remove(executorId)
+ }
+ invalidAllCache()
+ })
+ }
+
+ def executorIsBlacklisted(
+ executorId: String,
+ sched: TaskSchedulerImpl,
+ taskId: Long) : Boolean = {
+
+ executorBlacklist(sched, taskId).contains(executorId)
+ }
+
+ // If the node is in blacklist, all executors allocated on that node will
+ // also be put into executor blacklist.
+ // By default it's turned off, user can enable it in sparkConf.
+ private def speculationFailedExecutor(
+ sched: TaskSchedulerImpl): Set[String] = {
+ if (!enableBlacklistSpeculate) {
+ Set.empty[String]
+ } else {
+ nodeBlacklist().flatMap(sched.getExecutorsAliveOnHost(_)
+ .getOrElse(Set.empty[String])).toSet
+ }
+ }
+
+ def executorBlacklist(
+ sched: TaskSchedulerImpl,
+ taskId: Long): Set[String] = synchronized {
+ if (!isBlacklistExecutorCacheValid) {
+ reEvaluateExecutorBlacklistAndUpdateCache(sched, taskId)
+ } else {
+ getExecutorBlacklistFromCache(taskId).getOrElse {
+ reEvaluateExecutorBlacklistAndUpdateCache(sched, taskId)
+ }
+ }
+ }
+
+ private def reEvaluateExecutorBlacklistAndUpdateCache(
+ sched: TaskSchedulerImpl,
+ taskId: Long): Set[String] = {
+ val executors = speculationFailedExecutor(sched) ++
+ strategy.getExecutorBlacklist(executorIdToFailureStatus, taskId)
+ updateBlacklistExecutorCache(taskId, executors)
+ executors
+ }
+
+ // The actual implementation is delegated to strategy
+ def nodeBlacklist(): Set[String] = synchronized {
+ if (isBlacklistNodeCacheValid) {
+ getNodeBlacklistFromCache
+ } else {
+ val nodes = strategy.getNodeBlacklist(executorIdToFailureStatus)
+ updateBlacklistNodeCache(nodes)
+ nodes
+ }
+ }
+}
+
+// Hide cache details in this trait to make code clean and avoid operation mistake
+trait WithCache {
--- End diff --
can you rename to `BlacklistCache`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #8760: [Spark-8426] [scheduler] enhance blacklist mechanism
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/8760
This is made obsolete by #14079; @mwws could you close the PR please?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40164712
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.util.SystemClock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+trait BlacklistStrategy {
+ val expireTimeInMillisecond: Long
+
+ // TaskId is optional here because the BlacklistStrategy could be unrelated to TaskId
+ def getExecutorBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String]
+
+ def getNodeBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String]
+
+ // Default implementation to remove failure executors from HashMap based on given time period.
+ def expireExecutorsInBlackList(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]): Unit = {
+ val now = new SystemClock().getTimeMillis()
+ failedExecutorMap.retain((executorid, failureStatus) => {
+ (now - failureStatus.updatedTime) < expireTimeInMillisecond
+ })
+ }
+}
+
+/**
+ * The interface for taskId unrelated strategy. In this case, provided taskId will be ignored. The
+ * benefit for taskId unrelated strategy is different taskSets can learn experience from other
+ * taskSet to avoid allocating tasks to problematic executors.
+ */
+trait TaskUnrelatedBlacklistStrategy extends BlacklistStrategy with Logging{
+ def getExecutorBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String] = {
+ if (taskId.isDefined){
+ logWarning("TaskId unrelated strategy is applied, given taskId will be ignored")
+ }
+ getExecutorBlacklist(failedExecutorMap)
+ }
+
+ def getNodeBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String] = {
+ if (taskId.isDefined){
+ logWarning("TaskId unrelated strategy is applied, given taskId will be ignored")
+ }
+ getNodeBlacklist(failedExecutorMap)
+ }
+
+ // define new interface which remove taskId from parameter.
+ def getExecutorBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]): Set[String]
+ def getNodeBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]): Set[String]
+}
+
+/**
+ * This strategy is simply based on given threshold and is taskId unrelated. An executor will be
+ * in blacklist, if it failed more than "maxFailureTaskNumber" times. A node will be in blacklist,
+ * if there are more than "maxBlackExecutorNumber" executors on it in executor blacklist.
+ */
+class SimpleStrategy(
+ maxFailureTaskNumber: Int,
+ maxBlackExecutorNumber: Int,
+ val expireTimeInMillisecond: Long
+ )extends TaskUnrelatedBlacklistStrategy {
+
+ private def getSelectedExecutorMap(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]) = {
+ failedExecutorMap.filter{
+ case (id, failureStatus) => failureStatus.failureTimes > maxFailureTaskNumber
+ }
+ }
+
+ def getExecutorBlacklist(
+ failureExecutors: mutable.HashMap[String, FailureStatus]): Set[String] = {
+ getSelectedExecutorMap(failureExecutors).keys.toSet
+ }
+
+ def getNodeBlacklist(
+ failureExecutors: mutable.HashMap[String, FailureStatus]): Set[String] = {
+ getSelectedExecutorMap(failureExecutors)
+ .groupBy{case (id, failureStatus) => failureStatus.host}
+ .filter {case (host, failureExecutors) => failureExecutors.size > maxBlackExecutorNumber}
+ .keys.toSet
+ }
+}
+
+/**
+ * This strategy is applied as default to keep the same semantics as original.
--- End diff --
Sure, I will enhance the comment
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149076636
[Test build #43908 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43908/consoleFull) for PR 8760 at commit [`a2c8b40`](https://github.com/apache/spark/commit/a2c8b4021940606f17d122dc71ff444ca47c25a5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-149374004
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43935/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-154347405
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45200/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-152944165
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-187586614
**[Test build #51737 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51737/consoleFull)** for PR 8760 at commit [`ecad5ff`](https://github.com/apache/spark/commit/ecad5ffe2bcc02f9b181a209172f22294e6af4ef).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-154329217
@squito I have rebased this branch to the latest spark/master branch, so there should be no conflict to merge. About the new wiki page of BlacklistStrategy, could we first close this Jira and create a new one to follow up?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by mwws <gi...@git.apache.org>.
Github user mwws commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r40277864
--- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistStrategy.scala ---
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkConf
+import org.apache.spark.util.SystemClock
+
+/**
+ * The interface to determine executor blacklist and node blacklist.
+ */
+trait BlacklistStrategy {
+ val expireTimeInMillisecond: Long
+
+ // TaskId is optional here because the BlacklistStrategy could be unrelated to TaskId
+ def getExecutorBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String]
+
+ def getNodeBlacklist(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus],
+ taskId: Option[Long]): Set[String]
+
+ // Default implementation to remove failure executors from HashMap based on given time period.
+ def expireExecutorsInBlackList(
+ failedExecutorMap: mutable.HashMap[String, FailureStatus]): Unit = {
+ val now = new SystemClock().getTimeMillis()
+ failedExecutorMap.retain((executorid, failureStatus) => {
+ (now - failureStatus.updatedTime) < expireTimeInMillisecond
+ })
--- End diff --
For (b), Yes, there is logical flaw, and I agree with your suggestion. Thanks for point it out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-150732979
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org