You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by squito <gi...@git.apache.org> on 2016/06/10 16:55:31 UTC
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/13603
[SPARK-15865][CORE] Blacklist should not result in job hanging with less than 4 executors
## What changes were proposed in this pull request?
Before this change, when you turn on blacklisting with {{spark.scheduler.executorTaskBlacklistTime}}, but you have fewer than {{spark.task.maxFailures}} executors, you can end with a job "hung" after some task failures.
Whenever a taskset is unable to schedule anything on resourceOfferSingleTaskSet, we check whether the last pending task can be scheduled on *any* known executor. If not, the taskset (and any corresponding jobs) are failed.
* Worst case, this is O(numExecutors). But unless many executors are bad, this should be small
* This does not fail as fast as possible -- when a task becomes unschedulable, we keep scheduling other tasks. This is to avoid an O(numPendingTasks) operation
* Also, it is conceivable this fails too quickly. You may be 1 millisecond away from unblacklisting a place for a task to run, or acquiring a new executor.
Also, I scratched an itch and tightened up visibility throughout `TaskSetManager`. Can undo that if it clutters this change.
## How was this patch tested?
Added unit tests, ran via jenkins.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/squito/spark progress_w_few_execs_and_blacklist
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13603.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13603
----
commit 3f462750c1e99d9357106526ec724cebe639c561
Author: Imran Rashid <ir...@cloudera.com>
Date: 2016-06-10T05:23:03Z
if all executors have been blacklisted for a ask, abort stage (instead of just hanging)
commit bc80e8ceeee180c37fa9211ee1b85ce0a7a90ac7
Author: Imran Rashid <ir...@cloudera.com>
Date: 2016-06-10T06:43:23Z
tighten visibility throughout TaskSetManager
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r67773405
--- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -27,18 +29,48 @@ class FakeSchedulerBackend extends SchedulerBackend {
def defaultParallelism(): Int = 1
}
-class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext with Logging {
+class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext with BeforeAndAfterEach
+ with Logging {
- test("Scheduler does not always schedule tasks on the same workers") {
+
+ var failedTaskSetExc: Option[Throwable] = None
+ var failedTaskSetReason: String = null
+ var failedTaskSet = false
+
+ override def beforeEach(): Unit = {
+ super.beforeEach()
+ failedTaskSet = false
+ failedTaskSetExc = None
--- End diff --
super nit but could you write this out as "Exception"? I kept mentally reading it as "Executor" -- and it doesn't look like it will throw any lines over 100
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60964/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61061/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the issue:
https://github.com/apache/spark/pull/13603
actually just realized `TaskInfo.index` is *not* the same as `partitionId`, so I'll revert https://github.com/apache/spark/pull/13603/commits/6362b28468e752a61560bffdb2f383c69163ac36 and update
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #60952 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60952/consoleFull)** for PR 13603 at commit [`c1aabea`](https://github.com/apache/spark/commit/c1aabea4bdfceb1e528b7146faf062758b9493a1).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r68809060
--- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistIntegrationSuite.scala ---
@@ -91,6 +91,31 @@ class BlacklistIntegrationSuite extends SchedulerIntegrationSuite[MultiExecutorM
assertDataStructuresEmpty(noFailure = true)
}
+ // Make sure that if we've failed on all executors, but haven't hit task.maxFailures yet, the job
+ // doesn't hang
+ testScheduler(
+ "SPARK-15865 Progress with fewer executors than maxTaskFailures",
+ extraConfs = Seq(
+ // set this to something much longer than the test duration
+ "spark.scheduler.executorTaskBlacklistTime" -> "10000000",
+ "spark.testing.nHosts" -> "2",
+ "spark.testing.nExecutorsPerHost" -> "1",
+ "spark.testing.nCoresPerExecutor" -> "1"
+ )
+ ) {
+ def runBackend(): Unit = {
+ val (taskDescription, _) = backend.beginTask()
+ backend.taskFailed(taskDescription, new RuntimeException("test task failure"))
+ }
+ withBackend(runBackend _) {
+ val jobFuture = submit(new MockRDD(sc, 10, Nil), (0 until 10).toArray)
+ Await.ready(jobFuture, duration)
+ val pattern = ("Aborting TaskSet 0.0 because Task .* " +
+ "cannot be scheduled on any executor due to blacklists").r
--- End diff --
(just noting you'll need to update this if you change the message above)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r68809604
--- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -281,6 +281,96 @@ class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext with B
assert(!failedTaskSet)
}
+ test("abort stage if executor loss results in unschedulability from previously failed tasks") {
+ // Make sure we can detect when a taskset becomes unschedulability from a blacklisting. This
--- End diff --
unschedulability -> unschedulable
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/13603
This is looking good... at least with the current design, looks much cleaner to have the check in TaskSetManager. Having the test in TaskSetManagerSuite seems reasonable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #60955 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60955/consoleFull)** for PR 13603 at commit [`158775c`](https://github.com/apache/spark/commit/158775c13b16e6b303147c78443addce6e6aacd1).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r67774961
--- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -272,6 +261,55 @@ class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext with L
val taskDescriptions3 = taskScheduler.resourceOffers(e1Offers).flatten
assert(1 === taskDescriptions3.length)
assert("executor1" === taskDescriptions3(0).executorId)
+ assert(!failedTaskSet)
+ }
+
+ test("abort stage if executor loss results in unschedulability from previously failed tasks") {
+ // Make sure we can detect when a taskset becomes unschedulability from a blacklisting. This
+ // test explores a particular corner case -- you may have one task fail, but still be
+ // schedulable on another executor. However, that executor may fail later on, leaving the
+ // first task with no place to run.
+ val taskScheduler = setupScheduler(
+ // set this to something much longer than the test duration
+ "spark.scheduler.executorTaskBlacklistTime" -> "10000000"
+ )
+
+ val taskSet = FakeTask.createTaskSet(2)
+ taskScheduler.submitTasks(taskSet)
+ val tsm = taskScheduler.taskSetManagerForAttempt(taskSet.stageId, taskSet.stageAttemptId).get
+
+ val firstTasks = taskScheduler.resourceOffers(Seq(
--- End diff --
firstTaskAttempts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/13603
Ohh good point that makes sense re: lost executors. Given that, I agree that this approach seems like the right one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r67775386
--- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -272,6 +261,55 @@ class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext with L
val taskDescriptions3 = taskScheduler.resourceOffers(e1Offers).flatten
assert(1 === taskDescriptions3.length)
assert("executor1" === taskDescriptions3(0).executorId)
+ assert(!failedTaskSet)
+ }
+
+ test("abort stage if executor loss results in unschedulability from previously failed tasks") {
+ // Make sure we can detect when a taskset becomes unschedulability from a blacklisting. This
+ // test explores a particular corner case -- you may have one task fail, but still be
+ // schedulable on another executor. However, that executor may fail later on, leaving the
+ // first task with no place to run.
+ val taskScheduler = setupScheduler(
+ // set this to something much longer than the test duration
+ "spark.scheduler.executorTaskBlacklistTime" -> "10000000"
+ )
+
+ val taskSet = FakeTask.createTaskSet(2)
+ taskScheduler.submitTasks(taskSet)
+ val tsm = taskScheduler.taskSetManagerForAttempt(taskSet.stageId, taskSet.stageAttemptId).get
+
+ val firstTasks = taskScheduler.resourceOffers(Seq(
+ new WorkerOffer("executor0", "host0", 1),
+ new WorkerOffer("executor1", "host1", 1)
+ )).flatten
+ assert(Set("executor0", "executor1") === firstTasks.map{_.executorId}.toSet)
+
+ // fail one of the tasks, but leave the other running
+ val failedTask = firstTasks.find(_.executorId == "executor0").get
+ taskScheduler.handleFailedTask(tsm, failedTask.taskId, TaskState.FAILED, TaskResultLost)
+ // at this point, our failed task could run on the other executor, so don't give up the task
+ // set yet.
+
+ // but now we fail our second executor. The other task still has a failed executor it can
+ // run on, but our first failed task does not. So we should fail the task set.
+ taskScheduler.executorLost("executor1", SlaveLost("oops"))
+ val nextTasks =
--- End diff --
nextTaskAttempts? Also fix indentation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61378 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61378/consoleFull)** for PR 13603 at commit [`860ee1a`](https://github.com/apache/spark/commit/860ee1a39c8ad02023a0f539c2d3f0c4c6df09df).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r69032740
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -576,6 +576,59 @@ private[spark] class TaskSetManager(
}
/**
+ * Check whether the given task set has been blacklisted to the point that it can't run anywhere.
+ *
+ * It is possible that this taskset has become impossible to schedule *anywhere* due to the
+ * blacklist. The most common scenario would be if there are fewer executors than
+ * spark.task.maxFailures. We need to detect this so we can fail the task set, otherwise the job
+ * will hang.
+ *
+ * There's a tradeoff here: we could make sure all tasks in the task set are schedulable, but that
+ * would add extra time to each iteration of the scheduling loop. Here, we take the approach of
+ * making sure at least one of the unscheduled tasks is schedulable. This means we may not detect
+ * the hang as quickly as we could have, but we'll always detect the hang eventually, and the
+ * method is faster in the typical case. In the worst case, this method can take
+ * O(maxTaskFailures + numTasks) time, but it will be faster when there haven't been any task
+ * failures (this is because the method picks on unscheduled task, and then iterates through each
+ * executor until it finds one that the task hasn't failed on already).
+ */
+ private[scheduler] def abortIfCompletelyBlacklisted(executors: Iterable[String]): Unit = {
+
+ def pendingTask: Option[Int] = {
--- End diff --
what about
val pendingTask: Option[Int] = allPendingTasks.lastIndexWhere { indexInTaskSet =>
copiesRunning(indexInTaskSet) == 0 && !successful(indexInTaskSet)
}
(I realize we're really in the weeds here so whatever you prefer here is fine)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r67772866
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -576,6 +576,62 @@ private[spark] class TaskSetManager(
}
/**
+ * Check whether the given task set has been blacklisted to the point that it can't run anywhere.
+ *
+ * It is possible that this taskset has become impossible to schedule *anywhere* due to the
+ * blacklist. The most common scenario would be if there are fewer executors than
+ * spark.task.maxFailures. We need to detect this so we can fail the task set, otherwise the job
+ * will hang.
+ *
+ * The check here is a balance between being sure to catch the issue, but not wasting
+ * too much time inside the scheduling loop. Just check if the last task is schedulable
+ * on any of the available executors. So this is O(numExecutors) worst-case, but it'll
+ * really be fast unless you've got a bunch of things blacklisted. Its possible it won't detect
+ * the unschedulable task immediately, but if it returns false, there is at least *some* task
+ * that is schedulable, and after scheduling all of those, we'll eventually find the unschedulable
+ * task.
+ */
+ private[scheduler] def abortIfTaskSetCompletelyBlacklisted(
+ executorsByHost: HashMap[String, HashSet[String]]): Unit = {
+ // If no executors have registered yet, don't abort the stage, just wait. We probably
+ // got here because a task set was added before the executors registered.
+ if (executorsByHost.nonEmpty) {
+ // take any task that needs to be scheduled, and see if we can find some executor it *could*
+ // run on
+ pollPendingTask.foreach { task =>
+ executorsByHost.foreach { case (host, execs) =>
+ execs.foreach { exec =>
+ if (!executorIsBlacklisted(exec, task)) {
+ return
+ }
+ }
+ }
+ abort(s"Aborting ${taskSet} because it has a task which cannot be scheduled on any" +
+ s" executor due to blacklists.")
+ }
+ }
+ }
+
+ /**
+ * Return some task which is pending, but do not remove it from the list of pending tasks.
+ * Used as a simple way to test if this task set is schedulable anywhere, or if it has been
+ * completely blacklisted.
+ */
+ private def pollPendingTask: Option[Int] = {
+ // usually this will just take the last pending task, but because of the lazy removal
+ // from each list, we may need to go deeper in the list
+ var indexOffset = allPendingTasks.size
--- End diff --
Why do you start at the end here? Is the idea to find the task that's least likely to have been scheduled? (Can you add a brief comment to this effect?)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r68807958
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -576,6 +576,61 @@ private[spark] class TaskSetManager(
}
/**
+ * Check whether the given task set has been blacklisted to the point that it can't run anywhere.
+ *
+ * It is possible that this taskset has become impossible to schedule *anywhere* due to the
+ * blacklist. The most common scenario would be if there are fewer executors than
+ * spark.task.maxFailures. We need to detect this so we can fail the task set, otherwise the job
+ * will hang.
+ *
+ * There's a tradeoff here: we could make sure all tasks in the task set are schedulable, but that
+ * would add extra time to each iteration of the scheduling loop. Here, we take the approach of
+ * making sure at least one of the unscheduled tasks is schedulable. This means we may not detect
+ * the hang as quickly as we could have, but we'll always detect the hang eventually, and the
+ * method is faster in the typical case. In the worst case, this method can take
+ * O(maxTaskFailures + numTasks) time, but it will be faster when there haven't been any task
+ * failures (this is because the method picks on unscheduled task, and then iterates through each
+ * executor until it finds one that the task hasn't failed on already).
+ */
+ private[scheduler] def abortIfTaskSetCompletelyBlacklisted(
+ executorsByHost: HashMap[String, HashSet[String]]): Unit = {
+
+ def pendingTask: Option[Int] = {
+ // usually this will just take the last pending task, but because of the lazy removal
+ // from each list, we may need to go deeper in the list. We poll from the end because
+ // failed tasks are put back at the end of allPendingTasks, so we're more likely to find
+ // an unschedulable task this way.
+ var indexOffset = allPendingTasks.size
+ while (indexOffset > 0) {
+ indexOffset -= 1
+ val indexInTaskSet = allPendingTasks(indexOffset)
+ if (copiesRunning(indexInTaskSet) == 0 && !successful(indexInTaskSet)) {
+ return Some(indexInTaskSet)
+ }
+ }
+ None
+ }
+
+ // If no executors have registered yet, don't abort the stage, just wait. We probably
+ // got here because a task set was added before the executors registered.
+ if (executorsByHost.nonEmpty) {
+ // take any task that needs to be scheduled, and see if we can find some executor it *could*
+ // run on
+ pendingTask.foreach { taskId =>
+ executorsByHost.foreach { case (host, execs) =>
+ execs.foreach { exec =>
+ if (!executorIsBlacklisted(exec, taskId)) {
+ return
+ }
+ }
+ }
+ abort(s"Aborting ${taskSet} because Task $taskId (partition " +
+ s"${tasks(taskId).partitionId}) cannot be scheduled on any executor due to blacklists.")
--- End diff --
Maybe include the executors here (esp. since this is something users might see)?
s"Aborting ${taskSet} because task $taskId (partition ${tasks(taskId).partitionId}) has already failed on executors ${executorsByHost.values.flatten.mkString(",")}, and no other executors are available."
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r68808263
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -576,6 +576,61 @@ private[spark] class TaskSetManager(
}
/**
+ * Check whether the given task set has been blacklisted to the point that it can't run anywhere.
+ *
+ * It is possible that this taskset has become impossible to schedule *anywhere* due to the
+ * blacklist. The most common scenario would be if there are fewer executors than
+ * spark.task.maxFailures. We need to detect this so we can fail the task set, otherwise the job
+ * will hang.
+ *
+ * There's a tradeoff here: we could make sure all tasks in the task set are schedulable, but that
+ * would add extra time to each iteration of the scheduling loop. Here, we take the approach of
+ * making sure at least one of the unscheduled tasks is schedulable. This means we may not detect
+ * the hang as quickly as we could have, but we'll always detect the hang eventually, and the
+ * method is faster in the typical case. In the worst case, this method can take
+ * O(maxTaskFailures + numTasks) time, but it will be faster when there haven't been any task
+ * failures (this is because the method picks on unscheduled task, and then iterates through each
+ * executor until it finds one that the task hasn't failed on already).
+ */
+ private[scheduler] def abortIfTaskSetCompletelyBlacklisted(
--- End diff --
you could shorten this name to eliminate "TaskSet" (which I think is leftover from when it was in TaskSchedulerImpl)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #60952 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60952/consoleFull)** for PR 13603 at commit [`c1aabea`](https://github.com/apache/spark/commit/c1aabea4bdfceb1e528b7146faf062758b9493a1).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61073/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/13603
q: was blacklist merged in 2.0?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61340 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61340/consoleFull)** for PR 13603 at commit [`b5b0f3f`](https://github.com/apache/spark/commit/b5b0f3f114abefaa3372cdf98f32ac200a756a29).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r68819144
--- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -281,6 +281,96 @@ class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext with B
assert(!failedTaskSet)
}
+ test("abort stage if executor loss results in unschedulability from previously failed tasks") {
+ // Make sure we can detect when a taskset becomes unschedulability from a blacklisting. This
+ // test explores a particular corner case -- you may have one task fail, but still be
+ // schedulable on another executor. However, that executor may fail later on, leaving the
+ // first task with no place to run.
+ val taskScheduler = setupScheduler(
+ // set this to something much longer than the test duration
+ "spark.scheduler.executorTaskBlacklistTime" -> "10000000"
+ )
+
+ val taskSet = FakeTask.createTaskSet(2)
+ taskScheduler.submitTasks(taskSet)
+ val tsm = taskScheduler.taskSetManagerForAttempt(taskSet.stageId, taskSet.stageAttemptId).get
+
+ val firstTaskAttempts = taskScheduler.resourceOffers(Seq(
+ new WorkerOffer("executor0", "host0", 1),
+ new WorkerOffer("executor1", "host1", 1)
+ )).flatten
+ assert(Set("executor0", "executor1") === firstTaskAttempts.map(_.executorId).toSet)
+
+ // fail one of the tasks, but leave the other running
+ val failedTask = firstTaskAttempts.find(_.executorId == "executor0").get
+ taskScheduler.handleFailedTask(tsm, failedTask.taskId, TaskState.FAILED, TaskResultLost)
+ // at this point, our failed task could run on the other executor, so don't give up the task
+ // set yet.
+ assert(!failedTaskSet)
+
+ // Now we fail our second executor. The other task can still run on executor1, so make an offer
+ // on that executor, and make sure that the other task (not the failed one) is assigned there
+ taskScheduler.executorLost("executor1", SlaveLost("oops"))
+ val nextTaskAttempts =
+ taskScheduler.resourceOffers(Seq(new WorkerOffer("executor0", "host0", 1))).flatten
+ // Note: Its OK if some future change makes this already realize the taskset has become
+ // unschedulable at this point (though in the current implementation, we're sure it will not)
+ assert(nextTaskAttempts.size === 1)
+ assert(nextTaskAttempts.head.executorId === "executor0")
+ assert(nextTaskAttempts.head.attemptNumber === 1)
+ assert(nextTaskAttempts.head.index != failedTask.index)
+
+ // now we should definitely realize that our task set is unschedulable, because the only
+ // task left can't be scheduled on any executors due to the blacklist
+ taskScheduler.resourceOffers(Seq(new WorkerOffer("executor0", "host0", 1)))
+ sc.listenerBus.waitUntilEmpty(100000)
+ assert(tsm.isZombie)
+ assert(failedTaskSet)
+ val idx = failedTask.index
+ assert(failedTaskSetReason.contains(s"Aborting TaskSet 0.0 because Task $idx (partition $idx)" +
+ s" cannot be scheduled on any executor due to blacklists."))
+ }
+
+ test("don't abort if there is an executor available, though it hasn't had scheduled tasks yet") {
+ // interaction of SPARK-15865 & SPARK-16106
+ // if we have a small number of tasks, we might be able to schedule them all on the first
+ // executor. But if those tasks fail, we should still realize there is another executor
+ // available and not bail on the job
+
+ val taskScheduler = setupScheduler(
+ // set this to something much longer than the test duration
+ "spark.scheduler.executorTaskBlacklistTime" -> "10000000"
+ )
+
+ val taskSet = FakeTask.createTaskSet(2, (0 until 2).map { _ => Seq(TaskLocation("host0")) }: _*)
+ taskScheduler.submitTasks(taskSet)
+ val tsm = taskScheduler.taskSetManagerForAttempt(taskSet.stageId, taskSet.stageAttemptId).get
+
+ val offers = Seq(
+ // each offer has more than enough free cores for the entire task set, so we schedule
+ // all tasks on one executor
--- End diff --
yes, because of locality preference. I'll update the comment to make that clear
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #3125 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3125/consoleFull)** for PR 13603 at commit [`39fbf72`](https://github.com/apache/spark/commit/39fbf72df8ff9f4c61bc19e4a2fb4c7332109116).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r66647379
--- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistIntegrationSuite.scala ---
@@ -93,6 +94,30 @@ class BlacklistIntegrationSuite extends SchedulerIntegrationSuite[MultiExecutorM
assertDataStructuresEmpty(noFailure = true)
}
+ // Make sure that if we've failed on all executors, but haven't hit task.maxFailures yet, the job
+ // doesn't hang
+ testScheduler(
+ "SPARK-15865 Progress with fewer executors than maxTaskFailures",
--- End diff --
I'm pretty sure the previous flakiness in this test suite is unrelated to the test I'm adding here, but we should probably wait to merge this until after https://github.com/apache/spark/pull/13565 just to be safe. (at least, its unrelated to the delay-scheduling-revive-offers issue, which was the major previous flakiness -- potentially still suffers from the DAGScheduler / assertDataStructuresEmpty race, but that is quite rare.) I did run this test 5K times on my laptop and it passed (though that hasn't always been enough to discover some of the races).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r66646791
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -404,6 +404,25 @@ private[spark] class TaskSetManager(
}
/**
+ * Return some task which is pending, but do not remove it from the list of pending tasks.
+ * Used as a simple way to test if this task set is schedulable anywhere, or if it has been
+ * completely blacklisted.
+ */
+ def pollPendingTask: Option[Int] = {
+ // usually this will just take the last pending task, but because of the lazy removal
+ // from each list, we may need to go deeper in the list
+ var indexOffset = allPendingTasks.size
+ while (indexOffset > 0) {
+ indexOffset -= 1
+ val index = allPendingTasks(indexOffset)
+ if (copiesRunning(index) == 0 && !successful(index)) {
+ return Some(index)
+ }
--- End diff --
I'm *pretty* sure that we could add
```scala
else {
// this task has already been scheduled from one of our other task queues, so remove it
// from this one as well, even though we're not actually scheduling anything here.
allPendingTasks.remove(indexOffset)
}
```
But its shouldn't be *necessary* to do here, and I'm just nervous enough about adding it that I opted not to.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61385 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61385/consoleFull)** for PR 13603 at commit [`60cd959`](https://github.com/apache/spark/commit/60cd959cb37bf89af8f5b10dfb6d19bd5a46e627).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r67775656
--- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -27,18 +29,48 @@ class FakeSchedulerBackend extends SchedulerBackend {
def defaultParallelism(): Int = 1
}
-class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext with Logging {
+class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext with BeforeAndAfterEach
+ with Logging {
- test("Scheduler does not always schedule tasks on the same workers") {
+
+ var failedTaskSetExc: Option[Throwable] = None
+ var failedTaskSetReason: String = null
+ var failedTaskSet = false
+
+ override def beforeEach(): Unit = {
+ super.beforeEach()
+ failedTaskSet = false
+ failedTaskSetExc = None
+ failedTaskSetReason = null
+ }
+
+ def setupScheduler(confs: (String, String)*): TaskSchedulerImpl = {
--- End diff --
👍 for eliminating this redundancy
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r67776927
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -576,6 +576,62 @@ private[spark] class TaskSetManager(
}
/**
+ * Check whether the given task set has been blacklisted to the point that it can't run anywhere.
+ *
+ * It is possible that this taskset has become impossible to schedule *anywhere* due to the
+ * blacklist. The most common scenario would be if there are fewer executors than
+ * spark.task.maxFailures. We need to detect this so we can fail the task set, otherwise the job
+ * will hang.
+ *
--- End diff --
The next par. is a little hard to understand. How about something like (also is the below correct? It's a little different from what you had):
There's a tradeoff here: we could make sure all tasks in the task set are schedulable, but that would add extra time to each iteration of the scheduling loop. Here, we take the approach of making sure at least one of the unscheduled tasks is schedulable. This means we may not detect the hang as quickly as we could have, but we'll always detect the hang eventually, and the method is faster in the typical case. In the worst case, this method can take O(maxTaskFailures) time, but it will be faster when there haven't been any task failures (this is because the method picks on unscheduled task, and then iterates through each executor until it finds one that the task hasn't failed on already).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r68629355
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala ---
@@ -30,6 +30,7 @@ import org.apache.spark.annotation.DeveloperApi
@DeveloperApi
class TaskInfo(
val taskId: Long,
+ /** the index of this task in its TaskSet. *Not* necessarily the same as the partitionid */
--- End diff --
Confused by your comments on this -- your comment in the thread for this PR seemed to imply that these are indeed different?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61399 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61399/consoleFull)** for PR 13603 at commit [`9665029`](https://github.com/apache/spark/commit/96650292fdf02087cebe6b9efc1d921a66d35d6a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the issue:
https://github.com/apache/spark/pull/13603
@kayousterhout thanks for the feedback. Another round of updates. I actually don't think SPARK-16106 is a blocker here, it doesn't effect the task serializability tests, those still work, and I added some extra asserts on them. (the error was in another test, and I put a workaround for it in any case.)
In addition to your suggestions, I ended up making a couple of other changes:
* make sure that we consistently check the blacklist with the partitionId, not the index in the taskset (they are the same on initial stage submission, but not stage retry). Also updated the existing naming, since it used to say `taskId`: https://github.com/apache/spark/pull/13603/commits/6362b28468e752a61560bffdb2f383c69163ac36
* fixed `FakeTask` to have reasonable `partitionId`s (before, when you called `FakeTask.createTaskSet(10)`, each task had partitiionId = 0 but the *stage* went from 0 to n): https://github.com/apache/spark/pull/13603/commits/060dbfead7c5fe3ca57c1f2aaa3986455f34be09
* fix more leaked threads: https://github.com/apache/spark/pull/13603/commits/fd48403e6c94cf220c7ed2ad86b7d0cd5bd05e94
I'm also not sure what is going on in the [second half of the test for not serializable tasks]( https://github.com/apache/spark/blob/master/core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala#L123). Its resubmitting the exact same task set has the previous attempt, which never happens (should have another attemptId at least), and submitting another task set also with the same stageId and attemptId. I'm not sure what the expected behavior is in this case, I think that test case should probably be changes, but I've been playing enough whack-a-mole for now that I'm just going to leave it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the issue:
https://github.com/apache/spark/pull/13603
merged to master. thanks Kay!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61378/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r69024693
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -576,6 +576,59 @@ private[spark] class TaskSetManager(
}
/**
+ * Check whether the given task set has been blacklisted to the point that it can't run anywhere.
+ *
+ * It is possible that this taskset has become impossible to schedule *anywhere* due to the
+ * blacklist. The most common scenario would be if there are fewer executors than
+ * spark.task.maxFailures. We need to detect this so we can fail the task set, otherwise the job
+ * will hang.
+ *
+ * There's a tradeoff here: we could make sure all tasks in the task set are schedulable, but that
+ * would add extra time to each iteration of the scheduling loop. Here, we take the approach of
+ * making sure at least one of the unscheduled tasks is schedulable. This means we may not detect
+ * the hang as quickly as we could have, but we'll always detect the hang eventually, and the
+ * method is faster in the typical case. In the worst case, this method can take
+ * O(maxTaskFailures + numTasks) time, but it will be faster when there haven't been any task
+ * failures (this is because the method picks on unscheduled task, and then iterates through each
+ * executor until it finds one that the task hasn't failed on already).
+ */
+ private[scheduler] def abortIfCompletelyBlacklisted(executors: Iterable[String]): Unit = {
+
+ def pendingTask: Option[Int] = {
--- End diff --
I pushed a commit with changing it to a val, so you can see both options. easy enough to back out that last commit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61385/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r68641914
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala ---
@@ -30,6 +30,7 @@ import org.apache.spark.annotation.DeveloperApi
@DeveloperApi
class TaskInfo(
val taskId: Long,
+ /** the index of this task in its TaskSet. *Not* necessarily the same as the partitionid */
--- End diff --
yeah, sorry, at first I thought they were the same. and in fact I even made a bunch of changes to reflect that (which is probably what you are referring to).
Then the next morning I was poking around some more and discovered that I was completely wrong. They are definitely not the same, so I reverted those earlier changes and updated the names some more. (I dropped another comment later on the PR saying as much, but probably got lost in all the noise, sorry.)
The names here are pretty confusing. Part of the problem is that `taskIndex` makes me think of `taskId` aka `TID`, which it definitely is *not*. There are 4 different numbers identifying a task now:
a) TID / `taskId`
b) index within a taskset (sometimes called `index`, sometimes called `taskIndex`, sometimes called `taskId`, depending on where in the code)
c) attempt number
d) partition
In the logs, you'll see "task [b].[c]" eg [`TaskSetManager.handleSuccessfulTask`](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala#L630). [b] and [d] are *usually* exactly the same. However, they aren't the same when a stage is retried, and you only run selective partitions -- the new taskset still numbers from 0 to n, but the partitions will be arbitrary.
We don't have a good name for [b] now, I'd like to come up with something better. We could at least consistently use `taskIndex`, so that `taskId` stopped meaning two different things. Another option is `indexInTaskSet` but that's kinda long ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61335 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61335/consoleFull)** for PR 13603 at commit [`a5f7eb4`](https://github.com/apache/spark/commit/a5f7eb44504f81637d65f66af5af560b92aa3f68).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class ShowFunctionsCommand(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #60953 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60953/consoleFull)** for PR 13603 at commit [`9efd95d`](https://github.com/apache/spark/commit/9efd95d4fcac65aba2d6cc36712acb3667781818).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61322 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61322/consoleFull)** for PR 13603 at commit [`e8e4fe5`](https://github.com/apache/spark/commit/e8e4fe5989bf07693f49842253df792d70fb7286).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/13603
This LGTM -- just a bunch of cosmetic suggestions
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r69005354
--- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistIntegrationSuite.scala ---
@@ -91,6 +91,31 @@ class BlacklistIntegrationSuite extends SchedulerIntegrationSuite[MultiExecutorM
assertDataStructuresEmpty(noFailure = true)
}
+ // Make sure that if we've failed on all executors, but haven't hit task.maxFailures yet, the job
+ // doesn't hang
+ testScheduler(
+ "SPARK-15865 Progress with fewer executors than maxTaskFailures",
+ extraConfs = Seq(
+ // set this to something much longer than the test duration
--- End diff --
doh, I thought I had changed it everywhere earlier, sorry, done now
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61488 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61488/consoleFull)** for PR 13603 at commit [`96049cd`](https://github.com/apache/spark/commit/96049cd421d1092b202d832f79969952323d5eb0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61335 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61335/consoleFull)** for PR 13603 at commit [`a5f7eb4`](https://github.com/apache/spark/commit/a5f7eb44504f81637d65f66af5af560b92aa3f68).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #60851 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60851/consoleFull)** for PR 13603 at commit [`f870bde`](https://github.com/apache/spark/commit/f870bde39c4f0e0b83e079439291b432717a5a82).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61488 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61488/consoleFull)** for PR 13603 at commit [`96049cd`](https://github.com/apache/spark/commit/96049cd421d1092b202d832f79969952323d5eb0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the issue:
https://github.com/apache/spark/pull/13603
actually, calling it a flaky test is unfair -- really spark itself was flaky, the behavior was not reliable when you had locality + blacklists, you might think the taskSet was unschedulable, though it just hadn't scheduled anything on some executors yet due to locality. I added a test for that specifically: https://github.com/apache/spark/pull/13603/commits/635edec9072585d2232d97c085ef60f39fddaafa
I also just merged in the changes https://github.com/apache/spark/pull/13826, so that should be reviewed first.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61505 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61505/consoleFull)** for PR 13603 at commit [`ed413ce`](https://github.com/apache/spark/commit/ed413ceaecff8daa20c57e7272abb0f04ff6355c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r68804577
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -35,7 +35,7 @@ import org.apache.spark.scheduler.SchedulingMode.SchedulingMode
import org.apache.spark.scheduler.TaskLocality.TaskLocality
import org.apache.spark.scheduler.local.LocalSchedulerBackend
import org.apache.spark.storage.BlockManagerId
-import org.apache.spark.util.{AccumulatorV2, ThreadUtils, Utils}
+import org.apache.spark.util._
--- End diff --
undo this change? (nice to have the explicit imports as long as they're short)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61497/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60851/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61385 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61385/consoleFull)** for PR 13603 at commit [`60cd959`](https://github.com/apache/spark/commit/60cd959cb37bf89af8f5b10dfb6d19bd5a46e627).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61322/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r68808597
--- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistIntegrationSuite.scala ---
@@ -91,6 +91,31 @@ class BlacklistIntegrationSuite extends SchedulerIntegrationSuite[MultiExecutorM
assertDataStructuresEmpty(noFailure = true)
}
+ // Make sure that if we've failed on all executors, but haven't hit task.maxFailures yet, the job
+ // doesn't hang
+ testScheduler(
+ "SPARK-15865 Progress with fewer executors than maxTaskFailures",
+ extraConfs = Seq(
+ // set this to something much longer than the test duration
--- End diff --
add "so that executors don't get removed from the blacklist during the test"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61073 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61073/consoleFull)** for PR 13603 at commit [`9a63912`](https://github.com/apache/spark/commit/9a63912560c2fb0d46fc753e040bdd5ccef706c7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60300/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r68852047
--- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistIntegrationSuite.scala ---
@@ -91,6 +91,31 @@ class BlacklistIntegrationSuite extends SchedulerIntegrationSuite[MultiExecutorM
assertDataStructuresEmpty(noFailure = true)
}
+ // Make sure that if we've failed on all executors, but haven't hit task.maxFailures yet, the job
+ // doesn't hang
+ testScheduler(
+ "SPARK-15865 Progress with fewer executors than maxTaskFailures",
+ extraConfs = Seq(
+ // set this to something much longer than the test duration
--- End diff --
Looks like you added this in the other test -- I know it's redundant here but still helpful for reference I think
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r68851944
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -576,6 +576,59 @@ private[spark] class TaskSetManager(
}
/**
+ * Check whether the given task set has been blacklisted to the point that it can't run anywhere.
+ *
+ * It is possible that this taskset has become impossible to schedule *anywhere* due to the
+ * blacklist. The most common scenario would be if there are fewer executors than
+ * spark.task.maxFailures. We need to detect this so we can fail the task set, otherwise the job
+ * will hang.
+ *
+ * There's a tradeoff here: we could make sure all tasks in the task set are schedulable, but that
+ * would add extra time to each iteration of the scheduling loop. Here, we take the approach of
+ * making sure at least one of the unscheduled tasks is schedulable. This means we may not detect
+ * the hang as quickly as we could have, but we'll always detect the hang eventually, and the
+ * method is faster in the typical case. In the worst case, this method can take
+ * O(maxTaskFailures + numTasks) time, but it will be faster when there haven't been any task
+ * failures (this is because the method picks on unscheduled task, and then iterates through each
+ * executor until it finds one that the task hasn't failed on already).
+ */
+ private[scheduler] def abortIfCompletelyBlacklisted(executors: Iterable[String]): Unit = {
+
+ def pendingTask: Option[Int] = {
+ // usually this will just take the last pending task, but because of the lazy removal
+ // from each list, we may need to go deeper in the list. We poll from the end because
+ // failed tasks are put back at the end of allPendingTasks, so we're more likely to find
+ // an unschedulable task this way.
+ var indexOffset = allPendingTasks.size
+ while (indexOffset > 0) {
+ indexOffset -= 1
+ val indexInTaskSet = allPendingTasks(indexOffset)
+ if (copiesRunning(indexInTaskSet) == 0 && !successful(indexInTaskSet)) {
+ return Some(indexInTaskSet)
+ }
+ }
+ None
+ }
+
+ // If no executors have registered yet, don't abort the stage, just wait. We probably
+ // got here because a task set was added before the executors registered.
+ if (executors.nonEmpty) {
+ // take any task that needs to be scheduled, and see if we can find some executor it *could*
+ // run on
+ pendingTask.foreach { taskId =>
+ executors.foreach { exec =>
--- End diff --
now that executors is an iterable, just do "if (executors.find(executorIsBlacklisted(_, taskId)).isEmpty) { .. abort ...}" here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r67772507
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -576,6 +576,62 @@ private[spark] class TaskSetManager(
}
/**
+ * Check whether the given task set has been blacklisted to the point that it can't run anywhere.
+ *
+ * It is possible that this taskset has become impossible to schedule *anywhere* due to the
+ * blacklist. The most common scenario would be if there are fewer executors than
+ * spark.task.maxFailures. We need to detect this so we can fail the task set, otherwise the job
+ * will hang.
+ *
+ * The check here is a balance between being sure to catch the issue, but not wasting
+ * too much time inside the scheduling loop. Just check if the last task is schedulable
+ * on any of the available executors. So this is O(numExecutors) worst-case, but it'll
+ * really be fast unless you've got a bunch of things blacklisted. Its possible it won't detect
+ * the unschedulable task immediately, but if it returns false, there is at least *some* task
+ * that is schedulable, and after scheduling all of those, we'll eventually find the unschedulable
+ * task.
+ */
+ private[scheduler] def abortIfTaskSetCompletelyBlacklisted(
+ executorsByHost: HashMap[String, HashSet[String]]): Unit = {
+ // If no executors have registered yet, don't abort the stage, just wait. We probably
+ // got here because a task set was added before the executors registered.
+ if (executorsByHost.nonEmpty) {
+ // take any task that needs to be scheduled, and see if we can find some executor it *could*
+ // run on
+ pollPendingTask.foreach { task =>
+ executorsByHost.foreach { case (host, execs) =>
+ execs.foreach { exec =>
+ if (!executorIsBlacklisted(exec, task)) {
+ return
+ }
+ }
+ }
+ abort(s"Aborting ${taskSet} because it has a task which cannot be scheduled on any" +
--- End diff --
Can you include the task ID here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61320 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61320/consoleFull)** for PR 13603 at commit [`9684eb7`](https://github.com/apache/spark/commit/9684eb793c7137371bf428661a44cd73d548a289).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r67070153
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -280,10 +280,54 @@ private[spark] class TaskSchedulerImpl(
}
}
}
+ if (!launchedTask && isTaskSetCompletelyBlacklisted(taskSet)) {
+ taskSet.abort(s"Aborting ${taskSet.taskSet} because it has a task which cannot be scheduled" +
+ s" on any executor due to blacklists.")
+ }
return launchedTask
}
/**
+ * Check whether the given task set has been blacklisted to the point that it can't run anywhere.
+ *
+ * It is possible that this taskset has become impossible to schedule *anywhere* due to the
+ * blacklist. The most common scenario would be if there are fewer executors than
+ * spark.task.maxFailures. We need to detect this so we can fail the task set, otherwise the job
+ * will hang.
+ *
+ * The check here is a balance between being sure to catch the issue, but not wasting
+ * too much time inside the scheduling loop. Just check if the last task is schedulable
+ * on any of the available executors. So this is O(numExecutors) worst-case, but it'll
+ * really be fast unless you've got a bunch of things blacklisted. Its possible it won't detect
+ * the unschedulable task immediately, but if it returns false, there is at least *some* task
+ * that is schedulable, and after scheduling all of those, we'll eventually find the unschedulable
+ * task.
+ */
+ private[scheduler] def isTaskSetCompletelyBlacklisted(
--- End diff --
Ah got it -- agree that in the long term, given the new blacklistling, it might make sense here! But let's put it in the TaskSetManager for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the issue:
https://github.com/apache/spark/pull/13603
sigh, BlacklistIntegrationSuite becomes flaky with this, but without SPARK-16106, so I have included that change here just so tests will pass.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61116/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61340 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61340/consoleFull)** for PR 13603 at commit [`b5b0f3f`](https://github.com/apache/spark/commit/b5b0f3f114abefaa3372cdf98f32ac200a756a29).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r68642776
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala ---
@@ -30,6 +30,7 @@ import org.apache.spark.annotation.DeveloperApi
@DeveloperApi
class TaskInfo(
val taskId: Long,
+ /** the index of this task in its TaskSet. *Not* necessarily the same as the partitionid */
--- End diff --
BTW, at some point a while ago, I attempted to change the naming of these to make it less confusing, but than ran into issues because there are various naming artifacts from Mesos, which uses taskId to refer to our taskAttemptId, so I eventually gave up. It does seem like we should (separately from this PR) fix all the places the code uses taskId in place of index to say "index" instead for consistency.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61320/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61497 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61497/consoleFull)** for PR 13603 at commit [`ed71c99`](https://github.com/apache/spark/commit/ed71c99d886149b71a1123c014cf0004425edbf5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r69005566
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -576,6 +576,59 @@ private[spark] class TaskSetManager(
}
/**
+ * Check whether the given task set has been blacklisted to the point that it can't run anywhere.
+ *
+ * It is possible that this taskset has become impossible to schedule *anywhere* due to the
+ * blacklist. The most common scenario would be if there are fewer executors than
+ * spark.task.maxFailures. We need to detect this so we can fail the task set, otherwise the job
+ * will hang.
+ *
+ * There's a tradeoff here: we could make sure all tasks in the task set are schedulable, but that
+ * would add extra time to each iteration of the scheduling loop. Here, we take the approach of
+ * making sure at least one of the unscheduled tasks is schedulable. This means we may not detect
+ * the hang as quickly as we could have, but we'll always detect the hang eventually, and the
+ * method is faster in the typical case. In the worst case, this method can take
+ * O(maxTaskFailures + numTasks) time, but it will be faster when there haven't been any task
+ * failures (this is because the method picks on unscheduled task, and then iterates through each
+ * executor until it finds one that the task hasn't failed on already).
+ */
+ private[scheduler] def abortIfCompletelyBlacklisted(executors: Iterable[String]): Unit = {
+
+ def pendingTask: Option[Int] = {
+ // usually this will just take the last pending task, but because of the lazy removal
+ // from each list, we may need to go deeper in the list. We poll from the end because
+ // failed tasks are put back at the end of allPendingTasks, so we're more likely to find
+ // an unschedulable task this way.
+ var indexOffset = allPendingTasks.size
+ while (indexOffset > 0) {
+ indexOffset -= 1
+ val indexInTaskSet = allPendingTasks(indexOffset)
+ if (copiesRunning(indexInTaskSet) == 0 && !successful(indexInTaskSet)) {
+ return Some(indexInTaskSet)
+ }
+ }
+ None
+ }
+
+ // If no executors have registered yet, don't abort the stage, just wait. We probably
+ // got here because a task set was added before the executors registered.
+ if (executors.nonEmpty) {
+ // take any task that needs to be scheduled, and see if we can find some executor it *could*
+ // run on
+ pendingTask.foreach { taskId =>
+ executors.foreach { exec =>
--- End diff --
good point, in fact I can just use `executors.forall`. Sorry I keep working on the new blacklist version in between and sometimes don't see some of these obvious simplifications in this version, thanks for catching them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #60856 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60856/consoleFull)** for PR 13603 at commit [`3edb6fe`](https://github.com/apache/spark/commit/3edb6fe37f40e636d43340321066b11d23fbba97).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r66870624
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -280,10 +280,54 @@ private[spark] class TaskSchedulerImpl(
}
}
}
+ if (!launchedTask && isTaskSetCompletelyBlacklisted(taskSet)) {
+ taskSet.abort(s"Aborting ${taskSet.taskSet} because it has a task which cannot be scheduled" +
+ s" on any executor due to blacklists.")
+ }
return launchedTask
}
/**
+ * Check whether the given task set has been blacklisted to the point that it can't run anywhere.
+ *
+ * It is possible that this taskset has become impossible to schedule *anywhere* due to the
+ * blacklist. The most common scenario would be if there are fewer executors than
+ * spark.task.maxFailures. We need to detect this so we can fail the task set, otherwise the job
+ * will hang.
+ *
+ * The check here is a balance between being sure to catch the issue, but not wasting
+ * too much time inside the scheduling loop. Just check if the last task is schedulable
+ * on any of the available executors. So this is O(numExecutors) worst-case, but it'll
+ * really be fast unless you've got a bunch of things blacklisted. Its possible it won't detect
+ * the unschedulable task immediately, but if it returns false, there is at least *some* task
+ * that is schedulable, and after scheduling all of those, we'll eventually find the unschedulable
+ * task.
+ */
+ private[scheduler] def isTaskSetCompletelyBlacklisted(
--- End diff --
I think it would be cleaner to add this method to the TaskSetManager class (and then you don't need the pollPendingTask method) -- and then just pass in the executorsByHost map. That also makes things a little easier to change in the future, if there gets to be some easier way of checking if a particular task set is completely blacklisted.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61341 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61341/consoleFull)** for PR 13603 at commit [`f4e95c6`](https://github.com/apache/spark/commit/f4e95c624db60802d08fd0e64c68fa9e0593a086).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r68642421
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala ---
@@ -30,6 +30,7 @@ import org.apache.spark.annotation.DeveloperApi
@DeveloperApi
class TaskInfo(
val taskId: Long,
+ /** the index of this task in its TaskSet. *Not* necessarily the same as the partitionid */
--- End diff --
Ah ok got it. I saw one of your earlier comments but wasn't sure how to interpret what was striked-through in it. Since this is in the developer (so semi-user-facing), what about changing this comment to: "The index of this task within its task set. Not necessarily the same as the ID of the RDD partition that the task is computing."
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/13603
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61335/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r67899147
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -576,6 +576,62 @@ private[spark] class TaskSetManager(
}
/**
+ * Check whether the given task set has been blacklisted to the point that it can't run anywhere.
+ *
+ * It is possible that this taskset has become impossible to schedule *anywhere* due to the
+ * blacklist. The most common scenario would be if there are fewer executors than
+ * spark.task.maxFailures. We need to detect this so we can fail the task set, otherwise the job
+ * will hang.
+ *
+ * The check here is a balance between being sure to catch the issue, but not wasting
+ * too much time inside the scheduling loop. Just check if the last task is schedulable
+ * on any of the available executors. So this is O(numExecutors) worst-case, but it'll
+ * really be fast unless you've got a bunch of things blacklisted. Its possible it won't detect
+ * the unschedulable task immediately, but if it returns false, there is at least *some* task
+ * that is schedulable, and after scheduling all of those, we'll eventually find the unschedulable
+ * task.
+ */
+ private[scheduler] def abortIfTaskSetCompletelyBlacklisted(
+ executorsByHost: HashMap[String, HashSet[String]]): Unit = {
+ // If no executors have registered yet, don't abort the stage, just wait. We probably
+ // got here because a task set was added before the executors registered.
+ if (executorsByHost.nonEmpty) {
+ // take any task that needs to be scheduled, and see if we can find some executor it *could*
+ // run on
+ pollPendingTask.foreach { task =>
+ executorsByHost.foreach { case (host, execs) =>
+ execs.foreach { exec =>
+ if (!executorIsBlacklisted(exec, task)) {
+ return
+ }
+ }
+ }
+ abort(s"Aborting ${taskSet} because it has a task which cannot be scheduled on any" +
+ s" executor due to blacklists.")
+ }
+ }
+ }
+
+ /**
+ * Return some task which is pending, but do not remove it from the list of pending tasks.
+ * Used as a simple way to test if this task set is schedulable anywhere, or if it has been
+ * completely blacklisted.
+ */
+ private def pollPendingTask: Option[Int] = {
+ // usually this will just take the last pending task, but because of the lazy removal
+ // from each list, we may need to go deeper in the list
+ var indexOffset = allPendingTasks.size
--- End diff --
to be honest, I was mostly just copying `dequeueTaskFromList`. But now that I'm thinking about it, there is a tradeoff: If we check from the end of the list, we're more likely to find the unschedulable task (since failed tasks are put back on the end). If we check from the beginning, we'll find a task which passes `copiesRunning(index) == 0 && !successful(index)` more quickly, since the stacks of pending tasks are only lazily updated.
Given that we only run this check when we've failed to schedule any tasks, I think this version is fine, though I dont' feel very strongly about it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #60300 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60300/consoleFull)** for PR 13603 at commit [`bc80e8c`](https://github.com/apache/spark/commit/bc80e8ceeee180c37fa9211ee1b85ce0a7a90ac7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #60964 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60964/consoleFull)** for PR 13603 at commit [`e880495`](https://github.com/apache/spark/commit/e88049551c64c907b79a80754fb2469d108145ed).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61340/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/13603
I'll finish looking at this later today after #13826 gets merged, so it's easier to see what's new here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r68808784
--- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistIntegrationSuite.scala ---
@@ -91,6 +91,31 @@ class BlacklistIntegrationSuite extends SchedulerIntegrationSuite[MultiExecutorM
assertDataStructuresEmpty(noFailure = true)
}
+ // Make sure that if we've failed on all executors, but haven't hit task.maxFailures yet, the job
+ // doesn't hang
+ testScheduler(
+ "SPARK-15865 Progress with fewer executors than maxTaskFailures",
+ extraConfs = Seq(
+ // set this to something much longer than the test duration
+ "spark.scheduler.executorTaskBlacklistTime" -> "10000000",
+ "spark.testing.nHosts" -> "2",
+ "spark.testing.nExecutorsPerHost" -> "1",
+ "spark.testing.nCoresPerExecutor" -> "1"
+ )
+ ) {
--- End diff --
super nit but can you put these on the previous line? Was confused by the syntax here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r67772715
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -576,6 +576,62 @@ private[spark] class TaskSetManager(
}
/**
+ * Check whether the given task set has been blacklisted to the point that it can't run anywhere.
+ *
+ * It is possible that this taskset has become impossible to schedule *anywhere* due to the
+ * blacklist. The most common scenario would be if there are fewer executors than
+ * spark.task.maxFailures. We need to detect this so we can fail the task set, otherwise the job
+ * will hang.
+ *
+ * The check here is a balance between being sure to catch the issue, but not wasting
+ * too much time inside the scheduling loop. Just check if the last task is schedulable
+ * on any of the available executors. So this is O(numExecutors) worst-case, but it'll
+ * really be fast unless you've got a bunch of things blacklisted. Its possible it won't detect
+ * the unschedulable task immediately, but if it returns false, there is at least *some* task
+ * that is schedulable, and after scheduling all of those, we'll eventually find the unschedulable
+ * task.
+ */
+ private[scheduler] def abortIfTaskSetCompletelyBlacklisted(
+ executorsByHost: HashMap[String, HashSet[String]]): Unit = {
+ // If no executors have registered yet, don't abort the stage, just wait. We probably
+ // got here because a task set was added before the executors registered.
+ if (executorsByHost.nonEmpty) {
+ // take any task that needs to be scheduled, and see if we can find some executor it *could*
+ // run on
+ pollPendingTask.foreach { task =>
+ executorsByHost.foreach { case (host, execs) =>
+ execs.foreach { exec =>
+ if (!executorIsBlacklisted(exec, task)) {
+ return
+ }
+ }
+ }
+ abort(s"Aborting ${taskSet} because it has a task which cannot be scheduled on any" +
+ s" executor due to blacklists.")
+ }
+ }
+ }
+
+ /**
+ * Return some task which is pending, but do not remove it from the list of pending tasks.
+ * Used as a simple way to test if this task set is schedulable anywhere, or if it has been
+ * completely blacklisted.
+ */
+ private def pollPendingTask: Option[Int] = {
--- End diff --
Can you inline this in abortIfTaskSetCompletelyBlacklisted? It's a little confusing and I'd rather help prevent it from being used in other places (I think that also makes some of the commenting about why it's necessary unneeded)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61116 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61116/consoleFull)** for PR 13603 at commit [`2d52962`](https://github.com/apache/spark/commit/2d52962ccab92387b65d63418f6c20b3c66f6263).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61049/consoleFull)** for PR 13603 at commit [`39fbf72`](https://github.com/apache/spark/commit/39fbf72df8ff9f4c61bc19e4a2fb4c7332109116).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61061 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61061/consoleFull)** for PR 13603 at commit [`c606dc3`](https://github.com/apache/spark/commit/c606dc31de84d3bdfbbd2f00827206e150adf6b3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #60953 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60953/consoleFull)** for PR 13603 at commit [`9efd95d`](https://github.com/apache/spark/commit/9efd95d4fcac65aba2d6cc36712acb3667781818).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #60855 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60855/consoleFull)** for PR 13603 at commit [`64ab7fb`](https://github.com/apache/spark/commit/64ab7fbbd4c574881e5132e09998225a476bba42).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61341 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61341/consoleFull)** for PR 13603 at commit [`f4e95c6`](https://github.com/apache/spark/commit/f4e95c624db60802d08fd0e64c68fa9e0593a086).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r68810277
--- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -281,6 +281,96 @@ class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext with B
assert(!failedTaskSet)
}
+ test("abort stage if executor loss results in unschedulability from previously failed tasks") {
+ // Make sure we can detect when a taskset becomes unschedulability from a blacklisting. This
+ // test explores a particular corner case -- you may have one task fail, but still be
+ // schedulable on another executor. However, that executor may fail later on, leaving the
+ // first task with no place to run.
+ val taskScheduler = setupScheduler(
+ // set this to something much longer than the test duration
+ "spark.scheduler.executorTaskBlacklistTime" -> "10000000"
+ )
+
+ val taskSet = FakeTask.createTaskSet(2)
+ taskScheduler.submitTasks(taskSet)
+ val tsm = taskScheduler.taskSetManagerForAttempt(taskSet.stageId, taskSet.stageAttemptId).get
+
+ val firstTaskAttempts = taskScheduler.resourceOffers(Seq(
+ new WorkerOffer("executor0", "host0", 1),
+ new WorkerOffer("executor1", "host1", 1)
+ )).flatten
+ assert(Set("executor0", "executor1") === firstTaskAttempts.map(_.executorId).toSet)
+
+ // fail one of the tasks, but leave the other running
+ val failedTask = firstTaskAttempts.find(_.executorId == "executor0").get
+ taskScheduler.handleFailedTask(tsm, failedTask.taskId, TaskState.FAILED, TaskResultLost)
+ // at this point, our failed task could run on the other executor, so don't give up the task
+ // set yet.
+ assert(!failedTaskSet)
+
+ // Now we fail our second executor. The other task can still run on executor1, so make an offer
+ // on that executor, and make sure that the other task (not the failed one) is assigned there
+ taskScheduler.executorLost("executor1", SlaveLost("oops"))
+ val nextTaskAttempts =
+ taskScheduler.resourceOffers(Seq(new WorkerOffer("executor0", "host0", 1))).flatten
+ // Note: Its OK if some future change makes this already realize the taskset has become
+ // unschedulable at this point (though in the current implementation, we're sure it will not)
+ assert(nextTaskAttempts.size === 1)
+ assert(nextTaskAttempts.head.executorId === "executor0")
+ assert(nextTaskAttempts.head.attemptNumber === 1)
+ assert(nextTaskAttempts.head.index != failedTask.index)
+
+ // now we should definitely realize that our task set is unschedulable, because the only
+ // task left can't be scheduled on any executors due to the blacklist
+ taskScheduler.resourceOffers(Seq(new WorkerOffer("executor0", "host0", 1)))
+ sc.listenerBus.waitUntilEmpty(100000)
+ assert(tsm.isZombie)
+ assert(failedTaskSet)
+ val idx = failedTask.index
+ assert(failedTaskSetReason.contains(s"Aborting TaskSet 0.0 because Task $idx (partition $idx)" +
+ s" cannot be scheduled on any executor due to blacklists."))
+ }
+
+ test("don't abort if there is an executor available, though it hasn't had scheduled tasks yet") {
+ // interaction of SPARK-15865 & SPARK-16106
+ // if we have a small number of tasks, we might be able to schedule them all on the first
+ // executor. But if those tasks fail, we should still realize there is another executor
+ // available and not bail on the job
+
+ val taskScheduler = setupScheduler(
+ // set this to something much longer than the test duration
+ "spark.scheduler.executorTaskBlacklistTime" -> "10000000"
+ )
+
+ val taskSet = FakeTask.createTaskSet(2, (0 until 2).map { _ => Seq(TaskLocation("host0")) }: _*)
+ taskScheduler.submitTasks(taskSet)
+ val tsm = taskScheduler.taskSetManagerForAttempt(taskSet.stageId, taskSet.stageAttemptId).get
+
+ val offers = Seq(
+ // each offer has more than enough free cores for the entire task set, so we schedule
+ // all tasks on one executor
--- End diff --
We schedule them all on host0 b/c of the locality preference right? I thought we try to spread the tasks out across executors (even if they'd all fit on one) otherwise?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #3125 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3125/consoleFull)** for PR 13603 at commit [`39fbf72`](https://github.com/apache/spark/commit/39fbf72df8ff9f4c61bc19e4a2fb4c7332109116).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #60964 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60964/consoleFull)** for PR 13603 at commit [`e880495`](https://github.com/apache/spark/commit/e88049551c64c907b79a80754fb2469d108145ed).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60952/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61505 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61505/consoleFull)** for PR 13603 at commit [`ed413ce`](https://github.com/apache/spark/commit/ed413ceaecff8daa20c57e7272abb0f04ff6355c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61399 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61399/consoleFull)** for PR 13603 at commit [`9665029`](https://github.com/apache/spark/commit/96650292fdf02087cebe6b9efc1d921a66d35d6a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61116 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61116/consoleFull)** for PR 13603 at commit [`2d52962`](https://github.com/apache/spark/commit/2d52962ccab92387b65d63418f6c20b3c66f6263).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61073 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61073/consoleFull)** for PR 13603 at commit [`9a63912`](https://github.com/apache/spark/commit/9a63912560c2fb0d46fc753e040bdd5ccef706c7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #60855 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60855/consoleFull)** for PR 13603 at commit [`64ab7fb`](https://github.com/apache/spark/commit/64ab7fbbd4c574881e5132e09998225a476bba42).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r67939472
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -576,6 +576,62 @@ private[spark] class TaskSetManager(
}
/**
+ * Check whether the given task set has been blacklisted to the point that it can't run anywhere.
+ *
+ * It is possible that this taskset has become impossible to schedule *anywhere* due to the
+ * blacklist. The most common scenario would be if there are fewer executors than
+ * spark.task.maxFailures. We need to detect this so we can fail the task set, otherwise the job
+ * will hang.
+ *
--- End diff --
ah right, I forgot about the cost of pollPendingTask
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r69006676
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -576,6 +576,59 @@ private[spark] class TaskSetManager(
}
/**
+ * Check whether the given task set has been blacklisted to the point that it can't run anywhere.
+ *
+ * It is possible that this taskset has become impossible to schedule *anywhere* due to the
+ * blacklist. The most common scenario would be if there are fewer executors than
+ * spark.task.maxFailures. We need to detect this so we can fail the task set, otherwise the job
+ * will hang.
+ *
+ * There's a tradeoff here: we could make sure all tasks in the task set are schedulable, but that
+ * would add extra time to each iteration of the scheduling loop. Here, we take the approach of
+ * making sure at least one of the unscheduled tasks is schedulable. This means we may not detect
+ * the hang as quickly as we could have, but we'll always detect the hang eventually, and the
+ * method is faster in the typical case. In the worst case, this method can take
+ * O(maxTaskFailures + numTasks) time, but it will be faster when there haven't been any task
+ * failures (this is because the method picks on unscheduled task, and then iterates through each
+ * executor until it finds one that the task hasn't failed on already).
+ */
+ private[scheduler] def abortIfCompletelyBlacklisted(executors: Iterable[String]): Unit = {
+
+ def pendingTask: Option[Int] = {
--- End diff --
gah sorry one more tiny thing: can this just be a val?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r67893270
--- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -96,22 +120,13 @@ class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext with L
taskDescriptions = taskScheduler.resourceOffers(multiCoreWorkerOffers).flatten
assert(1 === taskDescriptions.length)
assert("executor0" === taskDescriptions(0).executorId)
+ assert(!failedTaskSet)
}
test("Scheduler does not crash when tasks are not serializable") {
- sc = new SparkContext("local", "TaskSchedulerImplSuite")
--- End diff --
opened https://issues.apache.org/jira/browse/SPARK-16106. was just surprised enough that I wanted a sanity check first :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #60300 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60300/consoleFull)** for PR 13603 at commit [`bc80e8c`](https://github.com/apache/spark/commit/bc80e8ceeee180c37fa9211ee1b85ce0a7a90ac7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61488/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61320 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61320/consoleFull)** for PR 13603 at commit [`9684eb7`](https://github.com/apache/spark/commit/9684eb793c7137371bf428661a44cd73d548a289).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60856/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r67740258
--- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -96,22 +120,13 @@ class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext with L
taskDescriptions = taskScheduler.resourceOffers(multiCoreWorkerOffers).flatten
assert(1 === taskDescriptions.length)
assert("executor0" === taskDescriptions(0).executorId)
+ assert(!failedTaskSet)
}
test("Scheduler does not crash when tasks are not serializable") {
- sc = new SparkContext("local", "TaskSchedulerImplSuite")
--- End diff --
Unfortunately, this case will also trigger a failure with the msg "Aborting TaskSet ... due to blacklist". I'm pretty sure this is a bug in `TaskSchedulerImpl`, but I am so shocked by it I'd like a quick sanity check. What's going on here is that no tasks have been accepted, so `executorsByHost` never actually adds the new executor in [`resourceOfferSingleTaskSet`](https://github.com/apache/spark/blob/9613424898fd2a586156bc4eb48e255749774f20/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala#L266). But `executorsByHost` has already added the host at the [beginning of `resourceOffers`, just not the executor](https://github.com/apache/spark/blob/9613424898fd2a586156bc4eb48e255749774f20/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala#L298).
But isn't there a bug in `resourceOffers` -- shouldn't that loop be updating `newExecAvailable` even if an executor is added to an already existing host? I expect this to actually be quite common under dynamic allocation. The end result is that locality preferences aren't properly updated, and `failedEpochs` aren't updated correctly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r68642206
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -83,7 +83,7 @@ private[spark] class TaskSetManager(
val copiesRunning = new Array[Int](numTasks)
val successful = new Array[Boolean](numTasks)
private val numFailures = new Array[Int](numTasks)
- // key is taskId, value is a Map of executor id to when it failed
+ // key is taskId (aka TaskInfo.index), value is a Map of executor id to when it failed
--- End diff --
here (and `executorIsBlacklisted`) is what got me confused about the meaning of these identifiers. I knew it was defintiely *not* the globally unique taskId, aka TID.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r67067563
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -280,10 +280,54 @@ private[spark] class TaskSchedulerImpl(
}
}
}
+ if (!launchedTask && isTaskSetCompletelyBlacklisted(taskSet)) {
+ taskSet.abort(s"Aborting ${taskSet.taskSet} because it has a task which cannot be scheduled" +
+ s" on any executor due to blacklists.")
+ }
return launchedTask
}
/**
+ * Check whether the given task set has been blacklisted to the point that it can't run anywhere.
+ *
+ * It is possible that this taskset has become impossible to schedule *anywhere* due to the
+ * blacklist. The most common scenario would be if there are fewer executors than
+ * spark.task.maxFailures. We need to detect this so we can fail the task set, otherwise the job
+ * will hang.
+ *
+ * The check here is a balance between being sure to catch the issue, but not wasting
+ * too much time inside the scheduling loop. Just check if the last task is schedulable
+ * on any of the available executors. So this is O(numExecutors) worst-case, but it'll
+ * really be fast unless you've got a bunch of things blacklisted. Its possible it won't detect
+ * the unschedulable task immediately, but if it returns false, there is at least *some* task
+ * that is schedulable, and after scheduling all of those, we'll eventually find the unschedulable
+ * task.
+ */
+ private[scheduler] def isTaskSetCompletelyBlacklisted(
--- End diff --
yeah I put it here b/c in the blacklisting stuff I'm still working on, I felt it made more sense outside TaskSetManager, since blacklisting extends beyond a single taskset (executor & node blacklisting). But I'll change it here, and we can revisit that discussion when looking at that change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the issue:
https://github.com/apache/spark/pull/13603
@kayousterhout sorry for the delays on my end, I've updated with the requested changes. The check is now inside `TaskSetManager`, and I added a test for the "unschedulablility" case I was describing. Though the behavior is ust in `TaskSetManager`, I put the test in `TaskSchedulerImplSuite`, because I'd like to push this test up to a slightly higher, more functional level, so its easier to refactor etc. and keep the test. But if that is opening up a larger discussion, I can just push it to `TaskSetManagerSuite` instead.
I also found something kinda weird about `TaskSchedulerImpl` I'll comment on inline, mostly unrelated to this change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61378 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61378/consoleFull)** for PR 13603 at commit [`860ee1a`](https://github.com/apache/spark/commit/860ee1a39c8ad02023a0f539c2d3f0c4c6df09df).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the issue:
https://github.com/apache/spark/pull/13603
@kayousterhout sure I'll pull the visibility stuff out.
I did consider trying to do a check on task failure instead. However, I don't think that is sufficient, because you can have an executor fail. Imagine you have task 1 on executor A & task 2 on executor B. Task 1 fails, gets blacklisted from executor A -- but it can still be scheduled on executor B so you don't fail the stage. Then executor B dies. Task 2 can run on executor A, so that isn't stuck. But task 1 now can't run anywhere.
Probably unlikely, but still having the job just hang is so bad that I think we really should avoid it. Plus it becomes much more likely w/ the new blacklisting I'm working on -- in that case, executor B gets blacklisted for the bad stage because of many task failures, and now there isn't any place for the first failed tasks to run. I actually ran into that case when testing an early iteration of that change.
This is subtle enough its probably worth codifying into a test -- I'll work on adding that.
(I agree with you that its OK to fail the task set even if a new executor is just about to launch. Even this version doesn't really avoid something like that.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/13603
@rxin no, this is an old (an un-documented) feature that was added a while ago as a band-aid until we did the more complete solution (which @squito is planning to do soon, but not targeted for 2.0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r67943657
--- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -96,22 +120,13 @@ class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext with L
taskDescriptions = taskScheduler.resourceOffers(multiCoreWorkerOffers).flatten
assert(1 === taskDescriptions.length)
assert("executor0" === taskDescriptions(0).executorId)
+ assert(!failedTaskSet)
}
test("Scheduler does not crash when tasks are not serializable") {
- sc = new SparkContext("local", "TaskSchedulerImplSuite")
--- End diff --
mentioned this below as well, but just to be clear -- I was mistaken, that bug doesn't effect the case where the tasks aren't serializable. That still correctly fails with an error about serialization. The error I was encountering is in a different case ("multiple CPUs per task", since there you never add the executors, just the hosts), and still needs a workaround for now, which I've added.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/13603
LGTM!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60855/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61497 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61497/consoleFull)** for PR 13603 at commit [`ed71c99`](https://github.com/apache/spark/commit/ed71c99d886149b71a1123c014cf0004425edbf5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60953/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61505/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61049 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61049/consoleFull)** for PR 13603 at commit [`39fbf72`](https://github.com/apache/spark/commit/39fbf72df8ff9f4c61bc19e4a2fb4c7332109116).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/13603
@squito if it's not too painful, would you mind moving the visibility stuff to a separate PR? (I suspect that PR can be merged almost immediately!).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #60856 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60856/consoleFull)** for PR 13603 at commit [`3edb6fe`](https://github.com/apache/spark/commit/3edb6fe37f40e636d43340321066b11d23fbba97).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61399/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #60851 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60851/consoleFull)** for PR 13603 at commit [`f870bde`](https://github.com/apache/spark/commit/f870bde39c4f0e0b83e079439291b432717a5a82).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r68808158
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -279,6 +279,9 @@ private[spark] class TaskSchedulerImpl(
}
}
}
+ if (!launchedTask) {
+ taskSet.abortIfTaskSetCompletelyBlacklisted(executorsByHost)
--- End diff --
Based on our discussion yesterday, should this use executorIdToHost instead of executorsByHost? (that would also be marginally easier to parse in abortIfTaskSetCompletelyBlacklisted)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/13603
Did you consider instead doing this when a task fails (on line 761 in TaskSetManager)? Instead of just checking if the number of failures is greater than maxTaskFailures, you could add a second check (if blacklisting is enabled) that checks whether the task that just failed could be scheduled anywhere, and if it can't be, fail the task set. This seems simpler to me.
The main drawback I see in that approach is that it could be the case that the task failure was caused by an executor failure, and the cluster manager is in the process of launching a new executor that the task could run on, so it's not correct to fail the task set. My sense is that it's OK-ish to fail in that case, since that seems like it will only happen for jobs that use a super small number of executors, in which case random-ish failures are less likely, so the failure is more likely to be a real issue with the job.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60955/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r67775501
--- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -272,6 +261,55 @@ class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext with L
val taskDescriptions3 = taskScheduler.resourceOffers(e1Offers).flatten
assert(1 === taskDescriptions3.length)
assert("executor1" === taskDescriptions3(0).executorId)
+ assert(!failedTaskSet)
+ }
+
+ test("abort stage if executor loss results in unschedulability from previously failed tasks") {
+ // Make sure we can detect when a taskset becomes unschedulability from a blacklisting. This
+ // test explores a particular corner case -- you may have one task fail, but still be
+ // schedulable on another executor. However, that executor may fail later on, leaving the
+ // first task with no place to run.
+ val taskScheduler = setupScheduler(
+ // set this to something much longer than the test duration
+ "spark.scheduler.executorTaskBlacklistTime" -> "10000000"
+ )
+
+ val taskSet = FakeTask.createTaskSet(2)
+ taskScheduler.submitTasks(taskSet)
+ val tsm = taskScheduler.taskSetManagerForAttempt(taskSet.stageId, taskSet.stageAttemptId).get
+
+ val firstTasks = taskScheduler.resourceOffers(Seq(
+ new WorkerOffer("executor0", "host0", 1),
+ new WorkerOffer("executor1", "host1", 1)
+ )).flatten
+ assert(Set("executor0", "executor1") === firstTasks.map{_.executorId}.toSet)
+
+ // fail one of the tasks, but leave the other running
+ val failedTask = firstTasks.find(_.executorId == "executor0").get
+ taskScheduler.handleFailedTask(tsm, failedTask.taskId, TaskState.FAILED, TaskResultLost)
+ // at this point, our failed task could run on the other executor, so don't give up the task
+ // set yet.
+
+ // but now we fail our second executor. The other task still has a failed executor it can
+ // run on, but our first failed task does not. So we should fail the task set.
--- End diff --
It's a little confusing that the comment here says "So we should fail the task set", because what happens in the next few lines is that the task set doesn't get failed. Maybe instead: "The other task can still run on executor1, so make an offer on that executor, and make sure that the other task (not the failed one) is assigned to run there"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r67774823
--- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -96,22 +120,13 @@ class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext with L
taskDescriptions = taskScheduler.resourceOffers(multiCoreWorkerOffers).flatten
assert(1 === taskDescriptions.length)
assert("executor0" === taskDescriptions(0).executorId)
+ assert(!failedTaskSet)
}
test("Scheduler does not crash when tasks are not serializable") {
- sc = new SparkContext("local", "TaskSchedulerImplSuite")
--- End diff --
I agree that this looks completely wrong, and that (1) resourceOffers should mark a new executor as available more times, not just when a new host is available -- and this means the HDFS cache locality hasn't been working when folks have multiple executors and (2) resourceOffers, and not resourceOffersSingleTaskSet, should add the executor to executorsByHost. Do you have time to file a JIRA / fix this? Seems like a quick fix, and would be nice to do before this PR, because of this weird failure (which seems like something that will be user-visible, since I'm guessing it's not uncommon, for new users, that their first task isn't serializable).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r67899538
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -576,6 +576,62 @@ private[spark] class TaskSetManager(
}
/**
+ * Check whether the given task set has been blacklisted to the point that it can't run anywhere.
+ *
+ * It is possible that this taskset has become impossible to schedule *anywhere* due to the
+ * blacklist. The most common scenario would be if there are fewer executors than
+ * spark.task.maxFailures. We need to detect this so we can fail the task set, otherwise the job
+ * will hang.
+ *
--- End diff --
yes, much better. Also good point that its only O(maxTaskFailures), I hadn't realized that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r69037911
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -576,6 +576,59 @@ private[spark] class TaskSetManager(
}
/**
+ * Check whether the given task set has been blacklisted to the point that it can't run anywhere.
+ *
+ * It is possible that this taskset has become impossible to schedule *anywhere* due to the
+ * blacklist. The most common scenario would be if there are fewer executors than
+ * spark.task.maxFailures. We need to detect this so we can fail the task set, otherwise the job
+ * will hang.
+ *
+ * There's a tradeoff here: we could make sure all tasks in the task set are schedulable, but that
+ * would add extra time to each iteration of the scheduling loop. Here, we take the approach of
+ * making sure at least one of the unscheduled tasks is schedulable. This means we may not detect
+ * the hang as quickly as we could have, but we'll always detect the hang eventually, and the
+ * method is faster in the typical case. In the worst case, this method can take
+ * O(maxTaskFailures + numTasks) time, but it will be faster when there haven't been any task
+ * failures (this is because the method picks on unscheduled task, and then iterates through each
+ * executor until it finds one that the task hasn't failed on already).
+ */
+ private[scheduler] def abortIfCompletelyBlacklisted(executors: Iterable[String]): Unit = {
+
+ def pendingTask: Option[Int] = {
--- End diff --
oh I didn't even know about lastIndexWhere! thanks, simpler, and despite being a minor point I appreciate learning something new :)
sorry I think I just got on the wrong track in this while thinking about doing the lazy-removal here as well, and when I decided against it never stepped back to simplify it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r69021897
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -576,6 +576,59 @@ private[spark] class TaskSetManager(
}
/**
+ * Check whether the given task set has been blacklisted to the point that it can't run anywhere.
+ *
+ * It is possible that this taskset has become impossible to schedule *anywhere* due to the
+ * blacklist. The most common scenario would be if there are fewer executors than
+ * spark.task.maxFailures. We need to detect this so we can fail the task set, otherwise the job
+ * will hang.
+ *
+ * There's a tradeoff here: we could make sure all tasks in the task set are schedulable, but that
+ * would add extra time to each iteration of the scheduling loop. Here, we take the approach of
+ * making sure at least one of the unscheduled tasks is schedulable. This means we may not detect
+ * the hang as quickly as we could have, but we'll always detect the hang eventually, and the
+ * method is faster in the typical case. In the worst case, this method can take
+ * O(maxTaskFailures + numTasks) time, but it will be faster when there haven't been any task
+ * failures (this is because the method picks on unscheduled task, and then iterates through each
+ * executor until it finds one that the task hasn't failed on already).
+ */
+ private[scheduler] def abortIfCompletelyBlacklisted(executors: Iterable[String]): Unit = {
+
+ def pendingTask: Option[Int] = {
--- End diff --
you can't just change this to a `val` with nothing else, b/c of the `return` when we find the task. Though you could make it a `val` by changing the inner-logic, with a `var keepGoing` in the while loop or something. I actually changed it once and couldn't really make up my mind which version was cleaner ... in the end I felt an inner function wasn't so bad, but happy to change it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61322 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61322/consoleFull)** for PR 13603 at commit [`e8e4fe5`](https://github.com/apache/spark/commit/e8e4fe5989bf07693f49842253df792d70fb7286).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13603
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61341/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r67904173
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -576,6 +576,62 @@ private[spark] class TaskSetManager(
}
/**
+ * Check whether the given task set has been blacklisted to the point that it can't run anywhere.
+ *
+ * It is possible that this taskset has become impossible to schedule *anywhere* due to the
+ * blacklist. The most common scenario would be if there are fewer executors than
+ * spark.task.maxFailures. We need to detect this so we can fail the task set, otherwise the job
+ * will hang.
+ *
--- End diff --
After further thought I think this is actually O(maxTaskFailures + # tasks) right? Since you might need to first iterate through all of the tasks to find one that hasn't started yet, and then iterate through some failed executors?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #60955 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60955/consoleFull)** for PR 13603 at commit [`158775c`](https://github.com/apache/spark/commit/158775c13b16e6b303147c78443addce6e6aacd1).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13603: [SPARK-15865][CORE] Blacklist should not result in job h...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13603
**[Test build #61061 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61061/consoleFull)** for PR 13603 at commit [`c606dc3`](https://github.com/apache/spark/commit/c606dc31de84d3bdfbbd2f00827206e150adf6b3).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13603: [SPARK-15865][CORE] Blacklist should not result i...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r67775034
--- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -272,6 +261,55 @@ class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext with L
val taskDescriptions3 = taskScheduler.resourceOffers(e1Offers).flatten
assert(1 === taskDescriptions3.length)
assert("executor1" === taskDescriptions3(0).executorId)
+ assert(!failedTaskSet)
+ }
+
+ test("abort stage if executor loss results in unschedulability from previously failed tasks") {
+ // Make sure we can detect when a taskset becomes unschedulability from a blacklisting. This
+ // test explores a particular corner case -- you may have one task fail, but still be
+ // schedulable on another executor. However, that executor may fail later on, leaving the
+ // first task with no place to run.
+ val taskScheduler = setupScheduler(
+ // set this to something much longer than the test duration
+ "spark.scheduler.executorTaskBlacklistTime" -> "10000000"
+ )
+
+ val taskSet = FakeTask.createTaskSet(2)
+ taskScheduler.submitTasks(taskSet)
+ val tsm = taskScheduler.taskSetManagerForAttempt(taskSet.stageId, taskSet.stageAttemptId).get
+
+ val firstTasks = taskScheduler.resourceOffers(Seq(
+ new WorkerOffer("executor0", "host0", 1),
+ new WorkerOffer("executor1", "host1", 1)
+ )).flatten
+ assert(Set("executor0", "executor1") === firstTasks.map{_.executorId}.toSet)
+
+ // fail one of the tasks, but leave the other running
+ val failedTask = firstTasks.find(_.executorId == "executor0").get
+ taskScheduler.handleFailedTask(tsm, failedTask.taskId, TaskState.FAILED, TaskResultLost)
+ // at this point, our failed task could run on the other executor, so don't give up the task
+ // set yet.
--- End diff --
does it make sense to verify !failedTaskSet here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org