You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/29 21:48:48 UTC
[GitHub] [spark] tgravescs opened a new pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
tgravescs opened a new pull request #29906:
URL: https://github.com/apache/spark/pull/29906
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
this PR renames the blacklisting feature. I ended up using "excludeOnFailure" or "excluded" in most cases but there is a mix. I renamed the BlacklistTracker to HealthTracker, but for the TaskSetBlacklist HealthTracker didn't make sense to me since its not the health of the taskset itself but rather tracking the things its excluded on so I renamed it to be TaskSetExcludeList. Everything else I tried to use the context and in most cases excluded made sense. It made more sense to me then blocked since you are basically excluding those executors and nodes from scheduling tasks on them. Then can be unexcluded later after timeouts and such. The configs I changed the name to use excludeOnFailure which I thought explained it.
I unfortunately couldn't get rid of some of them because its part of the event listener and history files. To keep backwards compatibility I kept the events and some of the parsing so that the history server would still properly read older history files. It is not forward compatible though - meaning a new application write the "Excluded" events so the older history server won't properly read display them as being blacklisted.
A few of the files below are showing up as deleted and recreated even though I did a git mv on them. I'm not sure why.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
get rid of problematic language
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Config name changes but the old configs still work but are deprecated.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
updated tests and also manually tested the UI changes and manually tested the history server reading older versions of history files and vice versa.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-715646423
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705901360
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r501046402
##########
File path: docs/monitoring.md
##########
@@ -1125,12 +1125,14 @@ This is the component with the largest amount of instrumented metrics
- stages.failedStages.count
- stages.skippedStages.count
- stages.completedStages.count
- - tasks.blackListedExecutors.count
+ - tasks.blackListedExecutors.count // deprecated use excludedExecutors instead
Review comment:
I think it is as until we come up with different way of deprecating only way to tell users
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705619969
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r506582624
##########
File path: core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala
##########
@@ -0,0 +1,478 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.atomic.AtomicReference
+
+import scala.collection.mutable.{ArrayBuffer, HashMap, HashSet}
+
+import org.apache.spark.{ExecutorAllocationClient, SparkConf, SparkContext}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config
+import org.apache.spark.util.{Clock, SystemClock, Utils}
+
+/**
+ * HealthTracker is designed to track problematic executors and nodes. It supports excluding
+ * executors and nodes across an entire application (with a periodic expiry). TaskSetManagers add
+ * additional logic for exclusion of executors and nodes for individual tasks and stages which
+ * works in concert with the logic here.
+ *
+ * The tracker needs to deal with a variety of workloads, eg.:
+ *
+ * * bad user code -- this may lead to many task failures, but that should not count against
+ * individual executors
+ * * many small stages -- this may prevent a bad executor for having many failures within one
+ * stage, but still many failures over the entire application
+ * * "flaky" executors -- they don't fail every task, but are still faulty enough to merit
+ * excluding
+ *
+ * See the design doc on SPARK-8425 for a more in-depth discussion.
+ *
+ * THREADING: As with most helpers of TaskSchedulerImpl, this is not thread-safe. Though it is
+ * called by multiple threads, callers must already have a lock on the TaskSchedulerImpl. The
+ * one exception is [[excludedNodeList()]], which can be called without holding a lock.
+ */
+private[scheduler] class HealthTracker (
+ private val listenerBus: LiveListenerBus,
+ conf: SparkConf,
+ allocationClient: Option[ExecutorAllocationClient],
+ clock: Clock = new SystemClock()) extends Logging {
+
+ def this(sc: SparkContext, allocationClient: Option[ExecutorAllocationClient]) = {
+ this(sc.listenerBus, sc.conf, allocationClient)
+ }
+
+ HealthTracker.validateExcludeOnFailureConfs(conf)
+ private val MAX_FAILURES_PER_EXEC = conf.get(config.MAX_FAILURES_PER_EXEC)
+ private val MAX_FAILED_EXEC_PER_NODE = conf.get(config.MAX_FAILED_EXEC_PER_NODE)
+ val EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS = HealthTracker.getExludeOnFailureTimeout(conf)
+ private val EXCLUDE_FETCH_FAILURE_ENABLED =
+ conf.get(config.EXCLUDE_ON_FAILURE_FETCH_FAILURE_ENABLED)
+
+ /**
+ * A map from executorId to information on task failures. Tracks the time of each task failure,
+ * so that we can avoid excluding executors due to failures that are very far apart. We do not
+ * actively remove from this as soon as tasks hit their timeouts, to avoid the time it would take
+ * to do so. But it will not grow too large, because as soon as an executor gets too many
+ * failures, we exclude the executor and remove its entry here.
+ */
+ private val executorIdToFailureList = new HashMap[String, ExecutorFailureList]()
+ val executorIdToExcludedStatus = new HashMap[String, ExcludedExecutor]()
+ val nodeIdToExcludedExpiryTime = new HashMap[String, Long]()
+ /**
+ * An immutable copy of the set of nodes that are currently excluded. Kept in an
+ * AtomicReference to make [[excludedNodeList()]] thread-safe.
+ */
+ private val _excludedNodeList = new AtomicReference[Set[String]](Set())
+ /**
+ * Time when the next excluded node will expire. Used as a shortcut to
+ * avoid iterating over all entries in the excludedNodeList when none will have expired.
+ */
+ var nextExpiryTime: Long = Long.MaxValue
+ /**
+ * Mapping from nodes to all of the executors that have been excluded on that node. We do *not*
+ * remove from this when executors are removed from spark, so we can track when we get multiple
+ * successive excluded executors on one node. Nonetheless, it will not grow too large because
+ * there cannot be many excluded executors on one node, before we stop requesting more
+ * executors on that node, and we clean up the list of exluded executors once an executor has
+ * been excluded for EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS.
+ */
+ val nodeToExcludedExecs = new HashMap[String, HashSet[String]]()
+
+ /**
+ * Include executors and nodes that have been excluded for at least
+ * EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS
+ */
+ def applyExcludeOnFailureTimeout(): Unit = {
+ val now = clock.getTimeMillis()
+ // quickly check if we've got anything to expire that is excluded -- if not,
+ // avoid doing any work
+ if (now > nextExpiryTime) {
+ // Apply the timeout to excluded nodes and executors
+ val execsToInclude = executorIdToExcludedStatus.filter(_._2.expiryTime < now).keys
+ if (execsToInclude.nonEmpty) {
+ // Include any executors that have been exluded longer than the excludeOnFailure timeout.
+ logInfo(s"Removing executors $execsToInclude from exclude list because the " +
+ s"the executors have reached the timed out")
+ execsToInclude.foreach { exec =>
+ val status = executorIdToExcludedStatus.remove(exec).get
+ val failedExecsOnNode = nodeToExcludedExecs(status.node)
+ listenerBus.post(SparkListenerExecutorUnexcluded(now, exec))
+ failedExecsOnNode.remove(exec)
+ if (failedExecsOnNode.isEmpty) {
+ nodeToExcludedExecs.remove(status.node)
+ }
+ }
+ }
+ val nodesToInclude = nodeIdToExcludedExpiryTime.filter(_._2 < now).keys
+ if (nodesToInclude.nonEmpty) {
+ // Include any nodes that have been excluded longer than the excludeOnFailure timeout.
+ logInfo(s"Removing nodes $nodesToInclude from exclude list because the " +
+ s"nodes have reached has timed out")
+ nodesToInclude.foreach { node =>
+ nodeIdToExcludedExpiryTime.remove(node)
+ listenerBus.post(SparkListenerNodeUnexcluded(now, node))
+ }
+ _excludedNodeList.set(nodeIdToExcludedExpiryTime.keySet.toSet)
+ }
+ updateNextExpiryTime()
+ }
+ }
+
+ private def updateNextExpiryTime(): Unit = {
+ val execMinExpiry = if (executorIdToExcludedStatus.nonEmpty) {
+ executorIdToExcludedStatus.map{_._2.expiryTime}.min
+ } else {
+ Long.MaxValue
+ }
+ val nodeMinExpiry = if (nodeIdToExcludedExpiryTime.nonEmpty) {
+ nodeIdToExcludedExpiryTime.values.min
+ } else {
+ Long.MaxValue
+ }
+ nextExpiryTime = math.min(execMinExpiry, nodeMinExpiry)
+ }
+
+ private def killExecutor(exec: String, msg: String): Unit = {
+ allocationClient match {
+ case Some(a) =>
+ logInfo(msg)
+ a.killExecutors(Seq(exec), adjustTargetNumExecutors = false, countFailures = false,
+ force = true)
+ case None =>
+ logInfo(s"Not attempting to kill excluded executor id $exec " +
+ s"since allocation client is not defined.")
+ }
+ }
+
+ private def killExcludedExecutor(exec: String): Unit = {
+ if (conf.get(config.EXCLUDE_ON_FAILURE_KILL_ENABLED)) {
+ killExecutor(exec, s"Killing excluded executor id $exec since " +
+ s"${config.EXCLUDE_ON_FAILURE_KILL_ENABLED.key} is set.")
+ }
+ }
+
+ private[scheduler] def killExcludedIdleExecutor(exec: String): Unit = {
+ killExecutor(exec,
+ s"Killing excluded idle executor id $exec because of task unschedulability and trying " +
+ "to acquire a new executor.")
+ }
+
+ private def killExecutorsOnExcludedNode(node: String): Unit = {
+ if (conf.get(config.EXCLUDE_ON_FAILURE_KILL_ENABLED)) {
+ allocationClient match {
+ case Some(a) =>
+ logInfo(s"Killing all executors on excluded host $node " +
+ s"since ${config.EXCLUDE_ON_FAILURE_KILL_ENABLED.key} is set.")
+ if (a.killExecutorsOnHost(node) == false) {
+ logError(s"Killing executors on node $node failed.")
+ }
+ case None =>
+ logWarning(s"Not attempting to kill executors on excluded host $node " +
+ s"since allocation client is not defined.")
+ }
+ }
+ }
+
+ def updateExcludedForFetchFailure(host: String, exec: String): Unit = {
+ if (EXCLUDE_FETCH_FAILURE_ENABLED) {
+ // If we exclude on fetch failures, we are implicitly saying that we believe the failure is
+ // non-transient, and can't be recovered from (even if this is the first fetch failure,
+ // stage is retried after just one failure, so we don't always get a chance to collect
+ // multiple fetch failures).
+ // If the external shuffle-service is on, then every other executor on this node would
+ // be suffering from the same issue, so we should exclude (and potentially kill) all
+ // of them immediately.
+
+ val now = clock.getTimeMillis()
+ val expiryTimeForNewExcludes = now + EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS
+
+ if (conf.get(config.SHUFFLE_SERVICE_ENABLED)) {
+ if (!nodeIdToExcludedExpiryTime.contains(host)) {
+ logInfo(s"excluding node $host due to fetch failure of external shuffle service")
+
+ nodeIdToExcludedExpiryTime.put(host, expiryTimeForNewExcludes)
+ listenerBus.post(SparkListenerNodeExcluded(now, host, 1))
+ _excludedNodeList.set(nodeIdToExcludedExpiryTime.keySet.toSet)
+ killExecutorsOnExcludedNode(host)
+ updateNextExpiryTime()
+ }
+ } else if (!executorIdToExcludedStatus.contains(exec)) {
+ logInfo(s"Excluding executor $exec due to fetch failure")
+
+ executorIdToExcludedStatus.put(exec, ExcludedExecutor(host, expiryTimeForNewExcludes))
+ // We hardcoded number of failure tasks to 1 for fetch failure, because there's no
+ // reattempt for such failure.
+ listenerBus.post(SparkListenerExecutorExcluded(now, exec, 1))
Review comment:
* Both `SparkListener` and `SparkFirehoseListener` are exposed classes : extending `SparkListenerInterface` (we missed adding `DeveloperApi` to `SparkFirehoseListener`).
* `sc.addSparkListener()` should work fine for these user supplied subclasses.
* We will have subclasses depending on these callbacks in user codebases - including consuming existing blacklist events this way.
* We can currently make forward compatible changes in `SparkListenerInterface`.
The issue we have here is a backwardly incompatible change - this is an issue for the exposed api.
While we can make backwardly incompatible changes to `SparkListenerInterface` directly (like dropping callbacks), the exposed subclasses should continue to preserve behavior while we go through deprecation cycle.
Until we remove support, this will mean we also rewire the new events to call the old handler methods until we drop support in 4.0 (?).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-712408158
changed to leave all the original blacklist events and also it now posts both the old and new events.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701517373
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33898/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r501165424
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala
##########
@@ -379,14 +379,15 @@ package object config extends Logging {
.stringConf
.createOptional
- /* YARN allocator-level blacklisting related config entries. */
- private[spark] val YARN_EXECUTOR_LAUNCH_BLACKLIST_ENABLED =
- ConfigBuilder("spark.yarn.blacklist.executor.launch.blacklisting.enabled")
+ /* YARN allocator-level excludeOnFailure related config entries. */
+ private[spark] val YARN_EXECUTOR_LAUNCH_EXCLUDE_ON_FAILURE_ENABLED =
+ ConfigBuilder("spark.yarn.executor.launch.excludeOnFailure.enabled")
Review comment:
Sure, makes sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r498369321
##########
File path: core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala
##########
@@ -0,0 +1,478 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.atomic.AtomicReference
+
+import scala.collection.mutable.{ArrayBuffer, HashMap, HashSet}
+
+import org.apache.spark.{ExecutorAllocationClient, SparkConf, SparkContext}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config
+import org.apache.spark.util.{Clock, SystemClock, Utils}
+
+/**
+ * HealthTracker is designed to track problematic executors and nodes. It supports excluding
+ * executors and nodes across an entire application (with a periodic expiry). TaskSetManagers add
+ * additional logic for exclusion of executors and nodes for individual tasks and stages which
+ * works in concert with the logic here.
+ *
+ * The tracker needs to deal with a variety of workloads, eg.:
+ *
+ * * bad user code -- this may lead to many task failures, but that should not count against
+ * individual executors
+ * * many small stages -- this may prevent a bad executor for having many failures within one
+ * stage, but still many failures over the entire application
+ * * "flaky" executors -- they don't fail every task, but are still faulty enough to merit
+ * excluding
+ *
+ * See the design doc on SPARK-8425 for a more in-depth discussion.
+ *
+ * THREADING: As with most helpers of TaskSchedulerImpl, this is not thread-safe. Though it is
+ * called by multiple threads, callers must already have a lock on the TaskSchedulerImpl. The
+ * one exception is [[excludedNodeList()]], which can be called without holding a lock.
+ */
+private[scheduler] class HealthTracker (
+ private val listenerBus: LiveListenerBus,
+ conf: SparkConf,
+ allocationClient: Option[ExecutorAllocationClient],
+ clock: Clock = new SystemClock()) extends Logging {
+
+ def this(sc: SparkContext, allocationClient: Option[ExecutorAllocationClient]) = {
+ this(sc.listenerBus, sc.conf, allocationClient)
+ }
+
+ HealthTracker.validateExcludeOnFailureConfs(conf)
+ private val MAX_FAILURES_PER_EXEC = conf.get(config.MAX_FAILURES_PER_EXEC)
+ private val MAX_FAILED_EXEC_PER_NODE = conf.get(config.MAX_FAILED_EXEC_PER_NODE)
+ val EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS = HealthTracker.getExludeOnFailureTimeout(conf)
+ private val EXCLUDE_FETCH_FAILURE_ENABLED =
+ conf.get(config.EXCLUDE_ON_FAILURE_FETCH_FAILURE_ENABLED)
+
+ /**
+ * A map from executorId to information on task failures. Tracks the time of each task failure,
+ * so that we can avoid excluding executors due to failures that are very far apart. We do not
+ * actively remove from this as soon as tasks hit their timeouts, to avoid the time it would take
+ * to do so. But it will not grow too large, because as soon as an executor gets too many
+ * failures, we exclude the executor and remove its entry here.
+ */
+ private val executorIdToFailureList = new HashMap[String, ExecutorFailureList]()
+ val executorIdToExcludedStatus = new HashMap[String, ExcludedExecutor]()
+ val nodeIdToExcludedExpiryTime = new HashMap[String, Long]()
+ /**
+ * An immutable copy of the set of nodes that are currently excluded. Kept in an
+ * AtomicReference to make [[excludedNodeList()]] thread-safe.
+ */
+ private val _excludedNodeList = new AtomicReference[Set[String]](Set())
+ /**
+ * Time when the next excluded node will expire. Used as a shortcut to
+ * avoid iterating over all entries in the excludedNodeList when none will have expired.
+ */
+ var nextExpiryTime: Long = Long.MaxValue
+ /**
+ * Mapping from nodes to all of the executors that have been excluded on that node. We do *not*
+ * remove from this when executors are removed from spark, so we can track when we get multiple
+ * successive excluded executors on one node. Nonetheless, it will not grow too large because
+ * there cannot be many excluded executors on one node, before we stop requesting more
+ * executors on that node, and we clean up the list of exluded executors once an executor has
+ * been excluded for EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS.
+ */
+ val nodeToExcludedExecs = new HashMap[String, HashSet[String]]()
+
+ /**
+ * Include executors and nodes that have been excluded for at least
+ * EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS
+ */
+ def applyExcludeOnFailureTimeout(): Unit = {
+ val now = clock.getTimeMillis()
+ // quickly check if we've got anything to expire that is excluded -- if not,
+ // avoid doing any work
+ if (now > nextExpiryTime) {
+ // Apply the timeout to excluded nodes and executors
+ val execsToInclude = executorIdToExcludedStatus.filter(_._2.expiryTime < now).keys
+ if (execsToInclude.nonEmpty) {
+ // Include any executors that have been exluded longer than the excludeOnFailure timeout.
+ logInfo(s"Removing executors $execsToInclude from exclude list because the " +
+ s"the executors have reached the timed out")
+ execsToInclude.foreach { exec =>
+ val status = executorIdToExcludedStatus.remove(exec).get
+ val failedExecsOnNode = nodeToExcludedExecs(status.node)
+ listenerBus.post(SparkListenerExecutorUnexcluded(now, exec))
+ failedExecsOnNode.remove(exec)
+ if (failedExecsOnNode.isEmpty) {
+ nodeToExcludedExecs.remove(status.node)
+ }
+ }
+ }
+ val nodesToInclude = nodeIdToExcludedExpiryTime.filter(_._2 < now).keys
+ if (nodesToInclude.nonEmpty) {
+ // Include any nodes that have been excluded longer than the excludeOnFailure timeout.
+ logInfo(s"Removing nodes $nodesToInclude from exclude list because the " +
+ s"nodes have reached has timed out")
+ nodesToInclude.foreach { node =>
+ nodeIdToExcludedExpiryTime.remove(node)
+ listenerBus.post(SparkListenerNodeUnexcluded(now, node))
+ }
+ _excludedNodeList.set(nodeIdToExcludedExpiryTime.keySet.toSet)
+ }
+ updateNextExpiryTime()
+ }
+ }
+
+ private def updateNextExpiryTime(): Unit = {
+ val execMinExpiry = if (executorIdToExcludedStatus.nonEmpty) {
+ executorIdToExcludedStatus.map{_._2.expiryTime}.min
+ } else {
+ Long.MaxValue
+ }
+ val nodeMinExpiry = if (nodeIdToExcludedExpiryTime.nonEmpty) {
+ nodeIdToExcludedExpiryTime.values.min
+ } else {
+ Long.MaxValue
+ }
+ nextExpiryTime = math.min(execMinExpiry, nodeMinExpiry)
+ }
+
+ private def killExecutor(exec: String, msg: String): Unit = {
+ allocationClient match {
+ case Some(a) =>
+ logInfo(msg)
+ a.killExecutors(Seq(exec), adjustTargetNumExecutors = false, countFailures = false,
+ force = true)
+ case None =>
+ logInfo(s"Not attempting to kill excluded executor id $exec " +
+ s"since allocation client is not defined.")
+ }
+ }
+
+ private def killExcludedExecutor(exec: String): Unit = {
+ if (conf.get(config.EXCLUDE_ON_FAILURE_KILL_ENABLED)) {
+ killExecutor(exec, s"Killing excluded executor id $exec since " +
+ s"${config.EXCLUDE_ON_FAILURE_KILL_ENABLED.key} is set.")
+ }
+ }
+
+ private[scheduler] def killExcludedIdleExecutor(exec: String): Unit = {
+ killExecutor(exec,
+ s"Killing excluded idle executor id $exec because of task unschedulability and trying " +
+ "to acquire a new executor.")
+ }
+
+ private def killExecutorsOnExcludedNode(node: String): Unit = {
+ if (conf.get(config.EXCLUDE_ON_FAILURE_KILL_ENABLED)) {
+ allocationClient match {
+ case Some(a) =>
+ logInfo(s"Killing all executors on excluded host $node " +
+ s"since ${config.EXCLUDE_ON_FAILURE_KILL_ENABLED.key} is set.")
+ if (a.killExecutorsOnHost(node) == false) {
+ logError(s"Killing executors on node $node failed.")
+ }
+ case None =>
+ logWarning(s"Not attempting to kill executors on excluded host $node " +
+ s"since allocation client is not defined.")
+ }
+ }
+ }
+
+ def updateExcludedForFetchFailure(host: String, exec: String): Unit = {
+ if (EXCLUDE_FETCH_FAILURE_ENABLED) {
+ // If we exclude on fetch failures, we are implicitly saying that we believe the failure is
+ // non-transient, and can't be recovered from (even if this is the first fetch failure,
+ // stage is retried after just one failure, so we don't always get a chance to collect
+ // multiple fetch failures).
+ // If the external shuffle-service is on, then every other executor on this node would
+ // be suffering from the same issue, so we should exclude (and potentially kill) all
+ // of them immediately.
+
+ val now = clock.getTimeMillis()
+ val expiryTimeForNewExcludes = now + EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS
+
+ if (conf.get(config.SHUFFLE_SERVICE_ENABLED)) {
+ if (!nodeIdToExcludedExpiryTime.contains(host)) {
+ logInfo(s"excluding node $host due to fetch failure of external shuffle service")
+
+ nodeIdToExcludedExpiryTime.put(host, expiryTimeForNewExcludes)
+ listenerBus.post(SparkListenerNodeExcluded(now, host, 1))
+ _excludedNodeList.set(nodeIdToExcludedExpiryTime.keySet.toSet)
+ killExecutorsOnExcludedNode(host)
+ updateNextExpiryTime()
+ }
+ } else if (!executorIdToExcludedStatus.contains(exec)) {
+ logInfo(s"Excluding executor $exec due to fetch failure")
+
+ executorIdToExcludedStatus.put(exec, ExcludedExecutor(host, expiryTimeForNewExcludes))
+ // We hardcoded number of failure tasks to 1 for fetch failure, because there's no
+ // reattempt for such failure.
+ listenerBus.post(SparkListenerExecutorExcluded(now, exec, 1))
Review comment:
Can we provide a default listener implementation that delegates the event to the old method?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701012133
**[Test build #129254 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129254/testReport)** for PR 29906 at commit [`6657785`](https://github.com/apache/spark/commit/6657785b249241232780117b4b10966a5d08aab0).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701012147
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-712441982
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-709514102
any other comments on this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705872036
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34171/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r497085607
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -381,7 +381,7 @@ private[spark] class TaskSchedulerImpl(
: (Boolean, Option[TaskLocality]) = {
var noDelayScheduleRejects = true
var minLaunchedLocality: Option[TaskLocality] = None
- // nodes and executors that are blacklisted for the entire application have already been
+ // nodes and executors that are blocked for the entire application have already been
Review comment:
excluded?
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedClusterMessage.scala
##########
@@ -122,7 +122,7 @@ private[spark] object CoarseGrainedClusterMessages {
resourceProfileToTotalExecs: Map[ResourceProfile, Int],
numLocalityAwareTasksPerResourceProfileId: Map[Int, Int],
hostToLocalTaskCount: Map[Int, Map[String, Int]],
- nodeBlacklist: Set[String])
+ nodeBlocklist: Set[String])
Review comment:
nodeExcludelist?
##########
File path: resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala
##########
@@ -523,9 +523,9 @@ class YarnAllocatorSuite extends SparkFunSuite with Matchers with BeforeAndAfter
handler.getNumUnexpectedContainerRelease should be (2)
}
- test("blacklisted nodes reflected in amClient requests") {
- // Internally we track the set of blacklisted nodes, but yarn wants us to send *changes*
- // to the blacklist. This makes sure we are sending the right updates.
+ test("excludeOnFailure nodes reflected in amClient requests") {
+ // Internally we track the set of excluded nodes, but yarn wants us to send *changes*
+ // to it. This makes sure we are sending the right updates.
Review comment:
I don't think the changes in `YarnAllocatorSuite` are correct -- I think this is testing the interaction with YARN's blacklisting feature (e.g. YARN-4576), rather than Spark's feature. I think it's the same for `YarnSchedulerBackendSuite` ?
##########
File path: sql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/auth/HiveAuthFactory.java
##########
@@ -291,14 +291,14 @@ public static TServerSocket getServerSSLSocket(String hiveHost, int portNum, Str
TServerSocket thriftServerSocket =
TSSLTransportFactory.getServerSocket(portNum, 0, serverAddress.getAddress(), params);
if (thriftServerSocket.getServerSocket() instanceof SSLServerSocket) {
- List<String> sslVersionBlacklistLocal = new ArrayList<String>();
- for (String sslVersion : sslVersionBlacklist) {
- sslVersionBlacklistLocal.add(sslVersion.trim().toLowerCase(Locale.ROOT));
+ List<String> sslVersionsExcludedLocal = new ArrayList<String>();
+ for (String sslVersion : sslVersionsExcluded) {
+ sslVersionsExcludedLocal.add(sslVersion.trim().toLowerCase(Locale.ROOT));
}
SSLServerSocket sslServerSocket = (SSLServerSocket) thriftServerSocket.getServerSocket();
List<String> enabledProtocols = new ArrayList<String>();
for (String protocol : sslServerSocket.getEnabledProtocols()) {
- if (sslVersionBlacklistLocal.contains(protocol.toLowerCase(Locale.ROOT))) {
+ if (sslVersionsExcludedLocal.contains(protocol.toLowerCase(Locale.ROOT))) {
Review comment:
Changes here (and in the other `sql/hive-thriftserver` files) shouldn't be included in this PR. Previously I was suggested not to adjust these since they are a direct copy of Hive files -- and either way it's not relevant to the excludeOnFailure feature.
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -515,15 +515,15 @@ private[spark] class TaskSchedulerImpl(
hostsByRack.getOrElseUpdate(rack, new HashSet[String]()) += host
}
- // Before making any offers, remove any nodes from the blacklist whose blacklist has expired. Do
+ // Before making any offers, remove any nodes from the blocklist whose blocklist has expired. Do
// this here to avoid a separate thread and added synchronization overhead, and also because
- // updating the blacklist is only relevant when task offers are being made.
- blacklistTrackerOpt.foreach(_.applyBlacklistTimeout())
+ // updating the blocklist is only relevant when task offers are being made.
Review comment:
blocklist -> excludeList?
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusSource.scala
##########
@@ -59,9 +59,17 @@ private[spark] class AppStatusSource extends Source {
val SKIPPED_TASKS = getCounter("tasks", "skippedTasks")
+ // this is private but user visible from metrics so just deprecate
+ // @deprecated("use excludedExecutors instead", "3.1.0")
Review comment:
Why is the deprecation commented out?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-702525542
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r513545836
##########
File path: core/src/main/scala/org/apache/spark/SparkConf.scala
##########
@@ -612,7 +612,29 @@ private[spark] object SparkConf extends Logging {
DeprecatedConfig("spark.yarn.credentials.file.retention.days", "2.4.0", "Not used anymore."),
DeprecatedConfig("spark.yarn.services", "3.0.0", "Feature no longer available."),
DeprecatedConfig("spark.executor.plugins", "3.0.0",
- "Feature replaced with new plugin API. See Monitoring documentation.")
+ "Feature replaced with new plugin API. See Monitoring documentation."),
+ DeprecatedConfig("spark.blacklist.enabled", "3.1.0",
+ "Please use spark.excludeOnFailure.enabled"),
+ DeprecatedConfig("spark.blacklist.task.maxTaskAttemptsPerExecutor", "3.1.0",
+ "Please use spark.excludeOnFailure.task.maxTaskAttemptsPerExecutor"),
+ DeprecatedConfig("spark.blacklist.task.maxTaskAttemptsPerNode", "3.1.0",
+ "Please use spark.excludeOnFailure.task.maxTaskAttemptsPerNode"),
+ DeprecatedConfig("spark.blacklist.application.maxFailedTasksPerExecutor", "3.1.0",
+ "Please use spark.excludeOnFailure.application.maxFailedTasksPerExecutor"),
+ DeprecatedConfig("spark.blacklist.stage.maxFailedTasksPerExecutor", "3.1.0",
+ "Please use spark.excludeOnFailure.stage.maxFailedTasksPerExecutor"),
+ DeprecatedConfig("spark.blacklist.application.maxFailedExecutorsPerNode", "3.1.0",
+ "Please use spark.excludeOnFailure.application.maxFailedExecutorsPerNode"),
+ DeprecatedConfig("spark.blacklist.stage.maxFailedExecutorsPerNode", "3.1.0",
+ "Please use spark.excludeOnFailure.stage.maxFailedExecutorsPerNode"),
+ DeprecatedConfig("spark.blacklist.timeout", "3.1.0",
+ "Please use spark.excludeOnFailure.timeout"),
+ DeprecatedConfig("spark.scheduler.executorTaskBlacklistTime", "3.1.0",
Review comment:
yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705620977
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705849704
test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705872047
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-712476650
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-715652629
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34820/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-717978929
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34974/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-715462608
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705259319
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r497506544
##########
File path: resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala
##########
@@ -523,9 +523,9 @@ class YarnAllocatorSuite extends SparkFunSuite with Matchers with BeforeAndAfter
handler.getNumUnexpectedContainerRelease should be (2)
}
- test("blacklisted nodes reflected in amClient requests") {
- // Internally we track the set of blacklisted nodes, but yarn wants us to send *changes*
- // to the blacklist. This makes sure we are sending the right updates.
+ test("excludeOnFailure nodes reflected in amClient requests") {
+ // Internally we track the set of excluded nodes, but yarn wants us to send *changes*
+ // to it. This makes sure we are sending the right updates.
Review comment:
In this case actually this is what I wanted. We internally track what nodes are excluded. This is from the excludeOnFailure list from the scheduler, from the spark.yarn.exclude.nodes config and the allocator logic for excluding nodes that have failures.. Yes we use the yarn blacklist list api to actually specify it to yarn but I don't see any reason internally we have to keep our code using the term blacklist except where we call into their api. I could add a comment about that in there to clarify in the Allocator itself until they can update their api.
--
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701448668
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705901360
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705619969
**[Test build #129558 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129558/testReport)** for PR 29906 at commit [`32fca2d`](https://github.com/apache/spark/commit/32fca2d9740f40784294d1bfd0584371c105c7d9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705620955
**[Test build #129558 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129558/testReport)** for PR 29906 at commit [`32fca2d`](https://github.com/apache/spark/commit/32fca2d9740f40784294d1bfd0584371c105c7d9).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705659379
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705700233
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705215424
**[Test build #129532 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129532/testReport)** for PR 29906 at commit [`0172aa4`](https://github.com/apache/spark/commit/0172aa4a8c64b6726a50e703e5e18462790c4949).
* This patch **fails Scala style tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r500600788
##########
File path: core/src/main/scala/org/apache/spark/status/api/v1/api.scala
##########
@@ -82,10 +82,11 @@ class ExecutorStageSummary private[spark](
val shuffleWriteRecords : Long,
val memoryBytesSpilled : Long,
val diskBytesSpilled : Long,
- val isBlacklistedForStage: Boolean,
+ val isBlacklistedForStage: Boolean, // deprecated
Review comment:
the problem with that is that this is the rest api and we can't really deprecate it without having some new api, which would like be a v2 instead of v1 or a different endpoint. Since we don't have that I don't think it makes sense to deprecate the entire structure. I think this is the first time this has happened as well so I'm open to ideas on how we want to handle.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701690413
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701570552
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129281/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-718041041
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130370/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705241902
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34138/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-717994903
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-712561183
Shall we add `DeveloperApi` annotation to `SparkFirehoseListener` since we all agree it missed before? @tgravescs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701481134
**[Test build #129281 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129281/testReport)** for PR 29906 at commit [`6da870b`](https://github.com/apache/spark/commit/6da870b4cd2f039345ddeb1792d6ddf0ccf19136).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705700233
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705252591
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34140/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705245033
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705215433
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r497595877
##########
File path: sql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/auth/HiveAuthFactory.java
##########
@@ -291,14 +291,14 @@ public static TServerSocket getServerSSLSocket(String hiveHost, int portNum, Str
TServerSocket thriftServerSocket =
TSSLTransportFactory.getServerSocket(portNum, 0, serverAddress.getAddress(), params);
if (thriftServerSocket.getServerSocket() instanceof SSLServerSocket) {
- List<String> sslVersionBlacklistLocal = new ArrayList<String>();
- for (String sslVersion : sslVersionBlacklist) {
- sslVersionBlacklistLocal.add(sslVersion.trim().toLowerCase(Locale.ROOT));
+ List<String> sslVersionsExcludedLocal = new ArrayList<String>();
+ for (String sslVersion : sslVersionsExcluded) {
+ sslVersionsExcludedLocal.add(sslVersion.trim().toLowerCase(Locale.ROOT));
}
SSLServerSocket sslServerSocket = (SSLServerSocket) thriftServerSocket.getServerSocket();
List<String> enabledProtocols = new ArrayList<String>();
for (String protocol : sslServerSocket.getEnabledProtocols()) {
- if (sslVersionBlacklistLocal.contains(protocol.toLowerCase(Locale.ROOT))) {
+ if (sslVersionsExcludedLocal.contains(protocol.toLowerCase(Locale.ROOT))) {
Review comment:
Best I can tell, it is copied from here: https://github.com/apache/hive/blob/branch-1.2/service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705237276
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34137/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701374627
thanks I started out with blocklist and must have missed converting a few, I'll fix those up. The yarn allocator side I obviously left the actual calls into the client but tried to rename the variables and such to something else. There is a hadoop jira to remove from there as well but I have no idea when it will be implemented so I was just trying to remove where it made sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701013285
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-720575364
Huge thanks for pushing this through @tgravescs ! It was no small effort!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701517394
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705245033
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705231539
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34137/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705245002
**[Test build #129535 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129535/testReport)** for PR 29906 at commit [`5ed326d`](https://github.com/apache/spark/commit/5ed326def00bc97807e24d1eb68a022c94ec6135).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r498437910
##########
File path: core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala
##########
@@ -0,0 +1,478 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.atomic.AtomicReference
+
+import scala.collection.mutable.{ArrayBuffer, HashMap, HashSet}
+
+import org.apache.spark.{ExecutorAllocationClient, SparkConf, SparkContext}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config
+import org.apache.spark.util.{Clock, SystemClock, Utils}
+
+/**
+ * HealthTracker is designed to track problematic executors and nodes. It supports excluding
+ * executors and nodes across an entire application (with a periodic expiry). TaskSetManagers add
+ * additional logic for exclusion of executors and nodes for individual tasks and stages which
+ * works in concert with the logic here.
+ *
+ * The tracker needs to deal with a variety of workloads, eg.:
+ *
+ * * bad user code -- this may lead to many task failures, but that should not count against
+ * individual executors
+ * * many small stages -- this may prevent a bad executor for having many failures within one
+ * stage, but still many failures over the entire application
+ * * "flaky" executors -- they don't fail every task, but are still faulty enough to merit
+ * excluding
+ *
+ * See the design doc on SPARK-8425 for a more in-depth discussion.
+ *
+ * THREADING: As with most helpers of TaskSchedulerImpl, this is not thread-safe. Though it is
+ * called by multiple threads, callers must already have a lock on the TaskSchedulerImpl. The
+ * one exception is [[excludedNodeList()]], which can be called without holding a lock.
+ */
+private[scheduler] class HealthTracker (
+ private val listenerBus: LiveListenerBus,
+ conf: SparkConf,
+ allocationClient: Option[ExecutorAllocationClient],
+ clock: Clock = new SystemClock()) extends Logging {
+
+ def this(sc: SparkContext, allocationClient: Option[ExecutorAllocationClient]) = {
+ this(sc.listenerBus, sc.conf, allocationClient)
+ }
+
+ HealthTracker.validateExcludeOnFailureConfs(conf)
+ private val MAX_FAILURES_PER_EXEC = conf.get(config.MAX_FAILURES_PER_EXEC)
+ private val MAX_FAILED_EXEC_PER_NODE = conf.get(config.MAX_FAILED_EXEC_PER_NODE)
+ val EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS = HealthTracker.getExludeOnFailureTimeout(conf)
+ private val EXCLUDE_FETCH_FAILURE_ENABLED =
+ conf.get(config.EXCLUDE_ON_FAILURE_FETCH_FAILURE_ENABLED)
+
+ /**
+ * A map from executorId to information on task failures. Tracks the time of each task failure,
+ * so that we can avoid excluding executors due to failures that are very far apart. We do not
+ * actively remove from this as soon as tasks hit their timeouts, to avoid the time it would take
+ * to do so. But it will not grow too large, because as soon as an executor gets too many
+ * failures, we exclude the executor and remove its entry here.
+ */
+ private val executorIdToFailureList = new HashMap[String, ExecutorFailureList]()
+ val executorIdToExcludedStatus = new HashMap[String, ExcludedExecutor]()
+ val nodeIdToExcludedExpiryTime = new HashMap[String, Long]()
+ /**
+ * An immutable copy of the set of nodes that are currently excluded. Kept in an
+ * AtomicReference to make [[excludedNodeList()]] thread-safe.
+ */
+ private val _excludedNodeList = new AtomicReference[Set[String]](Set())
+ /**
+ * Time when the next excluded node will expire. Used as a shortcut to
+ * avoid iterating over all entries in the excludedNodeList when none will have expired.
+ */
+ var nextExpiryTime: Long = Long.MaxValue
+ /**
+ * Mapping from nodes to all of the executors that have been excluded on that node. We do *not*
+ * remove from this when executors are removed from spark, so we can track when we get multiple
+ * successive excluded executors on one node. Nonetheless, it will not grow too large because
+ * there cannot be many excluded executors on one node, before we stop requesting more
+ * executors on that node, and we clean up the list of exluded executors once an executor has
+ * been excluded for EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS.
+ */
+ val nodeToExcludedExecs = new HashMap[String, HashSet[String]]()
+
+ /**
+ * Include executors and nodes that have been excluded for at least
+ * EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS
+ */
+ def applyExcludeOnFailureTimeout(): Unit = {
+ val now = clock.getTimeMillis()
+ // quickly check if we've got anything to expire that is excluded -- if not,
+ // avoid doing any work
+ if (now > nextExpiryTime) {
+ // Apply the timeout to excluded nodes and executors
+ val execsToInclude = executorIdToExcludedStatus.filter(_._2.expiryTime < now).keys
+ if (execsToInclude.nonEmpty) {
+ // Include any executors that have been exluded longer than the excludeOnFailure timeout.
+ logInfo(s"Removing executors $execsToInclude from exclude list because the " +
+ s"the executors have reached the timed out")
+ execsToInclude.foreach { exec =>
+ val status = executorIdToExcludedStatus.remove(exec).get
+ val failedExecsOnNode = nodeToExcludedExecs(status.node)
+ listenerBus.post(SparkListenerExecutorUnexcluded(now, exec))
+ failedExecsOnNode.remove(exec)
+ if (failedExecsOnNode.isEmpty) {
+ nodeToExcludedExecs.remove(status.node)
+ }
+ }
+ }
+ val nodesToInclude = nodeIdToExcludedExpiryTime.filter(_._2 < now).keys
+ if (nodesToInclude.nonEmpty) {
+ // Include any nodes that have been excluded longer than the excludeOnFailure timeout.
+ logInfo(s"Removing nodes $nodesToInclude from exclude list because the " +
+ s"nodes have reached has timed out")
+ nodesToInclude.foreach { node =>
+ nodeIdToExcludedExpiryTime.remove(node)
+ listenerBus.post(SparkListenerNodeUnexcluded(now, node))
+ }
+ _excludedNodeList.set(nodeIdToExcludedExpiryTime.keySet.toSet)
+ }
+ updateNextExpiryTime()
+ }
+ }
+
+ private def updateNextExpiryTime(): Unit = {
+ val execMinExpiry = if (executorIdToExcludedStatus.nonEmpty) {
+ executorIdToExcludedStatus.map{_._2.expiryTime}.min
+ } else {
+ Long.MaxValue
+ }
+ val nodeMinExpiry = if (nodeIdToExcludedExpiryTime.nonEmpty) {
+ nodeIdToExcludedExpiryTime.values.min
+ } else {
+ Long.MaxValue
+ }
+ nextExpiryTime = math.min(execMinExpiry, nodeMinExpiry)
+ }
+
+ private def killExecutor(exec: String, msg: String): Unit = {
+ allocationClient match {
+ case Some(a) =>
+ logInfo(msg)
+ a.killExecutors(Seq(exec), adjustTargetNumExecutors = false, countFailures = false,
+ force = true)
+ case None =>
+ logInfo(s"Not attempting to kill excluded executor id $exec " +
+ s"since allocation client is not defined.")
+ }
+ }
+
+ private def killExcludedExecutor(exec: String): Unit = {
+ if (conf.get(config.EXCLUDE_ON_FAILURE_KILL_ENABLED)) {
+ killExecutor(exec, s"Killing excluded executor id $exec since " +
+ s"${config.EXCLUDE_ON_FAILURE_KILL_ENABLED.key} is set.")
+ }
+ }
+
+ private[scheduler] def killExcludedIdleExecutor(exec: String): Unit = {
+ killExecutor(exec,
+ s"Killing excluded idle executor id $exec because of task unschedulability and trying " +
+ "to acquire a new executor.")
+ }
+
+ private def killExecutorsOnExcludedNode(node: String): Unit = {
+ if (conf.get(config.EXCLUDE_ON_FAILURE_KILL_ENABLED)) {
+ allocationClient match {
+ case Some(a) =>
+ logInfo(s"Killing all executors on excluded host $node " +
+ s"since ${config.EXCLUDE_ON_FAILURE_KILL_ENABLED.key} is set.")
+ if (a.killExecutorsOnHost(node) == false) {
+ logError(s"Killing executors on node $node failed.")
+ }
+ case None =>
+ logWarning(s"Not attempting to kill executors on excluded host $node " +
+ s"since allocation client is not defined.")
+ }
+ }
+ }
+
+ def updateExcludedForFetchFailure(host: String, exec: String): Unit = {
+ if (EXCLUDE_FETCH_FAILURE_ENABLED) {
+ // If we exclude on fetch failures, we are implicitly saying that we believe the failure is
+ // non-transient, and can't be recovered from (even if this is the first fetch failure,
+ // stage is retried after just one failure, so we don't always get a chance to collect
+ // multiple fetch failures).
+ // If the external shuffle-service is on, then every other executor on this node would
+ // be suffering from the same issue, so we should exclude (and potentially kill) all
+ // of them immediately.
+
+ val now = clock.getTimeMillis()
+ val expiryTimeForNewExcludes = now + EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS
+
+ if (conf.get(config.SHUFFLE_SERVICE_ENABLED)) {
+ if (!nodeIdToExcludedExpiryTime.contains(host)) {
+ logInfo(s"excluding node $host due to fetch failure of external shuffle service")
+
+ nodeIdToExcludedExpiryTime.put(host, expiryTimeForNewExcludes)
+ listenerBus.post(SparkListenerNodeExcluded(now, host, 1))
+ _excludedNodeList.set(nodeIdToExcludedExpiryTime.keySet.toSet)
+ killExecutorsOnExcludedNode(host)
+ updateNextExpiryTime()
+ }
+ } else if (!executorIdToExcludedStatus.contains(exec)) {
+ logInfo(s"Excluding executor $exec due to fetch failure")
+
+ executorIdToExcludedStatus.put(exec, ExcludedExecutor(host, expiryTimeForNewExcludes))
+ // We hardcoded number of failure tasks to 1 for fetch failure, because there's no
+ // reattempt for such failure.
+ listenerBus.post(SparkListenerExecutorExcluded(now, exec, 1))
Review comment:
so I left it in the listeners for 2 reasons.
1) SparkFirehoseListener is public, though I think that is mostly for testing purposes. Maybe this being public means SparkListenerExecutorExcluded is public so we should just make sure its sent.
2) for backwards compatibility of the history server reading the old event logs containing those events.
So really the only thing I got rid of is what you pointed out here with us posting it on new applications. I'm fine either way if we think its going to break a bunch of people we can post 2 messages. I think I would rather do that then have the default listener delegate just in case something weird happens there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701481134
**[Test build #129281 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129281/testReport)** for PR 29906 at commit [`6da870b`](https://github.com/apache/spark/commit/6da870b4cd2f039345ddeb1792d6ddf0ccf19136).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705657946
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34163/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-715665066
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705657936
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705240801
**[Test build #129535 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129535/testReport)** for PR 29906 at commit [`5ed326d`](https://github.com/apache/spark/commit/5ed326def00bc97807e24d1eb68a022c94ec6135).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r501166668
##########
File path: core/src/main/scala/org/apache/spark/status/api/v1/api.scala
##########
@@ -82,10 +82,11 @@ class ExecutorStageSummary private[spark](
val shuffleWriteRecords : Long,
val memoryBytesSpilled : Long,
val diskBytesSpilled : Long,
- val isBlacklistedForStage: Boolean,
+ val isBlacklistedForStage: Boolean, // deprecated
Review comment:
Hm. Maybe we have some misunderstanding. I don't mean marking `ExecutorStageSummary` as deprecated, I mean marking `isBlacklistedForStage` as deprecated. The alternative was added as `isExcludedForStage`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-710049939
@srowen @mridulm see the discussion back : https://github.com/apache/spark/pull/29906#discussion_r499125900
I'm fine either way. More input would be great so let me know what you think
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r506781069
##########
File path: core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala
##########
@@ -0,0 +1,478 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.atomic.AtomicReference
+
+import scala.collection.mutable.{ArrayBuffer, HashMap, HashSet}
+
+import org.apache.spark.{ExecutorAllocationClient, SparkConf, SparkContext}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config
+import org.apache.spark.util.{Clock, SystemClock, Utils}
+
+/**
+ * HealthTracker is designed to track problematic executors and nodes. It supports excluding
+ * executors and nodes across an entire application (with a periodic expiry). TaskSetManagers add
+ * additional logic for exclusion of executors and nodes for individual tasks and stages which
+ * works in concert with the logic here.
+ *
+ * The tracker needs to deal with a variety of workloads, eg.:
+ *
+ * * bad user code -- this may lead to many task failures, but that should not count against
+ * individual executors
+ * * many small stages -- this may prevent a bad executor for having many failures within one
+ * stage, but still many failures over the entire application
+ * * "flaky" executors -- they don't fail every task, but are still faulty enough to merit
+ * excluding
+ *
+ * See the design doc on SPARK-8425 for a more in-depth discussion.
+ *
+ * THREADING: As with most helpers of TaskSchedulerImpl, this is not thread-safe. Though it is
+ * called by multiple threads, callers must already have a lock on the TaskSchedulerImpl. The
+ * one exception is [[excludedNodeList()]], which can be called without holding a lock.
+ */
+private[scheduler] class HealthTracker (
+ private val listenerBus: LiveListenerBus,
+ conf: SparkConf,
+ allocationClient: Option[ExecutorAllocationClient],
+ clock: Clock = new SystemClock()) extends Logging {
+
+ def this(sc: SparkContext, allocationClient: Option[ExecutorAllocationClient]) = {
+ this(sc.listenerBus, sc.conf, allocationClient)
+ }
+
+ HealthTracker.validateExcludeOnFailureConfs(conf)
+ private val MAX_FAILURES_PER_EXEC = conf.get(config.MAX_FAILURES_PER_EXEC)
+ private val MAX_FAILED_EXEC_PER_NODE = conf.get(config.MAX_FAILED_EXEC_PER_NODE)
+ val EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS = HealthTracker.getExludeOnFailureTimeout(conf)
+ private val EXCLUDE_FETCH_FAILURE_ENABLED =
+ conf.get(config.EXCLUDE_ON_FAILURE_FETCH_FAILURE_ENABLED)
+
+ /**
+ * A map from executorId to information on task failures. Tracks the time of each task failure,
+ * so that we can avoid excluding executors due to failures that are very far apart. We do not
+ * actively remove from this as soon as tasks hit their timeouts, to avoid the time it would take
+ * to do so. But it will not grow too large, because as soon as an executor gets too many
+ * failures, we exclude the executor and remove its entry here.
+ */
+ private val executorIdToFailureList = new HashMap[String, ExecutorFailureList]()
+ val executorIdToExcludedStatus = new HashMap[String, ExcludedExecutor]()
+ val nodeIdToExcludedExpiryTime = new HashMap[String, Long]()
+ /**
+ * An immutable copy of the set of nodes that are currently excluded. Kept in an
+ * AtomicReference to make [[excludedNodeList()]] thread-safe.
+ */
+ private val _excludedNodeList = new AtomicReference[Set[String]](Set())
+ /**
+ * Time when the next excluded node will expire. Used as a shortcut to
+ * avoid iterating over all entries in the excludedNodeList when none will have expired.
+ */
+ var nextExpiryTime: Long = Long.MaxValue
+ /**
+ * Mapping from nodes to all of the executors that have been excluded on that node. We do *not*
+ * remove from this when executors are removed from spark, so we can track when we get multiple
+ * successive excluded executors on one node. Nonetheless, it will not grow too large because
+ * there cannot be many excluded executors on one node, before we stop requesting more
+ * executors on that node, and we clean up the list of exluded executors once an executor has
+ * been excluded for EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS.
+ */
+ val nodeToExcludedExecs = new HashMap[String, HashSet[String]]()
+
+ /**
+ * Include executors and nodes that have been excluded for at least
+ * EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS
+ */
+ def applyExcludeOnFailureTimeout(): Unit = {
+ val now = clock.getTimeMillis()
+ // quickly check if we've got anything to expire that is excluded -- if not,
+ // avoid doing any work
+ if (now > nextExpiryTime) {
+ // Apply the timeout to excluded nodes and executors
+ val execsToInclude = executorIdToExcludedStatus.filter(_._2.expiryTime < now).keys
+ if (execsToInclude.nonEmpty) {
+ // Include any executors that have been exluded longer than the excludeOnFailure timeout.
+ logInfo(s"Removing executors $execsToInclude from exclude list because the " +
+ s"the executors have reached the timed out")
+ execsToInclude.foreach { exec =>
+ val status = executorIdToExcludedStatus.remove(exec).get
+ val failedExecsOnNode = nodeToExcludedExecs(status.node)
+ listenerBus.post(SparkListenerExecutorUnexcluded(now, exec))
+ failedExecsOnNode.remove(exec)
+ if (failedExecsOnNode.isEmpty) {
+ nodeToExcludedExecs.remove(status.node)
+ }
+ }
+ }
+ val nodesToInclude = nodeIdToExcludedExpiryTime.filter(_._2 < now).keys
+ if (nodesToInclude.nonEmpty) {
+ // Include any nodes that have been excluded longer than the excludeOnFailure timeout.
+ logInfo(s"Removing nodes $nodesToInclude from exclude list because the " +
+ s"nodes have reached has timed out")
+ nodesToInclude.foreach { node =>
+ nodeIdToExcludedExpiryTime.remove(node)
+ listenerBus.post(SparkListenerNodeUnexcluded(now, node))
+ }
+ _excludedNodeList.set(nodeIdToExcludedExpiryTime.keySet.toSet)
+ }
+ updateNextExpiryTime()
+ }
+ }
+
+ private def updateNextExpiryTime(): Unit = {
+ val execMinExpiry = if (executorIdToExcludedStatus.nonEmpty) {
+ executorIdToExcludedStatus.map{_._2.expiryTime}.min
+ } else {
+ Long.MaxValue
+ }
+ val nodeMinExpiry = if (nodeIdToExcludedExpiryTime.nonEmpty) {
+ nodeIdToExcludedExpiryTime.values.min
+ } else {
+ Long.MaxValue
+ }
+ nextExpiryTime = math.min(execMinExpiry, nodeMinExpiry)
+ }
+
+ private def killExecutor(exec: String, msg: String): Unit = {
+ allocationClient match {
+ case Some(a) =>
+ logInfo(msg)
+ a.killExecutors(Seq(exec), adjustTargetNumExecutors = false, countFailures = false,
+ force = true)
+ case None =>
+ logInfo(s"Not attempting to kill excluded executor id $exec " +
+ s"since allocation client is not defined.")
+ }
+ }
+
+ private def killExcludedExecutor(exec: String): Unit = {
+ if (conf.get(config.EXCLUDE_ON_FAILURE_KILL_ENABLED)) {
+ killExecutor(exec, s"Killing excluded executor id $exec since " +
+ s"${config.EXCLUDE_ON_FAILURE_KILL_ENABLED.key} is set.")
+ }
+ }
+
+ private[scheduler] def killExcludedIdleExecutor(exec: String): Unit = {
+ killExecutor(exec,
+ s"Killing excluded idle executor id $exec because of task unschedulability and trying " +
+ "to acquire a new executor.")
+ }
+
+ private def killExecutorsOnExcludedNode(node: String): Unit = {
+ if (conf.get(config.EXCLUDE_ON_FAILURE_KILL_ENABLED)) {
+ allocationClient match {
+ case Some(a) =>
+ logInfo(s"Killing all executors on excluded host $node " +
+ s"since ${config.EXCLUDE_ON_FAILURE_KILL_ENABLED.key} is set.")
+ if (a.killExecutorsOnHost(node) == false) {
+ logError(s"Killing executors on node $node failed.")
+ }
+ case None =>
+ logWarning(s"Not attempting to kill executors on excluded host $node " +
+ s"since allocation client is not defined.")
+ }
+ }
+ }
+
+ def updateExcludedForFetchFailure(host: String, exec: String): Unit = {
+ if (EXCLUDE_FETCH_FAILURE_ENABLED) {
+ // If we exclude on fetch failures, we are implicitly saying that we believe the failure is
+ // non-transient, and can't be recovered from (even if this is the first fetch failure,
+ // stage is retried after just one failure, so we don't always get a chance to collect
+ // multiple fetch failures).
+ // If the external shuffle-service is on, then every other executor on this node would
+ // be suffering from the same issue, so we should exclude (and potentially kill) all
+ // of them immediately.
+
+ val now = clock.getTimeMillis()
+ val expiryTimeForNewExcludes = now + EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS
+
+ if (conf.get(config.SHUFFLE_SERVICE_ENABLED)) {
+ if (!nodeIdToExcludedExpiryTime.contains(host)) {
+ logInfo(s"excluding node $host due to fetch failure of external shuffle service")
+
+ nodeIdToExcludedExpiryTime.put(host, expiryTimeForNewExcludes)
+ listenerBus.post(SparkListenerNodeExcluded(now, host, 1))
+ _excludedNodeList.set(nodeIdToExcludedExpiryTime.keySet.toSet)
+ killExecutorsOnExcludedNode(host)
+ updateNextExpiryTime()
+ }
+ } else if (!executorIdToExcludedStatus.contains(exec)) {
+ logInfo(s"Excluding executor $exec due to fetch failure")
+
+ executorIdToExcludedStatus.put(exec, ExcludedExecutor(host, expiryTimeForNewExcludes))
+ // We hardcoded number of failure tasks to 1 for fetch failure, because there's no
+ // reattempt for such failure.
+ listenerBus.post(SparkListenerExecutorExcluded(now, exec, 1))
Review comment:
cc @gatorsmile
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701507195
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33898/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705725776
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129561/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r509854936
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -284,80 +284,127 @@ private[spark] class AppStatusListener(
}
override def onExecutorBlacklisted(event: SparkListenerExecutorBlacklisted): Unit = {
- updateBlackListStatus(event.executorId, true)
+ updateExclusionStatus(event.executorId, true)
+ }
+
+ override def onExecutorExcluded(event: SparkListenerExecutorExcluded): Unit = {
+ updateExclusionStatus(event.executorId, true)
}
override def onExecutorBlacklistedForStage(
event: SparkListenerExecutorBlacklistedForStage): Unit = {
- val now = System.nanoTime()
+ updateExclusionStatusForStage(event.stageId, event.stageAttemptId, event.executorId)
+ }
- Option(liveStages.get((event.stageId, event.stageAttemptId))).foreach { stage =>
- setStageBlackListStatus(stage, now, event.executorId)
- }
- liveExecutors.get(event.executorId).foreach { exec =>
- addBlackListedStageTo(exec, event.stageId, now)
- }
+ override def onExecutorExcludedForStage(
+ event: SparkListenerExecutorExcludedForStage): Unit = {
+ updateExclusionStatusForStage(event.stageId, event.stageAttemptId, event.executorId)
}
override def onNodeBlacklistedForStage(event: SparkListenerNodeBlacklistedForStage): Unit = {
- val now = System.nanoTime()
+ updateNodeExclusionStatusForStage(event.stageId, event.stageAttemptId, event.hostId)
+ }
- // Implicitly blacklist every available executor for the stage associated with this node
- Option(liveStages.get((event.stageId, event.stageAttemptId))).foreach { stage =>
- val executorIds = liveExecutors.values.filter(_.host == event.hostId).map(_.executorId).toSeq
- setStageBlackListStatus(stage, now, executorIds: _*)
- }
- liveExecutors.values.filter(_.hostname == event.hostId).foreach { exec =>
- addBlackListedStageTo(exec, event.stageId, now)
- }
+ override def onNodeExcludedForStage(event: SparkListenerNodeExcludedForStage): Unit = {
+ updateNodeExclusionStatusForStage(event.stageId, event.stageAttemptId, event.hostId)
}
- private def addBlackListedStageTo(exec: LiveExecutor, stageId: Int, now: Long): Unit = {
- exec.blacklistedInStages += stageId
+ private def addExcludedStageTo(exec: LiveExecutor, stageId: Int, now: Long): Unit = {
+ exec.excludedInStages += stageId
liveUpdate(exec, now)
}
private def setStageBlackListStatus(stage: LiveStage, now: Long, executorIds: String*): Unit = {
executorIds.foreach { executorId =>
val executorStageSummary = stage.executorSummary(executorId)
- executorStageSummary.isBlacklisted = true
+ executorStageSummary.isExcluded = true
+ maybeUpdate(executorStageSummary, now)
+ }
+ stage.excludedExecutors ++= executorIds
+ maybeUpdate(stage, now)
+ }
+
+ private def setStageExcludedStatus(stage: LiveStage, now: Long, executorIds: String*): Unit = {
+ executorIds.foreach { executorId =>
+ val executorStageSummary = stage.executorSummary(executorId)
+ executorStageSummary.isExcluded = true
maybeUpdate(executorStageSummary, now)
}
- stage.blackListedExecutors ++= executorIds
+ stage.excludedExecutors ++= executorIds
maybeUpdate(stage, now)
}
override def onExecutorUnblacklisted(event: SparkListenerExecutorUnblacklisted): Unit = {
- updateBlackListStatus(event.executorId, false)
+ updateExclusionStatus(event.executorId, false)
+ }
+
+ override def onExecutorUnexcluded(event: SparkListenerExecutorUnexcluded): Unit = {
+ updateExclusionStatus(event.executorId, false)
}
override def onNodeBlacklisted(event: SparkListenerNodeBlacklisted): Unit = {
- updateNodeBlackList(event.hostId, true)
+ updateNodeExcluded(event.hostId, true)
+ }
+
+ override def onNodeExcluded(event: SparkListenerNodeExcluded): Unit = {
+ updateNodeExcluded(event.hostId, true)
}
override def onNodeUnblacklisted(event: SparkListenerNodeUnblacklisted): Unit = {
- updateNodeBlackList(event.hostId, false)
+ updateNodeExcluded(event.hostId, false)
+ }
+
+ override def onNodeUnexcluded(event: SparkListenerNodeUnexcluded): Unit = {
+ updateNodeExcluded(event.hostId, false)
+ }
+
+ private def updateNodeExclusionStatusForStage(stageId: Int, stageAttemptId: Int,
+ hostId: String): Unit = {
+ val now = System.nanoTime()
+
+ // Implicitly exclude every available executor for the stage associated with this node
+ Option(liveStages.get((stageId, stageAttemptId))).foreach { stage =>
+ val executorIds = liveExecutors.values.filter(_.host == hostId).map(_.executorId).toSeq
+ setStageExcludedStatus(stage, now, executorIds: _*)
+ }
+ liveExecutors.values.filter(_.hostname == hostId).foreach { exec =>
+ addExcludedStageTo(exec, stageId, now)
+ }
+ }
+
+ private def updateExclusionStatusForStage(stageId: Int, stageAttemptId: Int,
+ execId: String): Unit = {
+ val now = System.nanoTime()
+
+ Option(liveStages.get((stageId, stageAttemptId))).foreach { stage =>
+ setStageExcludedStatus(stage, now, execId)
+ }
+ liveExecutors.get(execId).foreach { exec =>
+ addExcludedStageTo(exec, stageId, now)
+ }
}
- private def updateBlackListStatus(execId: String, blacklisted: Boolean): Unit = {
+ private def updateExclusionStatus(execId: String, excluded: Boolean): Unit = {
liveExecutors.get(execId).foreach { exec =>
- exec.isBlacklisted = blacklisted
- if (blacklisted) {
+ exec.isExcluded = excluded
+ if (excluded) {
appStatusSource.foreach(_.BLACKLISTED_EXECUTORS.inc())
+ appStatusSource.foreach(_.EXCLUDED_EXECUTORS.inc())
Review comment:
I wonder this could lead to the metrics overcounted since we always post two blacklist events?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705725069
**[Test build #129561 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129561/testReport)** for PR 29906 at commit [`3ebf881`](https://github.com/apache/spark/commit/3ebf8814299fb96f52df479a46d1be28d453a3ce).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705662640
**[Test build #129561 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129561/testReport)** for PR 29906 at commit [`3ebf881`](https://github.com/apache/spark/commit/3ebf8814299fb96f52df479a46d1be28d453a3ce).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705221126
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129533/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705220647
**[Test build #129533 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129533/testReport)** for PR 29906 at commit [`fcc3558`](https://github.com/apache/spark/commit/fcc3558bedbc62606d8432353e4e0027ed4418c9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705657925
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34163/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705240801
**[Test build #129535 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129535/testReport)** for PR 29906 at commit [`5ed326d`](https://github.com/apache/spark/commit/5ed326def00bc97807e24d1eb68a022c94ec6135).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-717949324
**[Test build #130370 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130370/testReport)** for PR 29906 at commit [`b38dd66`](https://github.com/apache/spark/commit/b38dd66500cdd4a12f78cca1eeacf116f0ea5b4e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-715425828
**[Test build #130210 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130210/testReport)** for PR 29906 at commit [`32ab73d`](https://github.com/apache/spark/commit/32ab73da672ad62192f811558961660fbc972c37).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r502915395
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -722,74 +722,83 @@ package object config {
.booleanConf
.createWithDefault(true)
- // Blacklist confs
- private[spark] val BLACKLIST_ENABLED =
- ConfigBuilder("spark.blacklist.enabled")
+ private[spark] val EXCLUDE_ON_FAILURE_ENABLED =
+ ConfigBuilder("spark.excludeOnFailure.enabled")
.version("2.1.0")
Review comment:
I think we need to update the version. I once renamed a configuration too and @dongjoon-hyun asked to update the version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r501041270
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -284,80 +284,138 @@ private[spark] class AppStatusListener(
}
override def onExecutorBlacklisted(event: SparkListenerExecutorBlacklisted): Unit = {
- updateBlackListStatus(event.executorId, true)
+ updateExcludedStatus(event.executorId, true)
+ }
+
+ override def onExecutorExcluded(event: SparkListenerExecutorExcluded): Unit = {
+ updateExcludedStatus(event.executorId, true)
}
override def onExecutorBlacklistedForStage(
- event: SparkListenerExecutorBlacklistedForStage): Unit = {
+ event: SparkListenerExecutorBlacklistedForStage): Unit = {
+ val now = System.nanoTime()
+
Review comment:
yeah I'll make common function
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701012152
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129254/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r497497185
##########
File path: sql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/auth/HiveAuthFactory.java
##########
@@ -291,14 +291,14 @@ public static TServerSocket getServerSSLSocket(String hiveHost, int portNum, Str
TServerSocket thriftServerSocket =
TSSLTransportFactory.getServerSocket(portNum, 0, serverAddress.getAddress(), params);
if (thriftServerSocket.getServerSocket() instanceof SSLServerSocket) {
- List<String> sslVersionBlacklistLocal = new ArrayList<String>();
- for (String sslVersion : sslVersionBlacklist) {
- sslVersionBlacklistLocal.add(sslVersion.trim().toLowerCase(Locale.ROOT));
+ List<String> sslVersionsExcludedLocal = new ArrayList<String>();
+ for (String sslVersion : sslVersionsExcluded) {
+ sslVersionsExcludedLocal.add(sslVersion.trim().toLowerCase(Locale.ROOT));
}
SSLServerSocket sslServerSocket = (SSLServerSocket) thriftServerSocket.getServerSocket();
List<String> enabledProtocols = new ArrayList<String>();
for (String protocol : sslServerSocket.getEnabledProtocols()) {
- if (sslVersionBlacklistLocal.contains(protocol.toLowerCase(Locale.ROOT))) {
+ if (sslVersionsExcludedLocal.contains(protocol.toLowerCase(Locale.ROOT))) {
Review comment:
ok if its an exact copy I'll revert, otherwise I don't really see the harm in renaming.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r503293173
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -722,74 +722,83 @@ package object config {
.booleanConf
.createWithDefault(true)
- // Blacklist confs
- private[spark] val BLACKLIST_ENABLED =
- ConfigBuilder("spark.blacklist.enabled")
+ private[spark] val EXCLUDE_ON_FAILURE_ENABLED =
+ ConfigBuilder("spark.excludeOnFailure.enabled")
.version("2.1.0")
Review comment:
ok, I guess if that is the case. Like I mentioned one of the other renames didn't change this. I think it depends on if we are tracking the name of the feature it controls with this? either way I think what we decide we should document. There is also no documentation on the .version function in the ConfigBuilder so we should really add something there ( I can if we agree up on something). It ends up showing up on the documentation page under Since Version so I guess looking at that we probably should update it as this rename won't work on older version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705237286
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701534523
hmm, YarnClusterSuite passes locally for me and other erros look unrelated so will rekick it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r497492321
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -381,7 +381,7 @@ private[spark] class TaskSchedulerImpl(
: (Boolean, Option[TaskLocality]) = {
var noDelayScheduleRejects = true
var minLaunchedLocality: Option[TaskLocality] = None
- // nodes and executors that are blacklisted for the entire application have already been
+ // nodes and executors that are blocked for the entire application have already been
Review comment:
yep
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705221105
**[Test build #129533 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129533/testReport)** for PR 29906 at commit [`fcc3558`](https://github.com/apache/spark/commit/fcc3558bedbc62606d8432353e4e0027ed4418c9).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701569800
**[Test build #129281 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129281/testReport)** for PR 29906 at commit [`6da870b`](https://github.com/apache/spark/commit/6da870b4cd2f039345ddeb1792d6ddf0ccf19136).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r501029076
##########
File path: core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala
##########
@@ -0,0 +1,478 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.atomic.AtomicReference
+
+import scala.collection.mutable.{ArrayBuffer, HashMap, HashSet}
+
+import org.apache.spark.{ExecutorAllocationClient, SparkConf, SparkContext}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config
+import org.apache.spark.util.{Clock, SystemClock, Utils}
+
+/**
+ * HealthTracker is designed to track problematic executors and nodes. It supports excluding
+ * executors and nodes across an entire application (with a periodic expiry). TaskSetManagers add
+ * additional logic for exclusion of executors and nodes for individual tasks and stages which
+ * works in concert with the logic here.
+ *
+ * The tracker needs to deal with a variety of workloads, eg.:
+ *
+ * * bad user code -- this may lead to many task failures, but that should not count against
+ * individual executors
+ * * many small stages -- this may prevent a bad executor for having many failures within one
+ * stage, but still many failures over the entire application
+ * * "flaky" executors -- they don't fail every task, but are still faulty enough to merit
+ * excluding
+ *
+ * See the design doc on SPARK-8425 for a more in-depth discussion.
+ *
+ * THREADING: As with most helpers of TaskSchedulerImpl, this is not thread-safe. Though it is
+ * called by multiple threads, callers must already have a lock on the TaskSchedulerImpl. The
+ * one exception is [[excludedNodeList()]], which can be called without holding a lock.
+ */
+private[scheduler] class HealthTracker (
+ private val listenerBus: LiveListenerBus,
+ conf: SparkConf,
+ allocationClient: Option[ExecutorAllocationClient],
+ clock: Clock = new SystemClock()) extends Logging {
+
+ def this(sc: SparkContext, allocationClient: Option[ExecutorAllocationClient]) = {
+ this(sc.listenerBus, sc.conf, allocationClient)
+ }
+
+ HealthTracker.validateExcludeOnFailureConfs(conf)
+ private val MAX_FAILURES_PER_EXEC = conf.get(config.MAX_FAILURES_PER_EXEC)
+ private val MAX_FAILED_EXEC_PER_NODE = conf.get(config.MAX_FAILED_EXEC_PER_NODE)
+ val EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS = HealthTracker.getExludeOnFailureTimeout(conf)
+ private val EXCLUDE_FETCH_FAILURE_ENABLED =
+ conf.get(config.EXCLUDE_ON_FAILURE_FETCH_FAILURE_ENABLED)
+
+ /**
+ * A map from executorId to information on task failures. Tracks the time of each task failure,
+ * so that we can avoid excluding executors due to failures that are very far apart. We do not
+ * actively remove from this as soon as tasks hit their timeouts, to avoid the time it would take
+ * to do so. But it will not grow too large, because as soon as an executor gets too many
+ * failures, we exclude the executor and remove its entry here.
+ */
+ private val executorIdToFailureList = new HashMap[String, ExecutorFailureList]()
+ val executorIdToExcludedStatus = new HashMap[String, ExcludedExecutor]()
+ val nodeIdToExcludedExpiryTime = new HashMap[String, Long]()
+ /**
+ * An immutable copy of the set of nodes that are currently excluded. Kept in an
+ * AtomicReference to make [[excludedNodeList()]] thread-safe.
+ */
+ private val _excludedNodeList = new AtomicReference[Set[String]](Set())
+ /**
+ * Time when the next excluded node will expire. Used as a shortcut to
+ * avoid iterating over all entries in the excludedNodeList when none will have expired.
+ */
+ var nextExpiryTime: Long = Long.MaxValue
+ /**
+ * Mapping from nodes to all of the executors that have been excluded on that node. We do *not*
+ * remove from this when executors are removed from spark, so we can track when we get multiple
+ * successive excluded executors on one node. Nonetheless, it will not grow too large because
+ * there cannot be many excluded executors on one node, before we stop requesting more
+ * executors on that node, and we clean up the list of exluded executors once an executor has
+ * been excluded for EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS.
+ */
+ val nodeToExcludedExecs = new HashMap[String, HashSet[String]]()
+
+ /**
+ * Include executors and nodes that have been excluded for at least
+ * EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS
+ */
+ def applyExcludeOnFailureTimeout(): Unit = {
+ val now = clock.getTimeMillis()
+ // quickly check if we've got anything to expire that is excluded -- if not,
+ // avoid doing any work
+ if (now > nextExpiryTime) {
+ // Apply the timeout to excluded nodes and executors
+ val execsToInclude = executorIdToExcludedStatus.filter(_._2.expiryTime < now).keys
+ if (execsToInclude.nonEmpty) {
+ // Include any executors that have been exluded longer than the excludeOnFailure timeout.
+ logInfo(s"Removing executors $execsToInclude from exclude list because the " +
+ s"the executors have reached the timed out")
+ execsToInclude.foreach { exec =>
+ val status = executorIdToExcludedStatus.remove(exec).get
+ val failedExecsOnNode = nodeToExcludedExecs(status.node)
+ listenerBus.post(SparkListenerExecutorUnexcluded(now, exec))
+ failedExecsOnNode.remove(exec)
+ if (failedExecsOnNode.isEmpty) {
+ nodeToExcludedExecs.remove(status.node)
+ }
+ }
+ }
+ val nodesToInclude = nodeIdToExcludedExpiryTime.filter(_._2 < now).keys
+ if (nodesToInclude.nonEmpty) {
+ // Include any nodes that have been excluded longer than the excludeOnFailure timeout.
+ logInfo(s"Removing nodes $nodesToInclude from exclude list because the " +
+ s"nodes have reached has timed out")
+ nodesToInclude.foreach { node =>
+ nodeIdToExcludedExpiryTime.remove(node)
+ listenerBus.post(SparkListenerNodeUnexcluded(now, node))
+ }
+ _excludedNodeList.set(nodeIdToExcludedExpiryTime.keySet.toSet)
+ }
+ updateNextExpiryTime()
+ }
+ }
+
+ private def updateNextExpiryTime(): Unit = {
+ val execMinExpiry = if (executorIdToExcludedStatus.nonEmpty) {
+ executorIdToExcludedStatus.map{_._2.expiryTime}.min
+ } else {
+ Long.MaxValue
+ }
+ val nodeMinExpiry = if (nodeIdToExcludedExpiryTime.nonEmpty) {
+ nodeIdToExcludedExpiryTime.values.min
+ } else {
+ Long.MaxValue
+ }
+ nextExpiryTime = math.min(execMinExpiry, nodeMinExpiry)
+ }
+
+ private def killExecutor(exec: String, msg: String): Unit = {
+ allocationClient match {
+ case Some(a) =>
+ logInfo(msg)
+ a.killExecutors(Seq(exec), adjustTargetNumExecutors = false, countFailures = false,
+ force = true)
+ case None =>
+ logInfo(s"Not attempting to kill excluded executor id $exec " +
+ s"since allocation client is not defined.")
+ }
+ }
+
+ private def killExcludedExecutor(exec: String): Unit = {
+ if (conf.get(config.EXCLUDE_ON_FAILURE_KILL_ENABLED)) {
+ killExecutor(exec, s"Killing excluded executor id $exec since " +
+ s"${config.EXCLUDE_ON_FAILURE_KILL_ENABLED.key} is set.")
+ }
+ }
+
+ private[scheduler] def killExcludedIdleExecutor(exec: String): Unit = {
+ killExecutor(exec,
+ s"Killing excluded idle executor id $exec because of task unschedulability and trying " +
+ "to acquire a new executor.")
+ }
+
+ private def killExecutorsOnExcludedNode(node: String): Unit = {
+ if (conf.get(config.EXCLUDE_ON_FAILURE_KILL_ENABLED)) {
+ allocationClient match {
+ case Some(a) =>
+ logInfo(s"Killing all executors on excluded host $node " +
+ s"since ${config.EXCLUDE_ON_FAILURE_KILL_ENABLED.key} is set.")
+ if (a.killExecutorsOnHost(node) == false) {
+ logError(s"Killing executors on node $node failed.")
+ }
+ case None =>
+ logWarning(s"Not attempting to kill executors on excluded host $node " +
+ s"since allocation client is not defined.")
+ }
+ }
+ }
+
+ def updateExcludedForFetchFailure(host: String, exec: String): Unit = {
+ if (EXCLUDE_FETCH_FAILURE_ENABLED) {
+ // If we exclude on fetch failures, we are implicitly saying that we believe the failure is
+ // non-transient, and can't be recovered from (even if this is the first fetch failure,
+ // stage is retried after just one failure, so we don't always get a chance to collect
+ // multiple fetch failures).
+ // If the external shuffle-service is on, then every other executor on this node would
+ // be suffering from the same issue, so we should exclude (and potentially kill) all
+ // of them immediately.
+
+ val now = clock.getTimeMillis()
+ val expiryTimeForNewExcludes = now + EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS
+
+ if (conf.get(config.SHUFFLE_SERVICE_ENABLED)) {
+ if (!nodeIdToExcludedExpiryTime.contains(host)) {
+ logInfo(s"excluding node $host due to fetch failure of external shuffle service")
+
+ nodeIdToExcludedExpiryTime.put(host, expiryTimeForNewExcludes)
+ listenerBus.post(SparkListenerNodeExcluded(now, host, 1))
+ _excludedNodeList.set(nodeIdToExcludedExpiryTime.keySet.toSet)
+ killExecutorsOnExcludedNode(host)
+ updateNextExpiryTime()
+ }
+ } else if (!executorIdToExcludedStatus.contains(exec)) {
+ logInfo(s"Excluding executor $exec due to fetch failure")
+
+ executorIdToExcludedStatus.put(exec, ExcludedExecutor(host, expiryTimeForNewExcludes))
+ // We hardcoded number of failure tasks to 1 for fetch failure, because there's no
+ // reattempt for such failure.
+ listenerBus.post(SparkListenerExecutorExcluded(now, exec, 1))
Review comment:
sounds good, I'll move the handling to OnOtherEvent
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-715502768
**[Test build #130210 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130210/testReport)** for PR 29906 at commit [`32ab73d`](https://github.com/apache/spark/commit/32ab73da672ad62192f811558961660fbc972c37).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-710051232
Before the comment I think to that is what I had -just deprecated, so it would be fairly easy to revert to it. Also cc @Ngone51 since he was part of that discussion
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705214880
**[Test build #129532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129532/testReport)** for PR 29906 at commit [`0172aa4`](https://github.com/apache/spark/commit/0172aa4a8c64b6726a50e703e5e18462790c4949).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-712441982
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-712870009
I don't know that we can change a public api to developer api in minor release. This may make someone think they can change it in like 3.2.0, when I think really we need to wait for major release. thoughts from others - @mridulm @srowen
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r497495449
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusSource.scala
##########
@@ -59,9 +59,17 @@ private[spark] class AppStatusSource extends Source {
val SKIPPED_TASKS = getCounter("tasks", "skippedTasks")
+ // this is private but user visible from metrics so just deprecate
+ // @deprecated("use excludedExecutors instead", "3.1.0")
Review comment:
not intentional
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-712441960
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34623/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-718039706
**[Test build #130370 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130370/testReport)** for PR 29906 at commit [`b38dd66`](https://github.com/apache/spark/commit/b38dd66500cdd4a12f78cca1eeacf116f0ea5b4e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-712412016
**[Test build #130016 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130016/testReport)** for PR 29906 at commit [`b344ef6`](https://github.com/apache/spark/commit/b344ef6a9213ac8ae0c1bd059976bfa1b58d9cde).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705214533
working on upmerging
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r505991668
##########
File path: core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala
##########
@@ -431,23 +481,23 @@ abstract class SparkListener extends SparkListenerInterface {
override def onExecutorRemoved(executorRemoved: SparkListenerExecutorRemoved): Unit = { }
- override def onExecutorBlacklisted(
- executorBlacklisted: SparkListenerExecutorBlacklisted): Unit = { }
+ override def onExecutorExcluded(
+ executorExcluded: SparkListenerExecutorExcluded): Unit = { }
- def onExecutorBlacklistedForStage(
- executorBlacklistedForStage: SparkListenerExecutorBlacklistedForStage): Unit = { }
+ override def onExecutorExcludedForStage(
+ executorExcludedForStage: SparkListenerExecutorExcludedForStage): Unit = { }
- def onNodeBlacklistedForStage(
- nodeBlacklistedForStage: SparkListenerNodeBlacklistedForStage): Unit = { }
+ override def onNodeExcludedForStage(
+ nodeExcludedForStage: SparkListenerNodeExcludedForStage): Unit = { }
- override def onExecutorUnblacklisted(
- executorUnblacklisted: SparkListenerExecutorUnblacklisted): Unit = { }
+ override def onExecutorUnexcluded(
+ executorUnexcluded: SparkListenerExecutorUnexcluded): Unit = { }
- override def onNodeBlacklisted(
- nodeBlacklisted: SparkListenerNodeBlacklisted): Unit = { }
+ override def onNodeExcluded(
+ nodeExcluded: SparkListenerNodeExcluded): Unit = { }
- override def onNodeUnblacklisted(
- nodeUnblacklisted: SparkListenerNodeUnblacklisted): Unit = { }
+ override def onNodeUnexcluded(
+ nodeUnexcluded: SparkListenerNodeUnexcluded): Unit = { }
Review comment:
I am catching up on this PR; this would be an incompatible source change right ?
Existing listeners which override `SparkListener`, particularly `override def onNodeBlacklisted` (for ex) will now fail compilation and will need to be updated to `override def onNodeExcluded` (and so on) - in a minor release.
Given we are not removing the event, do we want to keep both - but mark `*blaclisted` api's as deprecated and to be removed in next major release ?
Or are we doing this to force users to change ? In particular, to ensure that expectation of invocation of `*blaclisted` api's will not happen ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701012147
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-702525542
> I'm confused by why the build and test report test results says failed when everything passed. @dongjoon-hyun @HyukjinKwon have you seen that before, am I perhaps missing something?
Yes, I think I have seen this few times. The failure error message was somewhat not very deterministic so I was kept monitoring it. Seems like it fails when there are no JUnit XML generated, and there's nothing to report. Did the test failure like https://github.com/apache/spark/runs/1196184007? Let me take a quick look.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701450857
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701667505
**[Test build #129289 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129289/testReport)** for PR 29906 at commit [`6da870b`](https://github.com/apache/spark/commit/6da870b4cd2f039345ddeb1792d6ddf0ccf19136).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705620977
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705725764
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-715880777
Thanks for working on this Tom ! Looks good to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705657936
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-715665066
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-715503400
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130210/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r497510205
##########
File path: resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala
##########
@@ -523,9 +523,9 @@ class YarnAllocatorSuite extends SparkFunSuite with Matchers with BeforeAndAfter
handler.getNumUnexpectedContainerRelease should be (2)
}
- test("blacklisted nodes reflected in amClient requests") {
- // Internally we track the set of blacklisted nodes, but yarn wants us to send *changes*
- // to the blacklist. This makes sure we are sending the right updates.
+ test("excludeOnFailure nodes reflected in amClient requests") {
+ // Internally we track the set of excluded nodes, but yarn wants us to send *changes*
+ // to it. This makes sure we are sending the right updates.
Review comment:
Note the test case name actually should just be excluded nodes here so i'll update that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs edited a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs edited a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-710049939
@srowen @mridulm see the discussion back : https://github.com/apache/spark/pull/29906#discussion_r499125900
I'm fine either way. More input would be great so let me know what you think. Or if we know of people directly using these we may want to leave.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701736345
**[Test build #129289 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129289/testReport)** for PR 29906 at commit [`6da870b`](https://github.com/apache/spark/commit/6da870b4cd2f039345ddeb1792d6ddf0ccf19136).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705237286
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705259319
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-717994903
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705215433
Build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-715646698
**[Test build #130220 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130220/testReport)** for PR 29906 at commit [`32ab73d`](https://github.com/apache/spark/commit/32ab73da672ad62192f811558961660fbc972c37).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r501048450
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala
##########
@@ -379,14 +379,15 @@ package object config extends Logging {
.stringConf
.createOptional
- /* YARN allocator-level blacklisting related config entries. */
- private[spark] val YARN_EXECUTOR_LAUNCH_BLACKLIST_ENABLED =
- ConfigBuilder("spark.yarn.blacklist.executor.launch.blacklisting.enabled")
+ /* YARN allocator-level excludeOnFailure related config entries. */
+ private[spark] val YARN_EXECUTOR_LAUNCH_EXCLUDE_ON_FAILURE_ENABLED =
+ ConfigBuilder("spark.yarn.executor.launch.excludeOnFailure.enabled")
Review comment:
because its excluding due to executor launch failures. I'd prefer to leave this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701690413
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705620987
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129558/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705865452
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34171/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701448681
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701450868
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129279/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r498336779
##########
File path: core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala
##########
@@ -0,0 +1,478 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.atomic.AtomicReference
+
+import scala.collection.mutable.{ArrayBuffer, HashMap, HashSet}
+
+import org.apache.spark.{ExecutorAllocationClient, SparkConf, SparkContext}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config
+import org.apache.spark.util.{Clock, SystemClock, Utils}
+
+/**
+ * HealthTracker is designed to track problematic executors and nodes. It supports excluding
+ * executors and nodes across an entire application (with a periodic expiry). TaskSetManagers add
+ * additional logic for exclusion of executors and nodes for individual tasks and stages which
+ * works in concert with the logic here.
+ *
+ * The tracker needs to deal with a variety of workloads, eg.:
+ *
+ * * bad user code -- this may lead to many task failures, but that should not count against
+ * individual executors
+ * * many small stages -- this may prevent a bad executor for having many failures within one
+ * stage, but still many failures over the entire application
+ * * "flaky" executors -- they don't fail every task, but are still faulty enough to merit
+ * excluding
+ *
+ * See the design doc on SPARK-8425 for a more in-depth discussion.
+ *
+ * THREADING: As with most helpers of TaskSchedulerImpl, this is not thread-safe. Though it is
+ * called by multiple threads, callers must already have a lock on the TaskSchedulerImpl. The
+ * one exception is [[excludedNodeList()]], which can be called without holding a lock.
+ */
+private[scheduler] class HealthTracker (
+ private val listenerBus: LiveListenerBus,
+ conf: SparkConf,
+ allocationClient: Option[ExecutorAllocationClient],
+ clock: Clock = new SystemClock()) extends Logging {
+
+ def this(sc: SparkContext, allocationClient: Option[ExecutorAllocationClient]) = {
+ this(sc.listenerBus, sc.conf, allocationClient)
+ }
+
+ HealthTracker.validateExcludeOnFailureConfs(conf)
+ private val MAX_FAILURES_PER_EXEC = conf.get(config.MAX_FAILURES_PER_EXEC)
+ private val MAX_FAILED_EXEC_PER_NODE = conf.get(config.MAX_FAILED_EXEC_PER_NODE)
+ val EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS = HealthTracker.getExludeOnFailureTimeout(conf)
+ private val EXCLUDE_FETCH_FAILURE_ENABLED =
+ conf.get(config.EXCLUDE_ON_FAILURE_FETCH_FAILURE_ENABLED)
+
+ /**
+ * A map from executorId to information on task failures. Tracks the time of each task failure,
+ * so that we can avoid excluding executors due to failures that are very far apart. We do not
+ * actively remove from this as soon as tasks hit their timeouts, to avoid the time it would take
+ * to do so. But it will not grow too large, because as soon as an executor gets too many
+ * failures, we exclude the executor and remove its entry here.
+ */
+ private val executorIdToFailureList = new HashMap[String, ExecutorFailureList]()
+ val executorIdToExcludedStatus = new HashMap[String, ExcludedExecutor]()
+ val nodeIdToExcludedExpiryTime = new HashMap[String, Long]()
+ /**
+ * An immutable copy of the set of nodes that are currently excluded. Kept in an
+ * AtomicReference to make [[excludedNodeList()]] thread-safe.
+ */
+ private val _excludedNodeList = new AtomicReference[Set[String]](Set())
+ /**
+ * Time when the next excluded node will expire. Used as a shortcut to
+ * avoid iterating over all entries in the excludedNodeList when none will have expired.
+ */
+ var nextExpiryTime: Long = Long.MaxValue
+ /**
+ * Mapping from nodes to all of the executors that have been excluded on that node. We do *not*
+ * remove from this when executors are removed from spark, so we can track when we get multiple
+ * successive excluded executors on one node. Nonetheless, it will not grow too large because
+ * there cannot be many excluded executors on one node, before we stop requesting more
+ * executors on that node, and we clean up the list of exluded executors once an executor has
+ * been excluded for EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS.
+ */
+ val nodeToExcludedExecs = new HashMap[String, HashSet[String]]()
+
+ /**
+ * Include executors and nodes that have been excluded for at least
+ * EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS
+ */
+ def applyExcludeOnFailureTimeout(): Unit = {
+ val now = clock.getTimeMillis()
+ // quickly check if we've got anything to expire that is excluded -- if not,
+ // avoid doing any work
+ if (now > nextExpiryTime) {
+ // Apply the timeout to excluded nodes and executors
+ val execsToInclude = executorIdToExcludedStatus.filter(_._2.expiryTime < now).keys
+ if (execsToInclude.nonEmpty) {
+ // Include any executors that have been exluded longer than the excludeOnFailure timeout.
+ logInfo(s"Removing executors $execsToInclude from exclude list because the " +
+ s"the executors have reached the timed out")
+ execsToInclude.foreach { exec =>
+ val status = executorIdToExcludedStatus.remove(exec).get
+ val failedExecsOnNode = nodeToExcludedExecs(status.node)
+ listenerBus.post(SparkListenerExecutorUnexcluded(now, exec))
+ failedExecsOnNode.remove(exec)
+ if (failedExecsOnNode.isEmpty) {
+ nodeToExcludedExecs.remove(status.node)
+ }
+ }
+ }
+ val nodesToInclude = nodeIdToExcludedExpiryTime.filter(_._2 < now).keys
+ if (nodesToInclude.nonEmpty) {
+ // Include any nodes that have been excluded longer than the excludeOnFailure timeout.
+ logInfo(s"Removing nodes $nodesToInclude from exclude list because the " +
+ s"nodes have reached has timed out")
+ nodesToInclude.foreach { node =>
+ nodeIdToExcludedExpiryTime.remove(node)
+ listenerBus.post(SparkListenerNodeUnexcluded(now, node))
+ }
+ _excludedNodeList.set(nodeIdToExcludedExpiryTime.keySet.toSet)
+ }
+ updateNextExpiryTime()
+ }
+ }
+
+ private def updateNextExpiryTime(): Unit = {
+ val execMinExpiry = if (executorIdToExcludedStatus.nonEmpty) {
+ executorIdToExcludedStatus.map{_._2.expiryTime}.min
+ } else {
+ Long.MaxValue
+ }
+ val nodeMinExpiry = if (nodeIdToExcludedExpiryTime.nonEmpty) {
+ nodeIdToExcludedExpiryTime.values.min
+ } else {
+ Long.MaxValue
+ }
+ nextExpiryTime = math.min(execMinExpiry, nodeMinExpiry)
+ }
+
+ private def killExecutor(exec: String, msg: String): Unit = {
+ allocationClient match {
+ case Some(a) =>
+ logInfo(msg)
+ a.killExecutors(Seq(exec), adjustTargetNumExecutors = false, countFailures = false,
+ force = true)
+ case None =>
+ logInfo(s"Not attempting to kill excluded executor id $exec " +
+ s"since allocation client is not defined.")
+ }
+ }
+
+ private def killExcludedExecutor(exec: String): Unit = {
+ if (conf.get(config.EXCLUDE_ON_FAILURE_KILL_ENABLED)) {
+ killExecutor(exec, s"Killing excluded executor id $exec since " +
+ s"${config.EXCLUDE_ON_FAILURE_KILL_ENABLED.key} is set.")
+ }
+ }
+
+ private[scheduler] def killExcludedIdleExecutor(exec: String): Unit = {
+ killExecutor(exec,
+ s"Killing excluded idle executor id $exec because of task unschedulability and trying " +
+ "to acquire a new executor.")
+ }
+
+ private def killExecutorsOnExcludedNode(node: String): Unit = {
+ if (conf.get(config.EXCLUDE_ON_FAILURE_KILL_ENABLED)) {
+ allocationClient match {
+ case Some(a) =>
+ logInfo(s"Killing all executors on excluded host $node " +
+ s"since ${config.EXCLUDE_ON_FAILURE_KILL_ENABLED.key} is set.")
+ if (a.killExecutorsOnHost(node) == false) {
+ logError(s"Killing executors on node $node failed.")
+ }
+ case None =>
+ logWarning(s"Not attempting to kill executors on excluded host $node " +
+ s"since allocation client is not defined.")
+ }
+ }
+ }
+
+ def updateExcludedForFetchFailure(host: String, exec: String): Unit = {
+ if (EXCLUDE_FETCH_FAILURE_ENABLED) {
+ // If we exclude on fetch failures, we are implicitly saying that we believe the failure is
+ // non-transient, and can't be recovered from (even if this is the first fetch failure,
+ // stage is retried after just one failure, so we don't always get a chance to collect
+ // multiple fetch failures).
+ // If the external shuffle-service is on, then every other executor on this node would
+ // be suffering from the same issue, so we should exclude (and potentially kill) all
+ // of them immediately.
+
+ val now = clock.getTimeMillis()
+ val expiryTimeForNewExcludes = now + EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS
+
+ if (conf.get(config.SHUFFLE_SERVICE_ENABLED)) {
+ if (!nodeIdToExcludedExpiryTime.contains(host)) {
+ logInfo(s"excluding node $host due to fetch failure of external shuffle service")
+
+ nodeIdToExcludedExpiryTime.put(host, expiryTimeForNewExcludes)
+ listenerBus.post(SparkListenerNodeExcluded(now, host, 1))
+ _excludedNodeList.set(nodeIdToExcludedExpiryTime.keySet.toSet)
+ killExecutorsOnExcludedNode(host)
+ updateNextExpiryTime()
+ }
+ } else if (!executorIdToExcludedStatus.contains(exec)) {
+ logInfo(s"Excluding executor $exec due to fetch failure")
+
+ executorIdToExcludedStatus.put(exec, ExcludedExecutor(host, expiryTimeForNewExcludes))
+ // We hardcoded number of failure tasks to 1 for fetch failure, because there's no
+ // reattempt for such failure.
+ listenerBus.post(SparkListenerExecutorExcluded(now, exec, 1))
Review comment:
Looks like we no longer post `SparkListenerExecutorBlacklisted` after this change? So we'd break backward compatibility since `onExecutorBlacklisted` never receives it, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705662640
**[Test build #129561 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129561/testReport)** for PR 29906 at commit [`3ebf881`](https://github.com/apache/spark/commit/3ebf8814299fb96f52df479a46d1be28d453a3ce).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705687305
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34167/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701517394
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705620977
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705619969
**[Test build #129558 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129558/testReport)** for PR 29906 at commit [`32fca2d`](https://github.com/apache/spark/commit/32fca2d9740f40784294d1bfd0584371c105c7d9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-717949324
**[Test build #130370 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130370/testReport)** for PR 29906 at commit [`b38dd66`](https://github.com/apache/spark/commit/b38dd66500cdd4a12f78cca1eeacf116f0ea5b4e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701737070
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-702219531
Ya. I saw the same situation for `DOC` only commit. I thought that it's a bug of `Build and test / Report test results` GitHub Action job.
> I'm confused by why the build and test report test results says failed when everything passed. @dongjoon-hyun @HyukjinKwon have you seen that before, am I perhaps missing something?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-713034168
`SparkFirehoseListener` has been public right ? We just missed annotating it appropriately ?
Given this, I am comfortable fixing this oversight.
Thoughts @srowen, @dongjoon-hyun ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701446411
**[Test build #129279 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129279/testReport)** for PR 29906 at commit [`76714e7`](https://github.com/apache/spark/commit/76714e7ddf33cdf2058ecac2f1c6bb915fc008fb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r499077732
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -722,74 +722,83 @@ package object config {
.booleanConf
.createWithDefault(true)
- // Blacklist confs
- private[spark] val BLACKLIST_ENABLED =
- ConfigBuilder("spark.blacklist.enabled")
+ private[spark] val EXCLUDE_ON_FAILURE_ENABLED =
+ ConfigBuilder("spark.excludeOnFailure.enabled")
.version("2.1.0")
Review comment:
Do we need to update the "from" version strings here?
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -907,13 +908,13 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
protected def currentDelegationTokens: Array[Byte] = delegationTokens.get()
/**
- * Checks whether the executor is blacklisted. This is called when the executor tries to
- * register with the scheduler, and will deny registration if this method returns true.
+ * Checks whether the executor is excluded due to failure(s). This is called when the executor
+ * tries to register with the scheduler, and will deny registration if this method returns true.
Review comment:
minor nit: extra space at the start of the line
##########
File path: core/src/main/scala/org/apache/spark/status/api/v1/api.scala
##########
@@ -82,10 +82,11 @@ class ExecutorStageSummary private[spark](
val shuffleWriteRecords : Long,
val memoryBytesSpilled : Long,
val diskBytesSpilled : Long,
- val isBlacklistedForStage: Boolean,
+ val isBlacklistedForStage: Boolean, // deprecated
Review comment:
Can we `@deprecated` for this and others?
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -284,80 +284,138 @@ private[spark] class AppStatusListener(
}
override def onExecutorBlacklisted(event: SparkListenerExecutorBlacklisted): Unit = {
- updateBlackListStatus(event.executorId, true)
+ updateExcludedStatus(event.executorId, true)
+ }
+
+ override def onExecutorExcluded(event: SparkListenerExecutorExcluded): Unit = {
+ updateExcludedStatus(event.executorId, true)
}
override def onExecutorBlacklistedForStage(
- event: SparkListenerExecutorBlacklistedForStage): Unit = {
+ event: SparkListenerExecutorBlacklistedForStage): Unit = {
+ val now = System.nanoTime()
+
+ Option(liveStages.get((event.stageId, event.stageAttemptId))).foreach { stage =>
+ setStageExcludedStatus(stage, now, event.executorId)
+ }
+ liveExecutors.get(event.executorId).foreach { exec =>
+ addExcludedStageTo(exec, event.stageId, now)
+ }
+ }
+
+ override def onExecutorExcludedForStage(
+ event: SparkListenerExecutorExcludedForStage): Unit = {
val now = System.nanoTime()
Option(liveStages.get((event.stageId, event.stageAttemptId))).foreach { stage =>
- setStageBlackListStatus(stage, now, event.executorId)
+ setStageExcludedStatus(stage, now, event.executorId)
}
liveExecutors.get(event.executorId).foreach { exec =>
- addBlackListedStageTo(exec, event.stageId, now)
+ addExcludedStageTo(exec, event.stageId, now)
}
}
override def onNodeBlacklistedForStage(event: SparkListenerNodeBlacklistedForStage): Unit = {
val now = System.nanoTime()
- // Implicitly blacklist every available executor for the stage associated with this node
+ // Implicitly exclude every available executor for the stage associated with this node
Option(liveStages.get((event.stageId, event.stageAttemptId))).foreach { stage =>
val executorIds = liveExecutors.values.filter(_.host == event.hostId).map(_.executorId).toSeq
- setStageBlackListStatus(stage, now, executorIds: _*)
+ setStageExcludedStatus(stage, now, executorIds: _*)
}
liveExecutors.values.filter(_.hostname == event.hostId).foreach { exec =>
- addBlackListedStageTo(exec, event.stageId, now)
+ addExcludedStageTo(exec, event.stageId, now)
+ }
+ }
+
+ override def onNodeExcludedForStage(event: SparkListenerNodeExcludedForStage): Unit = {
+ val now = System.nanoTime()
+
+ // Implicitly exclude every available executor for the stage associated with this node
+ Option(liveStages.get((event.stageId, event.stageAttemptId))).foreach { stage =>
+ val executorIds = liveExecutors.values.filter(_.host == event.hostId).map(_.executorId).toSeq
+ setStageExcludedStatus(stage, now, executorIds: _*)
+ }
+ liveExecutors.values.filter(_.hostname == event.hostId).foreach { exec =>
+ addExcludedStageTo(exec, event.stageId, now)
}
}
private def addBlackListedStageTo(exec: LiveExecutor, stageId: Int, now: Long): Unit = {
- exec.blacklistedInStages += stageId
+ exec.excludedInStages += stageId
+ liveUpdate(exec, now)
+ }
+
+ private def addExcludedStageTo(exec: LiveExecutor, stageId: Int, now: Long): Unit = {
+ exec.excludedInStages += stageId
liveUpdate(exec, now)
}
private def setStageBlackListStatus(stage: LiveStage, now: Long, executorIds: String*): Unit = {
executorIds.foreach { executorId =>
val executorStageSummary = stage.executorSummary(executorId)
- executorStageSummary.isBlacklisted = true
+ executorStageSummary.isExcluded = true
maybeUpdate(executorStageSummary, now)
}
- stage.blackListedExecutors ++= executorIds
+ stage.excludedExecutors ++= executorIds
+ maybeUpdate(stage, now)
+ }
+
+ private def setStageExcludedStatus(stage: LiveStage, now: Long, executorIds: String*): Unit = {
+ executorIds.foreach { executorId =>
+ val executorStageSummary = stage.executorSummary(executorId)
+ executorStageSummary.isExcluded = true
+ maybeUpdate(executorStageSummary, now)
+ }
+ stage.excludedExecutors ++= executorIds
maybeUpdate(stage, now)
}
override def onExecutorUnblacklisted(event: SparkListenerExecutorUnblacklisted): Unit = {
- updateBlackListStatus(event.executorId, false)
+ updateExcludedStatus(event.executorId, false)
+ }
+
+ override def onExecutorUnexcluded(event: SparkListenerExecutorUnexcluded): Unit = {
+ updateExcludedStatus(event.executorId, false)
}
override def onNodeBlacklisted(event: SparkListenerNodeBlacklisted): Unit = {
- updateNodeBlackList(event.hostId, true)
+ updateNodeExcluded(event.hostId, true)
+ }
+
+ override def onNodeExcluded(event: SparkListenerNodeExcluded): Unit = {
+ updateNodeExcluded(event.hostId, true)
}
override def onNodeUnblacklisted(event: SparkListenerNodeUnblacklisted): Unit = {
- updateNodeBlackList(event.hostId, false)
+ updateNodeExcluded(event.hostId, false)
+ }
+
+ override def onNodeUnexcluded(event: SparkListenerNodeUnexcluded): Unit = {
+ updateNodeExcluded(event.hostId, false)
}
- private def updateBlackListStatus(execId: String, blacklisted: Boolean): Unit = {
+ private def updateExcludedStatus(execId: String, excluded: Boolean): Unit = {
Review comment:
Is `updateExclusionStatus` a better name? It reads more smoothly to me, but I'm not sure if it's less clear.
##########
File path: docs/monitoring.md
##########
@@ -1125,12 +1125,14 @@ This is the component with the largest amount of instrumented metrics
- stages.failedStages.count
- stages.skippedStages.count
- stages.completedStages.count
- - tasks.blackListedExecutors.count
+ - tasks.blackListedExecutors.count // deprecated use excludedExecutors instead
Review comment:
Is it even worth mentioning in the documentation?
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -284,80 +284,138 @@ private[spark] class AppStatusListener(
}
override def onExecutorBlacklisted(event: SparkListenerExecutorBlacklisted): Unit = {
- updateBlackListStatus(event.executorId, true)
+ updateExcludedStatus(event.executorId, true)
+ }
+
+ override def onExecutorExcluded(event: SparkListenerExecutorExcluded): Unit = {
+ updateExcludedStatus(event.executorId, true)
}
override def onExecutorBlacklistedForStage(
- event: SparkListenerExecutorBlacklistedForStage): Unit = {
+ event: SparkListenerExecutorBlacklistedForStage): Unit = {
+ val now = System.nanoTime()
+
+ Option(liveStages.get((event.stageId, event.stageAttemptId))).foreach { stage =>
+ setStageExcludedStatus(stage, now, event.executorId)
+ }
+ liveExecutors.get(event.executorId).foreach { exec =>
+ addExcludedStageTo(exec, event.stageId, now)
+ }
+ }
+
+ override def onExecutorExcludedForStage(
+ event: SparkListenerExecutorExcludedForStage): Unit = {
val now = System.nanoTime()
Option(liveStages.get((event.stageId, event.stageAttemptId))).foreach { stage =>
- setStageBlackListStatus(stage, now, event.executorId)
+ setStageExcludedStatus(stage, now, event.executorId)
}
liveExecutors.get(event.executorId).foreach { exec =>
- addBlackListedStageTo(exec, event.stageId, now)
+ addExcludedStageTo(exec, event.stageId, now)
}
}
override def onNodeBlacklistedForStage(event: SparkListenerNodeBlacklistedForStage): Unit = {
val now = System.nanoTime()
- // Implicitly blacklist every available executor for the stage associated with this node
+ // Implicitly exclude every available executor for the stage associated with this node
Option(liveStages.get((event.stageId, event.stageAttemptId))).foreach { stage =>
val executorIds = liveExecutors.values.filter(_.host == event.hostId).map(_.executorId).toSeq
- setStageBlackListStatus(stage, now, executorIds: _*)
+ setStageExcludedStatus(stage, now, executorIds: _*)
}
liveExecutors.values.filter(_.hostname == event.hostId).foreach { exec =>
- addBlackListedStageTo(exec, event.stageId, now)
+ addExcludedStageTo(exec, event.stageId, now)
+ }
+ }
+
+ override def onNodeExcludedForStage(event: SparkListenerNodeExcludedForStage): Unit = {
+ val now = System.nanoTime()
+
+ // Implicitly exclude every available executor for the stage associated with this node
+ Option(liveStages.get((event.stageId, event.stageAttemptId))).foreach { stage =>
+ val executorIds = liveExecutors.values.filter(_.host == event.hostId).map(_.executorId).toSeq
+ setStageExcludedStatus(stage, now, executorIds: _*)
+ }
+ liveExecutors.values.filter(_.hostname == event.hostId).foreach { exec =>
+ addExcludedStageTo(exec, event.stageId, now)
}
}
private def addBlackListedStageTo(exec: LiveExecutor, stageId: Int, now: Long): Unit = {
- exec.blacklistedInStages += stageId
+ exec.excludedInStages += stageId
+ liveUpdate(exec, now)
+ }
+
+ private def addExcludedStageTo(exec: LiveExecutor, stageId: Int, now: Long): Unit = {
+ exec.excludedInStages += stageId
liveUpdate(exec, now)
}
private def setStageBlackListStatus(stage: LiveStage, now: Long, executorIds: String*): Unit = {
executorIds.foreach { executorId =>
val executorStageSummary = stage.executorSummary(executorId)
- executorStageSummary.isBlacklisted = true
+ executorStageSummary.isExcluded = true
maybeUpdate(executorStageSummary, now)
}
- stage.blackListedExecutors ++= executorIds
+ stage.excludedExecutors ++= executorIds
+ maybeUpdate(stage, now)
+ }
+
+ private def setStageExcludedStatus(stage: LiveStage, now: Long, executorIds: String*): Unit = {
Review comment:
Is `setStageExclusionStatus` a better name? It reads more smoothly to me, but I'm not sure if it's less clear.
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -638,18 +638,19 @@ private[spark] class TaskSchedulerImpl(
}
} else {
// Abort Immediately
- logInfo("Cannot schedule any task because of complete blacklisting. No idle" +
- s" executors can be found to kill. Aborting stage ${taskSet.stageId}.")
- taskSet.abortSinceCompletelyBlacklisted(taskIndex)
+ logInfo("Cannot schedule any task because all executor excluded from " +
Review comment:
minor nit: `all executor excluded` -> `all executors excluded`
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -722,74 +722,83 @@ package object config {
.booleanConf
.createWithDefault(true)
- // Blacklist confs
- private[spark] val BLACKLIST_ENABLED =
- ConfigBuilder("spark.blacklist.enabled")
+ private[spark] val EXCLUDE_ON_FAILURE_ENABLED =
+ ConfigBuilder("spark.excludeOnFailure.enabled")
.version("2.1.0")
+ .withAlternative("spark.blacklist.enabled")
.booleanConf
.createOptional
private[spark] val MAX_TASK_ATTEMPTS_PER_EXECUTOR =
- ConfigBuilder("spark.blacklist.task.maxTaskAttemptsPerExecutor")
+ ConfigBuilder("spark.excludeOnFailure.task.maxTaskAttemptsPerExecutor")
.version("2.1.0")
+ .withAlternative("spark.blacklist.task.maxTaskAttemptsPerExecutor")
.intConf
.createWithDefault(1)
private[spark] val MAX_TASK_ATTEMPTS_PER_NODE =
- ConfigBuilder("spark.blacklist.task.maxTaskAttemptsPerNode")
+ ConfigBuilder("spark.excludeOnFailure.task.maxTaskAttemptsPerNode")
.version("2.1.0")
+ .withAlternative("spark.blacklist.task.maxTaskAttemptsPerNode")
.intConf
.createWithDefault(2)
private[spark] val MAX_FAILURES_PER_EXEC =
- ConfigBuilder("spark.blacklist.application.maxFailedTasksPerExecutor")
+ ConfigBuilder("spark.excludeOnFailure.application.maxFailedTasksPerExecutor")
.version("2.2.0")
+ .withAlternative("spark.blacklist.application.maxFailedTasksPerExecutor")
.intConf
.createWithDefault(2)
private[spark] val MAX_FAILURES_PER_EXEC_STAGE =
- ConfigBuilder("spark.blacklist.stage.maxFailedTasksPerExecutor")
+ ConfigBuilder("spark.excludeOnFailure.stage.maxFailedTasksPerExecutor")
.version("2.1.0")
+ .withAlternative("spark.blacklist.stage.maxFailedTasksPerExecutor")
.intConf
.createWithDefault(2)
private[spark] val MAX_FAILED_EXEC_PER_NODE =
- ConfigBuilder("spark.blacklist.application.maxFailedExecutorsPerNode")
+ ConfigBuilder("spark.excludeOnFailure.application.maxFailedExecutorsPerNode")
.version("2.2.0")
+ .withAlternative("spark.blacklist.application.maxFailedExecutorsPerNode")
.intConf
.createWithDefault(2)
private[spark] val MAX_FAILED_EXEC_PER_NODE_STAGE =
- ConfigBuilder("spark.blacklist.stage.maxFailedExecutorsPerNode")
+ ConfigBuilder("spark.excludeOnFailure.stage.maxFailedExecutorsPerNode")
.version("2.1.0")
+ .withAlternative("spark.blacklist.stage.maxFailedExecutorsPerNode")
.intConf
.createWithDefault(2)
- private[spark] val BLACKLIST_TIMEOUT_CONF =
- ConfigBuilder("spark.blacklist.timeout")
+ private[spark] val EXCLUDE_ON_FAILURE_TIMEOUT_CONF =
+ ConfigBuilder("spark.excludeOnFailure.timeout")
.version("2.1.0")
+ .withAlternative("spark.blacklist.timeout")
.timeConf(TimeUnit.MILLISECONDS)
.createOptional
- private[spark] val BLACKLIST_KILL_ENABLED =
- ConfigBuilder("spark.blacklist.killBlacklistedExecutors")
+ private[spark] val EXCLUDE_ON_FAILURE_KILL_ENABLED =
+ ConfigBuilder("spark.excludeOnFailure.killExcludedExecutors")
.version("2.2.0")
+ .withAlternative("spark.excludeOnFailure.timeout")
.booleanConf
.createWithDefault(false)
- private[spark] val BLACKLIST_LEGACY_TIMEOUT_CONF =
- ConfigBuilder("spark.scheduler.executorTaskBlacklistTime")
+ private[spark] val EXCLUDE_ON_FAILURE_LEGACY_TIMEOUT_CONF =
+ ConfigBuilder("spark.scheduler.executorTaskExcludeOnFailureTime")
.internal()
.version("1.0.0")
+ .withAlternative("spark.scheduler.executorTaskBlacklistTime")
.timeConf(TimeUnit.MILLISECONDS)
Review comment:
Given that this is already a legacy config marked as deprecated, it doesn't seem worth adding a new version. What do you think?
##########
File path: core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala
##########
@@ -0,0 +1,478 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.atomic.AtomicReference
+
+import scala.collection.mutable.{ArrayBuffer, HashMap, HashSet}
+
+import org.apache.spark.{ExecutorAllocationClient, SparkConf, SparkContext}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config
+import org.apache.spark.util.{Clock, SystemClock, Utils}
+
+/**
+ * HealthTracker is designed to track problematic executors and nodes. It supports excluding
+ * executors and nodes across an entire application (with a periodic expiry). TaskSetManagers add
+ * additional logic for exclusion of executors and nodes for individual tasks and stages which
+ * works in concert with the logic here.
+ *
+ * The tracker needs to deal with a variety of workloads, eg.:
+ *
+ * * bad user code -- this may lead to many task failures, but that should not count against
+ * individual executors
+ * * many small stages -- this may prevent a bad executor for having many failures within one
+ * stage, but still many failures over the entire application
+ * * "flaky" executors -- they don't fail every task, but are still faulty enough to merit
+ * excluding
+ *
+ * See the design doc on SPARK-8425 for a more in-depth discussion.
Review comment:
People might become confused when they go to SPARK-8425 and the name is completely different, should we also include a reference to SPARK-32037?
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -284,80 +284,138 @@ private[spark] class AppStatusListener(
}
override def onExecutorBlacklisted(event: SparkListenerExecutorBlacklisted): Unit = {
- updateBlackListStatus(event.executorId, true)
+ updateExcludedStatus(event.executorId, true)
+ }
+
+ override def onExecutorExcluded(event: SparkListenerExecutorExcluded): Unit = {
+ updateExcludedStatus(event.executorId, true)
}
override def onExecutorBlacklistedForStage(
- event: SparkListenerExecutorBlacklistedForStage): Unit = {
+ event: SparkListenerExecutorBlacklistedForStage): Unit = {
+ val now = System.nanoTime()
+
Review comment:
Should we consider delegating this implementation to `onExecutorExcludedForStage` to avoid code duplication? Same for others.
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocatorNodeHealthTracker.scala
##########
@@ -72,72 +73,76 @@ private[spark] class YarnAllocatorBlacklistTracker(
def handleResourceAllocationFailure(hostOpt: Option[String]): Unit = {
hostOpt match {
- case Some(hostname) if launchBlacklistEnabled =>
- // failures on an already blacklisted nodes are not even tracked.
+ case Some(hostname) if launchExcludeOnFailureEnabled =>
+ // failures on an already excluded node are not even tracked.
// otherwise, such failures could shutdown the application
// as resource requests are asynchronous
// and a late failure response could exceed MAX_EXECUTOR_FAILURES
- if (!schedulerBlacklist.contains(hostname) &&
- !allocatorBlacklist.contains(hostname)) {
+ if (!schedulerExcludedNodeList.contains(hostname) &&
+ !allocatorExcludedNodeList.contains(hostname)) {
failureTracker.registerFailureOnHost(hostname)
- updateAllocationBlacklistedNodes(hostname)
+ updateAllocationExcludedNodes(hostname)
}
case _ =>
failureTracker.registerExecutorFailure()
}
}
- private def updateAllocationBlacklistedNodes(hostname: String): Unit = {
+ private def updateAllocationExcludedNodes(hostname: String): Unit = {
val failuresOnHost = failureTracker.numFailuresOnHost(hostname)
if (failuresOnHost > maxFailuresPerHost) {
- logInfo(s"blacklisting $hostname as YARN allocation failed $failuresOnHost times")
- allocatorBlacklist.put(
+ logInfo(s"excluding $hostname as YARN allocation failed $failuresOnHost times")
+ allocatorExcludedNodeList.put(
hostname,
- failureTracker.clock.getTimeMillis() + blacklistTimeoutMillis)
- refreshBlacklistedNodes()
+ failureTracker.clock.getTimeMillis() + excludeOnFailureTimeoutMillis)
+ refreshExcludedNodes()
}
}
- def setSchedulerBlacklistedNodes(schedulerBlacklistedNodesWithExpiry: Set[String]): Unit = {
- this.schedulerBlacklist = schedulerBlacklistedNodesWithExpiry
- refreshBlacklistedNodes()
+ def setSchedulerExcludedNodes(schedulerExcludedNodesWithExpiry: Set[String]): Unit = {
+ this.schedulerExcludedNodeList = schedulerExcludedNodesWithExpiry
+ refreshExcludedNodes()
}
- def isAllNodeBlacklisted: Boolean = {
+ def isAllNodeExcluded: Boolean = {
if (numClusterNodes <= 0) {
logWarning("No available nodes reported, please check Resource Manager.")
false
} else {
- currentBlacklistedYarnNodes.size >= numClusterNodes
+ currentExcludededYarnNodes.size >= numClusterNodes
}
}
- private def refreshBlacklistedNodes(): Unit = {
- removeExpiredYarnBlacklistedNodes()
- val allBlacklistedNodes = excludeNodes ++ schedulerBlacklist ++ allocatorBlacklist.keySet
- synchronizeBlacklistedNodeWithYarn(allBlacklistedNodes)
+ private def refreshExcludedNodes(): Unit = {
+ removeExpiredYarnExcludedNodes()
+ val allExcludeddNodes =
Review comment:
minor nit: `allExcludeddNodes` -> `allExcludedNodes`
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala
##########
@@ -379,14 +379,15 @@ package object config extends Logging {
.stringConf
.createOptional
- /* YARN allocator-level blacklisting related config entries. */
- private[spark] val YARN_EXECUTOR_LAUNCH_BLACKLIST_ENABLED =
- ConfigBuilder("spark.yarn.blacklist.executor.launch.blacklisting.enabled")
+ /* YARN allocator-level excludeOnFailure related config entries. */
+ private[spark] val YARN_EXECUTOR_LAUNCH_EXCLUDE_ON_FAILURE_ENABLED =
+ ConfigBuilder("spark.yarn.executor.launch.excludeOnFailure.enabled")
Review comment:
I wonder if `spark.yarn.exclude.excludeOnFailure.enabled` or something might be better to bring it more in line with `spark.yarn.exclude.nodes`? I don't really see why "launch" appears in this config name...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701013285
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-702130902
I'm confused by why the build and test report test results says failed when everything passed. @dongjoon-hyun @HyukjinKwon have you seen that before, am I perhaps missing something?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705221120
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r499125900
##########
File path: core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala
##########
@@ -0,0 +1,478 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.atomic.AtomicReference
+
+import scala.collection.mutable.{ArrayBuffer, HashMap, HashSet}
+
+import org.apache.spark.{ExecutorAllocationClient, SparkConf, SparkContext}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config
+import org.apache.spark.util.{Clock, SystemClock, Utils}
+
+/**
+ * HealthTracker is designed to track problematic executors and nodes. It supports excluding
+ * executors and nodes across an entire application (with a periodic expiry). TaskSetManagers add
+ * additional logic for exclusion of executors and nodes for individual tasks and stages which
+ * works in concert with the logic here.
+ *
+ * The tracker needs to deal with a variety of workloads, eg.:
+ *
+ * * bad user code -- this may lead to many task failures, but that should not count against
+ * individual executors
+ * * many small stages -- this may prevent a bad executor for having many failures within one
+ * stage, but still many failures over the entire application
+ * * "flaky" executors -- they don't fail every task, but are still faulty enough to merit
+ * excluding
+ *
+ * See the design doc on SPARK-8425 for a more in-depth discussion.
+ *
+ * THREADING: As with most helpers of TaskSchedulerImpl, this is not thread-safe. Though it is
+ * called by multiple threads, callers must already have a lock on the TaskSchedulerImpl. The
+ * one exception is [[excludedNodeList()]], which can be called without holding a lock.
+ */
+private[scheduler] class HealthTracker (
+ private val listenerBus: LiveListenerBus,
+ conf: SparkConf,
+ allocationClient: Option[ExecutorAllocationClient],
+ clock: Clock = new SystemClock()) extends Logging {
+
+ def this(sc: SparkContext, allocationClient: Option[ExecutorAllocationClient]) = {
+ this(sc.listenerBus, sc.conf, allocationClient)
+ }
+
+ HealthTracker.validateExcludeOnFailureConfs(conf)
+ private val MAX_FAILURES_PER_EXEC = conf.get(config.MAX_FAILURES_PER_EXEC)
+ private val MAX_FAILED_EXEC_PER_NODE = conf.get(config.MAX_FAILED_EXEC_PER_NODE)
+ val EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS = HealthTracker.getExludeOnFailureTimeout(conf)
+ private val EXCLUDE_FETCH_FAILURE_ENABLED =
+ conf.get(config.EXCLUDE_ON_FAILURE_FETCH_FAILURE_ENABLED)
+
+ /**
+ * A map from executorId to information on task failures. Tracks the time of each task failure,
+ * so that we can avoid excluding executors due to failures that are very far apart. We do not
+ * actively remove from this as soon as tasks hit their timeouts, to avoid the time it would take
+ * to do so. But it will not grow too large, because as soon as an executor gets too many
+ * failures, we exclude the executor and remove its entry here.
+ */
+ private val executorIdToFailureList = new HashMap[String, ExecutorFailureList]()
+ val executorIdToExcludedStatus = new HashMap[String, ExcludedExecutor]()
+ val nodeIdToExcludedExpiryTime = new HashMap[String, Long]()
+ /**
+ * An immutable copy of the set of nodes that are currently excluded. Kept in an
+ * AtomicReference to make [[excludedNodeList()]] thread-safe.
+ */
+ private val _excludedNodeList = new AtomicReference[Set[String]](Set())
+ /**
+ * Time when the next excluded node will expire. Used as a shortcut to
+ * avoid iterating over all entries in the excludedNodeList when none will have expired.
+ */
+ var nextExpiryTime: Long = Long.MaxValue
+ /**
+ * Mapping from nodes to all of the executors that have been excluded on that node. We do *not*
+ * remove from this when executors are removed from spark, so we can track when we get multiple
+ * successive excluded executors on one node. Nonetheless, it will not grow too large because
+ * there cannot be many excluded executors on one node, before we stop requesting more
+ * executors on that node, and we clean up the list of exluded executors once an executor has
+ * been excluded for EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS.
+ */
+ val nodeToExcludedExecs = new HashMap[String, HashSet[String]]()
+
+ /**
+ * Include executors and nodes that have been excluded for at least
+ * EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS
+ */
+ def applyExcludeOnFailureTimeout(): Unit = {
+ val now = clock.getTimeMillis()
+ // quickly check if we've got anything to expire that is excluded -- if not,
+ // avoid doing any work
+ if (now > nextExpiryTime) {
+ // Apply the timeout to excluded nodes and executors
+ val execsToInclude = executorIdToExcludedStatus.filter(_._2.expiryTime < now).keys
+ if (execsToInclude.nonEmpty) {
+ // Include any executors that have been exluded longer than the excludeOnFailure timeout.
+ logInfo(s"Removing executors $execsToInclude from exclude list because the " +
+ s"the executors have reached the timed out")
+ execsToInclude.foreach { exec =>
+ val status = executorIdToExcludedStatus.remove(exec).get
+ val failedExecsOnNode = nodeToExcludedExecs(status.node)
+ listenerBus.post(SparkListenerExecutorUnexcluded(now, exec))
+ failedExecsOnNode.remove(exec)
+ if (failedExecsOnNode.isEmpty) {
+ nodeToExcludedExecs.remove(status.node)
+ }
+ }
+ }
+ val nodesToInclude = nodeIdToExcludedExpiryTime.filter(_._2 < now).keys
+ if (nodesToInclude.nonEmpty) {
+ // Include any nodes that have been excluded longer than the excludeOnFailure timeout.
+ logInfo(s"Removing nodes $nodesToInclude from exclude list because the " +
+ s"nodes have reached has timed out")
+ nodesToInclude.foreach { node =>
+ nodeIdToExcludedExpiryTime.remove(node)
+ listenerBus.post(SparkListenerNodeUnexcluded(now, node))
+ }
+ _excludedNodeList.set(nodeIdToExcludedExpiryTime.keySet.toSet)
+ }
+ updateNextExpiryTime()
+ }
+ }
+
+ private def updateNextExpiryTime(): Unit = {
+ val execMinExpiry = if (executorIdToExcludedStatus.nonEmpty) {
+ executorIdToExcludedStatus.map{_._2.expiryTime}.min
+ } else {
+ Long.MaxValue
+ }
+ val nodeMinExpiry = if (nodeIdToExcludedExpiryTime.nonEmpty) {
+ nodeIdToExcludedExpiryTime.values.min
+ } else {
+ Long.MaxValue
+ }
+ nextExpiryTime = math.min(execMinExpiry, nodeMinExpiry)
+ }
+
+ private def killExecutor(exec: String, msg: String): Unit = {
+ allocationClient match {
+ case Some(a) =>
+ logInfo(msg)
+ a.killExecutors(Seq(exec), adjustTargetNumExecutors = false, countFailures = false,
+ force = true)
+ case None =>
+ logInfo(s"Not attempting to kill excluded executor id $exec " +
+ s"since allocation client is not defined.")
+ }
+ }
+
+ private def killExcludedExecutor(exec: String): Unit = {
+ if (conf.get(config.EXCLUDE_ON_FAILURE_KILL_ENABLED)) {
+ killExecutor(exec, s"Killing excluded executor id $exec since " +
+ s"${config.EXCLUDE_ON_FAILURE_KILL_ENABLED.key} is set.")
+ }
+ }
+
+ private[scheduler] def killExcludedIdleExecutor(exec: String): Unit = {
+ killExecutor(exec,
+ s"Killing excluded idle executor id $exec because of task unschedulability and trying " +
+ "to acquire a new executor.")
+ }
+
+ private def killExecutorsOnExcludedNode(node: String): Unit = {
+ if (conf.get(config.EXCLUDE_ON_FAILURE_KILL_ENABLED)) {
+ allocationClient match {
+ case Some(a) =>
+ logInfo(s"Killing all executors on excluded host $node " +
+ s"since ${config.EXCLUDE_ON_FAILURE_KILL_ENABLED.key} is set.")
+ if (a.killExecutorsOnHost(node) == false) {
+ logError(s"Killing executors on node $node failed.")
+ }
+ case None =>
+ logWarning(s"Not attempting to kill executors on excluded host $node " +
+ s"since allocation client is not defined.")
+ }
+ }
+ }
+
+ def updateExcludedForFetchFailure(host: String, exec: String): Unit = {
+ if (EXCLUDE_FETCH_FAILURE_ENABLED) {
+ // If we exclude on fetch failures, we are implicitly saying that we believe the failure is
+ // non-transient, and can't be recovered from (even if this is the first fetch failure,
+ // stage is retried after just one failure, so we don't always get a chance to collect
+ // multiple fetch failures).
+ // If the external shuffle-service is on, then every other executor on this node would
+ // be suffering from the same issue, so we should exclude (and potentially kill) all
+ // of them immediately.
+
+ val now = clock.getTimeMillis()
+ val expiryTimeForNewExcludes = now + EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS
+
+ if (conf.get(config.SHUFFLE_SERVICE_ENABLED)) {
+ if (!nodeIdToExcludedExpiryTime.contains(host)) {
+ logInfo(s"excluding node $host due to fetch failure of external shuffle service")
+
+ nodeIdToExcludedExpiryTime.put(host, expiryTimeForNewExcludes)
+ listenerBus.post(SparkListenerNodeExcluded(now, host, 1))
+ _excludedNodeList.set(nodeIdToExcludedExpiryTime.keySet.toSet)
+ killExecutorsOnExcludedNode(host)
+ updateNextExpiryTime()
+ }
+ } else if (!executorIdToExcludedStatus.contains(exec)) {
+ logInfo(s"Excluding executor $exec due to fetch failure")
+
+ executorIdToExcludedStatus.put(exec, ExcludedExecutor(host, expiryTimeForNewExcludes))
+ // We hardcoded number of failure tasks to 1 for fetch failure, because there's no
+ // reattempt for such failure.
+ listenerBus.post(SparkListenerExecutorExcluded(now, exec, 1))
Review comment:
> 1. SparkFirehoseListener is public...
I feel like this a historical problem... The only way to add a custom listener is by using `sc.addSparkListener(listener: SparkListenerInterface)`. And `SparkListenerInterface` is decorated by `private[spark]`. So, it's impossible to keep backward compatibility of `SparkFirehoseListener` when `SparkListenerInterface` changes APIs. Therefore, it's meaningless to say that `SparkFirehoseListener` is public without considering `SparkListenerInterface`.
BTW, `SparkFirehoseListener` is actually a Java version of `SparkListener`. I think it should be developer API too but somehow we miss it.
> 2. for backwards compatibility of the history server reading the old event logs containing those events.
This's a good point. Then, I think we can remove the interface `onExecutorBlacklisted(executorBlacklisted: SparkListenerExecutorBlacklisted)` but can't remove the event `SparkListenerExecutorBlacklisted`. And for history server, I think we can override `onOtherEvent` to handle old `SparkListenerExecutorBlacklisted` specifically in order to keep backward compatibility.
> So really the only thing I got rid of is what you pointed out here with us posting it on new applications. I'm fine either way if we think its going to break a bunch of people we can post 2 messages. I think I would rather do that then have the default listener delegate just in case something weird happens there.
My original point is that it's meaningless to keep those APIs while we don't post `SparkListenerExecutorBlacklisted` anymore. We should either remove all things completely or still post old events with those APIs kept. I'm fine either way but just not stay in the middle. Though, I personally prefer to remove those APIs as mentioned above(Spark has no backward compatibility guarantee for the listener).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-712476056
**[Test build #130016 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130016/testReport)** for PR 29906 at commit [`b344ef6`](https://github.com/apache/spark/commit/b344ef6a9213ac8ae0c1bd059976bfa1b58d9cde).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701737070
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r500625642
##########
File path: core/src/main/scala/org/apache/spark/status/api/v1/api.scala
##########
@@ -82,10 +82,11 @@ class ExecutorStageSummary private[spark](
val shuffleWriteRecords : Long,
val memoryBytesSpilled : Long,
val diskBytesSpilled : Long,
- val isBlacklistedForStage: Boolean,
+ val isBlacklistedForStage: Boolean, // deprecated
Review comment:
Even if we can't remove it, we can at least mark it as deprecated so that consumers are aware that they're encouraged to migrate to the new field, right? Is your concern that a deprecation without plans for removal is pointless?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705643398
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34163/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701450819
**[Test build #129279 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129279/testReport)** for PR 29906 at commit [`76714e7`](https://github.com/apache/spark/commit/76714e7ddf33cdf2058ecac2f1c6bb915fc008fb).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701011427
**[Test build #129254 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129254/testReport)** for PR 29906 at commit [`6657785`](https://github.com/apache/spark/commit/6657785b249241232780117b4b10966a5d08aab0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-715425828
**[Test build #130210 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130210/testReport)** for PR 29906 at commit [`32ab73d`](https://github.com/apache/spark/commit/32ab73da672ad62192f811558961660fbc972c37).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-702412505
I wonder if its because I tried to cancel the workflow. I'll try rekicking once more
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r500528874
##########
File path: core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala
##########
@@ -0,0 +1,478 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.atomic.AtomicReference
+
+import scala.collection.mutable.{ArrayBuffer, HashMap, HashSet}
+
+import org.apache.spark.{ExecutorAllocationClient, SparkConf, SparkContext}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config
+import org.apache.spark.util.{Clock, SystemClock, Utils}
+
+/**
+ * HealthTracker is designed to track problematic executors and nodes. It supports excluding
+ * executors and nodes across an entire application (with a periodic expiry). TaskSetManagers add
+ * additional logic for exclusion of executors and nodes for individual tasks and stages which
+ * works in concert with the logic here.
+ *
+ * The tracker needs to deal with a variety of workloads, eg.:
+ *
+ * * bad user code -- this may lead to many task failures, but that should not count against
+ * individual executors
+ * * many small stages -- this may prevent a bad executor for having many failures within one
+ * stage, but still many failures over the entire application
+ * * "flaky" executors -- they don't fail every task, but are still faulty enough to merit
+ * excluding
+ *
+ * See the design doc on SPARK-8425 for a more in-depth discussion.
Review comment:
sounds good
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705872047
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-715656248
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34820/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-718041030
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r501269193
##########
File path: core/src/main/scala/org/apache/spark/status/api/v1/api.scala
##########
@@ -82,10 +82,11 @@ class ExecutorStageSummary private[spark](
val shuffleWriteRecords : Long,
val memoryBytesSpilled : Long,
val diskBytesSpilled : Long,
- val isBlacklistedForStage: Boolean,
+ val isBlacklistedForStage: Boolean, // deprecated
Review comment:
oh I see, yeah I think we can do that
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-712476650
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-715656257
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701463057
looks like some new files added, I'll upmerge to latest
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705900786
**[Test build #129565 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129565/testReport)** for PR 29906 at commit [`3ebf881`](https://github.com/apache/spark/commit/3ebf8814299fb96f52df479a46d1be28d453a3ce).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-715462586
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34810/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705236114
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34138/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r505997016
##########
File path: core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala
##########
@@ -431,23 +481,23 @@ abstract class SparkListener extends SparkListenerInterface {
override def onExecutorRemoved(executorRemoved: SparkListenerExecutorRemoved): Unit = { }
- override def onExecutorBlacklisted(
- executorBlacklisted: SparkListenerExecutorBlacklisted): Unit = { }
+ override def onExecutorExcluded(
+ executorExcluded: SparkListenerExecutorExcluded): Unit = { }
- def onExecutorBlacklistedForStage(
- executorBlacklistedForStage: SparkListenerExecutorBlacklistedForStage): Unit = { }
+ override def onExecutorExcludedForStage(
+ executorExcludedForStage: SparkListenerExecutorExcludedForStage): Unit = { }
- def onNodeBlacklistedForStage(
- nodeBlacklistedForStage: SparkListenerNodeBlacklistedForStage): Unit = { }
+ override def onNodeExcludedForStage(
+ nodeExcludedForStage: SparkListenerNodeExcludedForStage): Unit = { }
- override def onExecutorUnblacklisted(
- executorUnblacklisted: SparkListenerExecutorUnblacklisted): Unit = { }
+ override def onExecutorUnexcluded(
+ executorUnexcluded: SparkListenerExecutorUnexcluded): Unit = { }
- override def onNodeBlacklisted(
- nodeBlacklisted: SparkListenerNodeBlacklisted): Unit = { }
+ override def onNodeExcluded(
+ nodeExcluded: SparkListenerNodeExcluded): Unit = { }
- override def onNodeUnblacklisted(
- nodeUnblacklisted: SparkListenerNodeUnblacklisted): Unit = { }
+ override def onNodeUnexcluded(
+ nodeUnexcluded: SparkListenerNodeUnexcluded): Unit = { }
Review comment:
Also catching up, but, yes please merely deprecate things not remove them as part of this, where they could possibly be called or extended by apps
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r509853462
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -284,80 +284,127 @@ private[spark] class AppStatusListener(
}
override def onExecutorBlacklisted(event: SparkListenerExecutorBlacklisted): Unit = {
- updateBlackListStatus(event.executorId, true)
+ updateExclusionStatus(event.executorId, true)
+ }
+
+ override def onExecutorExcluded(event: SparkListenerExecutorExcluded): Unit = {
+ updateExclusionStatus(event.executorId, true)
}
override def onExecutorBlacklistedForStage(
event: SparkListenerExecutorBlacklistedForStage): Unit = {
- val now = System.nanoTime()
+ updateExclusionStatusForStage(event.stageId, event.stageAttemptId, event.executorId)
+ }
- Option(liveStages.get((event.stageId, event.stageAttemptId))).foreach { stage =>
- setStageBlackListStatus(stage, now, event.executorId)
- }
- liveExecutors.get(event.executorId).foreach { exec =>
- addBlackListedStageTo(exec, event.stageId, now)
- }
+ override def onExecutorExcludedForStage(
+ event: SparkListenerExecutorExcludedForStage): Unit = {
+ updateExclusionStatusForStage(event.stageId, event.stageAttemptId, event.executorId)
}
override def onNodeBlacklistedForStage(event: SparkListenerNodeBlacklistedForStage): Unit = {
- val now = System.nanoTime()
+ updateNodeExclusionStatusForStage(event.stageId, event.stageAttemptId, event.hostId)
+ }
- // Implicitly blacklist every available executor for the stage associated with this node
- Option(liveStages.get((event.stageId, event.stageAttemptId))).foreach { stage =>
- val executorIds = liveExecutors.values.filter(_.host == event.hostId).map(_.executorId).toSeq
- setStageBlackListStatus(stage, now, executorIds: _*)
- }
- liveExecutors.values.filter(_.hostname == event.hostId).foreach { exec =>
- addBlackListedStageTo(exec, event.stageId, now)
- }
+ override def onNodeExcludedForStage(event: SparkListenerNodeExcludedForStage): Unit = {
+ updateNodeExclusionStatusForStage(event.stageId, event.stageAttemptId, event.hostId)
}
- private def addBlackListedStageTo(exec: LiveExecutor, stageId: Int, now: Long): Unit = {
- exec.blacklistedInStages += stageId
+ private def addExcludedStageTo(exec: LiveExecutor, stageId: Int, now: Long): Unit = {
+ exec.excludedInStages += stageId
liveUpdate(exec, now)
}
private def setStageBlackListStatus(stage: LiveStage, now: Long, executorIds: String*): Unit = {
executorIds.foreach { executorId =>
val executorStageSummary = stage.executorSummary(executorId)
- executorStageSummary.isBlacklisted = true
+ executorStageSummary.isExcluded = true
+ maybeUpdate(executorStageSummary, now)
+ }
+ stage.excludedExecutors ++= executorIds
+ maybeUpdate(stage, now)
+ }
+
+ private def setStageExcludedStatus(stage: LiveStage, now: Long, executorIds: String*): Unit = {
+ executorIds.foreach { executorId =>
+ val executorStageSummary = stage.executorSummary(executorId)
+ executorStageSummary.isExcluded = true
maybeUpdate(executorStageSummary, now)
}
- stage.blackListedExecutors ++= executorIds
+ stage.excludedExecutors ++= executorIds
maybeUpdate(stage, now)
}
override def onExecutorUnblacklisted(event: SparkListenerExecutorUnblacklisted): Unit = {
- updateBlackListStatus(event.executorId, false)
+ updateExclusionStatus(event.executorId, false)
+ }
+
+ override def onExecutorUnexcluded(event: SparkListenerExecutorUnexcluded): Unit = {
+ updateExclusionStatus(event.executorId, false)
}
override def onNodeBlacklisted(event: SparkListenerNodeBlacklisted): Unit = {
Review comment:
IIUC, this is for new HS(history server) + old App compatibility? When a new HS reads old blacklist events for the old App, it can thus call these deprecated methods.
For old HS + new App case, I think it's fine since we always post new and old blacklist events, and thus EventLoggingListers writes two blacklist events as well. Therefore, old HS can parse the old blocklist events and ignore the new one.
I'm fine to leave them as they are. Maybe, add one more comment to explain why we still need it?
Another thought is, we can remove all those deprecated methods both in `AppStatusListener` and `EventLoggingListers` but keep compatibility by handling it in `onOtherEvent`. It's up to you. I'm fine either way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-717438995
@ngone51 any other comments?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r500527732
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -722,74 +722,83 @@ package object config {
.booleanConf
.createWithDefault(true)
- // Blacklist confs
- private[spark] val BLACKLIST_ENABLED =
- ConfigBuilder("spark.blacklist.enabled")
+ private[spark] val EXCLUDE_ON_FAILURE_ENABLED =
+ ConfigBuilder("spark.excludeOnFailure.enabled")
.version("2.1.0")
Review comment:
that is a good question, I left it because the config was just renamed but same functionality since 2.1.0. Also one of the other PRs for renaming slave we left this as well.
If others have thoughts or we have some standard on this let me know.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r498366360
##########
File path: core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala
##########
@@ -0,0 +1,478 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.atomic.AtomicReference
+
+import scala.collection.mutable.{ArrayBuffer, HashMap, HashSet}
+
+import org.apache.spark.{ExecutorAllocationClient, SparkConf, SparkContext}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config
+import org.apache.spark.util.{Clock, SystemClock, Utils}
+
+/**
+ * HealthTracker is designed to track problematic executors and nodes. It supports excluding
+ * executors and nodes across an entire application (with a periodic expiry). TaskSetManagers add
+ * additional logic for exclusion of executors and nodes for individual tasks and stages which
+ * works in concert with the logic here.
+ *
+ * The tracker needs to deal with a variety of workloads, eg.:
+ *
+ * * bad user code -- this may lead to many task failures, but that should not count against
+ * individual executors
+ * * many small stages -- this may prevent a bad executor for having many failures within one
+ * stage, but still many failures over the entire application
+ * * "flaky" executors -- they don't fail every task, but are still faulty enough to merit
+ * excluding
+ *
+ * See the design doc on SPARK-8425 for a more in-depth discussion.
+ *
+ * THREADING: As with most helpers of TaskSchedulerImpl, this is not thread-safe. Though it is
+ * called by multiple threads, callers must already have a lock on the TaskSchedulerImpl. The
+ * one exception is [[excludedNodeList()]], which can be called without holding a lock.
+ */
+private[scheduler] class HealthTracker (
+ private val listenerBus: LiveListenerBus,
+ conf: SparkConf,
+ allocationClient: Option[ExecutorAllocationClient],
+ clock: Clock = new SystemClock()) extends Logging {
+
+ def this(sc: SparkContext, allocationClient: Option[ExecutorAllocationClient]) = {
+ this(sc.listenerBus, sc.conf, allocationClient)
+ }
+
+ HealthTracker.validateExcludeOnFailureConfs(conf)
+ private val MAX_FAILURES_PER_EXEC = conf.get(config.MAX_FAILURES_PER_EXEC)
+ private val MAX_FAILED_EXEC_PER_NODE = conf.get(config.MAX_FAILED_EXEC_PER_NODE)
+ val EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS = HealthTracker.getExludeOnFailureTimeout(conf)
+ private val EXCLUDE_FETCH_FAILURE_ENABLED =
+ conf.get(config.EXCLUDE_ON_FAILURE_FETCH_FAILURE_ENABLED)
+
+ /**
+ * A map from executorId to information on task failures. Tracks the time of each task failure,
+ * so that we can avoid excluding executors due to failures that are very far apart. We do not
+ * actively remove from this as soon as tasks hit their timeouts, to avoid the time it would take
+ * to do so. But it will not grow too large, because as soon as an executor gets too many
+ * failures, we exclude the executor and remove its entry here.
+ */
+ private val executorIdToFailureList = new HashMap[String, ExecutorFailureList]()
+ val executorIdToExcludedStatus = new HashMap[String, ExcludedExecutor]()
+ val nodeIdToExcludedExpiryTime = new HashMap[String, Long]()
+ /**
+ * An immutable copy of the set of nodes that are currently excluded. Kept in an
+ * AtomicReference to make [[excludedNodeList()]] thread-safe.
+ */
+ private val _excludedNodeList = new AtomicReference[Set[String]](Set())
+ /**
+ * Time when the next excluded node will expire. Used as a shortcut to
+ * avoid iterating over all entries in the excludedNodeList when none will have expired.
+ */
+ var nextExpiryTime: Long = Long.MaxValue
+ /**
+ * Mapping from nodes to all of the executors that have been excluded on that node. We do *not*
+ * remove from this when executors are removed from spark, so we can track when we get multiple
+ * successive excluded executors on one node. Nonetheless, it will not grow too large because
+ * there cannot be many excluded executors on one node, before we stop requesting more
+ * executors on that node, and we clean up the list of exluded executors once an executor has
+ * been excluded for EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS.
+ */
+ val nodeToExcludedExecs = new HashMap[String, HashSet[String]]()
+
+ /**
+ * Include executors and nodes that have been excluded for at least
+ * EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS
+ */
+ def applyExcludeOnFailureTimeout(): Unit = {
+ val now = clock.getTimeMillis()
+ // quickly check if we've got anything to expire that is excluded -- if not,
+ // avoid doing any work
+ if (now > nextExpiryTime) {
+ // Apply the timeout to excluded nodes and executors
+ val execsToInclude = executorIdToExcludedStatus.filter(_._2.expiryTime < now).keys
+ if (execsToInclude.nonEmpty) {
+ // Include any executors that have been exluded longer than the excludeOnFailure timeout.
+ logInfo(s"Removing executors $execsToInclude from exclude list because the " +
+ s"the executors have reached the timed out")
+ execsToInclude.foreach { exec =>
+ val status = executorIdToExcludedStatus.remove(exec).get
+ val failedExecsOnNode = nodeToExcludedExecs(status.node)
+ listenerBus.post(SparkListenerExecutorUnexcluded(now, exec))
+ failedExecsOnNode.remove(exec)
+ if (failedExecsOnNode.isEmpty) {
+ nodeToExcludedExecs.remove(status.node)
+ }
+ }
+ }
+ val nodesToInclude = nodeIdToExcludedExpiryTime.filter(_._2 < now).keys
+ if (nodesToInclude.nonEmpty) {
+ // Include any nodes that have been excluded longer than the excludeOnFailure timeout.
+ logInfo(s"Removing nodes $nodesToInclude from exclude list because the " +
+ s"nodes have reached has timed out")
+ nodesToInclude.foreach { node =>
+ nodeIdToExcludedExpiryTime.remove(node)
+ listenerBus.post(SparkListenerNodeUnexcluded(now, node))
+ }
+ _excludedNodeList.set(nodeIdToExcludedExpiryTime.keySet.toSet)
+ }
+ updateNextExpiryTime()
+ }
+ }
+
+ private def updateNextExpiryTime(): Unit = {
+ val execMinExpiry = if (executorIdToExcludedStatus.nonEmpty) {
+ executorIdToExcludedStatus.map{_._2.expiryTime}.min
+ } else {
+ Long.MaxValue
+ }
+ val nodeMinExpiry = if (nodeIdToExcludedExpiryTime.nonEmpty) {
+ nodeIdToExcludedExpiryTime.values.min
+ } else {
+ Long.MaxValue
+ }
+ nextExpiryTime = math.min(execMinExpiry, nodeMinExpiry)
+ }
+
+ private def killExecutor(exec: String, msg: String): Unit = {
+ allocationClient match {
+ case Some(a) =>
+ logInfo(msg)
+ a.killExecutors(Seq(exec), adjustTargetNumExecutors = false, countFailures = false,
+ force = true)
+ case None =>
+ logInfo(s"Not attempting to kill excluded executor id $exec " +
+ s"since allocation client is not defined.")
+ }
+ }
+
+ private def killExcludedExecutor(exec: String): Unit = {
+ if (conf.get(config.EXCLUDE_ON_FAILURE_KILL_ENABLED)) {
+ killExecutor(exec, s"Killing excluded executor id $exec since " +
+ s"${config.EXCLUDE_ON_FAILURE_KILL_ENABLED.key} is set.")
+ }
+ }
+
+ private[scheduler] def killExcludedIdleExecutor(exec: String): Unit = {
+ killExecutor(exec,
+ s"Killing excluded idle executor id $exec because of task unschedulability and trying " +
+ "to acquire a new executor.")
+ }
+
+ private def killExecutorsOnExcludedNode(node: String): Unit = {
+ if (conf.get(config.EXCLUDE_ON_FAILURE_KILL_ENABLED)) {
+ allocationClient match {
+ case Some(a) =>
+ logInfo(s"Killing all executors on excluded host $node " +
+ s"since ${config.EXCLUDE_ON_FAILURE_KILL_ENABLED.key} is set.")
+ if (a.killExecutorsOnHost(node) == false) {
+ logError(s"Killing executors on node $node failed.")
+ }
+ case None =>
+ logWarning(s"Not attempting to kill executors on excluded host $node " +
+ s"since allocation client is not defined.")
+ }
+ }
+ }
+
+ def updateExcludedForFetchFailure(host: String, exec: String): Unit = {
+ if (EXCLUDE_FETCH_FAILURE_ENABLED) {
+ // If we exclude on fetch failures, we are implicitly saying that we believe the failure is
+ // non-transient, and can't be recovered from (even if this is the first fetch failure,
+ // stage is retried after just one failure, so we don't always get a chance to collect
+ // multiple fetch failures).
+ // If the external shuffle-service is on, then every other executor on this node would
+ // be suffering from the same issue, so we should exclude (and potentially kill) all
+ // of them immediately.
+
+ val now = clock.getTimeMillis()
+ val expiryTimeForNewExcludes = now + EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS
+
+ if (conf.get(config.SHUFFLE_SERVICE_ENABLED)) {
+ if (!nodeIdToExcludedExpiryTime.contains(host)) {
+ logInfo(s"excluding node $host due to fetch failure of external shuffle service")
+
+ nodeIdToExcludedExpiryTime.put(host, expiryTimeForNewExcludes)
+ listenerBus.post(SparkListenerNodeExcluded(now, host, 1))
+ _excludedNodeList.set(nodeIdToExcludedExpiryTime.keySet.toSet)
+ killExecutorsOnExcludedNode(host)
+ updateNextExpiryTime()
+ }
+ } else if (!executorIdToExcludedStatus.contains(exec)) {
+ logInfo(s"Excluding executor $exec due to fetch failure")
+
+ executorIdToExcludedStatus.put(exec, ExcludedExecutor(host, expiryTimeForNewExcludes))
+ // We hardcoded number of failure tasks to 1 for fetch failure, because there's no
+ // reattempt for such failure.
+ listenerBus.post(SparkListenerExecutorExcluded(now, exec, 1))
Review comment:
depends on what is looking for it. Nothing in spark is looking for it anymore. If others are listening for it and don't change then yes. This is a developer api though. I kind of hated to have to post 2 separate messages which is why I removed it.
Do you know of people actively relying on those listeners?
The only thing internal to Spark that it breaks is that older history servers don't see blacklisted on the new history files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705259321
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34140/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-712412016
**[Test build #130016 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130016/testReport)** for PR 29906 at commit [`b344ef6`](https://github.com/apache/spark/commit/b344ef6a9213ac8ae0c1bd059976bfa1b58d9cde).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705220647
**[Test build #129533 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129533/testReport)** for PR 29906 at commit [`fcc3558`](https://github.com/apache/spark/commit/fcc3558bedbc62606d8432353e4e0027ed4418c9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r513444687
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -722,74 +722,83 @@ package object config {
.booleanConf
.createWithDefault(true)
- // Blacklist confs
- private[spark] val BLACKLIST_ENABLED =
- ConfigBuilder("spark.blacklist.enabled")
- .version("2.1.0")
+ private[spark] val EXCLUDE_ON_FAILURE_ENABLED =
+ ConfigBuilder("spark.excludeOnFailure.enabled")
+ .version("3.1.0")
+ .withAlternative("spark.blacklist.enabled")
.booleanConf
.createOptional
private[spark] val MAX_TASK_ATTEMPTS_PER_EXECUTOR =
- ConfigBuilder("spark.blacklist.task.maxTaskAttemptsPerExecutor")
- .version("2.1.0")
+ ConfigBuilder("spark.excludeOnFailure.task.maxTaskAttemptsPerExecutor")
+ .version("3.1.0")
+ .withAlternative("spark.blacklist.task.maxTaskAttemptsPerExecutor")
.intConf
.createWithDefault(1)
private[spark] val MAX_TASK_ATTEMPTS_PER_NODE =
- ConfigBuilder("spark.blacklist.task.maxTaskAttemptsPerNode")
- .version("2.1.0")
+ ConfigBuilder("spark.excludeOnFailure.task.maxTaskAttemptsPerNode")
+ .version("3.1.0")
+ .withAlternative("spark.blacklist.task.maxTaskAttemptsPerNode")
.intConf
.createWithDefault(2)
private[spark] val MAX_FAILURES_PER_EXEC =
- ConfigBuilder("spark.blacklist.application.maxFailedTasksPerExecutor")
- .version("2.2.0")
+ ConfigBuilder("spark.excludeOnFailure.application.maxFailedTasksPerExecutor")
+ .version("3.1.0")
+ .withAlternative("spark.blacklist.application.maxFailedTasksPerExecutor")
.intConf
.createWithDefault(2)
private[spark] val MAX_FAILURES_PER_EXEC_STAGE =
- ConfigBuilder("spark.blacklist.stage.maxFailedTasksPerExecutor")
- .version("2.1.0")
+ ConfigBuilder("spark.excludeOnFailure.stage.maxFailedTasksPerExecutor")
+ .version("3.1.0")
+ .withAlternative("spark.blacklist.stage.maxFailedTasksPerExecutor")
.intConf
.createWithDefault(2)
private[spark] val MAX_FAILED_EXEC_PER_NODE =
- ConfigBuilder("spark.blacklist.application.maxFailedExecutorsPerNode")
- .version("2.2.0")
+ ConfigBuilder("spark.excludeOnFailure.application.maxFailedExecutorsPerNode")
+ .version("3.1.0")
+ .withAlternative("spark.blacklist.application.maxFailedExecutorsPerNode")
.intConf
.createWithDefault(2)
private[spark] val MAX_FAILED_EXEC_PER_NODE_STAGE =
- ConfigBuilder("spark.blacklist.stage.maxFailedExecutorsPerNode")
- .version("2.1.0")
+ ConfigBuilder("spark.excludeOnFailure.stage.maxFailedExecutorsPerNode")
+ .version("3.1.0")
+ .withAlternative("spark.blacklist.stage.maxFailedExecutorsPerNode")
.intConf
.createWithDefault(2)
- private[spark] val BLACKLIST_TIMEOUT_CONF =
- ConfigBuilder("spark.blacklist.timeout")
- .version("2.1.0")
+ private[spark] val EXCLUDE_ON_FAILURE_TIMEOUT_CONF =
+ ConfigBuilder("spark.excludeOnFailure.timeout")
+ .version("3.1.0")
+ .withAlternative("spark.blacklist.timeout")
.timeConf(TimeUnit.MILLISECONDS)
.createOptional
- private[spark] val BLACKLIST_KILL_ENABLED =
- ConfigBuilder("spark.blacklist.killBlacklistedExecutors")
Review comment:
yes thanks for catching
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701570539
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-718041030
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701684785
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33905/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705700219
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34167/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705245039
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129535/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705221120
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-715664806
**[Test build #130220 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130220/testReport)** for PR 29906 at commit [`32ab73d`](https://github.com/apache/spark/commit/32ab73da672ad62192f811558961660fbc972c37).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705619969
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701446281
thanks @xkrogen I made another pass through and think I updated all the reference to blocklist. Another set of eyes would be great as it is a lot of code.
cc @holdenk @srowen
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705659379
oops, put those in the wrong spot. its weird when I ran locally the mima script hung and I guess it was syntax issue. fixing it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-717994880
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34974/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701446411
**[Test build #129279 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129279/testReport)** for PR 29906 at commit [`76714e7`](https://github.com/apache/spark/commit/76714e7ddf33cdf2058ecac2f1c6bb915fc008fb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r513545930
##########
File path: core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala
##########
@@ -25,7 +25,7 @@ import scala.collection.Map
import com.fasterxml.jackson.annotation.JsonTypeInfo
import org.apache.spark.TaskEndReason
-import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.annotation.{DeveloperApi, Since}
Review comment:
yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701011427
**[Test build #129254 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129254/testReport)** for PR 29906 at commit [`6657785`](https://github.com/apache/spark/commit/6657785b249241232780117b4b10966a5d08aab0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-712476654
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130016/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705851486
**[Test build #129565 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129565/testReport)** for PR 29906 at commit [`3ebf881`](https://github.com/apache/spark/commit/3ebf8814299fb96f52df479a46d1be28d453a3ce).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r508026560
##########
File path: core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala
##########
@@ -431,23 +481,23 @@ abstract class SparkListener extends SparkListenerInterface {
override def onExecutorRemoved(executorRemoved: SparkListenerExecutorRemoved): Unit = { }
- override def onExecutorBlacklisted(
- executorBlacklisted: SparkListenerExecutorBlacklisted): Unit = { }
+ override def onExecutorExcluded(
+ executorExcluded: SparkListenerExecutorExcluded): Unit = { }
- def onExecutorBlacklistedForStage(
- executorBlacklistedForStage: SparkListenerExecutorBlacklistedForStage): Unit = { }
+ override def onExecutorExcludedForStage(
+ executorExcludedForStage: SparkListenerExecutorExcludedForStage): Unit = { }
- def onNodeBlacklistedForStage(
- nodeBlacklistedForStage: SparkListenerNodeBlacklistedForStage): Unit = { }
+ override def onNodeExcludedForStage(
+ nodeExcludedForStage: SparkListenerNodeExcludedForStage): Unit = { }
- override def onExecutorUnblacklisted(
- executorUnblacklisted: SparkListenerExecutorUnblacklisted): Unit = { }
+ override def onExecutorUnexcluded(
+ executorUnexcluded: SparkListenerExecutorUnexcluded): Unit = { }
- override def onNodeBlacklisted(
- nodeBlacklisted: SparkListenerNodeBlacklisted): Unit = { }
+ override def onNodeExcluded(
+ nodeExcluded: SparkListenerNodeExcluded): Unit = { }
- override def onNodeUnblacklisted(
- nodeUnblacklisted: SparkListenerNodeUnblacklisted): Unit = { }
+ override def onNodeUnexcluded(
+ nodeUnexcluded: SparkListenerNodeUnexcluded): Unit = { }
Review comment:
Updated to put back
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-712431242
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34623/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r508525385
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -284,80 +284,127 @@ private[spark] class AppStatusListener(
}
override def onExecutorBlacklisted(event: SparkListenerExecutorBlacklisted): Unit = {
- updateBlackListStatus(event.executorId, true)
+ updateExclusionStatus(event.executorId, true)
+ }
+
+ override def onExecutorExcluded(event: SparkListenerExecutorExcluded): Unit = {
+ updateExclusionStatus(event.executorId, true)
}
override def onExecutorBlacklistedForStage(
event: SparkListenerExecutorBlacklistedForStage): Unit = {
- val now = System.nanoTime()
+ updateExclusionStatusForStage(event.stageId, event.stageAttemptId, event.executorId)
+ }
- Option(liveStages.get((event.stageId, event.stageAttemptId))).foreach { stage =>
- setStageBlackListStatus(stage, now, event.executorId)
- }
- liveExecutors.get(event.executorId).foreach { exec =>
- addBlackListedStageTo(exec, event.stageId, now)
- }
+ override def onExecutorExcludedForStage(
+ event: SparkListenerExecutorExcludedForStage): Unit = {
+ updateExclusionStatusForStage(event.stageId, event.stageAttemptId, event.executorId)
}
override def onNodeBlacklistedForStage(event: SparkListenerNodeBlacklistedForStage): Unit = {
- val now = System.nanoTime()
+ updateNodeExclusionStatusForStage(event.stageId, event.stageAttemptId, event.hostId)
+ }
- // Implicitly blacklist every available executor for the stage associated with this node
- Option(liveStages.get((event.stageId, event.stageAttemptId))).foreach { stage =>
- val executorIds = liveExecutors.values.filter(_.host == event.hostId).map(_.executorId).toSeq
- setStageBlackListStatus(stage, now, executorIds: _*)
- }
- liveExecutors.values.filter(_.hostname == event.hostId).foreach { exec =>
- addBlackListedStageTo(exec, event.stageId, now)
- }
+ override def onNodeExcludedForStage(event: SparkListenerNodeExcludedForStage): Unit = {
+ updateNodeExclusionStatusForStage(event.stageId, event.stageAttemptId, event.hostId)
}
- private def addBlackListedStageTo(exec: LiveExecutor, stageId: Int, now: Long): Unit = {
- exec.blacklistedInStages += stageId
+ private def addExcludedStageTo(exec: LiveExecutor, stageId: Int, now: Long): Unit = {
+ exec.excludedInStages += stageId
liveUpdate(exec, now)
}
private def setStageBlackListStatus(stage: LiveStage, now: Long, executorIds: String*): Unit = {
executorIds.foreach { executorId =>
val executorStageSummary = stage.executorSummary(executorId)
- executorStageSummary.isBlacklisted = true
+ executorStageSummary.isExcluded = true
+ maybeUpdate(executorStageSummary, now)
+ }
+ stage.excludedExecutors ++= executorIds
+ maybeUpdate(stage, now)
+ }
+
+ private def setStageExcludedStatus(stage: LiveStage, now: Long, executorIds: String*): Unit = {
+ executorIds.foreach { executorId =>
+ val executorStageSummary = stage.executorSummary(executorId)
+ executorStageSummary.isExcluded = true
maybeUpdate(executorStageSummary, now)
}
- stage.blackListedExecutors ++= executorIds
+ stage.excludedExecutors ++= executorIds
maybeUpdate(stage, now)
}
override def onExecutorUnblacklisted(event: SparkListenerExecutorUnblacklisted): Unit = {
- updateBlackListStatus(event.executorId, false)
+ updateExclusionStatus(event.executorId, false)
+ }
+
+ override def onExecutorUnexcluded(event: SparkListenerExecutorUnexcluded): Unit = {
+ updateExclusionStatus(event.executorId, false)
}
override def onNodeBlacklisted(event: SparkListenerNodeBlacklisted): Unit = {
Review comment:
I left them to be completely backwards compatible. This way they are still written to event logs which means older history servers should work with newer files and if people have parsers for those it still works. I don't think we say the event log if compatible in that way but this gives more compatibility.
I'm fine with removing them if everyone else thinks its ok.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r510167682
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -284,80 +284,127 @@ private[spark] class AppStatusListener(
}
override def onExecutorBlacklisted(event: SparkListenerExecutorBlacklisted): Unit = {
- updateBlackListStatus(event.executorId, true)
+ updateExclusionStatus(event.executorId, true)
+ }
+
+ override def onExecutorExcluded(event: SparkListenerExecutorExcluded): Unit = {
+ updateExclusionStatus(event.executorId, true)
}
override def onExecutorBlacklistedForStage(
event: SparkListenerExecutorBlacklistedForStage): Unit = {
- val now = System.nanoTime()
+ updateExclusionStatusForStage(event.stageId, event.stageAttemptId, event.executorId)
+ }
- Option(liveStages.get((event.stageId, event.stageAttemptId))).foreach { stage =>
- setStageBlackListStatus(stage, now, event.executorId)
- }
- liveExecutors.get(event.executorId).foreach { exec =>
- addBlackListedStageTo(exec, event.stageId, now)
- }
+ override def onExecutorExcludedForStage(
+ event: SparkListenerExecutorExcludedForStage): Unit = {
+ updateExclusionStatusForStage(event.stageId, event.stageAttemptId, event.executorId)
}
override def onNodeBlacklistedForStage(event: SparkListenerNodeBlacklistedForStage): Unit = {
- val now = System.nanoTime()
+ updateNodeExclusionStatusForStage(event.stageId, event.stageAttemptId, event.hostId)
+ }
- // Implicitly blacklist every available executor for the stage associated with this node
- Option(liveStages.get((event.stageId, event.stageAttemptId))).foreach { stage =>
- val executorIds = liveExecutors.values.filter(_.host == event.hostId).map(_.executorId).toSeq
- setStageBlackListStatus(stage, now, executorIds: _*)
- }
- liveExecutors.values.filter(_.hostname == event.hostId).foreach { exec =>
- addBlackListedStageTo(exec, event.stageId, now)
- }
+ override def onNodeExcludedForStage(event: SparkListenerNodeExcludedForStage): Unit = {
+ updateNodeExclusionStatusForStage(event.stageId, event.stageAttemptId, event.hostId)
}
- private def addBlackListedStageTo(exec: LiveExecutor, stageId: Int, now: Long): Unit = {
- exec.blacklistedInStages += stageId
+ private def addExcludedStageTo(exec: LiveExecutor, stageId: Int, now: Long): Unit = {
+ exec.excludedInStages += stageId
liveUpdate(exec, now)
}
private def setStageBlackListStatus(stage: LiveStage, now: Long, executorIds: String*): Unit = {
executorIds.foreach { executorId =>
val executorStageSummary = stage.executorSummary(executorId)
- executorStageSummary.isBlacklisted = true
+ executorStageSummary.isExcluded = true
+ maybeUpdate(executorStageSummary, now)
+ }
+ stage.excludedExecutors ++= executorIds
+ maybeUpdate(stage, now)
+ }
+
+ private def setStageExcludedStatus(stage: LiveStage, now: Long, executorIds: String*): Unit = {
+ executorIds.foreach { executorId =>
+ val executorStageSummary = stage.executorSummary(executorId)
+ executorStageSummary.isExcluded = true
maybeUpdate(executorStageSummary, now)
}
- stage.blackListedExecutors ++= executorIds
+ stage.excludedExecutors ++= executorIds
maybeUpdate(stage, now)
}
override def onExecutorUnblacklisted(event: SparkListenerExecutorUnblacklisted): Unit = {
- updateBlackListStatus(event.executorId, false)
+ updateExclusionStatus(event.executorId, false)
+ }
+
+ override def onExecutorUnexcluded(event: SparkListenerExecutorUnexcluded): Unit = {
+ updateExclusionStatus(event.executorId, false)
}
override def onNodeBlacklisted(event: SparkListenerNodeBlacklisted): Unit = {
Review comment:
yes its for new HS reading old app event log compatibility.
I prefer to leave as is this way. I can add a comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r497591649
##########
File path: resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala
##########
@@ -523,9 +523,9 @@ class YarnAllocatorSuite extends SparkFunSuite with Matchers with BeforeAndAfter
handler.getNumUnexpectedContainerRelease should be (2)
}
- test("blacklisted nodes reflected in amClient requests") {
- // Internally we track the set of blacklisted nodes, but yarn wants us to send *changes*
- // to the blacklist. This makes sure we are sending the right updates.
+ test("excludeOnFailure nodes reflected in amClient requests") {
+ // Internally we track the set of excluded nodes, but yarn wants us to send *changes*
+ // to it. This makes sure we are sending the right updates.
Review comment:
Makes sense -- I will take another look with this in mind. Thanks for explaining!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-715503391
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705851486
**[Test build #129565 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129565/testReport)** for PR 29906 at commit [`3ebf881`](https://github.com/apache/spark/commit/3ebf8814299fb96f52df479a46d1be28d453a3ce).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705725764
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705241921
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r497506544
##########
File path: resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala
##########
@@ -523,9 +523,9 @@ class YarnAllocatorSuite extends SparkFunSuite with Matchers with BeforeAndAfter
handler.getNumUnexpectedContainerRelease should be (2)
}
- test("blacklisted nodes reflected in amClient requests") {
- // Internally we track the set of blacklisted nodes, but yarn wants us to send *changes*
- // to the blacklist. This makes sure we are sending the right updates.
+ test("excludeOnFailure nodes reflected in amClient requests") {
+ // Internally we track the set of excluded nodes, but yarn wants us to send *changes*
+ // to it. This makes sure we are sending the right updates.
Review comment:
In this case actually this is what I wanted. We internally track what nodes are excluded. This is from the excludeOnFailure list from the scheduler, from the spark.yarn.exclude.nodes config and the allocator logic for excluding nodes that have failures.. Yes we use the yarn blacklist list api to actually specify it to yarn but I don't see any reason internally we have to keep our code using the term blacklist except where we call into their api. I could add a comment about that in there to clarify in the Allocator itself until they can update their api.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705215441
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129532/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-715503391
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701690395
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33905/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705259300
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34140/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705872051
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34171/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701013294
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33871/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705620977
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-702525542
> I'm confused by why the build and test report test results says failed when everything passed. @dongjoon-hyun @HyukjinKwon have you seen that before, am I perhaps missing something?
Yes, I think I have seen this few times. The failure error message was somewhat not very deterministic so I was kept monitoring it. Feel free to just screen shot or copy/paste the error message and leave it in the comment when you guys see it before retriggering. I will take a look deeper.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-715462608
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701667505
**[Test build #129289 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129289/testReport)** for PR 29906 at commit [`6da870b`](https://github.com/apache/spark/commit/6da870b4cd2f039345ddeb1792d6ddf0ccf19136).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r498402082
##########
File path: core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala
##########
@@ -0,0 +1,478 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.atomic.AtomicReference
+
+import scala.collection.mutable.{ArrayBuffer, HashMap, HashSet}
+
+import org.apache.spark.{ExecutorAllocationClient, SparkConf, SparkContext}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config
+import org.apache.spark.util.{Clock, SystemClock, Utils}
+
+/**
+ * HealthTracker is designed to track problematic executors and nodes. It supports excluding
+ * executors and nodes across an entire application (with a periodic expiry). TaskSetManagers add
+ * additional logic for exclusion of executors and nodes for individual tasks and stages which
+ * works in concert with the logic here.
+ *
+ * The tracker needs to deal with a variety of workloads, eg.:
+ *
+ * * bad user code -- this may lead to many task failures, but that should not count against
+ * individual executors
+ * * many small stages -- this may prevent a bad executor for having many failures within one
+ * stage, but still many failures over the entire application
+ * * "flaky" executors -- they don't fail every task, but are still faulty enough to merit
+ * excluding
+ *
+ * See the design doc on SPARK-8425 for a more in-depth discussion.
+ *
+ * THREADING: As with most helpers of TaskSchedulerImpl, this is not thread-safe. Though it is
+ * called by multiple threads, callers must already have a lock on the TaskSchedulerImpl. The
+ * one exception is [[excludedNodeList()]], which can be called without holding a lock.
+ */
+private[scheduler] class HealthTracker (
+ private val listenerBus: LiveListenerBus,
+ conf: SparkConf,
+ allocationClient: Option[ExecutorAllocationClient],
+ clock: Clock = new SystemClock()) extends Logging {
+
+ def this(sc: SparkContext, allocationClient: Option[ExecutorAllocationClient]) = {
+ this(sc.listenerBus, sc.conf, allocationClient)
+ }
+
+ HealthTracker.validateExcludeOnFailureConfs(conf)
+ private val MAX_FAILURES_PER_EXEC = conf.get(config.MAX_FAILURES_PER_EXEC)
+ private val MAX_FAILED_EXEC_PER_NODE = conf.get(config.MAX_FAILED_EXEC_PER_NODE)
+ val EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS = HealthTracker.getExludeOnFailureTimeout(conf)
+ private val EXCLUDE_FETCH_FAILURE_ENABLED =
+ conf.get(config.EXCLUDE_ON_FAILURE_FETCH_FAILURE_ENABLED)
+
+ /**
+ * A map from executorId to information on task failures. Tracks the time of each task failure,
+ * so that we can avoid excluding executors due to failures that are very far apart. We do not
+ * actively remove from this as soon as tasks hit their timeouts, to avoid the time it would take
+ * to do so. But it will not grow too large, because as soon as an executor gets too many
+ * failures, we exclude the executor and remove its entry here.
+ */
+ private val executorIdToFailureList = new HashMap[String, ExecutorFailureList]()
+ val executorIdToExcludedStatus = new HashMap[String, ExcludedExecutor]()
+ val nodeIdToExcludedExpiryTime = new HashMap[String, Long]()
+ /**
+ * An immutable copy of the set of nodes that are currently excluded. Kept in an
+ * AtomicReference to make [[excludedNodeList()]] thread-safe.
+ */
+ private val _excludedNodeList = new AtomicReference[Set[String]](Set())
+ /**
+ * Time when the next excluded node will expire. Used as a shortcut to
+ * avoid iterating over all entries in the excludedNodeList when none will have expired.
+ */
+ var nextExpiryTime: Long = Long.MaxValue
+ /**
+ * Mapping from nodes to all of the executors that have been excluded on that node. We do *not*
+ * remove from this when executors are removed from spark, so we can track when we get multiple
+ * successive excluded executors on one node. Nonetheless, it will not grow too large because
+ * there cannot be many excluded executors on one node, before we stop requesting more
+ * executors on that node, and we clean up the list of exluded executors once an executor has
+ * been excluded for EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS.
+ */
+ val nodeToExcludedExecs = new HashMap[String, HashSet[String]]()
+
+ /**
+ * Include executors and nodes that have been excluded for at least
+ * EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS
+ */
+ def applyExcludeOnFailureTimeout(): Unit = {
+ val now = clock.getTimeMillis()
+ // quickly check if we've got anything to expire that is excluded -- if not,
+ // avoid doing any work
+ if (now > nextExpiryTime) {
+ // Apply the timeout to excluded nodes and executors
+ val execsToInclude = executorIdToExcludedStatus.filter(_._2.expiryTime < now).keys
+ if (execsToInclude.nonEmpty) {
+ // Include any executors that have been exluded longer than the excludeOnFailure timeout.
+ logInfo(s"Removing executors $execsToInclude from exclude list because the " +
+ s"the executors have reached the timed out")
+ execsToInclude.foreach { exec =>
+ val status = executorIdToExcludedStatus.remove(exec).get
+ val failedExecsOnNode = nodeToExcludedExecs(status.node)
+ listenerBus.post(SparkListenerExecutorUnexcluded(now, exec))
+ failedExecsOnNode.remove(exec)
+ if (failedExecsOnNode.isEmpty) {
+ nodeToExcludedExecs.remove(status.node)
+ }
+ }
+ }
+ val nodesToInclude = nodeIdToExcludedExpiryTime.filter(_._2 < now).keys
+ if (nodesToInclude.nonEmpty) {
+ // Include any nodes that have been excluded longer than the excludeOnFailure timeout.
+ logInfo(s"Removing nodes $nodesToInclude from exclude list because the " +
+ s"nodes have reached has timed out")
+ nodesToInclude.foreach { node =>
+ nodeIdToExcludedExpiryTime.remove(node)
+ listenerBus.post(SparkListenerNodeUnexcluded(now, node))
+ }
+ _excludedNodeList.set(nodeIdToExcludedExpiryTime.keySet.toSet)
+ }
+ updateNextExpiryTime()
+ }
+ }
+
+ private def updateNextExpiryTime(): Unit = {
+ val execMinExpiry = if (executorIdToExcludedStatus.nonEmpty) {
+ executorIdToExcludedStatus.map{_._2.expiryTime}.min
+ } else {
+ Long.MaxValue
+ }
+ val nodeMinExpiry = if (nodeIdToExcludedExpiryTime.nonEmpty) {
+ nodeIdToExcludedExpiryTime.values.min
+ } else {
+ Long.MaxValue
+ }
+ nextExpiryTime = math.min(execMinExpiry, nodeMinExpiry)
+ }
+
+ private def killExecutor(exec: String, msg: String): Unit = {
+ allocationClient match {
+ case Some(a) =>
+ logInfo(msg)
+ a.killExecutors(Seq(exec), adjustTargetNumExecutors = false, countFailures = false,
+ force = true)
+ case None =>
+ logInfo(s"Not attempting to kill excluded executor id $exec " +
+ s"since allocation client is not defined.")
+ }
+ }
+
+ private def killExcludedExecutor(exec: String): Unit = {
+ if (conf.get(config.EXCLUDE_ON_FAILURE_KILL_ENABLED)) {
+ killExecutor(exec, s"Killing excluded executor id $exec since " +
+ s"${config.EXCLUDE_ON_FAILURE_KILL_ENABLED.key} is set.")
+ }
+ }
+
+ private[scheduler] def killExcludedIdleExecutor(exec: String): Unit = {
+ killExecutor(exec,
+ s"Killing excluded idle executor id $exec because of task unschedulability and trying " +
+ "to acquire a new executor.")
+ }
+
+ private def killExecutorsOnExcludedNode(node: String): Unit = {
+ if (conf.get(config.EXCLUDE_ON_FAILURE_KILL_ENABLED)) {
+ allocationClient match {
+ case Some(a) =>
+ logInfo(s"Killing all executors on excluded host $node " +
+ s"since ${config.EXCLUDE_ON_FAILURE_KILL_ENABLED.key} is set.")
+ if (a.killExecutorsOnHost(node) == false) {
+ logError(s"Killing executors on node $node failed.")
+ }
+ case None =>
+ logWarning(s"Not attempting to kill executors on excluded host $node " +
+ s"since allocation client is not defined.")
+ }
+ }
+ }
+
+ def updateExcludedForFetchFailure(host: String, exec: String): Unit = {
+ if (EXCLUDE_FETCH_FAILURE_ENABLED) {
+ // If we exclude on fetch failures, we are implicitly saying that we believe the failure is
+ // non-transient, and can't be recovered from (even if this is the first fetch failure,
+ // stage is retried after just one failure, so we don't always get a chance to collect
+ // multiple fetch failures).
+ // If the external shuffle-service is on, then every other executor on this node would
+ // be suffering from the same issue, so we should exclude (and potentially kill) all
+ // of them immediately.
+
+ val now = clock.getTimeMillis()
+ val expiryTimeForNewExcludes = now + EXCLUDE_ON_FAILURE_TIMEOUT_MILLIS
+
+ if (conf.get(config.SHUFFLE_SERVICE_ENABLED)) {
+ if (!nodeIdToExcludedExpiryTime.contains(host)) {
+ logInfo(s"excluding node $host due to fetch failure of external shuffle service")
+
+ nodeIdToExcludedExpiryTime.put(host, expiryTimeForNewExcludes)
+ listenerBus.post(SparkListenerNodeExcluded(now, host, 1))
+ _excludedNodeList.set(nodeIdToExcludedExpiryTime.keySet.toSet)
+ killExecutorsOnExcludedNode(host)
+ updateNextExpiryTime()
+ }
+ } else if (!executorIdToExcludedStatus.contains(exec)) {
+ logInfo(s"Excluding executor $exec due to fetch failure")
+
+ executorIdToExcludedStatus.put(exec, ExcludedExecutor(host, expiryTimeForNewExcludes))
+ // We hardcoded number of failure tasks to 1 for fetch failure, because there's no
+ // reattempt for such failure.
+ listenerBus.post(SparkListenerExecutorExcluded(now, exec, 1))
Review comment:
Yeah, I'm talking about custom listeners. I don't exactly know how people rely on it but I guess it won't be used a lot.
Maybe, we can just remove it for simplicity as `SparkListener` (developer API) and `SparkListenerInterface` ( private spark) don't guarantee the backward compatibility?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701666882
Jenkins, test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r500556152
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -722,74 +722,83 @@ package object config {
.booleanConf
.createWithDefault(true)
- // Blacklist confs
- private[spark] val BLACKLIST_ENABLED =
- ConfigBuilder("spark.blacklist.enabled")
+ private[spark] val EXCLUDE_ON_FAILURE_ENABLED =
+ ConfigBuilder("spark.excludeOnFailure.enabled")
.version("2.1.0")
Review comment:
I guess it depends if you consider the "from version" to represent when the config string was added, or when the functionality was added. I think the former makes more sense, but I certainly wouldn't know if there was an established convention around this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r510159490
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -284,80 +284,127 @@ private[spark] class AppStatusListener(
}
override def onExecutorBlacklisted(event: SparkListenerExecutorBlacklisted): Unit = {
- updateBlackListStatus(event.executorId, true)
+ updateExclusionStatus(event.executorId, true)
+ }
+
+ override def onExecutorExcluded(event: SparkListenerExecutorExcluded): Unit = {
+ updateExclusionStatus(event.executorId, true)
}
override def onExecutorBlacklistedForStage(
event: SparkListenerExecutorBlacklistedForStage): Unit = {
- val now = System.nanoTime()
+ updateExclusionStatusForStage(event.stageId, event.stageAttemptId, event.executorId)
+ }
- Option(liveStages.get((event.stageId, event.stageAttemptId))).foreach { stage =>
- setStageBlackListStatus(stage, now, event.executorId)
- }
- liveExecutors.get(event.executorId).foreach { exec =>
- addBlackListedStageTo(exec, event.stageId, now)
- }
+ override def onExecutorExcludedForStage(
+ event: SparkListenerExecutorExcludedForStage): Unit = {
+ updateExclusionStatusForStage(event.stageId, event.stageAttemptId, event.executorId)
}
override def onNodeBlacklistedForStage(event: SparkListenerNodeBlacklistedForStage): Unit = {
- val now = System.nanoTime()
+ updateNodeExclusionStatusForStage(event.stageId, event.stageAttemptId, event.hostId)
+ }
- // Implicitly blacklist every available executor for the stage associated with this node
- Option(liveStages.get((event.stageId, event.stageAttemptId))).foreach { stage =>
- val executorIds = liveExecutors.values.filter(_.host == event.hostId).map(_.executorId).toSeq
- setStageBlackListStatus(stage, now, executorIds: _*)
- }
- liveExecutors.values.filter(_.hostname == event.hostId).foreach { exec =>
- addBlackListedStageTo(exec, event.stageId, now)
- }
+ override def onNodeExcludedForStage(event: SparkListenerNodeExcludedForStage): Unit = {
+ updateNodeExclusionStatusForStage(event.stageId, event.stageAttemptId, event.hostId)
}
- private def addBlackListedStageTo(exec: LiveExecutor, stageId: Int, now: Long): Unit = {
- exec.blacklistedInStages += stageId
+ private def addExcludedStageTo(exec: LiveExecutor, stageId: Int, now: Long): Unit = {
+ exec.excludedInStages += stageId
liveUpdate(exec, now)
}
private def setStageBlackListStatus(stage: LiveStage, now: Long, executorIds: String*): Unit = {
executorIds.foreach { executorId =>
val executorStageSummary = stage.executorSummary(executorId)
- executorStageSummary.isBlacklisted = true
+ executorStageSummary.isExcluded = true
+ maybeUpdate(executorStageSummary, now)
+ }
+ stage.excludedExecutors ++= executorIds
+ maybeUpdate(stage, now)
+ }
+
+ private def setStageExcludedStatus(stage: LiveStage, now: Long, executorIds: String*): Unit = {
+ executorIds.foreach { executorId =>
+ val executorStageSummary = stage.executorSummary(executorId)
+ executorStageSummary.isExcluded = true
maybeUpdate(executorStageSummary, now)
}
- stage.blackListedExecutors ++= executorIds
+ stage.excludedExecutors ++= executorIds
maybeUpdate(stage, now)
}
override def onExecutorUnblacklisted(event: SparkListenerExecutorUnblacklisted): Unit = {
- updateBlackListStatus(event.executorId, false)
+ updateExclusionStatus(event.executorId, false)
+ }
+
+ override def onExecutorUnexcluded(event: SparkListenerExecutorUnexcluded): Unit = {
+ updateExclusionStatus(event.executorId, false)
}
override def onNodeBlacklisted(event: SparkListenerNodeBlacklisted): Unit = {
- updateNodeBlackList(event.hostId, true)
+ updateNodeExcluded(event.hostId, true)
+ }
+
+ override def onNodeExcluded(event: SparkListenerNodeExcluded): Unit = {
+ updateNodeExcluded(event.hostId, true)
}
override def onNodeUnblacklisted(event: SparkListenerNodeUnblacklisted): Unit = {
- updateNodeBlackList(event.hostId, false)
+ updateNodeExcluded(event.hostId, false)
+ }
+
+ override def onNodeUnexcluded(event: SparkListenerNodeUnexcluded): Unit = {
+ updateNodeExcluded(event.hostId, false)
+ }
+
+ private def updateNodeExclusionStatusForStage(stageId: Int, stageAttemptId: Int,
+ hostId: String): Unit = {
+ val now = System.nanoTime()
+
+ // Implicitly exclude every available executor for the stage associated with this node
+ Option(liveStages.get((stageId, stageAttemptId))).foreach { stage =>
+ val executorIds = liveExecutors.values.filter(_.host == hostId).map(_.executorId).toSeq
+ setStageExcludedStatus(stage, now, executorIds: _*)
+ }
+ liveExecutors.values.filter(_.hostname == hostId).foreach { exec =>
+ addExcludedStageTo(exec, stageId, now)
+ }
+ }
+
+ private def updateExclusionStatusForStage(stageId: Int, stageAttemptId: Int,
+ execId: String): Unit = {
+ val now = System.nanoTime()
+
+ Option(liveStages.get((stageId, stageAttemptId))).foreach { stage =>
+ setStageExcludedStatus(stage, now, execId)
+ }
+ liveExecutors.get(execId).foreach { exec =>
+ addExcludedStageTo(exec, stageId, now)
+ }
}
- private def updateBlackListStatus(execId: String, blacklisted: Boolean): Unit = {
+ private def updateExclusionStatus(execId: String, excluded: Boolean): Unit = {
liveExecutors.get(execId).foreach { exec =>
- exec.isBlacklisted = blacklisted
- if (blacklisted) {
+ exec.isExcluded = excluded
+ if (excluded) {
appStatusSource.foreach(_.BLACKLISTED_EXECUTORS.inc())
+ appStatusSource.foreach(_.EXCLUDED_EXECUTORS.inc())
Review comment:
thanks for catching this one, I missed it. I had checked this and most are fine due to using set and just setting status which would already be set, but I somehow missed this one. I'll fix
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701570539
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-701448668
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-711467525
Ok, I'm fine to deprecate them first.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705214880
**[Test build #129532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129532/testReport)** for PR 29906 at commit [`0172aa4`](https://github.com/apache/spark/commit/0172aa4a8c64b6726a50e703e5e18462790c4949).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r501000476
##########
File path: core/src/main/scala/org/apache/spark/status/api/v1/api.scala
##########
@@ -82,10 +82,11 @@ class ExecutorStageSummary private[spark](
val shuffleWriteRecords : Long,
val memoryBytesSpilled : Long,
val diskBytesSpilled : Long,
- val isBlacklistedForStage: Boolean,
+ val isBlacklistedForStage: Boolean, // deprecated
Review comment:
deprecation without an alternative is pointless. So I think we would need a plan on what to replace it with. @Ngone51 any thoughts on the rest api deprecation?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-715656257
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-705241921
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-715447986
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34810/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r510979634
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -284,80 +284,127 @@ private[spark] class AppStatusListener(
}
override def onExecutorBlacklisted(event: SparkListenerExecutorBlacklisted): Unit = {
- updateBlackListStatus(event.executorId, true)
+ updateExclusionStatus(event.executorId, true)
+ }
+
+ override def onExecutorExcluded(event: SparkListenerExecutorExcluded): Unit = {
+ updateExclusionStatus(event.executorId, true)
}
override def onExecutorBlacklistedForStage(
event: SparkListenerExecutorBlacklistedForStage): Unit = {
- val now = System.nanoTime()
+ updateExclusionStatusForStage(event.stageId, event.stageAttemptId, event.executorId)
+ }
- Option(liveStages.get((event.stageId, event.stageAttemptId))).foreach { stage =>
- setStageBlackListStatus(stage, now, event.executorId)
- }
- liveExecutors.get(event.executorId).foreach { exec =>
- addBlackListedStageTo(exec, event.stageId, now)
- }
+ override def onExecutorExcludedForStage(
+ event: SparkListenerExecutorExcludedForStage): Unit = {
+ updateExclusionStatusForStage(event.stageId, event.stageAttemptId, event.executorId)
}
override def onNodeBlacklistedForStage(event: SparkListenerNodeBlacklistedForStage): Unit = {
- val now = System.nanoTime()
+ updateNodeExclusionStatusForStage(event.stageId, event.stageAttemptId, event.hostId)
+ }
- // Implicitly blacklist every available executor for the stage associated with this node
- Option(liveStages.get((event.stageId, event.stageAttemptId))).foreach { stage =>
- val executorIds = liveExecutors.values.filter(_.host == event.hostId).map(_.executorId).toSeq
- setStageBlackListStatus(stage, now, executorIds: _*)
- }
- liveExecutors.values.filter(_.hostname == event.hostId).foreach { exec =>
- addBlackListedStageTo(exec, event.stageId, now)
- }
+ override def onNodeExcludedForStage(event: SparkListenerNodeExcludedForStage): Unit = {
+ updateNodeExclusionStatusForStage(event.stageId, event.stageAttemptId, event.hostId)
}
- private def addBlackListedStageTo(exec: LiveExecutor, stageId: Int, now: Long): Unit = {
- exec.blacklistedInStages += stageId
+ private def addExcludedStageTo(exec: LiveExecutor, stageId: Int, now: Long): Unit = {
+ exec.excludedInStages += stageId
liveUpdate(exec, now)
}
private def setStageBlackListStatus(stage: LiveStage, now: Long, executorIds: String*): Unit = {
executorIds.foreach { executorId =>
val executorStageSummary = stage.executorSummary(executorId)
- executorStageSummary.isBlacklisted = true
+ executorStageSummary.isExcluded = true
+ maybeUpdate(executorStageSummary, now)
+ }
+ stage.excludedExecutors ++= executorIds
+ maybeUpdate(stage, now)
+ }
+
+ private def setStageExcludedStatus(stage: LiveStage, now: Long, executorIds: String*): Unit = {
+ executorIds.foreach { executorId =>
+ val executorStageSummary = stage.executorSummary(executorId)
+ executorStageSummary.isExcluded = true
maybeUpdate(executorStageSummary, now)
}
- stage.blackListedExecutors ++= executorIds
+ stage.excludedExecutors ++= executorIds
maybeUpdate(stage, now)
}
override def onExecutorUnblacklisted(event: SparkListenerExecutorUnblacklisted): Unit = {
- updateBlackListStatus(event.executorId, false)
+ updateExclusionStatus(event.executorId, false)
+ }
+
+ override def onExecutorUnexcluded(event: SparkListenerExecutorUnexcluded): Unit = {
+ updateExclusionStatus(event.executorId, false)
}
override def onNodeBlacklisted(event: SparkListenerNodeBlacklisted): Unit = {
- updateNodeBlackList(event.hostId, true)
+ updateNodeExcluded(event.hostId, true)
+ }
+
+ override def onNodeExcluded(event: SparkListenerNodeExcluded): Unit = {
+ updateNodeExcluded(event.hostId, true)
}
override def onNodeUnblacklisted(event: SparkListenerNodeUnblacklisted): Unit = {
- updateNodeBlackList(event.hostId, false)
+ updateNodeExcluded(event.hostId, false)
+ }
+
+ override def onNodeUnexcluded(event: SparkListenerNodeUnexcluded): Unit = {
+ updateNodeExcluded(event.hostId, false)
+ }
+
+ private def updateNodeExclusionStatusForStage(stageId: Int, stageAttemptId: Int,
+ hostId: String): Unit = {
+ val now = System.nanoTime()
+
+ // Implicitly exclude every available executor for the stage associated with this node
+ Option(liveStages.get((stageId, stageAttemptId))).foreach { stage =>
+ val executorIds = liveExecutors.values.filter(_.host == hostId).map(_.executorId).toSeq
+ setStageExcludedStatus(stage, now, executorIds: _*)
+ }
+ liveExecutors.values.filter(_.hostname == hostId).foreach { exec =>
+ addExcludedStageTo(exec, stageId, now)
+ }
+ }
+
+ private def updateExclusionStatusForStage(stageId: Int, stageAttemptId: Int,
+ execId: String): Unit = {
+ val now = System.nanoTime()
+
+ Option(liveStages.get((stageId, stageAttemptId))).foreach { stage =>
+ setStageExcludedStatus(stage, now, execId)
+ }
+ liveExecutors.get(execId).foreach { exec =>
+ addExcludedStageTo(exec, stageId, now)
+ }
}
- private def updateBlackListStatus(execId: String, blacklisted: Boolean): Unit = {
+ private def updateExclusionStatus(execId: String, excluded: Boolean): Unit = {
liveExecutors.get(execId).foreach { exec =>
- exec.isBlacklisted = blacklisted
- if (blacklisted) {
+ exec.isExcluded = excluded
+ if (excluded) {
appStatusSource.foreach(_.BLACKLISTED_EXECUTORS.inc())
+ appStatusSource.foreach(_.EXCLUDED_EXECUTORS.inc())
Review comment:
I updated this but I actually found a pre-existing bug where we weren't incrementing this when we excluded a node - which implicitly excludes the executors
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r513226852
##########
File path: core/src/main/scala/org/apache/spark/SparkConf.scala
##########
@@ -612,7 +612,29 @@ private[spark] object SparkConf extends Logging {
DeprecatedConfig("spark.yarn.credentials.file.retention.days", "2.4.0", "Not used anymore."),
DeprecatedConfig("spark.yarn.services", "3.0.0", "Feature no longer available."),
DeprecatedConfig("spark.executor.plugins", "3.0.0",
- "Feature replaced with new plugin API. See Monitoring documentation.")
+ "Feature replaced with new plugin API. See Monitoring documentation."),
+ DeprecatedConfig("spark.blacklist.enabled", "3.1.0",
+ "Please use spark.excludeOnFailure.enabled"),
+ DeprecatedConfig("spark.blacklist.task.maxTaskAttemptsPerExecutor", "3.1.0",
+ "Please use spark.excludeOnFailure.task.maxTaskAttemptsPerExecutor"),
+ DeprecatedConfig("spark.blacklist.task.maxTaskAttemptsPerNode", "3.1.0",
+ "Please use spark.excludeOnFailure.task.maxTaskAttemptsPerNode"),
+ DeprecatedConfig("spark.blacklist.application.maxFailedTasksPerExecutor", "3.1.0",
+ "Please use spark.excludeOnFailure.application.maxFailedTasksPerExecutor"),
+ DeprecatedConfig("spark.blacklist.stage.maxFailedTasksPerExecutor", "3.1.0",
+ "Please use spark.excludeOnFailure.stage.maxFailedTasksPerExecutor"),
+ DeprecatedConfig("spark.blacklist.application.maxFailedExecutorsPerNode", "3.1.0",
+ "Please use spark.excludeOnFailure.application.maxFailedExecutorsPerNode"),
+ DeprecatedConfig("spark.blacklist.stage.maxFailedExecutorsPerNode", "3.1.0",
+ "Please use spark.excludeOnFailure.stage.maxFailedExecutorsPerNode"),
+ DeprecatedConfig("spark.blacklist.timeout", "3.1.0",
+ "Please use spark.excludeOnFailure.timeout"),
+ DeprecatedConfig("spark.scheduler.executorTaskBlacklistTime", "3.1.0",
Review comment:
This one is duplicated? (see L605)
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -722,74 +722,83 @@ package object config {
.booleanConf
.createWithDefault(true)
- // Blacklist confs
- private[spark] val BLACKLIST_ENABLED =
- ConfigBuilder("spark.blacklist.enabled")
- .version("2.1.0")
+ private[spark] val EXCLUDE_ON_FAILURE_ENABLED =
+ ConfigBuilder("spark.excludeOnFailure.enabled")
+ .version("3.1.0")
+ .withAlternative("spark.blacklist.enabled")
.booleanConf
.createOptional
private[spark] val MAX_TASK_ATTEMPTS_PER_EXECUTOR =
- ConfigBuilder("spark.blacklist.task.maxTaskAttemptsPerExecutor")
- .version("2.1.0")
+ ConfigBuilder("spark.excludeOnFailure.task.maxTaskAttemptsPerExecutor")
+ .version("3.1.0")
+ .withAlternative("spark.blacklist.task.maxTaskAttemptsPerExecutor")
.intConf
.createWithDefault(1)
private[spark] val MAX_TASK_ATTEMPTS_PER_NODE =
- ConfigBuilder("spark.blacklist.task.maxTaskAttemptsPerNode")
- .version("2.1.0")
+ ConfigBuilder("spark.excludeOnFailure.task.maxTaskAttemptsPerNode")
+ .version("3.1.0")
+ .withAlternative("spark.blacklist.task.maxTaskAttemptsPerNode")
.intConf
.createWithDefault(2)
private[spark] val MAX_FAILURES_PER_EXEC =
- ConfigBuilder("spark.blacklist.application.maxFailedTasksPerExecutor")
- .version("2.2.0")
+ ConfigBuilder("spark.excludeOnFailure.application.maxFailedTasksPerExecutor")
+ .version("3.1.0")
+ .withAlternative("spark.blacklist.application.maxFailedTasksPerExecutor")
.intConf
.createWithDefault(2)
private[spark] val MAX_FAILURES_PER_EXEC_STAGE =
- ConfigBuilder("spark.blacklist.stage.maxFailedTasksPerExecutor")
- .version("2.1.0")
+ ConfigBuilder("spark.excludeOnFailure.stage.maxFailedTasksPerExecutor")
+ .version("3.1.0")
+ .withAlternative("spark.blacklist.stage.maxFailedTasksPerExecutor")
.intConf
.createWithDefault(2)
private[spark] val MAX_FAILED_EXEC_PER_NODE =
- ConfigBuilder("spark.blacklist.application.maxFailedExecutorsPerNode")
- .version("2.2.0")
+ ConfigBuilder("spark.excludeOnFailure.application.maxFailedExecutorsPerNode")
+ .version("3.1.0")
+ .withAlternative("spark.blacklist.application.maxFailedExecutorsPerNode")
.intConf
.createWithDefault(2)
private[spark] val MAX_FAILED_EXEC_PER_NODE_STAGE =
- ConfigBuilder("spark.blacklist.stage.maxFailedExecutorsPerNode")
- .version("2.1.0")
+ ConfigBuilder("spark.excludeOnFailure.stage.maxFailedExecutorsPerNode")
+ .version("3.1.0")
+ .withAlternative("spark.blacklist.stage.maxFailedExecutorsPerNode")
.intConf
.createWithDefault(2)
- private[spark] val BLACKLIST_TIMEOUT_CONF =
- ConfigBuilder("spark.blacklist.timeout")
- .version("2.1.0")
+ private[spark] val EXCLUDE_ON_FAILURE_TIMEOUT_CONF =
+ ConfigBuilder("spark.excludeOnFailure.timeout")
+ .version("3.1.0")
+ .withAlternative("spark.blacklist.timeout")
.timeConf(TimeUnit.MILLISECONDS)
.createOptional
- private[spark] val BLACKLIST_KILL_ENABLED =
- ConfigBuilder("spark.blacklist.killBlacklistedExecutors")
Review comment:
this one is missed in `DeprecatedConfig`?
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala
##########
@@ -379,14 +379,15 @@ package object config extends Logging {
.stringConf
.createOptional
- /* YARN allocator-level blacklisting related config entries. */
- private[spark] val YARN_EXECUTOR_LAUNCH_BLACKLIST_ENABLED =
- ConfigBuilder("spark.yarn.blacklist.executor.launch.blacklisting.enabled")
+ /* YARN allocator-level excludeOnFailure related config entries. */
+ private[spark] val YARN_EXECUTOR_LAUNCH_EXCLUDE_ON_FAILURE_ENABLED =
+ ConfigBuilder("spark.yarn.executor.launch.excludeOnFailure.enabled")
.version("2.4.0")
+ .withAlternative("spark.yarn.blacklist.executor.launch.blacklisting.enabled")
Review comment:
Shall we add it to deprecated config too?
##########
File path: core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala
##########
@@ -25,7 +25,7 @@ import scala.collection.Map
import com.fasterxml.jackson.annotation.JsonTypeInfo
import org.apache.spark.TaskEndReason
-import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.annotation.{DeveloperApi, Since}
Review comment:
Unused `Since`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29906:
URL: https://github.com/apache/spark/pull/29906#discussion_r508182684
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -284,80 +284,127 @@ private[spark] class AppStatusListener(
}
override def onExecutorBlacklisted(event: SparkListenerExecutorBlacklisted): Unit = {
- updateBlackListStatus(event.executorId, true)
+ updateExclusionStatus(event.executorId, true)
+ }
+
+ override def onExecutorExcluded(event: SparkListenerExecutorExcluded): Unit = {
+ updateExclusionStatus(event.executorId, true)
}
override def onExecutorBlacklistedForStage(
event: SparkListenerExecutorBlacklistedForStage): Unit = {
- val now = System.nanoTime()
+ updateExclusionStatusForStage(event.stageId, event.stageAttemptId, event.executorId)
+ }
- Option(liveStages.get((event.stageId, event.stageAttemptId))).foreach { stage =>
- setStageBlackListStatus(stage, now, event.executorId)
- }
- liveExecutors.get(event.executorId).foreach { exec =>
- addBlackListedStageTo(exec, event.stageId, now)
- }
+ override def onExecutorExcludedForStage(
+ event: SparkListenerExecutorExcludedForStage): Unit = {
+ updateExclusionStatusForStage(event.stageId, event.stageAttemptId, event.executorId)
}
override def onNodeBlacklistedForStage(event: SparkListenerNodeBlacklistedForStage): Unit = {
- val now = System.nanoTime()
+ updateNodeExclusionStatusForStage(event.stageId, event.stageAttemptId, event.hostId)
+ }
- // Implicitly blacklist every available executor for the stage associated with this node
- Option(liveStages.get((event.stageId, event.stageAttemptId))).foreach { stage =>
- val executorIds = liveExecutors.values.filter(_.host == event.hostId).map(_.executorId).toSeq
- setStageBlackListStatus(stage, now, executorIds: _*)
- }
- liveExecutors.values.filter(_.hostname == event.hostId).foreach { exec =>
- addBlackListedStageTo(exec, event.stageId, now)
- }
+ override def onNodeExcludedForStage(event: SparkListenerNodeExcludedForStage): Unit = {
+ updateNodeExclusionStatusForStage(event.stageId, event.stageAttemptId, event.hostId)
}
- private def addBlackListedStageTo(exec: LiveExecutor, stageId: Int, now: Long): Unit = {
- exec.blacklistedInStages += stageId
+ private def addExcludedStageTo(exec: LiveExecutor, stageId: Int, now: Long): Unit = {
+ exec.excludedInStages += stageId
liveUpdate(exec, now)
}
private def setStageBlackListStatus(stage: LiveStage, now: Long, executorIds: String*): Unit = {
executorIds.foreach { executorId =>
val executorStageSummary = stage.executorSummary(executorId)
- executorStageSummary.isBlacklisted = true
+ executorStageSummary.isExcluded = true
+ maybeUpdate(executorStageSummary, now)
+ }
+ stage.excludedExecutors ++= executorIds
+ maybeUpdate(stage, now)
+ }
+
+ private def setStageExcludedStatus(stage: LiveStage, now: Long, executorIds: String*): Unit = {
+ executorIds.foreach { executorId =>
+ val executorStageSummary = stage.executorSummary(executorId)
+ executorStageSummary.isExcluded = true
maybeUpdate(executorStageSummary, now)
}
- stage.blackListedExecutors ++= executorIds
+ stage.excludedExecutors ++= executorIds
maybeUpdate(stage, now)
}
override def onExecutorUnblacklisted(event: SparkListenerExecutorUnblacklisted): Unit = {
- updateBlackListStatus(event.executorId, false)
+ updateExclusionStatus(event.executorId, false)
+ }
+
+ override def onExecutorUnexcluded(event: SparkListenerExecutorUnexcluded): Unit = {
+ updateExclusionStatus(event.executorId, false)
}
override def onNodeBlacklisted(event: SparkListenerNodeBlacklisted): Unit = {
Review comment:
Do we need to implement these deprecated methods for internal listeners? I assume they are only used for external listeners.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29906:
URL: https://github.com/apache/spark/pull/29906#issuecomment-715646698
**[Test build #130220 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130220/testReport)** for PR 29906 at commit [`32ab73d`](https://github.com/apache/spark/commit/32ab73da672ad62192f811558961660fbc972c37).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org