You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Imran Rashid (JIRA)" <ji...@apache.org> on 2018/02/21 22:41:00 UTC

[jira] [Created] (SPARK-23485) Kubernetes should support node blacklist

Imran Rashid created SPARK-23485:
------------------------------------

             Summary: Kubernetes should support node blacklist
                 Key: SPARK-23485
                 URL: https://issues.apache.org/jira/browse/SPARK-23485
             Project: Spark
          Issue Type: New Feature
          Components: Kubernetes, Scheduler
    Affects Versions: 2.3.0
            Reporter: Imran Rashid


Spark's BlacklistTracker maintains a list of "bad nodes" which it will not use for running tasks (eg., because of bad hardware).  When running in yarn, this blacklist is used to avoid ever allocating resources on blacklisted nodes: https://github.com/apache/spark/blob/e836c27ce011ca9aef822bef6320b4a7059ec343/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala#L128

I'm just beginning to poke around the kubernetes code, so apologies if this is incorrect -- but I didn't see any references to {{scheduler.nodeBlacklist()}} in {{KubernetesClusterSchedulerBackend}} so it seems this is missing.  Thought of this while looking at SPARK-19755, a similar issue on mesos.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org