[jira] [Created] (SPARK-30689) Allow custom resource scheduling to work with Hadoop versions that don't support it

["Thomas Graves (Jira)" <ji...@apache.org>]

Thomas Graves created SPARK-30689:
-------------------------------------

             Summary: Allow custom resource scheduling to work with Hadoop versions that don't support it
                 Key: SPARK-30689
                 URL: https://issues.apache.org/jira/browse/SPARK-30689
             Project: Spark
          Issue Type: Bug
          Components: YARN
    Affects Versions: 3.0.0
            Reporter: Thomas Graves


Many people/companies will not be moving to Hadoop 3.0 that it supports custom resource scheduling for things like GPUs soon and have requested support for it in older hadoop 2.x versions. This also means that they may not have isolation enabled which is what the default behavior relies on.

right now the option is to write a custom discovery script to handle on their own. This is ok but has some limitation because the script runs as a separate process.  It also just a shell script.

I think we can make this a lot more flexible by making the entire resource discovery class pluggable. The default one would stay as is and call the discovery script, but if an advanced user wanted to replace the entire thing they could implement a pluggable class which they could write custom code on how to discovery resource addresses.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org