[GitHub] [spark] jiangxb1987 opened a new pull request #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling

[GitBox <gi...@apache.org>]

jiangxb1987 opened a new pull request #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling
URL: https://github.com/apache/spark/pull/24374
 
 
   ## What changes were proposed in this pull request?
   
   This PR adds support to schedule tasks with GPU resource requirements on executors with available GPU resources. A few major features introduced in this PR:
   
   1. Represent resources by `ResourceInformation`;
   2. Introduce new config `spark.task.resource.gpu.count` to specify number of GPUs required by each task;
   3. Introduce new configs to specify number of GPUs assigned to each executor/driver;
   4. Support two ways to pass in GPU resources to an executor:
   * Specify GPU devices by argument `--gpu-devices` passed to `CoarseGrainedExecutorBackend`;
   * Provide GPU devices auto discovery script through `spark.executor.resource.gpu.discoveryScript`;
   5. Provide `TaskContext.resources()` method so tasks can access available device indices allocated to them.
   
   ## How was this patch tested?
   
   * Added new end-to-end test cases in `SparkContextSuite`;
   * Added new test case in `CoarseGrainedSchedulerBackendSuite`;
   * Added new test case in `TaskSchedulerImplSuite`;
   * Added new test suite `ResourceDiscovererSuite`;
   * Updated existing tests.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org