You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/12/30 18:05:26 UTC

[GitHub] [spark] tgravescs opened a new pull request #27053: [WIP][SPARK-27495][Core][YARN][k8s] Stage Level Scheduling code for reference

tgravescs opened a new pull request #27053: [WIP][SPARK-27495][Core][YARN][k8s] Stage Level Scheduling code for reference
URL: https://github.com/apache/spark/pull/27053

### What changes were proposed in this pull request?

This is all the code for stage level scheduling feature - except for the UI changes.

This is meant to be for a reference when reviewing as I'm splitting this into mulitple prs with the intention its easier to review. Note that only YARN currently supports this and it requires dynamic allocation to be enabled because currently we get new executors that match the profile exactly. We do not try to fit tasks into executors that were acquired for a different profile.

At a high level in order to support having different stages with different ResourceProfiles the changes required include:

- Add a ResourceProfileManager that tracks the profiles and is used to map a Resource Profile Id to the actual Resource Profile. This allows us to pass around and store the id rather then the entire profile.
- Introduce the concept of a default profile. This is essentially the profile you get today without stage level scheduling from the application level configs.
- ImmutableResourceProfile - this is the actual resource profile used internally to Spark that is immutable. This is to allow the user to create and change a ResourceProfile in their code but as soon as they associated the profile with an RDD then spark internally uses the Immutable version so that it doesn't change.
- YARN cluster manager updated to handle the profiles and request the correct containers from YARN. I had to introduce using priorities here because YARN doesn't allow you to create containers with different resources within the same priority. Now we have the priority = ResourceProfile Id and its easy to match the container we get from Yarn to what ResourceProfile we requested it for.
- ExecutorAllocationManager, ExecutorMonitor, CoarseGrainedExecutorBackend - Updated to handle tracking the executors per ResourceProfile.
- Scheduler - updated to handle the ResourceProfile associated with an RDD. It creates the Stage with the appropriate ResourceProfile. It has logic for handling conflicting ResourceProfiles when multiple RDD are put into the same stage that have different ResourceProfiles. The default behavior is to throw an exception, but there is a config that will allow the scheduler to merge the profiles using the max value of each resource. The task scheduler was updated to make sure the resources of each executor meet the task resources for that profile and to assign them out properly.
- I updated all the locations that used the hardcoded task cpus or other global configs to use the ResourceProfile based configs.
- RDD api added and ResourceProfile, ExecutorResourceRequests, and TaskResourceRequests made public.

### Why are the changes needed?

Allow for different stages to use different executor/task resources

### Does this PR introduce any user-facing change?

Yes the RDD.withResources and ResourceProfile, ExecutorResourceRequest, TaskResourceRequest apis

### How was this patch tested?

Unit tests and manually.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org