[jira] [Comment Edited] (SPARK-30451) Stage level Sched: ExecutorResourceRequests/TaskResourceRequests should have functions to remove requests

["Jayadeep Jayaraman (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-30451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17939692#comment-17939692 ] 

Jayadeep Jayaraman edited comment on SPARK-30451 at 3/31/25 1:23 PM:
---------------------------------------------------------------------

Hi [~tgraves] - I would like to work on this request. As per the description my understanding is
{code:java}
class ExecutorResourceRequests() {code}
is a convenience method for the class 
{code:java}
class ExecutorResourceRequest(){code}
 Class `ExecutorResourceRequests() ` has multiple methods to add different resources
{code:java}
def offHeapMemory(amount: String): this.type

def memoryOverhead(amount: String): this.type

def memory(amount: String): this.type{code}
and they are stored in a mutable map
{code:java}
new ConcurrentHashMap[String, ExecutorResourceRequest](){code}
The goal of this PR is to offer an API that allows users to build a Request profile once as shown below
{code:java}
 
val executorReqs: ExecutorResourceRequests = ExecutorResourceRequests.builder() .memory("1g") .cores(4) .resource("gpu", 2) .build()
{code}
and then offer ways to add or remove specific resources as shown below
{code:java}
 
val anotherReqs: ExecutorResourceRequests = ExecutorResourceRequests.builder() .offHeapMemory("512m") .add(new ExecutorResourceRequest("fpga", 1)) .remove("memory") // This would have no effect as 'memory' wasn't added in this builder .build()
{code}
Kindly, let me know if my understanding is correct and I would like to work on this task

 


was (Author: jjayadeep):
Hi [~tgraves] - I would like to work on this request. As per the description my understanding is
{code:java}
class ExecutorResourceRequests() {code}
is a convenience method for the class 
{code:java}
class ExecutorResourceRequest(){code}
 Class `ExecutorResourceRequests() ` has multiple methods to add different resources
{code:java}
def offHeapMemory(amount: String): this.type

def memoryOverhead(amount: String): this.type

def memory(amount: String): this.type{code}
and they are stored in a mutable map
{code:java}
new ConcurrentHashMap[String, ExecutorResourceRequest](){code}
 

The goal of this PR is to offer an API that allows users to build a Request profile once as shown below
{code:java}
 
val executorReqs: ExecutorResourceRequests = ExecutorResourceRequests.builder() .memory("1g") .cores(4) .resource("gpu", 2) .build()
{code}
 

 

and then offer ways to add or remove specific resources as shown below
{code:java}
 
val anotherReqs: ExecutorResourceRequests = ExecutorResourceRequests.builder() .offHeapMemory("512m") .add(new ExecutorResourceRequest("fpga", 1)) .remove("memory") // This would have no effect as 'memory' wasn't added in this builder .build()
{code}
 

 

Kindly, let me know if my understanding is correct and I would like to work on this task

 

> Stage level Sched: ExecutorResourceRequests/TaskResourceRequests should have functions to remove requests
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-30451
>                 URL: https://issues.apache.org/jira/browse/SPARK-30451
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 3.1.0
>            Reporter: Thomas Graves
>            Priority: Major
>
> Stage level Sched: ExecutorResourceRequests/TaskResourceRequests should have functions to remove requests
> Currently in the design ExecutorResourceRequests and TaskREsourceRequests are mutable and users can update as they want. It would make sense to add api's to remove certain resource requirements from them. This would allow a user to create one ExecutorResourceRequests object and then if they want to just add/remove something from it they easily could without having to recreate all the requests in that.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org