Re: [PR] [SPARK-48040][CONNECT][WIP]Spark connect supports scheduler pool [spark]

["xieshuaihu (via GitHub)" <gi...@apache.org>]

xieshuaihu commented on PR #46278:
URL: https://github.com/apache/spark/pull/46278#issuecomment-2088566513

   @hvanhovell @HyukjinKwon 
   Thers are two reasons to support set scheduler pool in spark connect.
   
   1. Vanilla spark supports fair scheduler and pools, if server runs in a specific pool, then client cannot make full use of the fair scheduler ability. This missig feature could block uses to use connect in their environment. (At least we want this feature in our environment)
   2. In multi user environment, it's important to limit execution resources allocated to some jobs (or allocate more resources to some jobs). I think supporting multi-user environment is a selling point of connect, so it's better to bring scheduler pools to connect.
   
   Further discussions are expected.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org