You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "pan3793 (via GitHub)" <gi...@apache.org> on 2024/04/07 08:16:26 UTC

[I] [TASK][MEDIUM] Resource isolation in Spark Scala mode [kyuubi]

pan3793 opened a new issue, #6265:
URL: https://github.com/apache/kyuubi/issues/6265

   ### What's the level of this task?
   
   MEDIUM
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   
   
   ### Search before creating
   
   - [X] I have searched in the [task list](https://github.com/orgs/apache/projects/343) and found no similar tasks.
   
   
   ### Mentor
   
   - [ ] I have sufficient expertise on this task, and I volunteer to be a mentor of this task to guide contributors through the task.
   
   
   ### Skill requirements
   
   - Knowledge of Spark/Scala REPL
   - Knowledge of classloader
   
   ### Background and Goals
   
   The current Kyuubi Scala mode reuses the code of Spark Shell, while Spark Shell is designed for single session and client deploy mode, which has certain limitations for Kyuubis multi-tenant use cases.
   
   Thanks to Spark Connect, a similar mechanism is required to support isolated sessions (i.e. isolated resources/classloader) so that each REPL user does not conflict with other users on the same cluster.
   
   As of Spark 3.5, the core/sql already has infrastructure to support isolated resources/classloader, see details in
   
   - SPARK-44078
   - SPARK-44146
   - SPARK-45856
   
   ### Implementation steps
   
   Investigate how could Kyuubi Spark Scala mode leverage those infrastructures to support session-level isolated resources/classloader in Spark 3.5+
   
   ### Additional context
   
   Introduction of [2024H1 Kyuubi Code Contribution Program](https://github.com/apache/kyuubi/issues/6232)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [I] [TASK][MEDIUM] Resource isolation in Spark Scala mode [kyuubi]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #6265:
URL: https://github.com/apache/kyuubi/issues/6265#issuecomment-2041399712

   @beryllw thanks, assgined


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [I] [TASK][MEDIUM] Resource isolation in Spark Scala mode [kyuubi]

Posted by "beryllw (via GitHub)" <gi...@apache.org>.
beryllw commented on issue #6265:
URL: https://github.com/apache/kyuubi/issues/6265#issuecomment-2041391787

   Intersted in this, Please assign this task to me.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [I] [TASK][MEDIUM] Resource isolation in Spark Scala mode [kyuubi]

Posted by "zhaohehuhu (via GitHub)" <gi...@apache.org>.
zhaohehuhu commented on issue #6265:
URL: https://github.com/apache/kyuubi/issues/6265#issuecomment-2041894049

   Interesting task


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org