[GitHub] [arrow-datafusion] thinkharderdev commented on pull request #1924: Refactor ExecutionContext and related conf to support multi-tenancy configurations

[GitBox <gi...@apache.org>]

thinkharderdev commented on pull request #1924:
URL: https://github.com/apache/arrow-datafusion/pull/1924#issuecomment-1060547632


   I wonder if we are missing a layer here. My intuition is that a global `RuntimeEnv` might be a problem and not really support real multi-tenancy but a session-level `RuntimeEnv` may be too fine-grained. We already have the concept of a namespace in the state configurations so does it make sense to namespace the `RuntimeEnv`. This could be away to support proper resource-level multi-tenancy. That is each namespace is a proper "slice" of the total cluster resources and there is some level of isolation so that one resource-hungry query by one client (in theory at least) can not use all cluster resources. It would probably be a little complicated to do dynamic registration of namespaces but should be possible in principle (and we probably don't have to tackle this right away). 
   
   I have the same question as @yjshen with respect to mutli-scheduler environments. I think that will be a key consideration for Ballista in general. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org