You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/07/07 04:07:54 UTC

[GitHub] [spark] akshatb1 edited a comment on pull request #33135: [SPARK-35931][CORE][YARN] Ability to override Yarn Cluster Submit Class with Configuration

akshatb1 edited a comment on pull request #33135:
URL: https://github.com/apache/spark/pull/33135#issuecomment-875250761


   Thanks @tgravescs for looking into this. Please find my responses below.
   
   > Can you give more details about exactly what requires changing here?
   
   We have a custom implementation of setting up tokens for Yarn applications. This involves invoking some native libraries to generate different kinds of tokens and these needs to be set in AM Container launch context. Similarly, there are some changes required in Yarn configuration for running on multi-tenant Federated Yarn cluster. For this purpose, we are adding a new client similar to yarn/Client.scala along with a bunch of helper utility classes related to token handling.
   
   > If its a different yarn Client, if its way different it should just be a separate resource manager perhaps. If its something smaller (which it doesn't sound like from the description) perhaps something else needs to be more flexible.
   
   We are adding a new folder under /resource-managers/yarn such as /resource-managers/yarn/customyarn. This way it can leverage all existing internal classes and we can add custom implementation for Client in this new folder along with other utilities.
   
   > Another question - the ApplicationMaster doesn't need modified as well?
   
   ApplicationMaster does not change since only changes required are in how application submission is done.
   
   > These are all internal Spark classes as well, so making a config for it without some public or developer interface doesn't make a lot of sense.
   
   Will it be a good idea to make the private methods/fields in Client.scala to make protected to allow developers to extend the Client? Currently, it can not be extended since many methods are private.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org