[GitHub] [spark] liangyu-1 commented on pull request #42295: [SPARK-44581][YARN] Fix the bug that ShutdownHookManager get wrong hadoop user group information

["liangyu-1 (via GitHub)" <gi...@apache.org>]

liangyu-1 commented on PR #42295:
URL: https://github.com/apache/spark/pull/42295#issuecomment-1667554705

   The main cause is that ShutdownHook thread is created before we create the ugi in ApplicationMaster.
   
   When we set the config key _"hadoop.security.credential.provider.path"_, the ApplicationMaster will try to get a filesystem when generating SSLOptions, and when initialize the filesystem during which it will generate a new thread whose ugi is inherited from the current process (yarn).
   After this, it will generate a new ugi (SPARK_USER) in ApplicationMaster and execute the doAs() function.
   
   Here is the chain of the call:
   ApplicationMaster.<init>(ApplicationMaster.scala:83) -> org.apache.spark.SecurityManager.<init>(SecurityManager.scala:98) -> org.apache.spark.SSLOptions$.parse(SSLOptions.scala:188) -> org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2353) -> org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2434) -> org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:82)
    


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org