[GitHub] [spark] turboFei opened a new pull request #24412: [SPARK-27515][Delpy] When application master retry after a long time running, the hdfs delegation tokens may be expired

[GitBox <gi...@apache.org>]

turboFei opened a new pull request #24412: [SPARK-27515][Delpy] When application master retry after a long time running, the hdfs delegation tokens may be expired
URL: https://github.com/apache/spark/pull/24412
 
 
   ## What changes were proposed in this pull request?
   When submit  a spark yarn application,  we first create a container launch context and store the relative tokens.
   And for each attempt of applicationMaster, it would transfer origin tokens  to connect yarn.
   However, it also transfer origin hdfs delegation tokens.
   For a spark streaming application, if its applicationMaster failed when it has run for a long duration.
   The hdfs token stored in container launch context may be expired.
   When the new attempt applicationMaster prepareLocalResources, it would access the hdfs and failed  for token expired.
   This error occured when we rolling upgrading our cluster.
   
   In this PR, I renew hadoop delegation credentials and  transfer them to the new user.
   ## How was this patch tested?
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org