[GitHub] liupc edited a comment on issue #23786: [SPARK-26877]Support user-level app staging directory in yarn mode when spark.yarn…

[GitBox <gi...@apache.org>]

liupc edited a comment on issue #23786: [SPARK-26877]Support user-level app staging directory in yarn mode when spark.yarn…
URL: https://github.com/apache/spark/pull/23786#issuecomment-464051617
 
 
   @vanzin 
   I test the config you provide, but it seems only get the current username on the OS, but not the correct   kerberos name when hadoop security enabled.
   ```
   19/02/15 20:28:20 INFO yarn.Client: Uploading resource file:/tmp/spark-51e5acbe-0dee-49f3-a954-1de89b353c3f/__spark_libs__6081884678722264795.zip -> hdfs://zjyprc-hadoop/tmp/liupengcheng/staging/liupengcheng/.sparkStaging/application_1550196143681_22965/__spark_libs__6081884678722264795.zip
   ```
   
   I think `UGI.getUserName` is ok, this is verified on our production cluster. What's more, this kind of method call is used many places in spark:
   https://github.com/apache/spark/blob/71170e74df5c7ec657f61154212d1dc2ba7d0613/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L796
   
   In the code referred by above link, we can see that spark use this `UGI.getUserName` as the `SPARK_USER` in yarn mode.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org