You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2020/05/01 16:52:02 UTC
[jira] [Assigned] (SPARK-31551) createSparkUser lost user's
non-Hadoop credentials
[ https://issues.apache.org/jira/browse/SPARK-31551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-31551:
------------------------------------
Assignee: (was: Apache Spark)
> createSparkUser lost user's non-Hadoop credentials
> --------------------------------------------------
>
> Key: SPARK-31551
> URL: https://issues.apache.org/jira/browse/SPARK-31551
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.4, 2.4.5
> Reporter: Yuqi Wang
> Priority: Major
>
> See current *[createSparkUser|https://github.com/apache/spark/blob/263f04db865920d9c10251517b00a1b477b58ff1/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala#L66-L76]*:
> {code:java}
> def createSparkUser(): UserGroupInformation = {
> val user = Utils.getCurrentUserName()
> logDebug("creating UGI for user: " + user)
> val ugi = UserGroupInformation.createRemoteUser(user)
> transferCredentials(UserGroupInformation.getCurrentUser(), ugi)
> ugi
> }
> def transferCredentials(source: UserGroupInformation, dest: UserGroupInformation): Unit = {
> dest.addCredentials(source.getCredentials())
> }
> def getCurrentUserName(): String = {
> Option(System.getenv("SPARK_USER"))
> .getOrElse(UserGroupInformation.getCurrentUser().getShortUserName())
> }
> {code}
> The *transferCredentials* func can only transfer Hadoop creds such as Delegation Tokens.
> However, other creds stored in UGI.subject.getPrivateCredentials, will be lost here, such as:
> # Non-Hadoop creds:
> Such as, [Kafka creds |https://github.com/apache/kafka/blob/f3c8bff311b0e4c4d0e316ac949fe4491f9b107f/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/OAuthBearerLoginModule.java#L395]
> # Newly supported or 3rd party supported Hadoop creds:
> Such as to support OAuth/JWT token authn on Hadoop, we need to store the OAuth/JWT token into UGI.subject.getPrivateCredentials. However, these tokens are not supposed to be managed by Hadoop Credentials (currently it is only for Hadoop secret keys and delegation tokens)
> Another issue is that the *SPARK_USER* only gets the UserGroupInformation.getCurrentUser().getShortUserName() of the user, which may lost the user's fully qualified user name. We should better use the *getUserName* to get fully qualified user name in our client side, which is aligned to *[HADOOP_PROXY_USER|https://github.com/apache/hadoop/blob/30ef8d0f1a1463931fe581a46c739dad4c8260e4/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L716-L720]*.
> Related to https://issues.apache.org/jira/browse/SPARK-1051
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org