You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uniffle.apache.org by "kaijchen (via GitHub)" <gi...@apache.org> on 2023/02/17 11:04:41 UTC
[GitHub] [incubator-uniffle] kaijchen opened a new issue, #625: [Improvement] Replace Sun Kerberos binding with Apache Kerby
kaijchen opened a new issue, #625:
URL: https://github.com/apache/incubator-uniffle/issues/625
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### What would you like to be improved?
Sun Kerberos binding, i.e. package `sun.security.krb5` is blocking us to build Uniffle in Java 11 and Java 17.
```sh
# With JDK 11
mvn package -Djava.version=11
# With JDK 17
mvn package -Djava.version=17
```
### How should we improve?
> Apache Kerby, as an [Apache Directory](http://directory.apache.org/) sub project, is a Java Kerberos binding. It provides a rich, intuitive and interoperable implementation, library, KDC and various facilities that integrates PKI, OTP and token (OAuth2) as desired in modern environments such as cloud, Hadoop and mobile.
We can replace `sun.security.krb5` by [Apache Kerby](https://github.com/apache/directory-kerby).
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] advancedxy commented on issue #625: [Improvement] Replace Sun Kerberos binding with Apache Kerby
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on issue #625:
URL: https://github.com/apache/incubator-uniffle/issues/625#issuecomment-1437794225
> > @zuston would you like to take a look at this?
>
> Yes. But I don't have much time in this month, perhaps next month.
No worries. It's not urgent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] slfan1989 commented on issue #625: [Improvement] Replace Sun Kerberos binding with Apache Kerby
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on issue #625:
URL: https://github.com/apache/incubator-uniffle/issues/625#issuecomment-1465799874
> cc @zuston
@jerqi @zuston @kaijchen I still need to add some information. after the information is completed, please help to review it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] kaijchen commented on issue #625: [Improvement] Replace Sun Kerberos binding with Apache Kerby
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on issue #625:
URL: https://github.com/apache/incubator-uniffle/issues/625#issuecomment-1441344171
> @kaijchen Can you assign this to me?
Assigned, thanks for taking this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] advancedxy commented on issue #625: [Improvement] Replace Sun Kerberos binding with Apache Kerby
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on issue #625:
URL: https://github.com/apache/incubator-uniffle/issues/625#issuecomment-1437776313
@zuston would you like to take a look at this?
@kaijchen would you also post some compiling failure logs when with JDK 11 or JDK 17?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] slfan1989 commented on issue #625: [Improvement] Replace Sun Kerberos binding with Apache Kerby
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on issue #625:
URL: https://github.com/apache/incubator-uniffle/issues/625#issuecomment-1441341993
@kaijchen Can you assign this to me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi closed issue #625: [Improvement] Package sun.security.krb5 is not visible in Java 11 and 17
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi closed issue #625: [Improvement] Package sun.security.krb5 is not visible in Java 11 and 17
URL: https://github.com/apache/incubator-uniffle/issues/625
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #625: [Improvement] Replace Sun Kerberos binding with Apache Kerby
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on issue #625:
URL: https://github.com/apache/incubator-uniffle/issues/625#issuecomment-1467234664
Wow, really thanks for your great work. @slfan1989
> no need to use apache kerby
+1.
> But I still want to ask sun.security.krb5.Config.refresh(); the purpose of this.
This part of code is introduced in #184 . If I remember correctly, this is just for unit test to simulate setting the profile w/o, because all tests run in one JVM.
If test cases could pass, I prefer removing this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] kaijchen commented on issue #625: [Improvement] Replace Sun Kerberos binding with Apache Kerby
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on issue #625:
URL: https://github.com/apache/incubator-uniffle/issues/625#issuecomment-1467258688
> But I still want to ask `sun.security.krb5.Config.refresh();` the purpose of this.
Thanks @slfan1989 for the investigation, +1 for this proposal. I'll change the title of this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] slfan1989 commented on issue #625: [Improvement] Replace Sun Kerberos binding with Apache Kerby
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on issue #625:
URL: https://github.com/apache/incubator-uniffle/issues/625#issuecomment-1441344828
@kaijchen Thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] slfan1989 commented on issue #625: [Improvement] Replace Sun Kerberos binding with Apache Kerby
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on issue #625:
URL: https://github.com/apache/incubator-uniffle/issues/625#issuecomment-1467155576
@jerqi @zuston @kaijchen Can you please check my comment above? Thank you so much!
I think adding `--add-exports java.security.jgss/sun.security.krb5=ALL-UNNAMED` in the pom file is a better option. But I still want to ask `sun.security.krb5.Config.refresh();` the purpose of this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] slfan1989 commented on issue #625: [Improvement] Replace Sun Kerberos binding with Apache Kerby
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on issue #625:
URL: https://github.com/apache/incubator-uniffle/issues/625#issuecomment-1465578601
WIP
I read this part of the code carefully, and I summarized the relevant information as follows:
We have 2 ways to solve the issue.
- Remove `sun.security.krb5.Config.refresh();` code.
- Add `--add-exports java.security.jgss/sun.security.krb5=ALL-UNNAMED` in the pom.xml of the project.
> Remove `sun.security.krb5.Config.refresh();` code.
From my personal point of view, this part of the code can be removed, because this part of the code should only work when the location of the krb5.conf configuration file changes in the same JVM.
We use `UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytabFile)` to verify the identity of the user and ensure that legitimate users can access HDFS.
We can find that `refreshKrb5Config=true` is added when initializing access to Kerberos configuration in the Hadoop code.
UserGroupInformation#loginUserFromKeytabAndReturnUGI
```
public
static UserGroupInformation loginUserFromKeytabAndReturnUGI(String user,
String path
) throws IOException {
if (!isSecurityEnabled())
return UserGroupInformation.getCurrentUser();
LoginParams params = new LoginParams();
params.put(LoginParam.PRINCIPAL, user);
params.put(LoginParam.KEYTAB, path);
return doSubjectLogin(null, params);
}
```
UserGroupInformation#doSubjectLogin
```
private static UserGroupInformation doSubjectLogin(
Subject subject, LoginParams params) throws IOException {
ensureInitialized();
// initial default login.
if (subject == null && params == null) {
params = LoginParams.getDefaults();
}
HadoopConfiguration loginConf = new HadoopConfiguration(params);
try {
// *****
// We need to focus on this code
// *****
HadoopLoginContext login = newLoginContext(
authenticationMethod.getLoginAppName(), subject, loginConf);
login.login();
UserGroupInformation ugi = new UserGroupInformation(login.getSubject());
// attach login context for relogin unless this was a pre-existing
// subject.
if (subject == null) {
params.put(LoginParam.PRINCIPAL, ugi.getUserName());
ugi.setLogin(login);
ugi.setLastLogin(Time.now());
}
return ugi;
} catch (LoginException le) {
KerberosAuthException kae =
new KerberosAuthException(FAILURE_TO_LOGIN, le);
if (params != null) {
kae.setPrincipal(params.get(LoginParam.PRINCIPAL));
kae.setKeytabFile(params.get(LoginParam.KEYTAB));
kae.setTicketCacheFile(params.get(LoginParam.CCACHE));
}
throw kae;
}
}
```
HadoopConfiguration#new HadoopLoginContext()
```
private static HadoopLoginContext
newLoginContext(String appName, Subject subject,
HadoopConfiguration loginConf)
throws LoginException {
// Temporarily switch the thread's ContextClassLoader to match this
// class's classloader, so that we can properly load HadoopLoginModule
// from the JAAS libraries.
Thread t = Thread.currentThread();
ClassLoader oldCCL = t.getContextClassLoader();
t.setContextClassLoader(HadoopLoginModule.class.getClassLoader());
try {
return new HadoopLoginContext(appName, subject, loginConf);
} finally {
t.setContextClassLoader(oldCCL);
}
}
```
HadoopLoginContext#constructor
```
HadoopLoginContext(String appName, Subject subject,
HadoopConfiguration conf) throws LoginException {
super(appName, subject, null, conf);
this.appName = appName;
this.conf = conf;
}
```
LoginContext#constructor
Configuration config is HadoopConfiguration
```
public LoginContext(String name, Subject subject,
CallbackHandler callbackHandler,
Configuration config) throws LoginException {
this.config = config;
if (config != null) {
creatorAcc = java.security.AccessController.getContext();
}
//
init(name);
if (subject != null) {
this.subject = subject;
subjectProvided = true;
}
if (callbackHandler == null) {
loadDefaultCallbackHandler();
} else if (creatorAcc == null) {
this.callbackHandler = new SecureCallbackHandler
(java.security.AccessController.getContext(),
callbackHandler);
} else {
this.callbackHandler = callbackHandler;
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] kaijchen commented on issue #625: [Improvement] Replace Sun Kerberos binding with Apache Kerby
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on issue #625:
URL: https://github.com/apache/incubator-uniffle/issues/625#issuecomment-1437908327
> would you also post some compiling failure logs when with JDK 11 or JDK 17?
https://github.com/kaijchen/incubator-uniffle/actions/runs/4228912219/jobs/7344901292#step:5:2884
```
Error: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) on project rss-common: Compilation failure
Error: /home/runner/work/incubator-uniffle/incubator-uniffle/common/src/main/java/org/apache/uniffle/common/security/HadoopSecurityContext.java:[59,18] error: package sun.security.krb5 is not visible
Error: (package sun.security.krb5 is declared in module java.security.jgss, which does not export it to the unnamed module)
Error: -> [Help 1]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #625: [Improvement] Replace Sun Kerberos binding with Apache Kerby
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #625:
URL: https://github.com/apache/incubator-uniffle/issues/625#issuecomment-1465795702
cc @zuston
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #625: [Improvement] Replace Sun Kerberos binding with Apache Kerby
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on issue #625:
URL: https://github.com/apache/incubator-uniffle/issues/625#issuecomment-1437790888
> @zuston would you like to take a look at this?
Yes. But I don't have much time in this month, perhaps next month.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org