You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@orc.apache.org by Ryan Schachte <co...@gmail.com> on 2021/04/16 10:55:43 UTC
Disabling Kerberos when using ORC writer
Hi everyone,
I've spent many hours debugging this failure. I've written a small ORC
compactor using the Java libs. Everything works locally, but deployment to
cloud running in Docker is giving me Kerberos auth failures.
public FileSystem getHadoopFs() {
return new LocalFileSystem() {
@Override
public Configuration getConf() {
Configuration conf = new Configuration();
conf.set("hadoop.security.authorization", "false");
conf.set("hadoop.security.authentication", "simple");
return conf;
}
};
}
I've tried disabling Kerberos like so to no avail. Are there any
recommendations for how to go about this? This is just a standalone jar
running in Docker. Stack trace below.
Exception in thread "main"
org.apache.hadoop.security.KerberosAuthException: failure to login:
javax.security.auth.login.LoginException: java.lang.NullPointerException:
invalid null input: name
at
jdk.security.auth/com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:67)
at
jdk.security.auth/com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:129)
at
java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:726)
at
java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:665)
at
java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:663)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at
java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:663)
at
java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:574)
at
org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
at
org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)
at
org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3487)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3477)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3319)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:463)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
at com.orc.compaction.utils.FileDump.getReader(FileDump.java:72)
at com.compaction.utils.FileDump.inferSchemaFromBinary(FileDump.java:61)
at com.orc.compaction.utils.FileDump.inferSchemaFromBinary(FileDump.java:47)
at com.orc.compaction.utils.FileDump.inferSchemaFromFiles(FileDump.java:39)
at
com.orc.compaction.CompactMonkey.retrieveWritersPerSchema(CompactMonkey.java:46)
at com,orc.compaction.CompactMonkey.processCompaction(CompactMonkey.java:69)
at com.orc.compaction.App.main(App.java:43)
at
org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1847)
at
org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3487)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3477)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3319)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:463)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
My ORC writer is pretty standard.
public static Writer createOrcWriter(
Properties orcWriterProperties,
Configuration configuration,
Path orcOutputFile,
TypeDescription orcSchema,
FileSystem fs)
throws IOException {
OrcFile.WriterOptions opts =
OrcFile.writerOptions(orcWriterProperties, configuration)
.setSchema(orcSchema)
.useUTCTimestamp(true);
if (fs != null) {
fs.setConf(configuration);
opts.fileSystem(fs);
}
return OrcFile.createWriter(orcOutputFile, opts);
}
Re: Disabling Kerberos when using ORC writer
Posted by Ryan Schachte <co...@gmail.com>.
Just following up. I've added the core-default and core-site defaults from
hadoop onto the classpath with explicit overrides for kerberos and I seemed
to get around this issue.
The downside is, running this on Docker on Kubernetes, the localfilesystem
can't seem to find the path I'm trying to write to. Still debugging.
On Fri, Apr 16, 2021 at 3:55 AM Ryan Schachte <co...@gmail.com>
wrote:
> Hi everyone,
>
> I've spent many hours debugging this failure. I've written a small ORC
> compactor using the Java libs. Everything works locally, but deployment to
> cloud running in Docker is giving me Kerberos auth failures.
>
> public FileSystem getHadoopFs() {
> return new LocalFileSystem() {
> @Override
> public Configuration getConf() {
> Configuration conf = new Configuration();
> conf.set("hadoop.security.authorization", "false");
> conf.set("hadoop.security.authentication", "simple");
> return conf;
> }
> };
> }
>
>
> I've tried disabling Kerberos like so to no avail. Are there any
> recommendations for how to go about this? This is just a standalone jar
> running in Docker. Stack trace below.
>
> Exception in thread "main"
> org.apache.hadoop.security.KerberosAuthException: failure to login:
> javax.security.auth.login.LoginException: java.lang.NullPointerException:
> invalid null input: name
> at
> jdk.security.auth/com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:67)
> at
> jdk.security.auth/com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:129)
> at
> java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:726)
> at
> java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:665)
> at
> java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:663)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at
> java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:663)
> at
> java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:574)
> at
> org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
> at
> org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)
> at
> org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571)
> at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3487)
> at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3477)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3319)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:463)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
> at com.orc.compaction.utils.FileDump.getReader(FileDump.java:72)
> at com.compaction.utils.FileDump.inferSchemaFromBinary(FileDump.java:61)
> at
> com.orc.compaction.utils.FileDump.inferSchemaFromBinary(FileDump.java:47)
> at com.orc.compaction.utils.FileDump.inferSchemaFromFiles(FileDump.java:39)
> at
> com.orc.compaction.CompactMonkey.retrieveWritersPerSchema(CompactMonkey.java:46)
> at
> com,orc.compaction.CompactMonkey.processCompaction(CompactMonkey.java:69)
> at com.orc.compaction.App.main(App.java:43)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1847)
> at
> org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571)
> at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3487)
> at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3477)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3319)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:463)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
>
> My ORC writer is pretty standard.
>
> public static Writer createOrcWriter(
> Properties orcWriterProperties,
> Configuration configuration,
> Path orcOutputFile,
> TypeDescription orcSchema,
> FileSystem fs)
> throws IOException {
> OrcFile.WriterOptions opts =
> OrcFile.writerOptions(orcWriterProperties, configuration)
> .setSchema(orcSchema)
> .useUTCTimestamp(true);
>
> if (fs != null) {
> fs.setConf(configuration);
> opts.fileSystem(fs);
> }
> return OrcFile.createWriter(orcOutputFile, opts);
> }
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>