You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2021/09/06 17:35:00 UTC
[jira] [Commented] (HADOOP-17894)
CredentialProviderFactory.getProviders() recursion loading JCEKS file from
s3a
[ https://issues.apache.org/jira/browse/HADOOP-17894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17410754#comment-17410754 ]
Steve Loughran commented on HADOOP-17894:
-----------------------------------------
First part of PR, detect and report recursion, testable from CLI, but I can't replicate in ITests.
{code:java}
~/P/R/hadoop-3.4.0-SNAPSHOT bin/hadoop credential list -provider jceks://s3a@stevel-london/test/jceks/keystore.jceks
2021-09-06 18:33:03,834 [main] WARN fs.FileSystem (FileSystem.java:createFileSystem(3465)) - Failed to initialize fileystem s3a://stevel-london/test/jceks/keystore.jceks: java.io.IOException: Cannot find password option fs.s3a.bucket.stevel-london.fs.s3a.server-side-encryption-algorithm
java.io.IOException: Cannot find password option fs.s3a.bucket.stevel-london.fs.s3a.server-side-encryption-algorithm
at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:975)
at org.apache.hadoop.fs.s3a.S3AUtils.getPassword(S3AUtils.java:955)
at org.apache.hadoop.fs.s3a.S3AUtils.getPassword(S3AUtils.java:936)
at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:914)
at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:879)
at org.apache.hadoop.fs.s3a.S3AUtils.getEncryptionAlgorithm(S3AUtils.java:1605)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:432)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3461)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:173)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3566)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3513)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:366)
at org.apache.hadoop.security.alias.KeyStoreProvider.initFileSystem(KeyStoreProvider.java:84)
at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.<init>(AbstractJavaKeyStoreProvider.java:85)
at org.apache.hadoop.security.alias.KeyStoreProvider.<init>(KeyStoreProvider.java:49)
at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:42)
at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:35)
at org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:68)
at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:89)
at org.apache.hadoop.security.alias.CredentialShell$Command.getCredentialProvider(CredentialShell.java:147)
at org.apache.hadoop.security.alias.CredentialShell$ListCommand.validate(CredentialShell.java:198)
at org.apache.hadoop.tools.CommandShell.run(CommandShell.java:71)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
at org.apache.hadoop.security.alias.CredentialShell.main(CredentialShell.java:529)
Caused by: java.io.IOException: Configuration problem with provider path.
at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2459)
at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2378)
at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:970)
... 24 more
Caused by: org.apache.hadoop.fs.PathIOException: `jceks://s3a@stevel-london/test/jceks/keystore.jceks': Recursive load of credential provider!
at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:85)
at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2440)
... 26 more
There are no valid (non-transient) providers configured.
No action has been taken. Use the -provider option to specify
a provider. If you want to use a transient provider then you
MUST use the -provider argument.
list [-provider provider-path] [-strict]:
The list subcommand displays the aliases contained within
a particular provider - as configured in core-site.xml or
indicated through the -provider argument. If -strict is supplied,
fail immediately if the provider requires a password and none is
provided.
{code}
> CredentialProviderFactory.getProviders() recursion loading JCEKS file from s3a
> ------------------------------------------------------------------------------
>
> Key: HADOOP-17894
> URL: https://issues.apache.org/jira/browse/HADOOP-17894
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: conf, fs/s3
> Affects Versions: 3.3.1
> Reporter: Steve Loughran
> Priority: Major
>
> {{CredentialProviderFactory.getProviders()}} will be called recursively if the FS being instantiated as a source of a JCECKs file calls {{Configuration.getPassword()}}.
> S3A FileSystem does this when looking at encryption settings, even when authentication
> is not being done by hadoop config options.
> Recursion is straightforward to detect, leaving a choice of actions
> # Fail
> # Fix by having getPassword() return null; this implicitly skips use of the credential providers for the nested FS instance.
> # fix by skipping the specific credential provider factory with the problem (warn first?)
> My preference is for #3.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org