[jira] [Created] (HADOOP-16567) S3A Secret access to fall back to XML if credential provider raises IOE.

["Steve Loughran (Jira)" <ji...@apache.org>]

Steve Loughran created HADOOP-16567:
---------------------------------------

             Summary: S3A Secret access to fall back to XML if credential provider raises IOE.
                 Key: HADOOP-16567
                 URL: https://issues.apache.org/jira/browse/HADOOP-16567
             Project: Hadoop Common
          Issue Type: Sub-task
          Components: fs/s3
    Affects Versions: 3.1.2
            Reporter: Steve Loughran


This is hive related. Hive can put secrets into a JCEKS file which only hive may read.

S3A file systems created on behalf of a user do not have access to this file. Yet it is listed as the credential provider in the hadoop.credential.providers option in core-site -and that is marked as final. When the S3 a initializre() method looks up passwords and encryption keys, the failure to open the file raises an IOE -and the FS cannot be instantiated.

Proposed: {{S3AUtils.lookupPassword()}} to catch such exceptions, and fall back to using {{Configuration.get()}} and so retrieve any property in the XML. If there is one failing here, it is if the user did want to read from a credential provider, the failure to read the credential will be lost, and the filesystem will simply get the default value.

There is a side issue, that permission exceptions can surface as found not found exceptions, which are then wrapped as generic IOEs in Configuration. It will be hard and brittle to attempt to only respond to permission restrictions. We could look at improving {{Configuration.getPassword()}} but that class is so widely used, I am not in a rush to break things.

I think this means we have to add another option. Trying to be clever about when to fall back versus when to rethrow the exception is doomed.

If this works for S3A, we will need to consider replicating it for ABFS. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org