[jira] [Created] (HADOOP-19733) S3A: Credentials provider classes not found despite setting `fs.s3a.classloader.isolation` to `false`

["Brandon (Jira)" <ji...@apache.org>]

Brandon created HADOOP-19733:
--------------------------------

             Summary: S3A: Credentials provider classes not found despite setting `fs.s3a.classloader.isolation` to `false`
                 Key: HADOOP-19733
                 URL: https://issues.apache.org/jira/browse/HADOOP-19733
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs/s3
    Affects Versions: 3.4.2
            Reporter: Brandon


HADOOP-18993 added the option `fs.s3a.classloader.isolation` to support, for example, a Spark job using an AWS credentials provider class that is bundled into the Spark job JAR. In testing this, the AWS credentials provider classes are still not found.

I think the cause is:
 * `fs.s3a.classloader.isolation` is implemented by setting (or not setting) a classloader on the `Configuration`
 * However, code paths to load AWS credential provider call `S3AUtils.getInstanceFromReflection`, which uses the classloader that loaded the S3AUtils class. That's likely to be the built-in application classloader, which won't be able to load classes in a Spark job JAR.

And the fix seems small:
 * Change `S3AUtils.getInstanceFromReflection` to load classes using the `Configuration`'s classloader. Luckily we already have the Configuration in this method.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org