You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Brandon (Jira)" <ji...@apache.org> on 2025/10/22 20:50:00 UTC
[jira] [Comment Edited] (HADOOP-19733) S3A: Credentials provider classes not found despite setting `fs.s3a.classloader.isolation` to `false`
[ https://issues.apache.org/jira/browse/HADOOP-19733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18032249#comment-18032249 ]
Brandon edited comment on HADOOP-19733 at 10/22/25 8:49 PM:
------------------------------------------------------------
I haven't contributed to Hadoop or other Apache projects before, but this approachable for a first contribution. I'll open a PR.
was (Author: brandonvin):
I haven't contributed to Hadoop or other Apache projects before, but this looks like a good fit for a first try. I'll open a PR.
> S3A: Credentials provider classes not found despite setting `fs.s3a.classloader.isolation` to `false`
> -----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-19733
> URL: https://issues.apache.org/jira/browse/HADOOP-19733
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.4.2
> Reporter: Brandon
> Priority: Minor
>
> HADOOP-18993 added the option `fs.s3a.classloader.isolation` to support, for example, a Spark job using an AWS credentials provider class that is bundled into the Spark job JAR. In testing this, the AWS credentials provider classes are still not found.
> I think the cause is:
> * `fs.s3a.classloader.isolation` is implemented by setting (or not setting) a classloader on the `Configuration`
> * However, code paths to load AWS credential provider call `S3AUtils.getInstanceFromReflection`, which uses the classloader that loaded the S3AUtils class. That's likely to be the built-in application classloader, which won't be able to load classes in a Spark job JAR.
> And the fix seems small:
> * Change `S3AUtils.getInstanceFromReflection` to load classes using the `Configuration`'s classloader. Luckily we already have the Configuration in this method.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org