You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Brandon (Jira)" <ji...@apache.org> on 2025/10/22 06:04:00 UTC

[jira] [Comment Edited] (HADOOP-18993) S3A: Add option fs.s3a.classloader.isolation (#6301)

    [ https://issues.apache.org/jira/browse/HADOOP-18993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18031961#comment-18031961 ] 

Brandon edited comment on HADOOP-18993 at 10/22/25 6:03 AM:
------------------------------------------------------------

Tried this with Hadoop 3.4.1. It appears the `fs.s3a.classloader.isolation` property does not have the desired effect, because the credentials providers are loaded using the classloader of S3AUtils class:

[CredentialProviderListFactory#createAWSV2CredentialProvider|https://github.com/apache/hadoop/blob/4f9efcb3bc10d11d6c81457bc51e59dfe0791041/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/CredentialProviderListFactory.java#L299-L305] calls [S3AUtils#getInstanceFromReflection|https://github.com/apache/hadoop/blob/4f9efcb3bc10d11d6c81457bc51e59dfe0791041/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java#L642], which uses the classloader that loaded the S3AUtils class, not the classloader set on the Configuration


was (Author: brandonvin):
Tried this with Hadoop 3.4.1. It appears the `fs.s3a.classloader.isolation` property does not have the desired effect, because the credentials providers are loaded using the classloader of S3AUtils class instead of the Configuration class:

[[https://github.com/apache/hadoop/blob/4f9efcb3bc10d11d6c81457bc51e59dfe0791041/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/CredentialProviderListFactory.java#L299-L305|http://example.com]|https://github.com/apache/hadoop/blob/4f9efcb3bc10d11d6c81457bc51e59dfe0791041/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/CredentialProviderListFactory.java#L299-L305]

[https://github.com/apache/hadoop/blob/4f9efcb3bc10d11d6c81457bc51e59dfe0791041/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java#L642] 

> S3A: Add option fs.s3a.classloader.isolation (#6301)
> ----------------------------------------------------
>
>                 Key: HADOOP-18993
>                 URL: https://issues.apache.org/jira/browse/HADOOP-18993
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>    Affects Versions: 3.3.6
>            Reporter: Antonio Murgia
>            Assignee: Antonio Murgia
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 3.4.0, 3.5.0, 3.4.1
>
>
> In HADOOP-17372 the S3AFileSystem forces the configuration classloader to be the same as the one that loaded S3AFileSystem. This leads to the impossibility in Spark applications to load third party credentials providers as user jars.
> The option fs.s3a.classloader.isolation (default: true) can be set to false to disable s3a classloader isolation;
> This can assist in using custom credential providers and other extension points.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org