You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "ajfabbri (via GitHub)" <gi...@apache.org> on 2026/05/11 19:28:53 UTC
[PR] HADOOP-19890: s3a: fix isolated classloader test for jdk >=17 [hadoop]
ajfabbri opened a new pull request, #8491:
URL: https://github.com/apache/hadoop/pull/8491
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
The S3A isolated classloader test appears to be broken with newer JDKs'
restrictions on reflection access. The test fails with
`java.lang.IllegalAccessException: Attempt to lookup caller-sensitive method
using restricted lookup object`
### How was this patch tested?
```
./mvnw -pl :hadoop-aws -am -Dtest=none -Dit.test=ITestS3AFileSystemIsolatedClassloader.java clean verify
```
with Java 17 and 21.
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [x] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [na] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [na] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [na] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [x] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19890: s3a: fix isolated classloader test for jdk >=17 [hadoop]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #8491:
URL: https://github.com/apache/hadoop/pull/8491#issuecomment-4444207150
I run the command with both JDK 17 and 25, and I think the test does not involve a platform-dependent code path, so the OS should be irrelevant. The failed GHA job runs a different command than the one provided in the Jira and PR description. I will take a look tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19890: s3a: fix isolated classloader test for jdk >=17 [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8491:
URL: https://github.com/apache/hadoop/pull/8491#issuecomment-4425571979
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 50m 37s | | trunk passed |
| +1 :green_heart: | compile | 1m 8s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | compile | 1m 12s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | checkstyle | 1m 16s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 21s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javadoc | 1m 22s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | spotbugs | 1m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 31s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 28s | | hadoop-tools/hadoop-aws: The patch generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) |
| +1 :green_heart: | mvnsite | 0m 43s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | spotbugs | 1m 24s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 53s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 28s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not generate ASF License warnings. |
| | | 143m 3s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8491/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8491 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux ced61c76316c 5.15.0-174-generic #184-Ubuntu SMP Fri Mar 13 18:41:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 542ce78bd7590b6a52ea0bb9f29d645c4d96198a |
| Default Java | Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8491/2/testReport/ |
| Max. process+thread count | 570 (vs. ulimit of 10000) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8491/2/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19890: s3a: fix isolated classloader test for jdk >=17 [hadoop]
Posted by "ajfabbri (via GitHub)" <gi...@apache.org>.
ajfabbri commented on PR #8491:
URL: https://github.com/apache/hadoop/pull/8491#issuecomment-4455036394
> The failed GHA job runs a different command than the one provided in the Jira and PR description
Thanks for taking a look @pan3793 .. I'm guessing these are different default JVM options? The command I pasted in the Jira is the exact one I used to reproduce it locally. I'm using openjdk there.
Either way, this is low risk, being test code. Also Steve reviewed it--and he has done a lot more with reflection than I have. 😄 I'll merge this but feel free to comment if you have other thoughts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19890: s3a: fix isolated classloader test for jdk >=17 [hadoop]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #8491:
URL: https://github.com/apache/hadoop/pull/8491#discussion_r3221646534
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileSystemIsolatedClassloader.java:
##########
@@ -60,19 +59,25 @@ public AwsCredentials resolveCredentials() {
}
private static class CustomClassLoader extends ClassLoader {
- }
+ private final ClassLoader parent;
- private final ClassLoader customClassLoader = spy(new CustomClassLoader());
- {
- try {
- doReturn(CustomCredentialsProvider.class)
- .when(customClassLoader)
- .loadClass(customClassName);
- } catch (ClassNotFoundException ex) {
- throw new RuntimeException(ex);
+ CustomClassLoader(ClassLoader parent) {
+ super(parent);
+ this.parent = parent;
+ }
+
+ @Override
+ public Class<?> loadClass(String name) throws ClassNotFoundException {
+ if (customClassName.equals(name)) {
+ return CustomCredentialsProvider.class;
+ }
+ return parent.loadClass(name);
}
Review Comment:
Overriding `ClassLoader#loadClass(String)` and delegating directly to `parent.loadClass(name)` bypasses the standard `ClassLoader` loading flow (notably the built-in locking/caching path in `ClassLoader#loadClass(...)`). Since you already call `super(parent)`, you can drop the redundant `parent` field and delegate the non-special case to `super.loadClass(name)` (or `super.loadClass(name, false)` if you choose to override the protected method) to preserve standard classloading semantics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19890: s3a: fix isolated classloader test for jdk >=17 [hadoop]
Posted by "ajfabbri (via GitHub)" <gi...@apache.org>.
ajfabbri merged PR #8491:
URL: https://github.com/apache/hadoop/pull/8491
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19890: s3a: fix isolated classloader test for jdk >=17 [hadoop]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #8491:
URL: https://github.com/apache/hadoop/pull/8491#issuecomment-4427010390
seems I can not reproduce the test failure using the provided command
> ```
> ./mvnw -pl :hadoop-aws -am -Dtest=none -Dit.test=ITestS3AFileSystemIsolatedClassloader.java clean verify
> ```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19890: s3a: fix isolated classloader test for jdk >=17 [hadoop]
Posted by "ajfabbri (via GitHub)" <gi...@apache.org>.
ajfabbri commented on PR #8491:
URL: https://github.com/apache/hadoop/pull/8491#issuecomment-4443516866
> seems I can not reproduce the test failure using the provided command
Strange. Which Java version are you using? I think I failed to reproduce it once. Wonder if there are different JVM options enabled or something?
You can see this failure in CI here: [hadoop/actions/runs/25702760695/job/75466573534](https://github.com/apache/hadoop/actions/runs/25702760695/job/75466573534?pr=8467). This is why I want to merge that S3A CI PR before this one. Assuming my test modifications are correct, removing a fragile mock in favor of plain java overrides is a win here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19890: s3a: fix isolated classloader test for jdk >=17 [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8491:
URL: https://github.com/apache/hadoop/pull/8491#issuecomment-4425064019
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 46s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | checkstyle | 0m 44s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javadoc | 0m 38s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | spotbugs | 1m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 51s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 0m 23s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javac | 0m 23s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 16s | | hadoop-tools/hadoop-aws: The patch generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) |
| +1 :green_heart: | mvnsite | 0m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | spotbugs | 0m 53s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 49s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 32s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 26s | | The patch does not generate ASF License warnings. |
| | | 78m 41s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8491/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8491 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 8b0978a402a8 5.15.0-173-generic #183-Ubuntu SMP Fri Mar 6 13:29:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bd825f637278e6036cd860d69d33ec11dd2627db |
| Default Java | Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8491/1/testReport/ |
| Max. process+thread count | 616 (vs. ulimit of 10000) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8491/1/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org