You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "stoty (via GitHub)" <gi...@apache.org> on 2025/04/01 08:32:34 UTC
[PR] HDFS-17767. Skip tests that depend on custom SecurityManager when Java doesn't support it [hadoop]
stoty opened a new pull request, #7566:
URL: https://github.com/apache/hadoop/pull/7566
### Description of PR
Skip tests that depend on custom SecurityManager when Java doesn't support it
Required for recent Java versions.
### How was this patch tested?
Run on Java 24 on my Java 24 branch.
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17767. Skip tests that depend on custom SecurityManager when Java doesn't support it [hadoop]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #7566:
URL: https://github.com/apache/hadoop/pull/7566#discussion_r2061500341
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:
##########
@@ -574,28 +575,36 @@ public void run() {
};
//use SecurityManager to pause the copying of f1 and begin copying f2
- SecurityManager sm = System.getSecurityManager();
- System.out.println("SecurityManager = " + sm);
- System.setSecurityManager(new SecurityManager() {
- private boolean firstTime = true;
+ SecurityManager sm = null;
Review Comment:
Back in 2007, DFSClient won't create a file in NN before flushing the first block. As a result, the client 2 below will succeed but client 1 will fail with a file already created exception. We want s client 1 succeeds and client 2 fails since client 1 has started earlier.
1. Client 1 starts writing to a file but not yet flushes the first block.
2. After that client 2 starts writing to the same file and also flushes the first block.
[HADOOP-999](https://issues.apache.org/jira/browse/HADOOP-999) was to changing DFSClient to create the file in NN in the beginning of write. The test was to use SecurityManager to pause client 1 in order to delay flushing the first block.
Just check the existing test (before this PR), it actually not working since it tries to search "FileUtil.copyContent" in the stack trace but the `FileUtil.copyContent` method no longer exists, i.e. the search always fails.
I suggest simply changing the test as below
```diff
@@ -556,52 +586,25 @@ public void testPut() throws IOException {
final Path root = mkdir(dfs, new Path("/testPut"));
final Path dst = new Path(root, "dst");
- show("begin");
-
- final Thread copy2ndFileThread = new Thread() {
- @Override
- public void run() {
- try {
- show("copy local " + f2 + " to remote " + dst);
- dfs.copyFromLocalFile(false, false, new Path(f2.getPath()), dst);
- } catch (IOException ioe) {
- show("good " + StringUtils.stringifyException(ioe));
- return;
- }
- //should not be here, must got IOException
- assertTrue(false);
+ final String hello = "hello";
+ try (FSDataOutputStream out = dfs.create(dst, false)) {
+ // It should fail to create a new client writing to the same file.
+ try(DFSClient client = new DFSClient(dfs.getUri(), dfs.getConf())) {
+ final RemoteException e = Assertions.assertThrows(RemoteException.class,
+ () -> client.create(dst.toString(), false));
+ LOG.info("GOOD", e);
+ Assertions.assertEquals(AlreadyBeingCreatedException.class.getName(), e.getClassName());
}
- };
-
- //use SecurityManager to pause the copying of f1 and begin copying f2
- SecurityManager sm = System.getSecurityManager();
- System.out.println("SecurityManager = " + sm);
- System.setSecurityManager(new SecurityManager() {
- private boolean firstTime = true;
- @Override
- public void checkPermission(Permission perm) {
- if (firstTime) {
- Thread t = Thread.currentThread();
- if (!t.toString().contains("DataNode")) {
- String s = "" + Arrays.asList(t.getStackTrace());
- if (s.contains("FileUtil.copyContent")) {
- //pause at FileUtil.copyContent
-
- firstTime = false;
- copy2ndFileThread.start();
- try {Thread.sleep(5000);} catch (InterruptedException e) {}
- }
- }
- }
- }
- });
- show("copy local " + f1 + " to remote " + dst);
- dfs.copyFromLocalFile(false, false, new Path(f1.getPath()), dst);
- show("done");
+ // It should succeed to continue writing to the file.
+ out.writeUTF(hello);
+ }
- try {copy2ndFileThread.join();} catch (InterruptedException e) { }
- System.setSecurityManager(sm);
+ // Verify the file content.
+ try (FSDataInputStream in = dfs.open(dst)) {
+ final String read = in.readUTF();
+ assertEquals(hello, read);
+ }
// copy multiple files to destination directory
final Path destmultiple = mkdir(dfs, new Path(root, "putmultiple"));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17767. Skip tests that depend on custom SecurityManager when Java doesn't support it [hadoop]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #7566:
URL: https://github.com/apache/hadoop/pull/7566#issuecomment-2781898344
@ayushtkn @Hexiaoqiao Could you please review this pr? Thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17767. Skip tests that depend on custom SecurityManager when Java doesn't support it [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7566:
URL: https://github.com/apache/hadoop/pull/7566#issuecomment-2842826667
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 49s | | trunk passed |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 19s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 40s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 32s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 6s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 1m 2s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 2s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 56s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7566/3/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 130 unchanged - 4 fixed = 131 total (was 134) |
| +1 :green_heart: | mvnsite | 1m 7s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 30s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 14s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 256m 22s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7566/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not generate ASF License warnings. |
| | | 386m 39s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.viewfs.TestViewFileSystemLinkMergeSlash |
| | hadoop.fs.viewfs.TestViewFsAtHdfsRoot |
| | hadoop.fs.viewfs.TestViewFileSystemHdfs |
| | hadoop.fs.TestWebHdfsFileContextMainOperations |
| | hadoop.fs.viewfs.TestViewFsHdfs |
| | hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
| | hadoop.fs.viewfs.TestViewFileSystemOverloadSchemeHdfsFileSystemContract |
| | hadoop.hdfs.server.blockmanagement.TestBlockReportLease |
| | hadoop.fs.viewfs.TestViewFileSystemLinkRegex |
| | hadoop.fs.TestSWebHdfsFileContextMainOperations |
| | hadoop.fs.TestHDFSFileContextMainOperations |
| | hadoop.fs.viewfs.TestViewFileSystemLinkFallback |
| | hadoop.hdfs.server.blockmanagement.TestDatanodeManager |
| | hadoop.hdfs.TestDatanodeRegistration |
| | hadoop.fs.TestFcHdfsCreateMkdir |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.49 ServerAPI=1.49 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7566/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7566 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 17fb10fc7448 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 270d392394a187c23c24337b883a67f30bab0e7f |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7566/3/testReport/ |
| Max. process+thread count | 3203 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7566/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17767. Skip tests that depend on custom SecurityManager when Java doesn't support it [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7566:
URL: https://github.com/apache/hadoop/pull/7566#issuecomment-2768949930
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 9s | | trunk passed |
| +1 :green_heart: | compile | 1m 20s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 11s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 16s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 43s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 8s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 8s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 1s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 1s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 58s | | hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 132 unchanged - 2 fixed = 132 total (was 134) |
| +1 :green_heart: | mvnsite | 1m 9s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 36s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 23s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 33s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not generate ASF License warnings. |
| | | 131m 4s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7566/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7566 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3948f719ec9c 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2b9a26a001352cbb4931476c93e2b8b62c8c4480 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7566/1/testReport/ |
| Max. process+thread count | 555 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7566/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17767. Skip tests that depend on custom SecurityManager when Java doesn't support it [hadoop]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on code in PR #7566:
URL: https://github.com/apache/hadoop/pull/7566#discussion_r2032386035
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:
##########
@@ -576,26 +577,33 @@ public void run() {
//use SecurityManager to pause the copying of f1 and begin copying f2
SecurityManager sm = System.getSecurityManager();
Review Comment:
Thank you, I've moved getSecurityManager inside the try block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17767. Skip tests that depend on custom SecurityManager when Java doesn't support it [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7566:
URL: https://github.com/apache/hadoop/pull/7566#issuecomment-2853838506
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 19s | | trunk passed |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 11s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 17s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 39s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 9s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 9s | | the patch passed |
| +1 :green_heart: | compile | 1m 11s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 2s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 2s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 57s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7566/4/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 130 unchanged - 4 fixed = 131 total (was 134) |
| +1 :green_heart: | mvnsite | 1m 8s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 32s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 39s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 11m 53s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 141m 32s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.49 ServerAPI=1.49 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7566/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7566 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b9bdb48a8853 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 36f0dc4550041133a136f8c988c863cc472702dc |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7566/4/testReport/ |
| Max. process+thread count | 1371 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7566/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17767. Skip tests that depend on custom SecurityManager when Java doesn't support it [hadoop]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #7566:
URL: https://github.com/apache/hadoop/pull/7566#issuecomment-2841663766
I have applied @szetszwo 's changes.
Please check again @cnauroth .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17767. Skip tests that depend on custom SecurityManager when Java doesn't support it [hadoop]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #7566:
URL: https://github.com/apache/hadoop/pull/7566#discussion_r2061500341
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:
##########
@@ -574,28 +575,36 @@ public void run() {
};
//use SecurityManager to pause the copying of f1 and begin copying f2
- SecurityManager sm = System.getSecurityManager();
- System.out.println("SecurityManager = " + sm);
- System.setSecurityManager(new SecurityManager() {
- private boolean firstTime = true;
+ SecurityManager sm = null;
Review Comment:
Back in 2007, DFSClient won't create a file in NN before flushing the first block. As a result, the client 2 below will succeed but client 1 will fail with a file already created exception. We want s client 1 succeeds and client 2 fails since client 1 has started earlier.
1. Client 1 starts writing to a file but not yet flushes the first block.
2. After that, client 2 starts writing to the same file and also flushes the first block.
[HADOOP-999](https://issues.apache.org/jira/browse/HADOOP-999) was to change DFSClient to create the file in NN in the beginning of write. The test was to use SecurityManager to pause client 1 in order to delay flushing the first block.
Just check the existing test (before this PR), it actually not working since it tries to search "FileUtil.copyContent" in the stack trace but the `FileUtil.copyContent` method no longer exists, i.e. the search always fails.
I suggest simply changing the test as below
```diff
@@ -556,52 +586,25 @@ public void testPut() throws IOException {
final Path root = mkdir(dfs, new Path("/testPut"));
final Path dst = new Path(root, "dst");
- show("begin");
-
- final Thread copy2ndFileThread = new Thread() {
- @Override
- public void run() {
- try {
- show("copy local " + f2 + " to remote " + dst);
- dfs.copyFromLocalFile(false, false, new Path(f2.getPath()), dst);
- } catch (IOException ioe) {
- show("good " + StringUtils.stringifyException(ioe));
- return;
- }
- //should not be here, must got IOException
- assertTrue(false);
+ final String hello = "hello";
+ try (FSDataOutputStream out = dfs.create(dst, false)) {
+ // It should fail to create a new client writing to the same file.
+ try(DFSClient client = new DFSClient(dfs.getUri(), dfs.getConf())) {
+ final RemoteException e = Assertions.assertThrows(RemoteException.class,
+ () -> client.create(dst.toString(), false));
+ LOG.info("GOOD", e);
+ Assertions.assertEquals(AlreadyBeingCreatedException.class.getName(), e.getClassName());
}
- };
-
- //use SecurityManager to pause the copying of f1 and begin copying f2
- SecurityManager sm = System.getSecurityManager();
- System.out.println("SecurityManager = " + sm);
- System.setSecurityManager(new SecurityManager() {
- private boolean firstTime = true;
- @Override
- public void checkPermission(Permission perm) {
- if (firstTime) {
- Thread t = Thread.currentThread();
- if (!t.toString().contains("DataNode")) {
- String s = "" + Arrays.asList(t.getStackTrace());
- if (s.contains("FileUtil.copyContent")) {
- //pause at FileUtil.copyContent
-
- firstTime = false;
- copy2ndFileThread.start();
- try {Thread.sleep(5000);} catch (InterruptedException e) {}
- }
- }
- }
- }
- });
- show("copy local " + f1 + " to remote " + dst);
- dfs.copyFromLocalFile(false, false, new Path(f1.getPath()), dst);
- show("done");
+ // It should succeed to continue writing to the file.
+ out.writeUTF(hello);
+ }
- try {copy2ndFileThread.join();} catch (InterruptedException e) { }
- System.setSecurityManager(sm);
+ // Verify the file content.
+ try (FSDataInputStream in = dfs.open(dst)) {
+ final String read = in.readUTF();
+ assertEquals(hello, read);
+ }
// copy multiple files to destination directory
final Path destmultiple = mkdir(dfs, new Path(root, "putmultiple"));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17767. Skip tests that depend on custom SecurityManager when Java doesn't support it [hadoop]
Posted by "Hexiaoqiao (via GitHub)" <gi...@apache.org>.
Hexiaoqiao commented on code in PR #7566:
URL: https://github.com/apache/hadoop/pull/7566#discussion_r2031250158
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:
##########
@@ -576,26 +577,33 @@ public void run() {
//use SecurityManager to pause the copying of f1 and begin copying f2
SecurityManager sm = System.getSecurityManager();
Review Comment:
Thanks to involve me here. As #7567 mentioned, "System.getSecurityManager() will also be one Deprecate APIs based on JEP-411, Would you mind to process it together." Thanks. Others make sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17767. Skip tests that depend on custom SecurityManager when Java doesn't support it [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7566:
URL: https://github.com/apache/hadoop/pull/7566#issuecomment-2785500013
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 14m 28s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 52s | | trunk passed |
| +1 :green_heart: | compile | 1m 20s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 9s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 18s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 42s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 6s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 6s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 1m 2s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 2s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 56s | | hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 132 unchanged - 2 fixed = 132 total (was 134) |
| +1 :green_heart: | mvnsite | 1m 9s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 33s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 15s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 36s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not generate ASF License warnings. |
| | | 145m 9s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7566/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7566 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d39d13cafdec 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3b2195425fc053c863df4c16d8cab0597ccfc910 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7566/2/testReport/ |
| Max. process+thread count | 553 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7566/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17767. Skip tests that depend on custom SecurityManager when Java doesn't support it [hadoop]
Posted by "cnauroth (via GitHub)" <gi...@apache.org>.
cnauroth commented on code in PR #7566:
URL: https://github.com/apache/hadoop/pull/7566#discussion_r2061005400
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeRegistration.java:
##########
@@ -79,8 +80,12 @@ public void checkConnect(String host, int port) {
@Test
public void testDNSLookups() throws Exception {
MonitorDNS sm = new MonitorDNS();
- System.setSecurityManager(sm);
-
+ try {
+ System.setSecurityManager(sm);
+ } catch (UnsupportedOperationException e) {
+ assumeTrue("Test is skipped because SecurityManager cannot be set (JEP 411)", false);
Review Comment:
I would prefer to not lose test coverage, but I don't have a good idea for a different way to implement this.
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:
##########
@@ -574,28 +575,36 @@ public void run() {
};
//use SecurityManager to pause the copying of f1 and begin copying f2
- SecurityManager sm = System.getSecurityManager();
- System.out.println("SecurityManager = " + sm);
- System.setSecurityManager(new SecurityManager() {
- private boolean firstTime = true;
+ SecurityManager sm = null;
Review Comment:
I am a little unclear on what this part of the test is trying to achieve. @szetszwo , I think you introduced this all the way back in [HADOOP-999](https://issues.apache.org/jira/browse/HADOOP-999)! 😄 I don't want to lose test coverage, but I'm wondering if it's safe to remove the part of the code that works with `SecurityManager`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17767. Skip tests that depend on custom SecurityManager when Java doesn't support it [hadoop]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on code in PR #7566:
URL: https://github.com/apache/hadoop/pull/7566#discussion_r2068464042
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeRegistration.java:
##########
@@ -79,8 +80,12 @@ public void checkConnect(String host, int port) {
@Test
public void testDNSLookups() throws Exception {
MonitorDNS sm = new MonitorDNS();
- System.setSecurityManager(sm);
-
+ try {
+ System.setSecurityManager(sm);
+ } catch (UnsupportedOperationException e) {
+ assumeTrue("Test is skipped because SecurityManager cannot be set (JEP 411)", false);
Review Comment:
Neither did I.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org