You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/02/24 07:44:22 UTC
[GitHub] [hadoop] virajjasani opened a new pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
virajjasani opened a new pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028
### Description of PR
We should provide support for clients to set Http and Rpc ports of JournalNodes in MiniJournalCluster.
### How was this patch tested?
Using a downstreamer application as well as using UT.
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
virajjasani commented on pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#issuecomment-1054211465
> Hi @virajjasani , will this fixed port increase the probability of port occupation, resulting in the interruption of unit test?
@tomscut only one UT is going to use this ability, rest of the UTs are still going to use default mode (i.e. random port usage). Hence this is more like feature for JN downstream applications.
Thanks for taking a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#discussion_r818756871
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
##########
@@ -52,4 +60,92 @@ public void testStartStop() throws IOException {
c.shutdown();
}
}
+
+ @Test
+ public void testStartStopWithPorts() throws Exception {
+ Configuration conf = new Configuration();
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of http ports (1) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481).build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of rpc ports (2) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setRpcPorts(8481, 8482).build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of rpc ports (1) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 10000).setRpcPorts(8481)
+ .build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of http ports (4) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 1000, 2000)
+ .setRpcPorts(8481, 8482, 8483).build();
+ });
+
+ final Set<Integer> httpAndRpcPorts = NetUtils.getFreeSocketPorts(6);
+ LOG.info("Free socket ports: {}", httpAndRpcPorts);
+
+ final int[] httpPorts = new int[3];
+ final int[] rpcPorts = new int[3];
+ int httpPortIdx = 0;
+ int rpcPortIdx = 0;
+ for (Integer httpAndRpcPort : httpAndRpcPorts) {
+ if (httpPortIdx < 3) {
+ httpPorts[httpPortIdx++] = httpAndRpcPort;
+ } else {
+ rpcPorts[rpcPortIdx++] = httpAndRpcPort;
+ }
+ }
+
+ LOG.info("Http ports selected: {}", httpPorts);
+ LOG.info("Rpc ports selected: {}", rpcPorts);
+
+ for (int i = 0; i < 3; i++) {
+ assertNotEquals(0, rpcPorts[i]);
+ assertNotEquals(0, httpPorts[i]);
+ }
Review comment:
Done, Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#discussion_r817419407
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
##########
@@ -52,4 +59,82 @@ public void testStartStop() throws IOException {
c.shutdown();
}
}
+
+ @Test
+ public void testStartStopWithPorts() throws Exception {
+ Configuration conf = new Configuration();
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of http ports (1) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481).build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of rpc ports (2) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setRpcPorts(8481, 8482).build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of rpc ports (1) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 10000).setRpcPorts(8481)
+ .build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of http ports (4) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 1000, 2000)
+ .setRpcPorts(8481, 8482, 8483).build();
+ });
+
+ final int[] httpPorts = new int[] { NetUtils.getFreeSocketPort(), NetUtils.getFreeSocketPort(),
+ NetUtils.getFreeSocketPort() };
+ final int[] rpcPorts = new int[] { NetUtils.getFreeSocketPort(), NetUtils.getFreeSocketPort(),
+ NetUtils.getFreeSocketPort() };
Review comment:
@ayushtkn @tomscut There is nothing wrong with providing the utility as mentioned by Ayush above. I believe it is better to fail the test if we can not get 6 free ports for this UT and once we do get free ports, we can go ahead with the UT. I will make the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani edited a comment on pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
virajjasani edited a comment on pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#issuecomment-1056772509
@ayushtkn With the latest changes, here is the result of a stretch test on MacOS:
```
java.lang.IllegalStateException: 50000 free ports could not be acquired.
at org.apache.hadoop.net.NetUtils.getFreeSocketPorts(NetUtils.java:1072)
at org.apache.hadoop.hdfs.qjournal.TestMiniJournalCluster.testStartStopWithPorts(TestMiniJournalCluster.java:102)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
```
The patch used:
```
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
index daff40d898f..d35536a37a0 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
@@ -99,7 +99,7 @@ public void testStartStopWithPorts() throws Exception {
.setRpcPorts(8481, 8482, 8483).build();
});
- final Set<Integer> httpAndRpcPorts = NetUtils.getFreeSocketPorts(6);
+ final Set<Integer> httpAndRpcPorts = NetUtils.getFreeSocketPorts(50000);
LOG.info("Free socket ports: {}", httpAndRpcPorts);
final int[] httpPorts = new int[3];
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#issuecomment-1058708707
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 35s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 3s | | trunk passed |
| +1 :green_heart: | compile | 24m 27s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 48s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 48s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 13s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 16s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 17s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 9s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 18s | | the patch passed |
| +1 :green_heart: | compile | 23m 39s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 23m 39s | | the patch passed |
| +1 :green_heart: | compile | 20m 49s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 3m 45s | | root: The patch generated 0 new + 23 unchanged - 1 fixed = 23 total (was 24) |
| +1 :green_heart: | mvnsite | 3m 10s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 15s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 19s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 15s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 29s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 322m 47s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 58s | | The patch does not generate ASF License warnings. |
| | | 560m 8s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4028 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 337880d17ba8 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 84e430e4cc436387f66b860d49a6d6d14b7378e3 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/8/testReport/ |
| Max. process+thread count | 1910 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/8/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#issuecomment-1050004062
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 5s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 34s | | trunk passed |
| +1 :green_heart: | compile | 1m 47s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 37s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 28m 57s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 20s | | the patch passed |
| +1 :green_heart: | compile | 1m 26s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 26s | | the patch passed |
| +1 :green_heart: | compile | 1m 25s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 53s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2) |
| +1 :green_heart: | mvnsite | 1m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 31s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 45s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 45s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 376m 16s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not generate ASF License warnings. |
| | | 492m 43s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.mover.TestMover |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4028 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 1e8e24d1b39f 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 948305f889420b9b502a9777879227178171cf7b |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/1/testReport/ |
| Max. process+thread count | 2135 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#discussion_r816829536
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
##########
@@ -52,4 +58,86 @@ public void testStartStop() throws IOException {
c.shutdown();
}
}
+
+ @Test
+ public void testStartStopWithPorts() throws Exception {
+ Configuration conf = new Configuration();
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of http ports (1) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481).build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of rpc ports (2) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setRpcPorts(8481, 8482).build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of rpc ports (1) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 10000).setRpcPorts(8481)
+ .build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of http ports (4) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 1000, 2000)
+ .setRpcPorts(8481, 8482, 8483).build();
+ });
+
+ final int[] httpPorts = new int[3];
+ final int[] rpcPorts = new int[3];
+ try (MiniJournalCluster miniJournalCluster = new MiniJournalCluster.Builder(conf).build()) {
+ miniJournalCluster.waitActive();
+
+ for (int i = 0; i < 3; i++) {
+ httpPorts[i] = miniJournalCluster.getJournalNode(i).getHttpAddress().getPort();
+ }
+
+ for (int i = 0; i < 3; i++) {
+ rpcPorts[i] = miniJournalCluster.getJournalNode(i).getRpcServer().getAddress().getPort();
+ }
+ }
+
+ LOG.info("Http ports selected: {}", httpPorts);
+ LOG.info("Rpc ports selected: {}", rpcPorts);
Review comment:
Sounds good. Done, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#discussion_r816932592
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
##########
@@ -52,4 +59,82 @@ public void testStartStop() throws IOException {
c.shutdown();
}
}
+
+ @Test
+ public void testStartStopWithPorts() throws Exception {
+ Configuration conf = new Configuration();
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of http ports (1) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481).build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of rpc ports (2) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setRpcPorts(8481, 8482).build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of rpc ports (1) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 10000).setRpcPorts(8481)
+ .build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of http ports (4) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 1000, 2000)
+ .setRpcPorts(8481, 8482, 8483).build();
+ });
+
+ final int[] httpPorts = new int[] { NetUtils.getFreeSocketPort(), NetUtils.getFreeSocketPort(),
+ NetUtils.getFreeSocketPort() };
+ final int[] rpcPorts = new int[] { NetUtils.getFreeSocketPort(), NetUtils.getFreeSocketPort(),
+ NetUtils.getFreeSocketPort() };
Review comment:
I see this output:
```
2022-03-01 20:08:22,504 [Listener at localhost/58862] INFO qjournal.TestMiniJournalCluster (TestMiniJournalCluster.java:testStartStopWithPorts(106)) - Http ports selected: [58873, 58874, 58875]
2022-03-01 20:08:22,504 [Listener at localhost/58862] INFO qjournal.TestMiniJournalCluster (TestMiniJournalCluster.java:testStartStopWithPorts(107)) - Rpc ports selected: [58876, 58877, 58878]
2022-03-01 21:53:55,768 [Listener at localhost/59292] INFO qjournal.TestMiniJournalCluster (TestMiniJournalCluster.java:testStartStopWithPorts(106)) - Http ports selected: [59302, 59303, 59304]
2022-03-01 21:53:55,768 [Listener at localhost/59292] INFO qjournal.TestMiniJournalCluster (TestMiniJournalCluster.java:testStartStopWithPorts(107)) - Rpc ports selected: [59305, 59306, 59307]
```
And so on..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani edited a comment on pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
virajjasani edited a comment on pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#issuecomment-1056772509
@ayushtkn @tomscut With the latest changes, here is the result of a stretch test on MacOS:
```
java.lang.IllegalStateException: 50000 free ports could not be acquired.
at org.apache.hadoop.net.NetUtils.getFreeSocketPorts(NetUtils.java:1072)
at org.apache.hadoop.hdfs.qjournal.TestMiniJournalCluster.testStartStopWithPorts(TestMiniJournalCluster.java:102)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
```
The patch used:
```
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
index daff40d898f..d35536a37a0 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
@@ -99,7 +99,7 @@ public void testStartStopWithPorts() throws Exception {
.setRpcPorts(8481, 8482, 8483).build();
});
- final Set<Integer> httpAndRpcPorts = NetUtils.getFreeSocketPorts(6);
+ final Set<Integer> httpAndRpcPorts = NetUtils.getFreeSocketPorts(50000);
LOG.info("Free socket ports: {}", httpAndRpcPorts);
final int[] httpPorts = new int[3];
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#discussion_r816694773
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
##########
@@ -52,4 +53,70 @@ public void testStartStop() throws IOException {
c.shutdown();
}
}
+
+ @Test
+ public void testStartStopWithPorts() throws IOException {
+ Configuration conf = new Configuration();
+
+ try {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481).build();
+ fail("Should not reach here");
+ } catch (IllegalArgumentException e) {
+ assertEquals("Num of http ports (1) should match num of JournalNodes (3)", e.getMessage());
+ }
+
+ try {
+ new MiniJournalCluster.Builder(conf).setRpcPorts(8481, 8482)
+ .build();
+ fail("Should not reach here");
+ } catch (IllegalArgumentException e) {
+ assertEquals("Num of rpc ports (2) should match num of JournalNodes (3)", e.getMessage());
+ }
+
+ try {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 10000).setRpcPorts(8481)
+ .build();
+ fail("Should not reach here");
+ } catch (IllegalArgumentException e) {
+ assertEquals("Num of rpc ports (1) should match num of JournalNodes (3)", e.getMessage());
+ }
+
+ try {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 1000, 2000)
+ .setRpcPorts(8481, 8482, 8483)
+ .build();
+ fail("Should not reach here");
+ } catch (IllegalArgumentException e) {
+ assertEquals("Num of http ports (4) should match num of JournalNodes (3)", e.getMessage());
+ }
+
+ MiniJournalCluster miniJournalCluster =
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481, 8482, 8483)
+ .setRpcPorts(8491, 8492, 8493).build();
Review comment:
Thanks. @ayushtkn I just wanted to see if this UT with hardcoded ports pass in Jenkins build and since it did, is it good enough for us to keep? If not, it's fine and I can try to randomize the ports generation (similar to default case) and assert that randomly chosen port is the one being used by MiniJournal cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on a change in pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#discussion_r816704591
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
##########
@@ -52,4 +53,70 @@ public void testStartStop() throws IOException {
c.shutdown();
}
}
+
+ @Test
+ public void testStartStopWithPorts() throws IOException {
+ Configuration conf = new Configuration();
+
+ try {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481).build();
+ fail("Should not reach here");
+ } catch (IllegalArgumentException e) {
+ assertEquals("Num of http ports (1) should match num of JournalNodes (3)", e.getMessage());
+ }
+
+ try {
+ new MiniJournalCluster.Builder(conf).setRpcPorts(8481, 8482)
+ .build();
+ fail("Should not reach here");
+ } catch (IllegalArgumentException e) {
+ assertEquals("Num of rpc ports (2) should match num of JournalNodes (3)", e.getMessage());
+ }
+
+ try {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 10000).setRpcPorts(8481)
+ .build();
+ fail("Should not reach here");
+ } catch (IllegalArgumentException e) {
+ assertEquals("Num of rpc ports (1) should match num of JournalNodes (3)", e.getMessage());
+ }
+
+ try {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 1000, 2000)
+ .setRpcPorts(8481, 8482, 8483)
+ .build();
+ fail("Should not reach here");
+ } catch (IllegalArgumentException e) {
+ assertEquals("Num of http ports (4) should match num of JournalNodes (3)", e.getMessage());
+ }
+
+ MiniJournalCluster miniJournalCluster =
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481, 8482, 8483)
+ .setRpcPorts(8491, 8492, 8493).build();
Review comment:
We can not rely completely on Jenkins, I remember seeing an issue with a RBF test, where we had a port hard-coded, and for some organisation, there internal build was failing for that test, because they had something running on that port.
May be first finding some free ports, and then putting them into this conf should do. Or if that isn't possible, second option is try some randomisation, get a set of some random ports in a specified range, if they work great, if not loop back find another set and so on for some specified amount of iterations.
If above two doesn't work, we can try some skipping mechanism, like ports are occupied so skip the test or so, But this would be the last and the worst thing to do.
And in Jenkins the tests run in parallel, so results might change depending on what tests are running together & what ports they randomly choose.
My general experience so far, such controversial tests don't fail in the actual PR, but in other folks PR. :-P
Better we play safe. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on a change in pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#discussion_r816760498
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
##########
@@ -52,4 +58,86 @@ public void testStartStop() throws IOException {
c.shutdown();
}
}
+
+ @Test
+ public void testStartStopWithPorts() throws Exception {
+ Configuration conf = new Configuration();
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of http ports (1) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481).build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of rpc ports (2) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setRpcPorts(8481, 8482).build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of rpc ports (1) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 10000).setRpcPorts(8481)
+ .build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of http ports (4) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 1000, 2000)
+ .setRpcPorts(8481, 8482, 8483).build();
+ });
+
+ final int[] httpPorts = new int[3];
+ final int[] rpcPorts = new int[3];
+ try (MiniJournalCluster miniJournalCluster = new MiniJournalCluster.Builder(conf).build()) {
+ miniJournalCluster.waitActive();
+
+ for (int i = 0; i < 3; i++) {
+ httpPorts[i] = miniJournalCluster.getJournalNode(i).getHttpAddress().getPort();
+ }
+
+ for (int i = 0; i < 3; i++) {
+ rpcPorts[i] = miniJournalCluster.getJournalNode(i).getRpcServer().getAddress().getPort();
+ }
+ }
+
+ LOG.info("Http ports selected: {}", httpPorts);
+ LOG.info("Rpc ports selected: {}", rpcPorts);
Review comment:
Is this to just get the free ports?
Like start the cluster with 0 -> get the ports -> Shutdown the cluster(Ports are free)-> Now use them explicitly
If that is the case, You can explore using ``NetUtils.getFreeSocketPort()``, that would fetch you free ports, which you can pass as arguments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#discussion_r816590867
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
##########
@@ -52,4 +53,70 @@ public void testStartStop() throws IOException {
c.shutdown();
}
}
+
+ @Test
+ public void testStartStopWithPorts() throws IOException {
+ Configuration conf = new Configuration();
+
+ try {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481).build();
+ fail("Should not reach here");
+ } catch (IllegalArgumentException e) {
+ assertEquals("Num of http ports (1) should match num of JournalNodes (3)", e.getMessage());
+ }
+
+ try {
+ new MiniJournalCluster.Builder(conf).setRpcPorts(8481, 8482)
+ .build();
+ fail("Should not reach here");
+ } catch (IllegalArgumentException e) {
+ assertEquals("Num of rpc ports (2) should match num of JournalNodes (3)", e.getMessage());
+ }
+
+ try {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 10000).setRpcPorts(8481)
+ .build();
+ fail("Should not reach here");
+ } catch (IllegalArgumentException e) {
+ assertEquals("Num of rpc ports (1) should match num of JournalNodes (3)", e.getMessage());
+ }
+
+ try {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 1000, 2000)
+ .setRpcPorts(8481, 8482, 8483)
+ .build();
+ fail("Should not reach here");
+ } catch (IllegalArgumentException e) {
+ assertEquals("Num of http ports (4) should match num of JournalNodes (3)", e.getMessage());
+ }
+
+ MiniJournalCluster miniJournalCluster =
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481, 8482, 8483)
+ .setRpcPorts(8491, 8492, 8493).build();
Review comment:
@tomscut this is the only UT that will use `setHttpPorts` and `setRpcPorts` to set custom ports and then asserts that correct ports are used, hence this UT should not collide with any other UTs. Besides, we must have some UT to ensure both `setHttpPorts` and `setRpcPorts` are working as expected, correct?
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#issuecomment-1050170528
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| -1 :x: | mvninstall | 20m 36s | [/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/2/artifact/out/branch-mvninstall-root.txt) | root in trunk failed. |
| -1 :x: | compile | 0m 27s | [/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/2/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt) | hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | compile | 0m 27s | [/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/2/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-hdfs in trunk failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -0 :warning: | checkstyle | 0m 26s | [/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/2/artifact/out/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt) | The patch fails to run checkstyle in hadoop-hdfs |
| -1 :x: | mvnsite | 0m 28s | [/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/2/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in trunk failed. |
| -1 :x: | javadoc | 0m 28s | [/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/2/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt) | hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | javadoc | 0m 27s | [/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/2/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-hdfs in trunk failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | spotbugs | 0m 28s | [/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/2/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in trunk failed. |
| +1 :green_heart: | shadedclient | 3m 17s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 22s | [/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/2/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch failed. |
| -1 :x: | compile | 0m 22s | [/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt) | hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | javac | 0m 22s | [/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt) | hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | compile | 0m 23s | [/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javac | 0m 23s | [/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 20s | [/buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/2/artifact/out/buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt) | The patch fails to run checkstyle in hadoop-hdfs |
| -1 :x: | mvnsite | 0m 22s | [/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/2/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch failed. |
| -1 :x: | javadoc | 0m 22s | [/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/2/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt) | hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | javadoc | 0m 22s | [/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/2/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | spotbugs | 0m 23s | [/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/2/artifact/out/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch failed. |
| +1 :green_heart: | shadedclient | 4m 27s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 0m 23s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch failed. |
| +0 :ok: | asflicense | 0m 24s | | ASF License check generated no output? |
| | | 34m 0s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4028 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 9ee192d7cf97 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5833956b6f9d4c3116897d027efe6d4cabdbd1c9 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/2/testReport/ |
| Max. process+thread count | 93 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/2/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#issuecomment-1050482214
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 51s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 2s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 47s | | trunk passed |
| +1 :green_heart: | compile | 2m 6s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 55s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 16s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 1s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 27s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 54s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 16s | | trunk passed |
| +1 :green_heart: | shadedclient | 32m 39s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 52s | | the patch passed |
| +1 :green_heart: | compile | 1m 59s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 59s | | the patch passed |
| +1 :green_heart: | compile | 1m 40s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 40s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 3s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 30s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 35s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 38s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 356m 37s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not generate ASF License warnings. |
| | | 485m 9s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4028 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux a8a1c7074c87 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5833956b6f9d4c3116897d027efe6d4cabdbd1c9 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/3/testReport/ |
| Max. process+thread count | 2083 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
tomscut commented on pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#issuecomment-1054899948
> > Hi @virajjasani , will this fixed port increase the probability of port occupation, resulting in the interruption of unit test?
>
> @tomscut only one UT is going to use this ability, rest of the UTs are still going to use default mode (i.e. random port usage). Hence this is more like feature for JN downstream applications. Thanks for taking a look.
Thank you @virajjasani for your explanation. I'm not sure whether the UT you mentioned can be verified in other ways.
For example:
1. Get a random port in the default way.
2. Fetch the port from the process for assert after the process starts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on a change in pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#discussion_r816970241
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
##########
@@ -52,4 +59,82 @@ public void testStartStop() throws IOException {
c.shutdown();
}
}
+
+ @Test
+ public void testStartStopWithPorts() throws Exception {
+ Configuration conf = new Configuration();
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of http ports (1) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481).build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of rpc ports (2) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setRpcPorts(8481, 8482).build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of rpc ports (1) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 10000).setRpcPorts(8481)
+ .build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of http ports (4) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 1000, 2000)
+ .setRpcPorts(8481, 8482, 8483).build();
+ });
+
+ final int[] httpPorts = new int[] { NetUtils.getFreeSocketPort(), NetUtils.getFreeSocketPort(),
+ NetUtils.getFreeSocketPort() };
+ final int[] rpcPorts = new int[] { NetUtils.getFreeSocketPort(), NetUtils.getFreeSocketPort(),
+ NetUtils.getFreeSocketPort() };
Review comment:
I am not sure about it, would need a second opinion, from the javadoc of the comment, it feels like it can return same ports.
And logically say port X is the only free port, will it not return port X, if called again. Not sure how does this behave in different OS.
Tried something like this:
```
HashSet<Integer> hs = new LinkedHashSet();
while(true) {
int port = NetUtils.getFreeSocketPort();
if(!hs.add(port)) {
throw new IOException("Dupe Port "+ port + " ports : " + hs + " size " + hs.size());
}
```
it threw me an exception on MacOs with higher hs.size and on Ubuntu with lesser. not sure if there is some logic behind it or not, or just machine specific. Let me see if I can ask some N/W expert internally
@tomscut in case you have any opinion here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#issuecomment-1055821013
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 9s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 6s | | trunk passed |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 4s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 39s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 40s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 20s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 20s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 51s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/5/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 1 unchanged - 1 fixed = 4 total (was 2) |
| +1 :green_heart: | mvnsite | 1m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 29s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 22s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 56s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 230m 45s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not generate ASF License warnings. |
| | | 335m 5s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4028 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 3a275b4ad14f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 24fa464e7aecb792b291c70ab99fb8aaa4d64610 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/5/testReport/ |
| Max. process+thread count | 3158 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on a change in pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#discussion_r818311796
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java
##########
@@ -1053,6 +1053,25 @@ public static int getFreeSocketPort() {
return port;
}
+ /**
+ * Return free ports. There is no guarantee they will remain free, so
+ * ports should be used immediately. The number of free ports returned by
+ * this method should match argument {@code numOfPorts}.
+ *
+ * @param numOfPorts Number of free ports to acquire.
+ * @return Free ports for binding a local socket.
+ */
+ public static Set<Integer> getFreeSocketPorts(int numOfPorts) {
+ final Set<Integer> freePorts = new HashSet<>(numOfPorts);
+ for (int i = 0; i < numOfPorts * 5; i++) {
+ freePorts.add(getFreeSocketPort());
Review comment:
``getFreeSocketPort()`` as per code can return 0 as well, we should avoid that? We can't classify 0 as a unique free port.
if it returns 0, then your test would also fail. you would be expecting the port to be 0, but it would be different
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#issuecomment-1059331962
Thanx @tomscut & @virajjasani
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn merged pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
ayushtkn merged pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#discussion_r816722818
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
##########
@@ -52,4 +53,70 @@ public void testStartStop() throws IOException {
c.shutdown();
}
}
+
+ @Test
+ public void testStartStopWithPorts() throws IOException {
+ Configuration conf = new Configuration();
+
+ try {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481).build();
+ fail("Should not reach here");
+ } catch (IllegalArgumentException e) {
+ assertEquals("Num of http ports (1) should match num of JournalNodes (3)", e.getMessage());
+ }
+
+ try {
+ new MiniJournalCluster.Builder(conf).setRpcPorts(8481, 8482)
+ .build();
+ fail("Should not reach here");
+ } catch (IllegalArgumentException e) {
+ assertEquals("Num of rpc ports (2) should match num of JournalNodes (3)", e.getMessage());
+ }
+
+ try {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 10000).setRpcPorts(8481)
+ .build();
+ fail("Should not reach here");
+ } catch (IllegalArgumentException e) {
+ assertEquals("Num of rpc ports (1) should match num of JournalNodes (3)", e.getMessage());
+ }
+
+ try {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 1000, 2000)
+ .setRpcPorts(8481, 8482, 8483)
+ .build();
+ fail("Should not reach here");
+ } catch (IllegalArgumentException e) {
+ assertEquals("Num of http ports (4) should match num of JournalNodes (3)", e.getMessage());
+ }
+
+ MiniJournalCluster miniJournalCluster =
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481, 8482, 8483)
+ .setRpcPorts(8491, 8492, 8493).build();
Review comment:
> We can not rely completely on Jenkins, I remember seeing an issue with a RBF test, where we had a port hard-coded, and for some organisation, there internal build was failing for that test, because they had something running on that port.
Valid point!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a change in pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
tomscut commented on a change in pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#discussion_r817298583
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
##########
@@ -52,4 +59,82 @@ public void testStartStop() throws IOException {
c.shutdown();
}
}
+
+ @Test
+ public void testStartStopWithPorts() throws Exception {
+ Configuration conf = new Configuration();
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of http ports (1) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481).build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of rpc ports (2) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setRpcPorts(8481, 8482).build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of rpc ports (1) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 10000).setRpcPorts(8481)
+ .build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of http ports (4) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 1000, 2000)
+ .setRpcPorts(8481, 8482, 8483).build();
+ });
+
+ final int[] httpPorts = new int[] { NetUtils.getFreeSocketPort(), NetUtils.getFreeSocketPort(),
+ NetUtils.getFreeSocketPort() };
+ final int[] rpcPorts = new int[] { NetUtils.getFreeSocketPort(), NetUtils.getFreeSocketPort(),
+ NetUtils.getFreeSocketPort() };
Review comment:
Thanks @ayushtkn for ping me and your advice.
`NetUtils.getFreeSocketPort()` returns the currently freeport, which may be occupied if not immediately used.
If we set the port to 0, the system will automatically assign a free port for us. Then we'll see which ports are actually being used and do assert. Is that ok? As @virajjasani mentioned [here](https://github.com/apache/hadoop/pull/4028#discussion_r816694773) .
Please point out if I understand wrong. Thanks. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
virajjasani commented on pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#issuecomment-1056772509
@ayushtkn With the latest changes, here is the result of a stretch test on MacOS:
```
java.lang.IllegalStateException: 50000 free ports could not be acquired.
at org.apache.hadoop.net.NetUtils.getFreeSocketPorts(NetUtils.java:1072)
at org.apache.hadoop.hdfs.qjournal.TestMiniJournalCluster.testStartStopWithPorts(TestMiniJournalCluster.java:102)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#issuecomment-1055833072
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 33s | | trunk passed |
| +1 :green_heart: | compile | 1m 44s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 42s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 27m 54s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 21s | | the patch passed |
| +1 :green_heart: | compile | 1m 27s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 27s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 50s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/4/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 1 unchanged - 1 fixed = 2 total (was 2) |
| +1 :green_heart: | mvnsite | 1m 21s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 21s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 24s | | the patch passed |
| +1 :green_heart: | shadedclient | 27m 17s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 340m 0s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not generate ASF License warnings. |
| | | 453m 40s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4028 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 19ec2f7a5543 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e70c3115d9500a1f9e1cf433050f4c47445284eb |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/4/testReport/ |
| Max. process+thread count | 2194 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
virajjasani commented on pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#issuecomment-1055208592
@ayushtkn could you please take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
virajjasani commented on pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#issuecomment-1050610723
@ferhui @goiri could you please review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#discussion_r816932592
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
##########
@@ -52,4 +59,82 @@ public void testStartStop() throws IOException {
c.shutdown();
}
}
+
+ @Test
+ public void testStartStopWithPorts() throws Exception {
+ Configuration conf = new Configuration();
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of http ports (1) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481).build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of rpc ports (2) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setRpcPorts(8481, 8482).build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of rpc ports (1) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 10000).setRpcPorts(8481)
+ .build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of http ports (4) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 1000, 2000)
+ .setRpcPorts(8481, 8482, 8483).build();
+ });
+
+ final int[] httpPorts = new int[] { NetUtils.getFreeSocketPort(), NetUtils.getFreeSocketPort(),
+ NetUtils.getFreeSocketPort() };
+ final int[] rpcPorts = new int[] { NetUtils.getFreeSocketPort(), NetUtils.getFreeSocketPort(),
+ NetUtils.getFreeSocketPort() };
Review comment:
I see the exact increments in the ports as per the logs:
```
2022-03-01 20:08:22,504 [Listener at localhost/58862] INFO qjournal.TestMiniJournalCluster (TestMiniJournalCluster.java:testStartStopWithPorts(106)) - Http ports selected: [58873, 58874, 58875]
2022-03-01 20:08:22,504 [Listener at localhost/58862] INFO qjournal.TestMiniJournalCluster (TestMiniJournalCluster.java:testStartStopWithPorts(107)) - Rpc ports selected: [58876, 58877, 58878]
2022-03-01 21:53:55,768 [Listener at localhost/59292] INFO qjournal.TestMiniJournalCluster (TestMiniJournalCluster.java:testStartStopWithPorts(106)) - Http ports selected: [59302, 59303, 59304]
2022-03-01 21:53:55,768 [Listener at localhost/59292] INFO qjournal.TestMiniJournalCluster (TestMiniJournalCluster.java:testStartStopWithPorts(107)) - Rpc ports selected: [59305, 59306, 59307]
```
And so on..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#discussion_r818338290
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java
##########
@@ -1053,6 +1053,25 @@ public static int getFreeSocketPort() {
return port;
}
+ /**
+ * Return free ports. There is no guarantee they will remain free, so
+ * ports should be used immediately. The number of free ports returned by
+ * this method should match argument {@code numOfPorts}.
+ *
+ * @param numOfPorts Number of free ports to acquire.
+ * @return Free ports for binding a local socket.
+ */
+ public static Set<Integer> getFreeSocketPorts(int numOfPorts) {
+ final Set<Integer> freePorts = new HashSet<>(numOfPorts);
+ for (int i = 0; i < numOfPorts * 5; i++) {
+ freePorts.add(getFreeSocketPort());
Review comment:
Yes, let me take care of this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
tomscut commented on pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#issuecomment-1058846529
`NetUtils. GetFreeSocketPort()` can greatly reduce the probability of port occupy. The latest version looks good to me. Thanks @ayushtkn and @virajjasani .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on a change in pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#discussion_r816685678
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
##########
@@ -52,4 +53,70 @@ public void testStartStop() throws IOException {
c.shutdown();
}
}
+
+ @Test
+ public void testStartStopWithPorts() throws IOException {
+ Configuration conf = new Configuration();
+
+ try {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481).build();
+ fail("Should not reach here");
+ } catch (IllegalArgumentException e) {
+ assertEquals("Num of http ports (1) should match num of JournalNodes (3)", e.getMessage());
+ }
+
+ try {
+ new MiniJournalCluster.Builder(conf).setRpcPorts(8481, 8482)
+ .build();
+ fail("Should not reach here");
+ } catch (IllegalArgumentException e) {
+ assertEquals("Num of rpc ports (2) should match num of JournalNodes (3)", e.getMessage());
+ }
+
+ try {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 10000).setRpcPorts(8481)
+ .build();
+ fail("Should not reach here");
+ } catch (IllegalArgumentException e) {
+ assertEquals("Num of rpc ports (1) should match num of JournalNodes (3)", e.getMessage());
+ }
+
+ try {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 1000, 2000)
+ .setRpcPorts(8481, 8482, 8483)
+ .build();
+ fail("Should not reach here");
+ } catch (IllegalArgumentException e) {
+ assertEquals("Num of http ports (4) should match num of JournalNodes (3)", e.getMessage());
+ }
+
+ MiniJournalCluster miniJournalCluster =
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481, 8482, 8483)
+ .setRpcPorts(8491, 8492, 8493).build();
Review comment:
> hence this UT should not collide with any other UTs.
I have similar concerns, We are hard coding the ports here, this test can fail in specific environments where these ports are already occupied. We have seen such cases in past..
--> Just had a cursory look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on a change in pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#discussion_r816927981
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
##########
@@ -52,4 +59,82 @@ public void testStartStop() throws IOException {
c.shutdown();
}
}
+
+ @Test
+ public void testStartStopWithPorts() throws Exception {
+ Configuration conf = new Configuration();
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of http ports (1) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481).build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of rpc ports (2) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setRpcPorts(8481, 8482).build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of rpc ports (1) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 10000).setRpcPorts(8481)
+ .build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of http ports (4) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 1000, 2000)
+ .setRpcPorts(8481, 8482, 8483).build();
+ });
+
+ final int[] httpPorts = new int[] { NetUtils.getFreeSocketPort(), NetUtils.getFreeSocketPort(),
+ NetUtils.getFreeSocketPort() };
+ final int[] rpcPorts = new int[] { NetUtils.getFreeSocketPort(), NetUtils.getFreeSocketPort(),
+ NetUtils.getFreeSocketPort() };
Review comment:
This won't work in all cases, the same port can be returned, say you are calling ``NetUtils.getFreeSocketPort()`` 6 times, it is possible it won't return 6 different ports, it can have dupes.
Can have a util method added in NetUtils, Something like this(Just for idea, can improve/fix)
```
/**
* Return a set of free ports. There is no guarantee it will remain free, so
* * it should be used immediately.
* @param numPorts number of free ports required.
* @return a set of unique ports.
* @throws IOException in case unable to find the required number of free ports.
*/
public static Set<Integer> getFreeSocketPorts(int numPorts)
throws IOException {
Set<Integer> ports = new HashSet<>();
for (int numAttempt = 0;
ports.size() != numPorts && numAttempt < 10 * numPorts; numAttempt++) {
int port = getFreeSocketPort();
if (port > 0) {
ports.add(port);
}
}
if (ports.size() != numPorts) {
throw new IOException("Couldn't find " + numPorts + " free ports");
}
return ports;
}
```
And we can consume the output of this method.
If possible, Please extend a test in ``TestNetUtils`` as well. A simple test should do:
```
@Test
public void testGetFreePorts() throws IOException {
assertEquals(6, NetUtils.getFreeSocketPorts(6).size());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#discussion_r817419407
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
##########
@@ -52,4 +59,82 @@ public void testStartStop() throws IOException {
c.shutdown();
}
}
+
+ @Test
+ public void testStartStopWithPorts() throws Exception {
+ Configuration conf = new Configuration();
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of http ports (1) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481).build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of rpc ports (2) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setRpcPorts(8481, 8482).build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of rpc ports (1) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 10000).setRpcPorts(8481)
+ .build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of http ports (4) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 1000, 2000)
+ .setRpcPorts(8481, 8482, 8483).build();
+ });
+
+ final int[] httpPorts = new int[] { NetUtils.getFreeSocketPort(), NetUtils.getFreeSocketPort(),
+ NetUtils.getFreeSocketPort() };
+ final int[] rpcPorts = new int[] { NetUtils.getFreeSocketPort(), NetUtils.getFreeSocketPort(),
+ NetUtils.getFreeSocketPort() };
Review comment:
@ayushtkn @tomscut There is nothing wrong with providing the utility as mentioned by Ayush above. I believe it is better to fail the test if we can not get 6 free ports for this UT. I will make the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#issuecomment-1057331063
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 2s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 45s | | trunk passed |
| +1 :green_heart: | compile | 24m 15s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 47s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 10s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 17s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 24s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 5s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 18s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 25s | | the patch passed |
| +1 :green_heart: | compile | 23m 32s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 23m 32s | | the patch passed |
| +1 :green_heart: | compile | 20m 41s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 3m 42s | | root: The patch generated 0 new + 23 unchanged - 1 fixed = 23 total (was 24) |
| +1 :green_heart: | mvnsite | 3m 7s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 12s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 21s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 25s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 19s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 5s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 332m 51s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 58s | | The patch does not generate ASF License warnings. |
| | | 571m 56s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4028 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 3bd38eb1e358 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bcf8a15d444a92575f341ca6b69858c403667421 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/6/testReport/ |
| Max. process+thread count | 2623 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on a change in pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#discussion_r818587730
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
##########
@@ -52,4 +60,92 @@ public void testStartStop() throws IOException {
c.shutdown();
}
}
+
+ @Test
+ public void testStartStopWithPorts() throws Exception {
+ Configuration conf = new Configuration();
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of http ports (1) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481).build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of rpc ports (2) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setRpcPorts(8481, 8482).build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of rpc ports (1) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 10000).setRpcPorts(8481)
+ .build();
+ });
+
+ LambdaTestUtils.intercept(
+ IllegalArgumentException.class,
+ "Num of http ports (4) should match num of JournalNodes (3)",
+ "MiniJournalCluster port validation failed",
+ () -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 1000, 2000)
+ .setRpcPorts(8481, 8482, 8483).build();
+ });
+
+ final Set<Integer> httpAndRpcPorts = NetUtils.getFreeSocketPorts(6);
+ LOG.info("Free socket ports: {}", httpAndRpcPorts);
+
+ final int[] httpPorts = new int[3];
+ final int[] rpcPorts = new int[3];
+ int httpPortIdx = 0;
+ int rpcPortIdx = 0;
+ for (Integer httpAndRpcPort : httpAndRpcPorts) {
+ if (httpPortIdx < 3) {
+ httpPorts[httpPortIdx++] = httpAndRpcPort;
+ } else {
+ rpcPorts[rpcPortIdx++] = httpAndRpcPort;
+ }
+ }
+
+ LOG.info("Http ports selected: {}", httpPorts);
+ LOG.info("Rpc ports selected: {}", rpcPorts);
+
+ for (int i = 0; i < 3; i++) {
+ assertNotEquals(0, rpcPorts[i]);
+ assertNotEquals(0, httpPorts[i]);
+ }
Review comment:
you could have asserted directly on httpAndRpcPorts. :-)
```
assertFalse(httpAndRpcPorts.contains(0));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4028: HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#issuecomment-1058227393
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 36s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 8s | | trunk passed |
| +1 :green_heart: | compile | 24m 37s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 21m 31s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 16s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 23s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 14s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 28s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 25s | | the patch passed |
| +1 :green_heart: | compile | 25m 42s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 25m 42s | | the patch passed |
| +1 :green_heart: | compile | 22m 5s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 22m 5s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 3m 49s | | root: The patch generated 0 new + 23 unchanged - 1 fixed = 23 total (was 24) |
| +1 :green_heart: | mvnsite | 3m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 16s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 25s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 25s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 25s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 36s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 370m 20s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 24s | | The patch does not generate ASF License warnings. |
| | | 613m 57s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4028 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 0691f6cfd6d2 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 68e7e2eb60f236b7b59a383ab2b8da06e724648d |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/7/testReport/ |
| Max. process+thread count | 3046 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org