You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/11/02 04:09:24 UTC
[GitHub] [hadoop] ZanderXu opened a new pull request, #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
ZanderXu opened a new pull request, #5098:
URL: https://github.com/apache/hadoop/pull/5098
### Description of PR
[HDFS-16831](https://issues.apache.org/jira/browse/HDFS-16831)
The method `getNamenodesForNameserviceId` in `MembershipNamenodeResolver.class` should shuffle Observer NameNodes every time. The current logic will return the cached list and will caused all of read requests are forwarding to the first observer namenode.
The related code as bellow:
```
@Override
public List<? extends FederationNamenodeContext> getNamenodesForNameserviceId(
final String nsId, boolean listObserversFirst) throws IOException {
List<? extends FederationNamenodeContext> ret = cacheNS.get(Pair.of(nsId, listObserversFirst));
if (ret != null) {
return ret;
}
...
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
ZanderXu commented on PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#issuecomment-1360641561
@simbadzina Master, can you help me review this PR when you are available?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#issuecomment-1301705989
@ZanderXu Thanks for your contribution, we should fix the checkstyle issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
ZanderXu commented on PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#issuecomment-1323707490
@goiri Sir, can help me review this PR again? Thanks so much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
ZanderXu commented on PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#issuecomment-1363655232
Merged, thanks @ashutoshcipher @simbadzina @slfan1989 @goiri for your review. Thanks so much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on a diff in pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
ZanderXu commented on code in PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#discussion_r1017802434
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MembershipNamenodeResolver.java:
##########
@@ -189,13 +189,45 @@ private void updateNameNodeState(final String nsId,
}
}
+ private <T extends FederationNamenodeContext> List<T> shuffleObserverNN(
Review Comment:
@goiri Sir, I have updated it, please help me review it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#discussion_r1054720868
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestNamenodeResolver.java:
##########
@@ -90,6 +90,83 @@ public void setup() throws IOException, InterruptedException {
assertTrue(cleared);
}
+ @Test
+ public void testShuffleObserverNNs() throws Exception {
+ // Add an active entry to the store
+ NamenodeStatusReport activeReport = createNamenodeReport(
+ NAMESERVICES[0], NAMENODES[0], HAServiceState.ACTIVE);
+ assertTrue(namenodeResolver.registerNamenode(activeReport));
+
+ // Add a standby entry to the store
+ NamenodeStatusReport standbyReport = createNamenodeReport(
+ NAMESERVICES[0], NAMENODES[1], HAServiceState.STANDBY);
+ assertTrue(namenodeResolver.registerNamenode(standbyReport));
+
+ // Load cache
+ stateStore.refreshCaches(true);
+
+ // Get namenodes from state store.
+ List<? extends FederationNamenodeContext> withoutObserver =
+ namenodeResolver.getNamenodesForNameserviceId(NAMESERVICES[0], true);
+ assertEquals(2, withoutObserver.size());
+ assertEquals(FederationNamenodeServiceState.ACTIVE, withoutObserver.get(0).getState());
+ assertEquals(FederationNamenodeServiceState.STANDBY, withoutObserver.get(1).getState());
+
+ // Get namenodes from cache.
+ withoutObserver = namenodeResolver.getNamenodesForNameserviceId(NAMESERVICES[0], true);
+ assertEquals(2, withoutObserver.size());
+ assertEquals(FederationNamenodeServiceState.ACTIVE, withoutObserver.get(0).getState());
+ assertEquals(FederationNamenodeServiceState.STANDBY, withoutObserver.get(1).getState());
+
+ // Add an observer entry to the store
+ NamenodeStatusReport observerReport1 = createNamenodeReport(
+ NAMESERVICES[0], NAMENODES[2], HAServiceState.OBSERVER);
+ assertTrue(namenodeResolver.registerNamenode(observerReport1));
+
+ // Load cache
+ stateStore.refreshCaches(true);
+
+ // Get namenodes from state store.
+ List<? extends FederationNamenodeContext> observerList =
+ namenodeResolver.getNamenodesForNameserviceId(NAMESERVICES[0], true);
+ assertEquals(3, observerList.size());
+ assertEquals(FederationNamenodeServiceState.OBSERVER, observerList.get(0).getState());
+ assertEquals(FederationNamenodeServiceState.ACTIVE, observerList.get(1).getState());
+ assertEquals(FederationNamenodeServiceState.STANDBY, observerList.get(2).getState());
+
+ // Get namenodes from cache.
+ observerList = namenodeResolver.getNamenodesForNameserviceId(NAMESERVICES[0], true);
+ assertEquals(3, observerList.size());
+ assertEquals(FederationNamenodeServiceState.OBSERVER, observerList.get(0).getState());
+ assertEquals(FederationNamenodeServiceState.ACTIVE, observerList.get(1).getState());
+ assertEquals(FederationNamenodeServiceState.STANDBY, observerList.get(2).getState());
+
+ // Add one new observer entry to the store
+ NamenodeStatusReport observerReport2 = createNamenodeReport(
+ NAMESERVICES[0], NAMENODES[3], HAServiceState.OBSERVER);
+ assertTrue(namenodeResolver.registerNamenode(observerReport2));
+
+ // Load cache
+ stateStore.refreshCaches(true);
+
+ // Get namenodes from state store.
+ List<? extends FederationNamenodeContext> observerList2 =
+ namenodeResolver.getNamenodesForNameserviceId(NAMESERVICES[0], true);
+ assertEquals(4, observerList2.size());
+ assertEquals(FederationNamenodeServiceState.OBSERVER, observerList2.get(0).getState());
+ assertEquals(FederationNamenodeServiceState.OBSERVER, observerList2.get(1).getState());
+ assertEquals(FederationNamenodeServiceState.ACTIVE, observerList2.get(2).getState());
+ assertEquals(FederationNamenodeServiceState.STANDBY, observerList2.get(3).getState());
+
+ // Get namenodes from cache.
+ observerList2 = namenodeResolver.getNamenodesForNameserviceId(NAMESERVICES[0], true);
+ assertEquals(4, observerList2.size());
+ assertEquals(FederationNamenodeServiceState.OBSERVER, observerList2.get(0).getState());
+ assertEquals(FederationNamenodeServiceState.OBSERVER, observerList2.get(1).getState());
+ assertEquals(FederationNamenodeServiceState.ACTIVE, observerList2.get(2).getState());
+ assertEquals(FederationNamenodeServiceState.STANDBY, observerList2.get(3).getState());
Review Comment:
To test the shuffling, one approach would be to call namenodeResolver.getNamenodesForNameserviceId() 100 times in a loop and check if ever the two observers swap positions in the list. If there is never a swap we fail the test.
The test will be probabilistic but the chance of it flaking will be extremely low 0.5^100.
> List<? extends FederationNamenodeContext> observerList3 ;
> for(int i = 0; i < 100; i++) {
> observerList3 = namenodeResolver.getNamenodesForNameserviceId(NAMESERVICES[0], true);
> assertEquals(FederationNamenodeServiceState.OBSERVER, observerList3.get(0).getState());
> assertEquals(FederationNamenodeServiceState.OBSERVER, observerList3.get(1).getState());
> if (observerList3.get(0).getNamenodeId().equals(observerList2.get(1).getNamenodeId()) &&
> observerList3.get(1).getNamenodeId().equals(observerList2.get(0).getNamenodeId())) {
> return;
> }
> }
> Assert.fail("Observer order never changed.");
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
ZanderXu commented on PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#issuecomment-1307007852
@ashutoshcipher @goiri @slfan1989 Sir, I have add one UT to test it. Please help me review it again. Thanks so much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#discussion_r1016957155
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MembershipNamenodeResolver.java:
##########
@@ -189,13 +189,45 @@ private void updateNameNodeState(final String nsId,
}
}
+ private <T extends FederationNamenodeContext> List<T> shuffleObserverNN(
+ List<T> inputNameNodes, boolean listObserversFirst) {
+ if (!listObserversFirst) {
+ return inputNameNodes;
+ }
+ // Get Observers first.
+ List<T> observerList = new ArrayList<>();
+ for (T t : inputNameNodes) {
+ if (t.getState() == OBSERVER) {
+ observerList.add(t);
+ } else {
+ // The inputNameNodes are already sorted, so it can break
Review Comment:
Can we make this assumption?
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MembershipNamenodeResolver.java:
##########
@@ -189,13 +189,45 @@ private void updateNameNodeState(final String nsId,
}
}
+ private <T extends FederationNamenodeContext> List<T> shuffleObserverNN(
Review Comment:
javadoc with purpose and general approach
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
ZanderXu commented on PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#issuecomment-1299925408
@omalley @simbadzina Master, please help me review this PR too. It works well locally.
The method `getNamenodesForNameserviceId` in `MembershipNamenodeResolver.class` should shuffle Observer NameNodes every time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#issuecomment-1301644555
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 20s | | trunk passed |
| +1 :green_heart: | compile | 0m 52s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 49s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 44s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 24s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 20s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) | hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 52s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 31s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 51s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 41m 57s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not generate ASF License warnings. |
| | | 146m 54s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5098 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux fa858f60893c 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a2b59c388f6e1cf8dbbd51cade5417780e3b8c22 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/2/testReport/ |
| Max. process+thread count | 3581 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ZanderXu merged pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
ZanderXu merged PR #5098:
URL: https://github.com/apache/hadoop/pull/5098
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#discussion_r1054720868
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestNamenodeResolver.java:
##########
@@ -90,6 +90,83 @@ public void setup() throws IOException, InterruptedException {
assertTrue(cleared);
}
+ @Test
+ public void testShuffleObserverNNs() throws Exception {
+ // Add an active entry to the store
+ NamenodeStatusReport activeReport = createNamenodeReport(
+ NAMESERVICES[0], NAMENODES[0], HAServiceState.ACTIVE);
+ assertTrue(namenodeResolver.registerNamenode(activeReport));
+
+ // Add a standby entry to the store
+ NamenodeStatusReport standbyReport = createNamenodeReport(
+ NAMESERVICES[0], NAMENODES[1], HAServiceState.STANDBY);
+ assertTrue(namenodeResolver.registerNamenode(standbyReport));
+
+ // Load cache
+ stateStore.refreshCaches(true);
+
+ // Get namenodes from state store.
+ List<? extends FederationNamenodeContext> withoutObserver =
+ namenodeResolver.getNamenodesForNameserviceId(NAMESERVICES[0], true);
+ assertEquals(2, withoutObserver.size());
+ assertEquals(FederationNamenodeServiceState.ACTIVE, withoutObserver.get(0).getState());
+ assertEquals(FederationNamenodeServiceState.STANDBY, withoutObserver.get(1).getState());
+
+ // Get namenodes from cache.
+ withoutObserver = namenodeResolver.getNamenodesForNameserviceId(NAMESERVICES[0], true);
+ assertEquals(2, withoutObserver.size());
+ assertEquals(FederationNamenodeServiceState.ACTIVE, withoutObserver.get(0).getState());
+ assertEquals(FederationNamenodeServiceState.STANDBY, withoutObserver.get(1).getState());
+
+ // Add an observer entry to the store
+ NamenodeStatusReport observerReport1 = createNamenodeReport(
+ NAMESERVICES[0], NAMENODES[2], HAServiceState.OBSERVER);
+ assertTrue(namenodeResolver.registerNamenode(observerReport1));
+
+ // Load cache
+ stateStore.refreshCaches(true);
+
+ // Get namenodes from state store.
+ List<? extends FederationNamenodeContext> observerList =
+ namenodeResolver.getNamenodesForNameserviceId(NAMESERVICES[0], true);
+ assertEquals(3, observerList.size());
+ assertEquals(FederationNamenodeServiceState.OBSERVER, observerList.get(0).getState());
+ assertEquals(FederationNamenodeServiceState.ACTIVE, observerList.get(1).getState());
+ assertEquals(FederationNamenodeServiceState.STANDBY, observerList.get(2).getState());
+
+ // Get namenodes from cache.
+ observerList = namenodeResolver.getNamenodesForNameserviceId(NAMESERVICES[0], true);
+ assertEquals(3, observerList.size());
+ assertEquals(FederationNamenodeServiceState.OBSERVER, observerList.get(0).getState());
+ assertEquals(FederationNamenodeServiceState.ACTIVE, observerList.get(1).getState());
+ assertEquals(FederationNamenodeServiceState.STANDBY, observerList.get(2).getState());
+
+ // Add one new observer entry to the store
+ NamenodeStatusReport observerReport2 = createNamenodeReport(
+ NAMESERVICES[0], NAMENODES[3], HAServiceState.OBSERVER);
+ assertTrue(namenodeResolver.registerNamenode(observerReport2));
+
+ // Load cache
+ stateStore.refreshCaches(true);
+
+ // Get namenodes from state store.
+ List<? extends FederationNamenodeContext> observerList2 =
+ namenodeResolver.getNamenodesForNameserviceId(NAMESERVICES[0], true);
+ assertEquals(4, observerList2.size());
+ assertEquals(FederationNamenodeServiceState.OBSERVER, observerList2.get(0).getState());
+ assertEquals(FederationNamenodeServiceState.OBSERVER, observerList2.get(1).getState());
+ assertEquals(FederationNamenodeServiceState.ACTIVE, observerList2.get(2).getState());
+ assertEquals(FederationNamenodeServiceState.STANDBY, observerList2.get(3).getState());
+
+ // Get namenodes from cache.
+ observerList2 = namenodeResolver.getNamenodesForNameserviceId(NAMESERVICES[0], true);
+ assertEquals(4, observerList2.size());
+ assertEquals(FederationNamenodeServiceState.OBSERVER, observerList2.get(0).getState());
+ assertEquals(FederationNamenodeServiceState.OBSERVER, observerList2.get(1).getState());
+ assertEquals(FederationNamenodeServiceState.ACTIVE, observerList2.get(2).getState());
+ assertEquals(FederationNamenodeServiceState.STANDBY, observerList2.get(3).getState());
Review Comment:
To test the shuffling, one approach would be to call namenodeResolver.getNamenodesForNameserviceId() 100 times in a loop and check if ever the two observers swap positions in the list. If there is never a swap we fail the test.
The test will be probabilistic but the chance of it flaking will be extremely low 0.5^100.
> List<? extends FederationNamenodeContext> observerList3 ;
> for(int i = 0; i < 100; i++) {
> observerList3 = namenodeResolver.getNamenodesForNameserviceId(NAMESERVICES[0], true);
> assertEquals(FederationNamenodeServiceState.OBSERVER, observerList3.get(0).getState());
> assertEquals(FederationNamenodeServiceState.OBSERVER, observerList3.get(1).getState());
> if (observerList3.get(0).getNamenodeId().equals(observerList2.get(1).getNamenodeId()) &&
> observerList3.get(1).getNamenodeId().equals(observerList2.get(0).getNamenodeId())) {
> return;
> }
> }
> Assert.fail("Observer order never changed.");
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
ZanderXu commented on PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#issuecomment-1363586358
@simbadzina master, I have modified the UT, please help me review again, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#discussion_r1054720868
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestNamenodeResolver.java:
##########
@@ -90,6 +90,83 @@ public void setup() throws IOException, InterruptedException {
assertTrue(cleared);
}
+ @Test
+ public void testShuffleObserverNNs() throws Exception {
+ // Add an active entry to the store
+ NamenodeStatusReport activeReport = createNamenodeReport(
+ NAMESERVICES[0], NAMENODES[0], HAServiceState.ACTIVE);
+ assertTrue(namenodeResolver.registerNamenode(activeReport));
+
+ // Add a standby entry to the store
+ NamenodeStatusReport standbyReport = createNamenodeReport(
+ NAMESERVICES[0], NAMENODES[1], HAServiceState.STANDBY);
+ assertTrue(namenodeResolver.registerNamenode(standbyReport));
+
+ // Load cache
+ stateStore.refreshCaches(true);
+
+ // Get namenodes from state store.
+ List<? extends FederationNamenodeContext> withoutObserver =
+ namenodeResolver.getNamenodesForNameserviceId(NAMESERVICES[0], true);
+ assertEquals(2, withoutObserver.size());
+ assertEquals(FederationNamenodeServiceState.ACTIVE, withoutObserver.get(0).getState());
+ assertEquals(FederationNamenodeServiceState.STANDBY, withoutObserver.get(1).getState());
+
+ // Get namenodes from cache.
+ withoutObserver = namenodeResolver.getNamenodesForNameserviceId(NAMESERVICES[0], true);
+ assertEquals(2, withoutObserver.size());
+ assertEquals(FederationNamenodeServiceState.ACTIVE, withoutObserver.get(0).getState());
+ assertEquals(FederationNamenodeServiceState.STANDBY, withoutObserver.get(1).getState());
+
+ // Add an observer entry to the store
+ NamenodeStatusReport observerReport1 = createNamenodeReport(
+ NAMESERVICES[0], NAMENODES[2], HAServiceState.OBSERVER);
+ assertTrue(namenodeResolver.registerNamenode(observerReport1));
+
+ // Load cache
+ stateStore.refreshCaches(true);
+
+ // Get namenodes from state store.
+ List<? extends FederationNamenodeContext> observerList =
+ namenodeResolver.getNamenodesForNameserviceId(NAMESERVICES[0], true);
+ assertEquals(3, observerList.size());
+ assertEquals(FederationNamenodeServiceState.OBSERVER, observerList.get(0).getState());
+ assertEquals(FederationNamenodeServiceState.ACTIVE, observerList.get(1).getState());
+ assertEquals(FederationNamenodeServiceState.STANDBY, observerList.get(2).getState());
+
+ // Get namenodes from cache.
+ observerList = namenodeResolver.getNamenodesForNameserviceId(NAMESERVICES[0], true);
+ assertEquals(3, observerList.size());
+ assertEquals(FederationNamenodeServiceState.OBSERVER, observerList.get(0).getState());
+ assertEquals(FederationNamenodeServiceState.ACTIVE, observerList.get(1).getState());
+ assertEquals(FederationNamenodeServiceState.STANDBY, observerList.get(2).getState());
+
+ // Add one new observer entry to the store
+ NamenodeStatusReport observerReport2 = createNamenodeReport(
+ NAMESERVICES[0], NAMENODES[3], HAServiceState.OBSERVER);
+ assertTrue(namenodeResolver.registerNamenode(observerReport2));
+
+ // Load cache
+ stateStore.refreshCaches(true);
+
+ // Get namenodes from state store.
+ List<? extends FederationNamenodeContext> observerList2 =
+ namenodeResolver.getNamenodesForNameserviceId(NAMESERVICES[0], true);
+ assertEquals(4, observerList2.size());
+ assertEquals(FederationNamenodeServiceState.OBSERVER, observerList2.get(0).getState());
+ assertEquals(FederationNamenodeServiceState.OBSERVER, observerList2.get(1).getState());
+ assertEquals(FederationNamenodeServiceState.ACTIVE, observerList2.get(2).getState());
+ assertEquals(FederationNamenodeServiceState.STANDBY, observerList2.get(3).getState());
+
+ // Get namenodes from cache.
+ observerList2 = namenodeResolver.getNamenodesForNameserviceId(NAMESERVICES[0], true);
+ assertEquals(4, observerList2.size());
+ assertEquals(FederationNamenodeServiceState.OBSERVER, observerList2.get(0).getState());
+ assertEquals(FederationNamenodeServiceState.OBSERVER, observerList2.get(1).getState());
+ assertEquals(FederationNamenodeServiceState.ACTIVE, observerList2.get(2).getState());
+ assertEquals(FederationNamenodeServiceState.STANDBY, observerList2.get(3).getState());
Review Comment:
To test the shuffling, one approach would be to call namenodeResolver.getNamenodesForNameserviceId() 100 times in a loop and check if ever the two observers swap positions in the list. If there is never a swap we fail the test.
The test will be probabilistic but the chance of it flaking will be extremely low 0.5^100.
> List<? extends FederationNamenodeContext> observerList3 ;
> for(int i = 0; i < 100; i++) {
> observerList3 =
> namenodeResolver.getNamenodesForNameserviceId(NAMESERVICES[0], true);
> assertEquals(FederationNamenodeServiceState.OBSERVER, observerList3.get(0).getState());
> assertEquals(FederationNamenodeServiceState.OBSERVER, observerList3.get(1).getState());
> if (observerList3.get(0).getNamenodeId().equals(observerList2.get(1).getNamenodeId()) &&
> observerList3.get(1).getNamenodeId().equals(observerList2.get(0).getNamenodeId())) {
> return;
> }
> }
> Assert.fail("Observer order never changed.");
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#issuecomment-1307203257
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 10s | | trunk passed |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 39s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 52s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 0s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 7s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 42s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 44s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 40m 32s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not generate ASF License warnings. |
| | | 145m 59s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5098 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5d7d54321b74 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ebadfb1c0c5c834b355d6b9b777a8b3f4a3fb62e |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/4/testReport/ |
| Max. process+thread count | 2288 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#issuecomment-1308772686
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 24s | | trunk passed |
| +1 :green_heart: | compile | 1m 0s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 54s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 43s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 1s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 13s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 0s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 18s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 31s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 2s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 42m 45s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not generate ASF License warnings. |
| | | 150m 14s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5098 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux ee7402af6c08 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4451a8438835a2598a111411570cefa033daec07 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/5/testReport/ |
| Max. process+thread count | 2377 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#discussion_r1012254353
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MembershipNamenodeResolver.java:
##########
@@ -189,13 +190,43 @@ private void updateNameNodeState(final String nsId,
}
}
+ @VisibleForTesting
+ public <T extends FederationNamenodeContext> List<T> shuffleObserverNN(
+ List<T> inputNameNodes, boolean listObserversFirst) {
+ if (!listObserversFirst) {
+ return inputNameNodes;
+ } else {
+ List<T> observerNNList = new ArrayList<>();
+ List<T> activeAndStandbyList = new ArrayList<>();
+ for (T t : inputNameNodes) {
+ if (t.getState() == OBSERVER) {
+ observerNNList.add(t);
+ } else {
+ activeAndStandbyList.add(t);
+ }
+ }
+
+ if (observerNNList.size() <= 1) {
Review Comment:
We could've done this check earlier.
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MembershipNamenodeResolver.java:
##########
@@ -189,13 +190,43 @@ private void updateNameNodeState(final String nsId,
}
}
+ @VisibleForTesting
+ public <T extends FederationNamenodeContext> List<T> shuffleObserverNN(
+ List<T> inputNameNodes, boolean listObserversFirst) {
+ if (!listObserversFirst) {
+ return inputNameNodes;
+ } else {
Review Comment:
No ned for elses when we do the return.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
ZanderXu commented on PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#issuecomment-1362283212
\
> To test the shuffling, one approach would be to call namenodeResolver.getNamenodesForNameserviceId() 100 times in a loop and check if ever the two observers swap positions in the list. If there is never a swap we fail the test.
>
> The test will be probabilistic but the chance of it flaking will be extremely low 0.5^100.
>
> ```
> List<? extends FederationNamenodeContext> observerList3 ;
> for(int i = 0; i < 100; i++) {
> observerList3 = namenodeResolver.getNamenodesForNameserviceId(NAMESERVICES[0], true);
> assertEquals(FederationNamenodeServiceState.OBSERVER, observerList3.get(0).getState());
> assertEquals(FederationNamenodeServiceState.OBSERVER, observerList3.get(1).getState());
> if (observerList3.get(0).getNamenodeId().equals(observerList2.get(1).getNamenodeId()) &&
> observerList3.get(1).getNamenodeId().equals(observerList2.get(0).getNamenodeId())) {
> return;
> }
> }
> Assert.fail("Observer order never changed.");
> ```
@simbadzina Thanks for your nice suggestion, I will do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
ZanderXu commented on PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#issuecomment-1305225730
> +1 LGTM after last commit. @ZanderXu - Do you think we need to add a UT ?
@ashutoshcipher Sir, thanks for your review. I will add one UT to test it. Do you have some good ideas to test shuffling result?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
ZanderXu commented on PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#issuecomment-1301579817
@goiri Sir, thanks for your review. I have updated this PR, please help me review it again when you are available. Thanks so much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on a diff in pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
ZanderXu commented on code in PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#discussion_r1017802115
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MembershipNamenodeResolver.java:
##########
@@ -189,13 +189,45 @@ private void updateNameNodeState(final String nsId,
}
}
+ private <T extends FederationNamenodeContext> List<T> shuffleObserverNN(
+ List<T> inputNameNodes, boolean listObserversFirst) {
+ if (!listObserversFirst) {
+ return inputNameNodes;
+ }
+ // Get Observers first.
+ List<T> observerList = new ArrayList<>();
+ for (T t : inputNameNodes) {
+ if (t.getState() == OBSERVER) {
+ observerList.add(t);
+ } else {
+ // The inputNameNodes are already sorted, so it can break
Review Comment:
Yes. If `listObserversFirst` is true, all Observers will be placed at the front of the `inputNameNodes` which been processed by `getRecentRegistrationForQuery` method.
If we want to make this method more common, I can ignore it and loop all the inputNameNodes to find the Observers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#issuecomment-1299591441
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| -1 :x: | mvninstall | 3m 40s | [/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/1/artifact/out/branch-mvninstall-root.txt) | root in trunk failed. |
| +1 :green_heart: | compile | 3m 48s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 50s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 28m 29s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 24s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 0m 19s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) | hadoop-hdfs-rbf in the patch failed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not generate ASF License warnings. |
| | | 73m 39s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5098 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5ecda8112937 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 15470ab94a91c304b21992590a0c3f1c837957ed |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/1/testReport/ |
| Max. process+thread count | 604 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#issuecomment-1302122207
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 25s | | trunk passed |
| +1 :green_heart: | compile | 0m 50s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 38s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 11s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 33s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 42m 0s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not generate ASF License warnings. |
| | | 146m 28s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5098 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 4ae9d29d7aa3 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2a0cff36d95301c31bf8c77d49f7192718c84aa0 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/3/testReport/ |
| Max. process+thread count | 3623 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
ZanderXu commented on PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#issuecomment-1301939535
> @ZanderXu Thanks for your contribution, we should fix the checkstyle issue.
@slfan1989 Sir, thanks for your review. I have fixed the checkstyle in the last commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#issuecomment-1308738120
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 3s | | trunk passed |
| +1 :green_heart: | compile | 0m 55s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 54s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 42s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 59s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 0s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 15s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 26s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 44s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 41s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 39s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 23m 19s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not generate ASF License warnings. |
| | | 121m 31s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5098 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 9fb903214903 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2329cb79ca390d2275ec623ebb0b96276fc5e4ef |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/6/testReport/ |
| Max. process+thread count | 2456 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ashutoshcipher commented on pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
ashutoshcipher commented on PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#issuecomment-1307702912
Thanks @ZanderXu . I am +1 along with @goiri comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#issuecomment-1307966061
> Thanks @ZanderXu . I am +1 along with @goiri comments.
+1 along with @goiri comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
simbadzina commented on PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#issuecomment-1361899500
> @simbadzina Master, can you help me review this PR when you are available?
Code generally looks good to me. Just one small recommended change to test the shuffling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5098: HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5098:
URL: https://github.com/apache/hadoop/pull/5098#issuecomment-1363641858
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 49s | | trunk passed |
| +1 :green_heart: | compile | 0m 46s | | trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 50s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 27s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 22m 35s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not generate ASF License warnings. |
| | | 117m 29s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5098 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux cbf532c9a828 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d154c049d9fd2419eb8c98bb45d95070d4ba14f5 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/7/testReport/ |
| Max. process+thread count | 2242 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5098/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org