You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/05/16 02:26:50 UTC
[GitHub] [hadoop] simbadzina opened a new pull request, #4311: HDFS-13522: IPC changes to support observer reads through routers.
simbadzina opened a new pull request, #4311:
URL: https://github.com/apache/hadoop/pull/4311
### Description of PR
IPC changes so that clients send their txn number to routers and in turn routers will forward this to the namenodes. The code to direct routers to the observer namenodes will be in a follow up PR.
### How was this patch tested?
Ran unit tests in module in Intellij
### For code changes:
- [ x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1128418276
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 53s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 51s | | trunk passed |
| +1 :green_heart: | compile | 25m 13s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 21m 12s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 46s | | trunk passed |
| -1 :x: | javadoc | 1m 32s | [/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/2/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-common in trunk failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| +1 :green_heart: | javadoc | 7m 5s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 22s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 14s | | the patch passed |
| +1 :green_heart: | compile | 24m 36s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 24m 36s | | the patch passed |
| +1 :green_heart: | compile | 22m 17s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 22m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 6s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/2/artifact/out/results-checkstyle-root.txt) | root: The patch generated 2 new + 199 unchanged - 1 fixed = 201 total (was 200) |
| +1 :green_heart: | mvnsite | 6m 57s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. |
| -1 :x: | javadoc | 1m 33s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/2/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-common in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| +1 :green_heart: | javadoc | 7m 1s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 31s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 50s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 45s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 11s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 266m 16s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 24m 56s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 45s | | The patch does not generate ASF License warnings. |
| | | 579m 40s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux e0dcb1f2bee6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2b51acbedc942e184b4df1599b41010d32a4f8f6 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/2/testReport/ |
| Max. process+thread count | 3124 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1130213231
This RB is followed up by https://github.com/apache/hadoop/pull/4127
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1218198032
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 2s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 2s | | detect-secrets was not available. |
| +0 :ok: | buf | 0m 2s | | buf was not available. |
| +0 :ok: | buf | 0m 2s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 30s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 2s | | trunk passed |
| +1 :green_heart: | compile | 24m 3s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 22m 16s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 21s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 19s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 43s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 13s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 33s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 6s | | the patch passed |
| +1 :green_heart: | compile | 22m 21s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | cc | 22m 21s | | the patch passed |
| +1 :green_heart: | javac | 22m 21s | | the patch passed |
| +1 :green_heart: | compile | 20m 43s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 20m 43s | | the patch passed |
| -1 :x: | javac | 20m 43s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/21/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 2643 unchanged - 1 fixed = 2644 total (was 2644) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 12s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/21/artifact/out/results-checkstyle-root.txt) | root: The patch generated 15 new + 396 unchanged - 2 fixed = 411 total (was 398) |
| +1 :green_heart: | mvnsite | 7m 34s | | the patch passed |
| -1 :x: | javadoc | 1m 38s | [/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/21/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) |
| +1 :green_heart: | javadoc | 6m 41s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 53s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 43s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 46s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 16s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 244m 6s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 23m 19s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 39s | | The patch does not generate ASF License warnings. |
| | | 550m 54s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/21/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat |
| uname | Linux c35afcdd324d 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9705e9943e52dfe7b355665ef81ae0a29b992438 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/21/testReport/ |
| Max. process+thread count | 3321 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/21/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r923567042
##########
hadoop-common-project/hadoop-common/src/main/proto/RpcHeader.proto:
##########
@@ -91,6 +91,7 @@ message RpcRequestHeaderProto { // the header for the RpcRequest
optional RPCTraceInfoProto traceInfo = 6; // tracing info
optional RPCCallerContextProto callerContext = 7; // call context
optional int64 stateId = 8; // The last seen Global State ID
+ map<string, int64> nameserviceStateIds = 9; // Last seen state IDs for multiple nameservices.
Review Comment:
On my laptop this compiles okay with protoc-2.5.0.
https://developers.google.com/protocol-buffers/docs/proto#maps
The protobuf v2 spec also show the map syntax.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r966172679
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java:
##########
@@ -40,6 +40,14 @@ public class ClientGSIContext implements AlignmentContext {
private final LongAccumulator lastSeenStateId =
new LongAccumulator(Math::max, Long.MIN_VALUE);
+ public void disableObserverRead() {
+ if(lastSeenStateId.get() > -1L) {
+ throw new IllegalStateException(
+ "Can't disable observer read after communicate.");
+ }
+ lastSeenStateId.accumulate(-1L);
Review Comment:
I've removed this special value now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1197766744
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 3 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 3s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 26s | | trunk passed |
| +1 :green_heart: | compile | 26m 7s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 22m 59s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 57s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 18s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 10s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 28s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 48s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 32s | | the patch passed |
| +1 :green_heart: | compile | 25m 6s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | cc | 25m 6s | | the patch passed |
| +1 :green_heart: | javac | 25m 6s | | the patch passed |
| +1 :green_heart: | compile | 21m 36s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 21m 36s | | the patch passed |
| +1 :green_heart: | javac | 21m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 31s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/18/artifact/out/results-checkstyle-root.txt) | root: The patch generated 4 new + 177 unchanged - 1 fixed = 181 total (was 178) |
| +1 :green_heart: | mvnsite | 6m 59s | | the patch passed |
| -1 :x: | javadoc | 1m 31s | [/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/18/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 18 new + 0 unchanged - 0 fixed = 18 total (was 0) |
| +1 :green_heart: | javadoc | 6m 54s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | spotbugs | 3m 14s | [/new-spotbugs-hadoop-common-project_hadoop-common.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/18/artifact/out/new-spotbugs-hadoop-common-project_hadoop-common.html) | hadoop-common-project/hadoop-common generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 22m 37s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 40s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 18s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 246m 3s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 25m 10s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 50s | | The patch does not generate ASF License warnings. |
| | | 562m 24s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-common-project/hadoop-common |
| | Class org.apache.hadoop.hdfs.server.federation.router.RouterFederatedState$RouterFederatedStateProto defines non-transient non-serializable instance field namespaceStateIds_ In RouterFederatedState.java:instance field namespaceStateIds_ In RouterFederatedState.java |
| | Useless control flow in org.apache.hadoop.hdfs.server.federation.router.RouterFederatedState$RouterFederatedStateProto$Builder.maybeForceBuilderInitialization() At RouterFederatedState.java: At RouterFederatedState.java:[line 483] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/18/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat |
| uname | Linux 7be0714fa3e0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 44de9c6aae0cc65381b4a8a98a2aa3ba2ad8e66e |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/18/testReport/ |
| Max. process+thread count | 3468 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/18/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1186053275
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 3 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 39s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 2s | | trunk passed |
| +1 :green_heart: | compile | 23m 4s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 37s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 30s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 49s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 38s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 23m 9s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 10s | | the patch passed |
| +1 :green_heart: | compile | 22m 8s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | cc | 22m 8s | | the patch passed |
| +1 :green_heart: | javac | 22m 8s | | the patch passed |
| +1 :green_heart: | compile | 20m 32s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 20m 32s | | the patch passed |
| +1 :green_heart: | javac | 20m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 11s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/14/artifact/out/results-checkstyle-root.txt) | root: The patch generated 3 new + 177 unchanged - 1 fixed = 180 total (was 178) |
| +1 :green_heart: | mvnsite | 7m 42s | | the patch passed |
| +1 :green_heart: | javadoc | 6m 23s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 49s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 13m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 45s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 2s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 16s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 239m 14s | | hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 2m 6s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/14/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) | hadoop-hdfs-rbf in the patch failed. |
| +1 :green_heart: | asflicense | 1m 59s | | The patch does not generate ASF License warnings. |
| | | 523m 28s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/14/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat |
| uname | Linux dd2c2838f076 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 094d0db1e4e73cf6d620b3676471503ea74cf488 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/14/testReport/ |
| Max. process+thread count | 3158 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/14/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
ZanderXu commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r893033565
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java:
##########
@@ -1091,6 +1091,10 @@ public void setDeferredResponse(Writable response) {
public void setDeferredError(Throwable t) {
}
+
+ public long getTimestampNanos() {
Review Comment:
It is best to replace all timestampNanos to getTimestampNanos().
Like, line583 and line3039.
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java:
##########
@@ -203,9 +210,16 @@ public ConnectionContext getConnection(UserGroupInformation ugi,
try {
pool = this.pools.get(connectionId);
if (pool == null) {
+ if(!alignmentContexts.containsKey(nsId)) {
+ synchronized (alignmentContexts) {
+ if(!alignmentContexts.containsKey(nsId)) {
+ alignmentContexts.put(nsId, new ClientGSIContext());
+ }
+ }
+ }
Review Comment:
+1
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java:
##########
@@ -349,6 +349,18 @@ public static ClientProtocol createProxyWithAlignmentContext(
boolean withRetries, AtomicBoolean fallbackToSimpleAuth,
AlignmentContext alignmentContext)
throws IOException {
+ if (!conf.getBoolean(HdfsClientConfigKeys.DFS_OBSERVER_READ_ENABLE,
Review Comment:
I have an idea on how to disable Observer Read.
- Proxy always with AlignmentContext, like ClientGSIContext or RouterGSIContext
- Always update the lastSeenStateId from active
- ClientGSIContext has a boolean flag, like disableOberverRead
- If disableObserverRead is true, getLastSeenStateId() will always return a special number, like -100
- And if we want dynamically enable ObserverRead for one ns, we just need to set the disableObserverRead flag to false.
Our cluster with 5w+ DNs used this solution, and it is convenient to control ObserverRead.
@simbadzina How do you think about this idea?
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterStateIdContext.java:
##########
@@ -0,0 +1,87 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.lang.reflect.Method;
+import java.util.HashSet;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.hdfs.protocol.ClientProtocol;
+import org.apache.hadoop.hdfs.server.namenode.ha.ReadOnly;
+import org.apache.hadoop.ipc.AlignmentContext;
+import org.apache.hadoop.ipc.RetriableException;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RpcRequestHeaderProto;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RpcResponseHeaderProto;
+
+/**
+ * This is the router implementation responsible for passing
+ * client state id to next level.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Evolving
+class RouterStateIdContext implements AlignmentContext {
Review Comment:
No used in this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1127518474
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 12 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 4s | | trunk passed |
| +1 :green_heart: | compile | 23m 8s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 33s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 40s | | trunk passed |
| -1 :x: | javadoc | 1m 51s | [/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/1/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-common in trunk failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| +1 :green_heart: | javadoc | 7m 48s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 34s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 40s | [/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/1/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) | hadoop-hdfs-rbf in the patch failed. |
| -1 :x: | compile | 13m 44s | [/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/1/artifact/out/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | root in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javac | 13m 44s | [/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/1/artifact/out/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | root in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | compile | 11m 53s | [/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javac | 11m 53s | [/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 42s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/1/artifact/out/results-checkstyle-root.txt) | root: The patch generated 2 new + 340 unchanged - 1 fixed = 342 total (was 341) |
| -1 :x: | mvnsite | 0m 54s | [/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/1/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) | hadoop-hdfs-rbf in the patch failed. |
| +1 :green_heart: | xml | 0m 3s | | The patch has no ill-formed XML file. |
| -1 :x: | javadoc | 1m 8s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/1/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-common in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| +1 :green_heart: | javadoc | 5m 28s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | spotbugs | 0m 53s | [/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/1/artifact/out/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) | hadoop-hdfs-rbf in the patch failed. |
| +1 :green_heart: | shadedclient | 22m 39s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 53s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 38s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 250m 44s | | hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 1m 15s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) | hadoop-hdfs-rbf in the patch failed. |
| +1 :green_heart: | asflicense | 1m 18s | | The patch does not generate ASF License warnings. |
| | | 502m 44s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 6bcb704d3094 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4d719a20b5800ee95458be76182e642e72ef387d |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/1/testReport/ |
| Max. process+thread count | 3601 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
ZanderXu commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1155951845
@simbadzina @omalley @melissayou I create a draft [PR](https://github.com/apache/hadoop/pull/4441), and have some different points, like:
- Proxy always with AlignmentContext, like RouterGSIContext which contains a boolean flag, like enableObserverRead
- Always update the lastSeenStateId from active
- If enableObserverRead is false, updateRequestState will not set lastSeenStateId in RPCHeader
- It's easily to dynamically reconfigure enableObserverRead.
I'm hope that we can continue push this Feature forward. Please help me review it.
@simbadzina If you want to continue work for this feature, feel free to take over it. Thanks~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r898452709
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java:
##########
@@ -349,6 +349,18 @@ public static ClientProtocol createProxyWithAlignmentContext(
boolean withRetries, AtomicBoolean fallbackToSimpleAuth,
AlignmentContext alignmentContext)
throws IOException {
+ if (!conf.getBoolean(HdfsClientConfigKeys.DFS_OBSERVER_READ_ENABLE,
+ HdfsClientConfigKeys.DFS_OBSERVER_READ_ENABLE_DEFAULT)) {
+ //Disabled observer read
+ if (alignmentContext == null) {
+ alignmentContext = new ClientGSIContext();
+ }
+ if (alignmentContext instanceof ClientGSIContext) {
Review Comment:
Even when alignment context is not equal to null, we still disable observer reads in this if condition. The not null condition occurs when the ObserverReadProxyProvider is being used.
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java:
##########
@@ -40,6 +40,14 @@ public class ClientGSIContext implements AlignmentContext {
private final LongAccumulator lastSeenStateId =
new LongAccumulator(Math::max, Long.MIN_VALUE);
+ public void disableObserverRead() {
+ if(lastSeenStateId.get() > -1L) {
+ throw new IllegalStateException(
+ "Can't disable observer read after communicate.");
+ }
+ lastSeenStateId.accumulate(-1L);
Review Comment:
I do not expect available use cases but I can document that this is a reserved value for disabling router reads.
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterStateIdContext.java:
##########
@@ -0,0 +1,87 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.lang.reflect.Method;
+import java.util.HashSet;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.hdfs.protocol.ClientProtocol;
+import org.apache.hadoop.hdfs.server.namenode.ha.ReadOnly;
+import org.apache.hadoop.ipc.AlignmentContext;
+import org.apache.hadoop.ipc.RetriableException;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RpcRequestHeaderProto;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RpcResponseHeaderProto;
+
+/**
+ * This is the router implementation responsible for passing
+ * client state id to next level.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Evolving
+class RouterStateIdContext implements AlignmentContext {
Review Comment:
This means the client's header state ID accessible in the router. But yes, the value is not used in this PR. https://github.com/apache/hadoop/pull/4127 is the one which then uses the value.
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java:
##########
@@ -172,11 +178,12 @@ public void close() {
* @param ugi User group information.
* @param nnAddress Namenode address for the connection.
* @param protocol Protocol for the connection.
+ * @param nsId Nameservice Identify.
Review Comment:
Fixed
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java:
##########
@@ -349,6 +349,18 @@ public static ClientProtocol createProxyWithAlignmentContext(
boolean withRetries, AtomicBoolean fallbackToSimpleAuth,
AlignmentContext alignmentContext)
throws IOException {
+ if (!conf.getBoolean(HdfsClientConfigKeys.DFS_OBSERVER_READ_ENABLE,
+ HdfsClientConfigKeys.DFS_OBSERVER_READ_ENABLE_DEFAULT)) {
+ //Disabled observer read
+ if (alignmentContext == null) {
+ alignmentContext = new ClientGSIContext();
Review Comment:
With the current approach, the router will perform observer reads unless clients explicitly opt out by sending the -1 in the RPC header. Old clients without this change will use observer reads.
If we switch to setting alignment context to null, INT64.MIN (default header stateID value) would become the new special value. We then won't be able to distinguish between when clients are explicitly opting out, versus times when the alignment context is just not set. AlignmentContext is null unless the ObserverReadProxyProvider is configured.
Another thing to consider is that when communicating with routers, the state ID has no meaning since the routers cover multiple namespaces while the client only stores one state ID. So we'd still need a special value or range to indicate clients care about alignment.
I think having a value like -1L which is not an expected state ID is better for debugging than say using >=0 because the state ID could then be interpreted as being a particular namenode's state.
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java:
##########
@@ -73,6 +75,8 @@ public class ConnectionManager {
/** Queue for creating new connections. */
private final BlockingQueue<ConnectionPool> creatorQueue;
+
+ private final Map<String, AlignmentContext> alignmentContexts;
Review Comment:
Added a comment. The keys are the nameservice names.
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java:
##########
@@ -40,6 +40,14 @@ public class ClientGSIContext implements AlignmentContext {
private final LongAccumulator lastSeenStateId =
new LongAccumulator(Math::max, Long.MIN_VALUE);
+ public void disableObserverRead() {
+ if(lastSeenStateId.get() > -1L) {
Review Comment:
The transation ID on the namenode starts from zero and only goes up. So negative values can be used to indicate error conditions.
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml:
##########
@@ -6446,4 +6446,11 @@
frequently than this time, the client will give up waiting.
</description>
</property>
+ <property>
+ <name>dfs.observer.read.enable</name>
+ <value>true</value>
+ <description>
+ Enable observer read for client with router
Review Comment:
I just realized setting this to false breaks direct reads to the observer namenodes.
I've add documentation to alert user of this. I'll investigate further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r964308328
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.hdfs.NamespaceStateId;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+import org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos.RouterFederatedStateProto;
+import org.apache.hadoop.thirdparty.protobuf.InvalidProtocolBufferException;
+import org.apache.hadoop.thirdparty.protobuf.ByteString;
+
+
+/**
+ * Collection of last-seen namespace state Ids for a set of namespaces.
+ * A single NamespaceStateId is shared by all outgoing connections to a particular namespace.
+ * Router clients share and query the entire collection.
+ */
+public class FederatedNamespaceIds {
+ private final Map<String, NamespaceStateId> namespaceIdMap = new ConcurrentHashMap<>();
+ private final ReentrantLock lock = new ReentrantLock();
Review Comment:
You are right. I was concerned about the iterator (foreach) I have on the map.
However, the weak consistency provided is sufficient for our needs.
https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/package-summary.html#Weakly:~:text=Most%20concurrent%20Collection%20implementations%20(including%20most%20Queues)%20also%20differ%20from%20the%20usual%20java.util%20conventions%20in%20that%20their%20Iterators%20and%20Spliterators%20provide%20weakly%20consistent%20rather%20than%20fast%2Dfail%20traversal%3A
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r964314207
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java:
##########
@@ -66,15 +75,24 @@ public void updateResponseState(RpcResponseHeaderProto.Builder header) {
*/
@Override
public void receiveResponseState(RpcResponseHeaderProto header) {
Review Comment:
Fixed.
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java:
##########
@@ -66,15 +75,24 @@ public void updateResponseState(RpcResponseHeaderProto.Builder header) {
*/
@Override
public void receiveResponseState(RpcResponseHeaderProto header) {
- lastSeenStateId.accumulate(header.getStateId());
+ if (header.hasRouterFederatedState()) {
+ routerFederatedState = header.getRouterFederatedState();
+ } else {
+ lastSeenStateId.update(header.getStateId());
+ }
}
/**
* Client side implementation for providing state alignment info in requests.
*/
@Override
public void updateRequestState(RpcRequestHeaderProto.Builder header) {
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r964131419
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.hdfs.NamespaceStateId;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+import org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos.RouterFederatedStateProto;
+import org.apache.hadoop.thirdparty.protobuf.InvalidProtocolBufferException;
+import org.apache.hadoop.thirdparty.protobuf.ByteString;
+
+
+/**
+ * Collection of last-seen namespace state Ids for a set of namespaces.
+ * A single NamespaceStateId is shared by all outgoing connections to a particular namespace.
+ * Router clients share and query the entire collection.
+ */
+public class FederatedNamespaceIds {
+ private final Map<String, NamespaceStateId> namespaceIdMap = new ConcurrentHashMap<>();
+ private final ReentrantLock lock = new ReentrantLock();
+
+ public void updateStateUsingRequestHeader(RpcHeaderProtos.RpcRequestHeaderProto header) {
+ if (header.hasRouterFederatedState()) {
+ RouterFederatedStateProto federatedState;
+ try {
+ federatedState = RouterFederatedStateProto.parseFrom(header.getRouterFederatedState());
+ } catch (InvalidProtocolBufferException e) {
+ throw new RuntimeException(e);
+ }
+ lock.lock();
+ try {
+ federatedState.getNamespaceStateIdsMap().forEach((nsId, stateId) -> {
+ if (!namespaceIdMap.containsKey(nsId)) {
+ namespaceIdMap.putIfAbsent(nsId, new NamespaceStateId());
+ }
+ namespaceIdMap.get(nsId).update(stateId);
+ });
+ } finally {
+ lock.unlock();
+ }
+
+ }
+ }
+
+ public void setResponseHeaderState(RpcHeaderProtos.RpcResponseHeaderProto.Builder headerBuilder) {
+ RouterFederatedStateProto.Builder federatedStateBuilder =
+ RouterFederatedStateProto.newBuilder();
+ lock.lock();
+ try {
+ namespaceIdMap.forEach((k, v) -> federatedStateBuilder.putNamespaceStateIds(k, v.get()));
+ } finally {
+ lock.unlock();
+ }
+ headerBuilder.setRouterFederatedState(federatedStateBuilder.build().toByteString());
+ }
+
+ public NamespaceStateId getNamespaceId(String nsId) {
+ lock.lock();
+ try {
+ namespaceIdMap.putIfAbsent(nsId, new NamespaceStateId());
+ } finally {
+ lock.unlock();
+ }
+ return namespaceIdMap.get(nsId);
+ }
+
+ public void removeNamespaceId(String nsId) {
+ lock.lock();
+ try {
+ namespaceIdMap.remove(nsId);
+ } finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Utility function to view state of routerFederatedState field in RPC headers.
+ */
+ @VisibleForTesting
+ public static Map<String, Long> getRouterFederatedStateMap(ByteString byteString) {
Review Comment:
I now use this class in the RouterRPCClient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r898503159
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java:
##########
@@ -349,6 +349,18 @@ public static ClientProtocol createProxyWithAlignmentContext(
boolean withRetries, AtomicBoolean fallbackToSimpleAuth,
AlignmentContext alignmentContext)
throws IOException {
+ if (!conf.getBoolean(HdfsClientConfigKeys.DFS_OBSERVER_READ_ENABLE,
Review Comment:
I'll add commented on your draft RB https://github.com/apache/hadoop/pull/4441
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java:
##########
@@ -349,6 +349,18 @@ public static ClientProtocol createProxyWithAlignmentContext(
boolean withRetries, AtomicBoolean fallbackToSimpleAuth,
AlignmentContext alignmentContext)
throws IOException {
+ if (!conf.getBoolean(HdfsClientConfigKeys.DFS_OBSERVER_READ_ENABLE,
Review Comment:
I'll add comments on your draft RB https://github.com/apache/hadoop/pull/4441
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1220105115
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 22s | | trunk passed |
| +1 :green_heart: | compile | 23m 9s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 42s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 22s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 42s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 34s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 9s | | the patch passed |
| +1 :green_heart: | compile | 22m 23s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | cc | 22m 23s | | the patch passed |
| +1 :green_heart: | javac | 22m 23s | | the patch passed |
| +1 :green_heart: | compile | 20m 40s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 20m 40s | | the patch passed |
| -1 :x: | javac | 20m 40s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/23/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 2643 unchanged - 1 fixed = 2644 total (was 2644) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 16s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/23/artifact/out/results-checkstyle-root.txt) | root: The patch generated 4 new + 396 unchanged - 2 fixed = 400 total (was 398) |
| +1 :green_heart: | mvnsite | 7m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 6m 17s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 47s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 48s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 36s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 44s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 16s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 240m 37s | | hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 2m 6s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/23/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) | hadoop-hdfs-rbf in the patch failed. |
| +1 :green_heart: | asflicense | 1m 55s | | The patch does not generate ASF License warnings. |
| | | 525m 38s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/23/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat |
| uname | Linux bf3844e9b81f 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ef4d26c6494927abc801b82af897583a62f2b727 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/23/testReport/ |
| Max. process+thread count | 3238 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/23/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r923272813
##########
hadoop-common-project/hadoop-common/src/main/proto/RpcHeader.proto:
##########
@@ -91,6 +91,7 @@ message RpcRequestHeaderProto { // the header for the RpcRequest
optional RPCTraceInfoProto traceInfo = 6; // tracing info
optional RPCCallerContextProto callerContext = 7; // call context
optional int64 stateId = 8; // The last seen Global State ID
+ map<string, int64> nameserviceStateIds = 9; // Last seen state IDs for multiple nameservices.
Review Comment:
Should we not use map in proto?
It is not compatibility to old hadoop version which is based on protobuf2.5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r966175313
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.hdfs.NamespaceStateId;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+import org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos.RouterFederatedStateProto;
+import org.apache.hadoop.thirdparty.protobuf.InvalidProtocolBufferException;
+import org.apache.hadoop.thirdparty.protobuf.ByteString;
+
+
+/**
+ * Collection of last-seen namespace state Ids for a set of namespaces.
+ * A single NamespaceStateId is shared by all outgoing connections to a particular namespace.
+ * Router clients share and query the entire collection.
+ */
+public class FederatedNamespaceIds {
+ private final Map<String, NamespaceStateId> namespaceIdMap = new ConcurrentHashMap<>();
+ private final ReentrantLock lock = new ReentrantLock();
+
+ public void updateStateUsingRequestHeader(RpcHeaderProtos.RpcRequestHeaderProto header) {
+ if (header.hasRouterFederatedState()) {
+ RouterFederatedStateProto federatedState;
+ try {
+ federatedState = RouterFederatedStateProto.parseFrom(header.getRouterFederatedState());
+ } catch (InvalidProtocolBufferException e) {
+ throw new RuntimeException(e);
+ }
+ lock.lock();
+ try {
+ federatedState.getNamespaceStateIdsMap().forEach((nsId, stateId) -> {
+ if (!namespaceIdMap.containsKey(nsId)) {
+ namespaceIdMap.putIfAbsent(nsId, new NamespaceStateId());
+ }
+ namespaceIdMap.get(nsId).update(stateId);
+ });
Review Comment:
I now only update the global map with values from the namenodes. Clients sharing a connection pool can still poison each other though. Working on address that in a later review comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r966164910
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java:
##########
@@ -252,21 +252,21 @@ public class RouterRpcServer extends AbstractService implements ClientProtocol,
/**
* Construct a router RPC server.
*
- * @param configuration HDFS Configuration.
+ * @param conf HDFS Configuration.
* @param router A router using this RPC server.
* @param nnResolver The NN resolver instance to determine active NNs in HA.
- * @param fileResolver File resolver to resolve file paths to subclusters.
+ * @param fResolver File resolver to resolve file paths to subclusters.
* @throws IOException If the RPC server could not be created.
*/
- public RouterRpcServer(Configuration configuration, Router router,
- ActiveNamenodeResolver nnResolver, FileSubclusterResolver fileResolver)
+ public RouterRpcServer(Configuration conf, Router router,
+ ActiveNamenodeResolver nnResolver, FileSubclusterResolver fResolver)
Review Comment:
I wanted to remove a line length warning. I'll change it back, "fileResolver" is easier to read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r966184747
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java:
##########
@@ -349,6 +349,9 @@ public static ClientProtocol createProxyWithAlignmentContext(
boolean withRetries, AtomicBoolean fallbackToSimpleAuth,
AlignmentContext alignmentContext)
throws IOException {
+ if (alignmentContext == null) {
Review Comment:
Clients need to be using the "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider" for them to have an alignment context in the ProxyProvider.
I think we need a new flag on the client to enable observer reads. And overriding the null would be behind this flag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r966302346
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml:
##########
@@ -6446,4 +6446,11 @@
frequently than this time, the client will give up waiting.
</description>
</property>
+ <property>
+ <name>dfs.observer.read.enable</name>
+ <value>true</value>
+ <description>
+ Enable observer read for client with router
Review Comment:
I no longer have this setting in the client. We may add it back in a future PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] melissayou commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
melissayou commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r892646952
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java:
##########
@@ -349,6 +349,18 @@ public static ClientProtocol createProxyWithAlignmentContext(
boolean withRetries, AtomicBoolean fallbackToSimpleAuth,
AlignmentContext alignmentContext)
throws IOException {
+ if (!conf.getBoolean(HdfsClientConfigKeys.DFS_OBSERVER_READ_ENABLE,
+ HdfsClientConfigKeys.DFS_OBSERVER_READ_ENABLE_DEFAULT)) {
+ //Disabled observer read
+ if (alignmentContext == null) {
+ alignmentContext = new ClientGSIContext();
+ }
+ if (alignmentContext instanceof ClientGSIContext) {
Review Comment:
I might lack context, so we will disable observer read only when alignmentContext is null and the observer read is set to false. Can you probably explain why we we care to check alignmentContext not null first? If it's originally not null, what does that mean? Thank you.
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java:
##########
@@ -40,6 +40,14 @@ public class ClientGSIContext implements AlignmentContext {
private final LongAccumulator lastSeenStateId =
new LongAccumulator(Math::max, Long.MIN_VALUE);
+ public void disableObserverRead() {
+ if(lastSeenStateId.get() > -1L) {
+ throw new IllegalStateException(
+ "Can't disable observer read after communicate.");
+ }
+ lastSeenStateId.accumulate(-1L);
Review Comment:
nitpick: Is -1L reserved only for disabled observer read situation or will it be used for other (future) cases? If we expect future cases use -2L, -3L, etc, will it be helpful to add a comment/doc to explicitly state -1L is reserved for disabled observer read situation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r943730742
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RouterFederatedStateProto;
+import org.apache.hadoop.thirdparty.protobuf.InvalidProtocolBufferException;
+import org.apache.hadoop.thirdparty.protobuf.ByteString;
+
+
+/** Collection of last-seen namespace state Ids for a set of namespaces. */
Review Comment:
Added a comment
> A single NamespaceStateId is shared by all outgoing connections to a particular namespace.
> Router clients share and query the entire collection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r966172015
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcConstants.java:
##########
@@ -37,7 +37,9 @@ private RpcConstants() {
public static final int INVALID_RETRY_COUNT = -1;
-
+ // Special state ID value to indicate client request header has routerFederatedState set.
Review Comment:
Yes it is. I made a mistake when cherrypicking my change. I'll remove this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r966506216
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NamespaceStateId.java:
##########
@@ -0,0 +1,42 @@
+/**
Review Comment:
I've removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1172117867
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 29m 26s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 17s | | trunk passed |
| +1 :green_heart: | compile | 22m 54s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 28s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 30s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 56s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 29s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 10s | | the patch passed |
| +1 :green_heart: | compile | 22m 19s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 22m 19s | | the patch passed |
| +1 :green_heart: | compile | 20m 41s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 12s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/11/artifact/out/results-checkstyle-root.txt) | root: The patch generated 2 new + 198 unchanged - 1 fixed = 200 total (was 199) |
| +1 :green_heart: | mvnsite | 7m 42s | | the patch passed |
| +1 :green_heart: | javadoc | 6m 23s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 51s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 13m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 44s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 39s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 16s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 239m 49s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 22m 41s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 2m 0s | | The patch does not generate ASF License warnings. |
| | | 559m 46s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 688a8160d4d2 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 97bd6920cacb0d642b4eb15f43fee8bd8c778942 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/11/testReport/ |
| Max. process+thread count | 3430 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/11/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1240167877
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 22m 21s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 5s | | trunk passed |
| +1 :green_heart: | compile | 25m 17s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 22m 52s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 30s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 50s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 14m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 32s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 8s | | the patch passed |
| +1 :green_heart: | compile | 25m 16s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 25m 16s | | the patch passed |
| +1 :green_heart: | compile | 22m 58s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 22m 58s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/27/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt) | root-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 generated 1 new + 2645 unchanged - 1 fixed = 2646 total (was 2646) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 58s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/27/artifact/out/results-checkstyle-root.txt) | root: The patch generated 2 new + 175 unchanged - 8 fixed = 177 total (was 183) |
| +1 :green_heart: | mvnsite | 6m 49s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 31s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 6m 1s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 13m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 7s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 35s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 6s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 249m 6s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 25m 19s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 44s | | The patch does not generate ASF License warnings. |
| | | 572m 20s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/27/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux d637fe4f6ec8 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bbaf062f4bff89c79e005f53b39812ad55a9c425 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/27/testReport/ |
| Max. process+thread count | 3135 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/27/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1162614384
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 40s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 8s | | trunk passed |
| +1 :green_heart: | compile | 23m 12s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 36s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 21s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 28s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 50s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 29s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 11s | | the patch passed |
| +1 :green_heart: | compile | 22m 4s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 22m 4s | | the patch passed |
| +1 :green_heart: | compile | 20m 20s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 13s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/10/artifact/out/results-checkstyle-root.txt) | root: The patch generated 2 new + 198 unchanged - 1 fixed = 200 total (was 199) |
| +1 :green_heart: | mvnsite | 7m 45s | | the patch passed |
| +1 :green_heart: | javadoc | 6m 21s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 50s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 57s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 42s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 30s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 16s | | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 239m 55s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/10/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 23m 0s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 2m 0s | | The patch does not generate ASF License warnings. |
| | | 544m 30s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux e0ad7bb622c3 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7cad62f38d6d6db534b54d5076babf9ab5f59e6e |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/10/testReport/ |
| Max. process+thread count | 3620 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/10/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1218609173
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 12s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 59s | | trunk passed |
| +1 :green_heart: | compile | 25m 41s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 34s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 21s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 20s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 48s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 36s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 7s | | the patch passed |
| +1 :green_heart: | compile | 22m 48s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | cc | 22m 48s | | the patch passed |
| +1 :green_heart: | javac | 22m 48s | | the patch passed |
| +1 :green_heart: | compile | 22m 1s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 22m 1s | | the patch passed |
| -1 :x: | javac | 22m 1s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/22/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 2643 unchanged - 1 fixed = 2644 total (was 2644) |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 23s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/22/artifact/out/results-checkstyle-root.txt) | root: The patch generated 13 new + 396 unchanged - 2 fixed = 409 total (was 398) |
| +1 :green_heart: | mvnsite | 6m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 43s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 5m 57s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 24s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 47s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 34s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 0s | | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 255m 18s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/22/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 24m 11s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 57s | | The patch does not generate ASF License warnings. |
| | | 563m 37s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/22/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat |
| uname | Linux a2cd995cbd73 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / aef5e288068cc7d68724919fd2bc2106b3c4f8f4 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/22/testReport/ |
| Max. process+thread count | 3146 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/22/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hchaverri commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hchaverri commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r892717426
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java:
##########
@@ -40,6 +40,14 @@ public class ClientGSIContext implements AlignmentContext {
private final LongAccumulator lastSeenStateId =
new LongAccumulator(Math::max, Long.MIN_VALUE);
+ public void disableObserverRead() {
+ if(lastSeenStateId.get() > -1L) {
Review Comment:
Should we be repurposing the accumulator to define the state? Could -1L be a valid value on the header stateId that might break this? Should we use a boolean instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hchaverri commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hchaverri commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r892722863
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java:
##########
@@ -172,11 +178,12 @@ public void close() {
* @param ugi User group information.
* @param nnAddress Namenode address for the connection.
* @param protocol Protocol for the connection.
+ * @param nsId Nameservice Identify.
Review Comment:
typo: identity
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1158361679
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 44m 39s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 44s | | trunk passed |
| +1 :green_heart: | compile | 22m 49s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 27s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 44s | | trunk passed |
| -1 :x: | javadoc | 1m 45s | [/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/8/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-hdfs in trunk failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| +1 :green_heart: | javadoc | 6m 50s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 27s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 10s | | the patch passed |
| +1 :green_heart: | compile | 22m 3s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 22m 3s | | the patch passed |
| +1 :green_heart: | compile | 20m 26s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 26s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/8/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 13s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/8/artifact/out/results-checkstyle-root.txt) | root: The patch generated 3 new + 198 unchanged - 1 fixed = 201 total (was 199) |
| +1 :green_heart: | mvnsite | 7m 42s | | the patch passed |
| -1 :x: | javadoc | 1m 45s | [/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/8/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-hdfs in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| +1 :green_heart: | javadoc | 6m 49s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 54s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 52s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 42s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 14s | | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 256m 19s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/8/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 23m 12s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 2m 0s | | The patch does not generate ASF License warnings. |
| | | 590m 36s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.cli.TestHDFSCLI |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 7200bc04ac4a 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 82a59d04438670a5135badf54be0cf49d0e3b338 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/8/testReport/ |
| Max. process+thread count | 3321 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/8/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by "simbadzina (via GitHub)" <gi...@apache.org>.
simbadzina commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1715929140
> The design doc https://issues.apache.org/jira/secure/attachment/13047193/observer_reads_in_rbf_proposal_simbadzina_v2.pdf is very helpful . But the Third party communication(pdf) in the observer_reads_in_rbf_proposal_simbadzina_v2.pdf is too blurry to read , and the pdf link https://drive.google.com/file/d/1851oaVeayiKKbNFSa93G5CS-bXXZY7BM/view?usp=sharing is does not exist.
>
> Can anyone put it again？ Thanks very much
I've replaced the file with one with refreshed links. https://issues.apache.org/jira/secure/attachment/13062835/observer_reads_in_rbf_proposal_simbadzina_v2.pdf
I've uploaded the third party reads pdf to Jira as well: https://issues.apache.org/jira/secure/attachment/13062830/rbf_observer_reads%20-%20third_party_reads.pdf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1239327788
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 12s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 49s | | trunk passed |
| +1 :green_heart: | compile | 23m 11s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 20m 32s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 10s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 37s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 58s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 11m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 6s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 4s | | the patch passed |
| +1 :green_heart: | compile | 22m 35s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 22m 35s | | the patch passed |
| +1 :green_heart: | compile | 20m 31s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 20m 31s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/26/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt) | root-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 generated 1 new + 2645 unchanged - 1 fixed = 2646 total (was 2646) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 2s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/26/artifact/out/results-checkstyle-root.txt) | root: The patch generated 8 new + 175 unchanged - 8 fixed = 183 total (was 183) |
| +1 :green_heart: | mvnsite | 6m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 14s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 6m 2s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 11m 50s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 48s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 7s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 5s | | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 241m 23s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/26/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 24m 15s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/26/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) | hadoop-hdfs-rbf in the patch passed. |
| -1 :x: | asflicense | 1m 38s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/26/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 540m 39s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.federation.router.TestRBFConfigFields |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/26/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 2c215e481077 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 856f63a974666da8ef022bbe9a43fe916132ed55 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/26/testReport/ |
| Max. process+thread count | 3398 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/26/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1242630498
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 4s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 31s | | trunk passed |
| +1 :green_heart: | compile | 23m 7s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 20m 34s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 1s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 37s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 46s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 11m 27s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 36s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 58s | | the patch passed |
| +1 :green_heart: | compile | 22m 21s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 22m 21s | | the patch passed |
| +1 :green_heart: | compile | 20m 38s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 38s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 6s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/30/artifact/out/results-checkstyle-root.txt) | root: The patch generated 4 new + 175 unchanged - 8 fixed = 179 total (was 183) |
| +1 :green_heart: | mvnsite | 6m 45s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 19s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| -1 :x: | javadoc | 1m 44s | [/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/30/artifact/out/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt) | hadoop-hdfs-project_hadoop-hdfs-rbf-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) |
| -1 :x: | spotbugs | 1m 2s | [/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/30/artifact/out/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) | hadoop-hdfs-rbf in the patch failed. |
| +1 :green_heart: | shadedclient | 24m 12s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 33s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 3s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 236m 0s | | hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 1m 22s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/30/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) | hadoop-hdfs-rbf in the patch failed. |
| +1 :green_heart: | asflicense | 1m 38s | | The patch does not generate ASF License warnings. |
| | | 509m 0s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/30/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 9a1fead4a5bd 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / faa6493401e587517562de9c7fab9f3d717503db |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/30/testReport/ |
| Max. process+thread count | 3275 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/30/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1240172942
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 22m 55s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 23s | | trunk passed |
| +1 :green_heart: | compile | 26m 11s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 22m 31s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 19s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 40s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 11m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 5s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 42s | | the patch passed |
| +1 :green_heart: | compile | 25m 38s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 25m 38s | | the patch passed |
| +1 :green_heart: | compile | 22m 20s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 22m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 24s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/28/artifact/out/results-checkstyle-root.txt) | root: The patch generated 2 new + 175 unchanged - 8 fixed = 177 total (was 183) |
| +1 :green_heart: | mvnsite | 7m 11s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 28s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 37s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 19s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 7s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 6s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 250m 39s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 24m 49s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 36s | | The patch does not generate ASF License warnings. |
| | | 571m 12s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/28/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux cc3a7c2f4d93 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 30497ad460cf879e940859548341b95d18d42b03 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/28/testReport/ |
| Max. process+thread count | 3262 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/28/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1183856828
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 28s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 29s | | trunk passed |
| +1 :green_heart: | compile | 24m 8s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 21m 16s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 58s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 19s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 4m 14s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 8m 27s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 34s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 49s | | the patch passed |
| +1 :green_heart: | compile | 23m 7s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | cc | 23m 7s | | the patch passed |
| +1 :green_heart: | javac | 23m 7s | | the patch passed |
| +1 :green_heart: | compile | 22m 16s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 22m 16s | | the patch passed |
| +1 :green_heart: | javac | 22m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 15s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/12/artifact/out/results-checkstyle-root.txt) | root: The patch generated 3 new + 168 unchanged - 1 fixed = 171 total (was 169) |
| +1 :green_heart: | mvnsite | 5m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 37s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 4m 10s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 8m 53s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 59s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 51s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 14s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 22m 38s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 32s | | The patch does not generate ASF License warnings. |
| | | 287m 11s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat |
| uname | Linux a3ea99a2a6f0 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3bb040826cb39de09b720f5cdcf54d3be7581c95 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/12/testReport/ |
| Max. process+thread count | 2807 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/12/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1183855423
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 1s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 23s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 9s | | trunk passed |
| +1 :green_heart: | compile | 22m 56s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 31s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 48s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 4m 36s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 8m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 26s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 45s | | the patch passed |
| +1 :green_heart: | compile | 22m 12s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | cc | 22m 12s | | the patch passed |
| +1 :green_heart: | javac | 22m 12s | | the patch passed |
| +1 :green_heart: | compile | 20m 34s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 20m 34s | | the patch passed |
| +1 :green_heart: | javac | 20m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 18s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/13/artifact/out/results-checkstyle-root.txt) | root: The patch generated 3 new + 168 unchanged - 1 fixed = 171 total (was 169) |
| +1 :green_heart: | mvnsite | 5m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 52s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 4m 37s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 8m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 44s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 39s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 16s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 23m 24s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 38s | | The patch does not generate ASF License warnings. |
| | | 282m 40s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/13/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat |
| uname | Linux 0ff2b355fdfa 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3bb040826cb39de09b720f5cdcf54d3be7581c95 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/13/testReport/ |
| Max. process+thread count | 2638 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/13/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] omalley commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
omalley commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r897101862
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java:
##########
@@ -349,6 +349,18 @@ public static ClientProtocol createProxyWithAlignmentContext(
boolean withRetries, AtomicBoolean fallbackToSimpleAuth,
AlignmentContext alignmentContext)
throws IOException {
+ if (!conf.getBoolean(HdfsClientConfigKeys.DFS_OBSERVER_READ_ENABLE,
+ HdfsClientConfigKeys.DFS_OBSERVER_READ_ENABLE_DEFAULT)) {
+ //Disabled observer read
+ if (alignmentContext == null) {
+ alignmentContext = new ClientGSIContext();
Review Comment:
The using code seems to handle alignment = null as not caring about alignment. Therefore, I think all you need to do is assign the variable to null to have the same results as disabling it. You'll remove the need for -1 as a special value and this code.
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml:
##########
@@ -6446,4 +6446,11 @@
frequently than this time, the client will give up waiting.
</description>
</property>
+ <property>
+ <name>dfs.observer.read.enable</name>
+ <value>true</value>
+ <description>
+ Enable observer read for client with router
Review Comment:
This is for all HDFS clients, right? Not just the routers.
I'd propose something like, "Enable the client to use observer NN for read operations. It RBF, it enables the routers to use the observer NN."
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java:
##########
@@ -1091,6 +1091,10 @@ public void setDeferredResponse(Writable response) {
public void setDeferredError(Throwable t) {
}
+
+ public long getTimestampNanos() {
Review Comment:
I think changing the visibility to private does most of the good. I'm less interested in making all of the other methods in the class use this accessor.
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java:
##########
@@ -73,6 +75,8 @@ public class ConnectionManager {
/** Queue for creating new connections. */
private final BlockingQueue<ConnectionPool> creatorQueue;
+
+ private final Map<String, AlignmentContext> alignmentContexts;
Review Comment:
It would be good to give an example of what the key here is. Is the NN address canonicalized?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r924006263
##########
hadoop-common-project/hadoop-common/src/main/proto/RpcHeader.proto:
##########
@@ -91,6 +91,7 @@ message RpcRequestHeaderProto { // the header for the RpcRequest
optional RPCTraceInfoProto traceInfo = 6; // tracing info
optional RPCCallerContextProto callerContext = 7; // call context
optional int64 stateId = 8; // The last seen Global State ID
+ map<string, int64> nameserviceStateIds = 9; // Last seen state IDs for multiple nameservices.
Review Comment:
But it is not supported in my Mac. My proto compile version as below:
```
% protoc --version
libprotoc 2.5.0
```
In github release,I found it is supported after v3.0.0-alpha-2. See link: https://github.com/protocolbuffers/protobuf/releases?page=10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r923612743
##########
hadoop-common-project/hadoop-common/src/main/proto/RpcHeader.proto:
##########
@@ -91,6 +91,7 @@ message RpcRequestHeaderProto { // the header for the RpcRequest
optional RPCTraceInfoProto traceInfo = 6; // tracing info
optional RPCCallerContextProto callerContext = 7; // call context
optional int64 stateId = 8; // The last seen Global State ID
+ map<string, int64> nameserviceStateIds = 9; // Last seen state IDs for multiple nameservices.
Review Comment:
I'm moving the protocol changes to HDFS-16669
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r943725929
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java:
##########
@@ -66,15 +79,23 @@ public void updateResponseState(RpcResponseHeaderProto.Builder header) {
*/
@Override
public void receiveResponseState(RpcResponseHeaderProto header) {
- lastSeenStateId.accumulate(header.getStateId());
+ lastSeenStateId.update(header.getStateId());
Review Comment:
I agree. Fixed.
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java:
##########
@@ -37,14 +37,27 @@
@InterfaceStability.Evolving
public class ClientGSIContext implements AlignmentContext {
- private final LongAccumulator lastSeenStateId =
- new LongAccumulator(Math::max, Long.MIN_VALUE);
+ private final NamespaceStateId lastSeenStateId;
+ private ByteString routerFederatedState;
+
+ public ClientGSIContext() {
+ this(new NamespaceStateId());
+ }
+
+ public ClientGSIContext(NamespaceStateId lastSeenStateId) {
+ this.lastSeenStateId = lastSeenStateId;
+ routerFederatedState = null;
+ }
@Override
public long getLastSeenStateId() {
return lastSeenStateId.get();
}
+ public void updateLastSeenStateID(Long stateId) {
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r964156460
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterStateIdContext.java:
##########
@@ -0,0 +1,94 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.lang.reflect.Method;
+import java.util.HashSet;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.hdfs.protocol.ClientProtocol;
+import org.apache.hadoop.hdfs.server.namenode.ha.ReadOnly;
+import org.apache.hadoop.ipc.AlignmentContext;
+import org.apache.hadoop.ipc.RetriableException;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RpcRequestHeaderProto;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RpcResponseHeaderProto;
+
+import static org.apache.hadoop.ipc.RpcConstants.REQUEST_HEADER_NAMESPACE_STATEIDS_SET;
+
+/**
+ * This is the router implementation responsible for passing
+ * client state id to next level.
Review Comment:
It holds data for all namespaces. I'll add more info to the comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r943730742
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RouterFederatedStateProto;
+import org.apache.hadoop.thirdparty.protobuf.InvalidProtocolBufferException;
+import org.apache.hadoop.thirdparty.protobuf.ByteString;
+
+
+/** Collection of last-seen namespace state Ids for a set of namespaces. */
Review Comment:
Added a comment
> A single NamespaceStateId is shared by all outgoing connections to a particular namespace.
> Router clients query the entire collection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1143079362
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 38s | | trunk passed |
| -1 :x: | compile | 4m 5s | [/branch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/5/artifact/out/branch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | root in trunk failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | compile | 3m 38s | [/branch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/5/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root in trunk failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | checkstyle | 3m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 47s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 30s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 4m 54s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 10m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 10s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 9s | | the patch passed |
| -1 :x: | compile | 3m 52s | [/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/5/artifact/out/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | root in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javac | 3m 52s | [/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/5/artifact/out/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | root in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | compile | 3m 22s | [/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/5/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javac | 3m 22s | [/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/5/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 33s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/5/artifact/out/results-checkstyle-root.txt) | root: The patch generated 2 new + 196 unchanged - 1 fixed = 198 total (was 197) |
| +1 :green_heart: | mvnsite | 4m 51s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 33s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 4m 1s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 10m 8s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 45s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 50s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 38s | | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 248m 40s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 23m 32s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 18s | | The patch does not generate ASF License warnings. |
| | | 454m 0s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux a291c8a97c1d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 43853ef5f599af4fbe488b4392bdb8fb3e513f95 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/5/testReport/ |
| Max. process+thread count | 3152 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1139245763
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 35s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 24s | | trunk passed |
| -1 :x: | compile | 4m 5s | [/branch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/4/artifact/out/branch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | root in trunk failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | compile | 3m 39s | [/branch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/4/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root in trunk failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | checkstyle | 3m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 35s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 4m 53s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 10m 29s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 56s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 9s | | the patch passed |
| -1 :x: | compile | 3m 48s | [/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/4/artifact/out/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | root in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javac | 3m 48s | [/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/4/artifact/out/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | root in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | compile | 3m 21s | [/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/4/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javac | 3m 21s | [/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/4/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 24s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/4/artifact/out/results-checkstyle-root.txt) | root: The patch generated 2 new + 196 unchanged - 1 fixed = 198 total (was 197) |
| +1 :green_heart: | mvnsite | 4m 50s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 30s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 3m 58s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 10m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 44s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 46s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 38s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 250m 46s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 22m 36s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 16s | | The patch does not generate ASF License warnings. |
| | | 454m 35s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 5b9011855fce 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4a3b31a8bbcd7caeb6661d8b6bec2d455e47470b |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/4/testReport/ |
| Max. process+thread count | 3567 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1146545896
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 8s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 46s | | trunk passed |
| +1 :green_heart: | compile | 22m 54s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 25s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 28s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 43s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 30s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 12s | | the patch passed |
| +1 :green_heart: | compile | 22m 7s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 22m 7s | | the patch passed |
| +1 :green_heart: | compile | 20m 26s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 5m 26s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/6/artifact/out/results-checkstyle-root.txt) | root: The patch generated 2 new + 196 unchanged - 1 fixed = 198 total (was 197) |
| +1 :green_heart: | mvnsite | 7m 59s | | the patch passed |
| +1 :green_heart: | javadoc | 6m 20s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 45s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 45s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 30s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 14s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 252m 50s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 22m 46s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 57s | | The patch does not generate ASF License warnings. |
| | | 555m 9s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 6c979e06d791 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 41312cb88c2f23e0620d9240e5d49bc0590be0b8 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/6/testReport/ |
| Max. process+thread count | 3586 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hchaverri commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hchaverri commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r892732891
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java:
##########
@@ -203,9 +210,16 @@ public ConnectionContext getConnection(UserGroupInformation ugi,
try {
pool = this.pools.get(connectionId);
if (pool == null) {
+ if(!alignmentContexts.containsKey(nsId)) {
+ synchronized (alignmentContexts) {
+ if(!alignmentContexts.containsKey(nsId)) {
+ alignmentContexts.put(nsId, new ClientGSIContext());
+ }
+ }
+ }
Review Comment:
ConcurrentHashMap#putIfAbsent should help with this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1187777645
> https://issues.apache.org/jira/browse/HADOOP-13144
> Could we have a HADOOP JIRA just for the changes in commons? It could be something like: https://issues.apache.org/jira/browse/HADOOP-13144
Ok. Moving these to HDFS-16669
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r924020781
##########
hadoop-common-project/hadoop-common/src/main/proto/RpcHeader.proto:
##########
@@ -91,6 +91,7 @@ message RpcRequestHeaderProto { // the header for the RpcRequest
optional RPCTraceInfoProto traceInfo = 6; // tracing info
optional RPCCallerContextProto callerContext = 7; // call context
optional int64 stateId = 8; // The last seen Global State ID
+ map<string, int64> nameserviceStateIds = 9; // Last seen state IDs for multiple nameservices.
Review Comment:
Yeah, it does not matter. Our internal version we just used another message to replace map.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r943726586
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RouterFederatedStateProto;
+import org.apache.hadoop.thirdparty.protobuf.InvalidProtocolBufferException;
+import org.apache.hadoop.thirdparty.protobuf.ByteString;
+
+
+/** Collection of last-seen namespace state Ids for a set of namespaces. */
+public class FederatedNamespaceIds {
Review Comment:
Added locking around iterations and modifications to the hashMap.
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RouterFederatedStateProto;
+import org.apache.hadoop.thirdparty.protobuf.InvalidProtocolBufferException;
+import org.apache.hadoop.thirdparty.protobuf.ByteString;
+
+
+/** Collection of last-seen namespace state Ids for a set of namespaces. */
+public class FederatedNamespaceIds {
+ private final Map<String, NamespaceStateId> namespaceIdMap = new ConcurrentHashMap<>();
+
+ public void updateStateUsingRequestHeader(RpcHeaderProtos.RpcRequestHeaderProto header) {
+ if (header.hasRouterFederatedState()) {
+ RouterFederatedStateProto federatedState = null;
+ try {
+ federatedState = RouterFederatedStateProto.parseFrom(header.getRouterFederatedState());
+ } catch (InvalidProtocolBufferException e) {
+ throw new RuntimeException(e);
+ }
+ federatedState.getNamespaceStateIdsMap().forEach(this::updateNamespaceState);
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1226876683
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 7s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 51s | | trunk passed |
| +1 :green_heart: | compile | 23m 16s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 36s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 8s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 53s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 5m 42s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 11m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 55s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 10s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 55s | | the patch passed |
| +1 :green_heart: | compile | 22m 39s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 22m 39s | | the patch passed |
| +1 :green_heart: | compile | 20m 34s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 20m 34s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/25/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 2643 unchanged - 1 fixed = 2644 total (was 2644) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 4s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/25/artifact/out/results-checkstyle-root.txt) | root: The patch generated 4 new + 14 unchanged - 9 fixed = 18 total (was 23) |
| +1 :green_heart: | mvnsite | 6m 46s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 26s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 5m 44s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 13s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 24s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 26s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 56s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 236m 53s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 23m 49s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 36s | | The patch does not generate ASF License warnings. |
| | | 534m 56s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/25/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 2b526f6d7f64 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 283ee928af7994513974fd81d159e97678fc63b5 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/25/testReport/ |
| Max. process+thread count | 3113 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/25/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] omalley commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
omalley commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r967381075
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java:
##########
@@ -18,14 +18,15 @@
package org.apache.hadoop.hdfs;
+import java.util.concurrent.atomic.LongAccumulator;
Review Comment:
Can you move the import back where it was?
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcConstants.java:
##########
@@ -37,7 +37,7 @@ private RpcConstants() {
public static final int INVALID_RETRY_COUNT = -1;
-
Review Comment:
Please undo the space change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1197410192
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 0s | | Docker mode activated. |
| -1 :x: | patch | 0m 46s | | https://github.com/apache/hadoop/pull/4311 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/17/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1157387445
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 42s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 39s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 42s | | trunk passed |
| +1 :green_heart: | compile | 26m 34s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 23m 28s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 39s | | trunk passed |
| -1 :x: | javadoc | 1m 36s | [/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/7/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-hdfs in trunk failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| +1 :green_heart: | javadoc | 6m 37s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 53s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 4s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 30s | | the patch passed |
| +1 :green_heart: | compile | 24m 50s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 24m 50s | | the patch passed |
| +1 :green_heart: | compile | 22m 57s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 22m 57s | | the patch passed |
| -1 :x: | blanks | 0m 1s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/7/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 18s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/7/artifact/out/results-checkstyle-root.txt) | root: The patch generated 3 new + 198 unchanged - 1 fixed = 201 total (was 199) |
| +1 :green_heart: | mvnsite | 7m 20s | | the patch passed |
| -1 :x: | javadoc | 1m 41s | [/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/7/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-hdfs in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| +1 :green_heart: | javadoc | 6m 28s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 13m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 42s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 25s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 9s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 258m 53s | | hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 23m 32s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 2m 0s | | The patch does not generate ASF License warnings. |
| | | 580m 49s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.federation.router.TestConnectionManager |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux be4aec47dab0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a02081acfc8cb79e7e3e61c45594b0a5bf034141 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/7/testReport/ |
| Max. process+thread count | 3152 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] omalley commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
omalley commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r957851531
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java:
##########
@@ -37,8 +37,17 @@
@InterfaceStability.Evolving
public class ClientGSIContext implements AlignmentContext {
- private final LongAccumulator lastSeenStateId =
- new LongAccumulator(Math::max, Long.MIN_VALUE);
+ private final NamespaceStateId lastSeenStateId;
Review Comment:
I'd replace this with a straight long. We need synchronization for the routerFederatedState also, so we might as well use a lock for both of them.
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NamespaceStateId.java:
##########
@@ -0,0 +1,42 @@
+/**
Review Comment:
This class doesn't add much value. Do we actually need it?
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java:
##########
@@ -66,15 +75,24 @@ public void updateResponseState(RpcResponseHeaderProto.Builder header) {
*/
@Override
public void receiveResponseState(RpcResponseHeaderProto header) {
- lastSeenStateId.accumulate(header.getStateId());
+ if (header.hasRouterFederatedState()) {
+ routerFederatedState = header.getRouterFederatedState();
+ } else {
+ lastSeenStateId.update(header.getStateId());
+ }
}
/**
* Client side implementation for providing state alignment info in requests.
*/
@Override
public void updateRequestState(RpcRequestHeaderProto.Builder header) {
Review Comment:
this one too
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.hdfs.NamespaceStateId;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+import org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos.RouterFederatedStateProto;
+import org.apache.hadoop.thirdparty.protobuf.InvalidProtocolBufferException;
+import org.apache.hadoop.thirdparty.protobuf.ByteString;
+
+
+/**
+ * Collection of last-seen namespace state Ids for a set of namespaces.
+ * A single NamespaceStateId is shared by all outgoing connections to a particular namespace.
+ * Router clients share and query the entire collection.
+ */
+public class FederatedNamespaceIds {
+ private final Map<String, NamespaceStateId> namespaceIdMap = new ConcurrentHashMap<>();
+ private final ReentrantLock lock = new ReentrantLock();
+
+ public void updateStateUsingRequestHeader(RpcHeaderProtos.RpcRequestHeaderProto header) {
+ if (header.hasRouterFederatedState()) {
+ RouterFederatedStateProto federatedState;
+ try {
+ federatedState = RouterFederatedStateProto.parseFrom(header.getRouterFederatedState());
+ } catch (InvalidProtocolBufferException e) {
+ throw new RuntimeException(e);
+ }
+ lock.lock();
+ try {
+ federatedState.getNamespaceStateIdsMap().forEach((nsId, stateId) -> {
+ if (!namespaceIdMap.containsKey(nsId)) {
+ namespaceIdMap.putIfAbsent(nsId, new NamespaceStateId());
+ }
+ namespaceIdMap.get(nsId).update(stateId);
+ });
+ } finally {
+ lock.unlock();
+ }
+
+ }
+ }
+
+ public void setResponseHeaderState(RpcHeaderProtos.RpcResponseHeaderProto.Builder headerBuilder) {
+ RouterFederatedStateProto.Builder federatedStateBuilder =
+ RouterFederatedStateProto.newBuilder();
+ lock.lock();
+ try {
+ namespaceIdMap.forEach((k, v) -> federatedStateBuilder.putNamespaceStateIds(k, v.get()));
+ } finally {
+ lock.unlock();
+ }
+ headerBuilder.setRouterFederatedState(federatedStateBuilder.build().toByteString());
+ }
+
+ public NamespaceStateId getNamespaceId(String nsId) {
+ lock.lock();
+ try {
+ namespaceIdMap.putIfAbsent(nsId, new NamespaceStateId());
Review Comment:
This is another place where we should use computeIfAbsent.
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.hdfs.NamespaceStateId;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+import org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos.RouterFederatedStateProto;
+import org.apache.hadoop.thirdparty.protobuf.InvalidProtocolBufferException;
+import org.apache.hadoop.thirdparty.protobuf.ByteString;
+
+
+/**
+ * Collection of last-seen namespace state Ids for a set of namespaces.
+ * A single NamespaceStateId is shared by all outgoing connections to a particular namespace.
+ * Router clients share and query the entire collection.
+ */
+public class FederatedNamespaceIds {
+ private final Map<String, NamespaceStateId> namespaceIdMap = new ConcurrentHashMap<>();
+ private final ReentrantLock lock = new ReentrantLock();
+
+ public void updateStateUsingRequestHeader(RpcHeaderProtos.RpcRequestHeaderProto header) {
+ if (header.hasRouterFederatedState()) {
+ RouterFederatedStateProto federatedState;
+ try {
+ federatedState = RouterFederatedStateProto.parseFrom(header.getRouterFederatedState());
+ } catch (InvalidProtocolBufferException e) {
+ throw new RuntimeException(e);
+ }
+ lock.lock();
+ try {
+ federatedState.getNamespaceStateIdsMap().forEach((nsId, stateId) -> {
+ if (!namespaceIdMap.containsKey(nsId)) {
+ namespaceIdMap.putIfAbsent(nsId, new NamespaceStateId());
+ }
+ namespaceIdMap.get(nsId).update(stateId);
+ });
+ } finally {
+ lock.unlock();
+ }
+
+ }
+ }
+
+ public void setResponseHeaderState(RpcHeaderProtos.RpcResponseHeaderProto.Builder headerBuilder) {
+ RouterFederatedStateProto.Builder federatedStateBuilder =
+ RouterFederatedStateProto.newBuilder();
+ lock.lock();
+ try {
+ namespaceIdMap.forEach((k, v) -> federatedStateBuilder.putNamespaceStateIds(k, v.get()));
+ } finally {
+ lock.unlock();
+ }
+ headerBuilder.setRouterFederatedState(federatedStateBuilder.build().toByteString());
+ }
+
+ public NamespaceStateId getNamespaceId(String nsId) {
+ lock.lock();
+ try {
+ namespaceIdMap.putIfAbsent(nsId, new NamespaceStateId());
+ } finally {
+ lock.unlock();
+ }
+ return namespaceIdMap.get(nsId);
+ }
+
+ public void removeNamespaceId(String nsId) {
+ lock.lock();
+ try {
+ namespaceIdMap.remove(nsId);
+ } finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Utility function to view state of routerFederatedState field in RPC headers.
+ */
+ @VisibleForTesting
+ public static Map<String, Long> getRouterFederatedStateMap(ByteString byteString) {
Review Comment:
Since this is just used for testing, could we move it to one of the test classes?
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java:
##########
@@ -37,8 +37,17 @@
@InterfaceStability.Evolving
public class ClientGSIContext implements AlignmentContext {
- private final LongAccumulator lastSeenStateId =
- new LongAccumulator(Math::max, Long.MIN_VALUE);
+ private final NamespaceStateId lastSeenStateId;
Review Comment:
You will also need to add a Math.max to the update.
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.hdfs.NamespaceStateId;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+import org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos.RouterFederatedStateProto;
+import org.apache.hadoop.thirdparty.protobuf.InvalidProtocolBufferException;
+import org.apache.hadoop.thirdparty.protobuf.ByteString;
+
+
+/**
+ * Collection of last-seen namespace state Ids for a set of namespaces.
+ * A single NamespaceStateId is shared by all outgoing connections to a particular namespace.
+ * Router clients share and query the entire collection.
+ */
+public class FederatedNamespaceIds {
+ private final Map<String, NamespaceStateId> namespaceIdMap = new ConcurrentHashMap<>();
+ private final ReentrantLock lock = new ReentrantLock();
+
+ public void updateStateUsingRequestHeader(RpcHeaderProtos.RpcRequestHeaderProto header) {
+ if (header.hasRouterFederatedState()) {
+ RouterFederatedStateProto federatedState;
+ try {
+ federatedState = RouterFederatedStateProto.parseFrom(header.getRouterFederatedState());
+ } catch (InvalidProtocolBufferException e) {
+ throw new RuntimeException(e);
+ }
+ lock.lock();
+ try {
+ federatedState.getNamespaceStateIdsMap().forEach((nsId, stateId) -> {
+ if (!namespaceIdMap.containsKey(nsId)) {
+ namespaceIdMap.putIfAbsent(nsId, new NamespaceStateId());
+ }
+ namespaceIdMap.get(nsId).update(stateId);
+ });
Review Comment:
One other problem with this, is that since values from the clients are updating the global map, it would be easy for a malicious client to poison a router with a high value that would effectively disable a namespace for that router.
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java:
##########
@@ -66,15 +75,24 @@ public void updateResponseState(RpcResponseHeaderProto.Builder header) {
*/
@Override
public void receiveResponseState(RpcResponseHeaderProto header) {
Review Comment:
this method needs to be synchronized
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.hdfs.NamespaceStateId;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+import org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos.RouterFederatedStateProto;
+import org.apache.hadoop.thirdparty.protobuf.InvalidProtocolBufferException;
+import org.apache.hadoop.thirdparty.protobuf.ByteString;
+
+
+/**
+ * Collection of last-seen namespace state Ids for a set of namespaces.
+ * A single NamespaceStateId is shared by all outgoing connections to a particular namespace.
+ * Router clients share and query the entire collection.
+ */
+public class FederatedNamespaceIds {
+ private final Map<String, NamespaceStateId> namespaceIdMap = new ConcurrentHashMap<>();
+ private final ReentrantLock lock = new ReentrantLock();
Review Comment:
With the concurrenthashmap, I don't think we need the explicit lock.
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.hdfs.NamespaceStateId;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+import org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos.RouterFederatedStateProto;
+import org.apache.hadoop.thirdparty.protobuf.InvalidProtocolBufferException;
+import org.apache.hadoop.thirdparty.protobuf.ByteString;
+
+
+/**
+ * Collection of last-seen namespace state Ids for a set of namespaces.
+ * A single NamespaceStateId is shared by all outgoing connections to a particular namespace.
+ * Router clients share and query the entire collection.
+ */
+public class FederatedNamespaceIds {
+ private final Map<String, NamespaceStateId> namespaceIdMap = new ConcurrentHashMap<>();
+ private final ReentrantLock lock = new ReentrantLock();
+
+ public void updateStateUsingRequestHeader(RpcHeaderProtos.RpcRequestHeaderProto header) {
+ if (header.hasRouterFederatedState()) {
+ RouterFederatedStateProto federatedState;
+ try {
+ federatedState = RouterFederatedStateProto.parseFrom(header.getRouterFederatedState());
+ } catch (InvalidProtocolBufferException e) {
+ throw new RuntimeException(e);
+ }
+ lock.lock();
+ try {
+ federatedState.getNamespaceStateIdsMap().forEach((nsId, stateId) -> {
+ if (!namespaceIdMap.containsKey(nsId)) {
+ namespaceIdMap.putIfAbsent(nsId, new NamespaceStateId());
+ }
+ namespaceIdMap.get(nsId).update(stateId);
Review Comment:
We should use namespaceIdMap.computeIfAbsent(...) instead of get and make the new NamespaceStateId as a lambda.
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.hdfs.NamespaceStateId;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+import org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos.RouterFederatedStateProto;
+import org.apache.hadoop.thirdparty.protobuf.InvalidProtocolBufferException;
+import org.apache.hadoop.thirdparty.protobuf.ByteString;
+
+
+/**
+ * Collection of last-seen namespace state Ids for a set of namespaces.
+ * A single NamespaceStateId is shared by all outgoing connections to a particular namespace.
+ * Router clients share and query the entire collection.
+ */
+public class FederatedNamespaceIds {
+ private final Map<String, NamespaceStateId> namespaceIdMap = new ConcurrentHashMap<>();
+ private final ReentrantLock lock = new ReentrantLock();
+
+ public void updateStateUsingRequestHeader(RpcHeaderProtos.RpcRequestHeaderProto header) {
+ if (header.hasRouterFederatedState()) {
+ RouterFederatedStateProto federatedState;
+ try {
+ federatedState = RouterFederatedStateProto.parseFrom(header.getRouterFederatedState());
+ } catch (InvalidProtocolBufferException e) {
+ throw new RuntimeException(e);
+ }
+ lock.lock();
+ try {
+ federatedState.getNamespaceStateIdsMap().forEach((nsId, stateId) -> {
+ if (!namespaceIdMap.containsKey(nsId)) {
+ namespaceIdMap.putIfAbsent(nsId, new NamespaceStateId());
+ }
+ namespaceIdMap.get(nsId).update(stateId);
+ });
Review Comment:
To transfer the information, we could probably use a thread local to pass the value to send.
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.hdfs.NamespaceStateId;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+import org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos.RouterFederatedStateProto;
+import org.apache.hadoop.thirdparty.protobuf.InvalidProtocolBufferException;
+import org.apache.hadoop.thirdparty.protobuf.ByteString;
+
+
+/**
+ * Collection of last-seen namespace state Ids for a set of namespaces.
+ * A single NamespaceStateId is shared by all outgoing connections to a particular namespace.
+ * Router clients share and query the entire collection.
+ */
+public class FederatedNamespaceIds {
+ private final Map<String, NamespaceStateId> namespaceIdMap = new ConcurrentHashMap<>();
+ private final ReentrantLock lock = new ReentrantLock();
+
+ public void updateStateUsingRequestHeader(RpcHeaderProtos.RpcRequestHeaderProto header) {
+ if (header.hasRouterFederatedState()) {
+ RouterFederatedStateProto federatedState;
+ try {
+ federatedState = RouterFederatedStateProto.parseFrom(header.getRouterFederatedState());
+ } catch (InvalidProtocolBufferException e) {
+ throw new RuntimeException(e);
+ }
+ lock.lock();
+ try {
+ federatedState.getNamespaceStateIdsMap().forEach((nsId, stateId) -> {
+ if (!namespaceIdMap.containsKey(nsId)) {
+ namespaceIdMap.putIfAbsent(nsId, new NamespaceStateId());
+ }
+ namespaceIdMap.get(nsId).update(stateId);
+ });
Review Comment:
I guess one approach would be:
- values from the NN update the map.
- we compute the value to send to the NN with the larger of the value from the client and the global map.
Thoughts?
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterStateIdContext.java:
##########
@@ -0,0 +1,94 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.lang.reflect.Method;
+import java.util.HashSet;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.hdfs.protocol.ClientProtocol;
+import org.apache.hadoop.hdfs.server.namenode.ha.ReadOnly;
+import org.apache.hadoop.ipc.AlignmentContext;
+import org.apache.hadoop.ipc.RetriableException;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RpcRequestHeaderProto;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RpcResponseHeaderProto;
+
+import static org.apache.hadoop.ipc.RpcConstants.REQUEST_HEADER_NAMESPACE_STATEIDS_SET;
+
+/**
+ * This is the router implementation responsible for passing
+ * client state id to next level.
Review Comment:
I assume this is the one that is limited to a single namespace?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] omalley closed pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
omalley closed pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
URL: https://github.com/apache/hadoop/pull/4311
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r964306464
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.hdfs.NamespaceStateId;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+import org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos.RouterFederatedStateProto;
+import org.apache.hadoop.thirdparty.protobuf.InvalidProtocolBufferException;
+import org.apache.hadoop.thirdparty.protobuf.ByteString;
+
+
+/**
+ * Collection of last-seen namespace state Ids for a set of namespaces.
+ * A single NamespaceStateId is shared by all outgoing connections to a particular namespace.
+ * Router clients share and query the entire collection.
+ */
+public class FederatedNamespaceIds {
+ private final Map<String, NamespaceStateId> namespaceIdMap = new ConcurrentHashMap<>();
+ private final ReentrantLock lock = new ReentrantLock();
+
+ public void updateStateUsingRequestHeader(RpcHeaderProtos.RpcRequestHeaderProto header) {
+ if (header.hasRouterFederatedState()) {
+ RouterFederatedStateProto federatedState;
+ try {
+ federatedState = RouterFederatedStateProto.parseFrom(header.getRouterFederatedState());
+ } catch (InvalidProtocolBufferException e) {
+ throw new RuntimeException(e);
+ }
+ lock.lock();
+ try {
+ federatedState.getNamespaceStateIdsMap().forEach((nsId, stateId) -> {
+ if (!namespaceIdMap.containsKey(nsId)) {
+ namespaceIdMap.putIfAbsent(nsId, new NamespaceStateId());
+ }
+ namespaceIdMap.get(nsId).update(stateId);
Review Comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r966303059
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,78 @@
+/**
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r966200688
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcClient.java:
##########
@@ -368,8 +370,20 @@ private ConnectionContext getConnection(UserGroupInformation ugi, String nsId,
connUGI = UserGroupInformation.createProxyUser(
ugi.getUserName(), routerUser);
}
+
Review Comment:
Good catch. I've refactored the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r966173059
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java:
##########
@@ -40,6 +40,14 @@ public class ClientGSIContext implements AlignmentContext {
private final LongAccumulator lastSeenStateId =
new LongAccumulator(Math::max, Long.MIN_VALUE);
+ public void disableObserverRead() {
+ if(lastSeenStateId.get() > -1L) {
Review Comment:
I've removed the special value now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r924006263
##########
hadoop-common-project/hadoop-common/src/main/proto/RpcHeader.proto:
##########
@@ -91,6 +91,7 @@ message RpcRequestHeaderProto { // the header for the RpcRequest
optional RPCTraceInfoProto traceInfo = 6; // tracing info
optional RPCCallerContextProto callerContext = 7; // call context
optional int64 stateId = 8; // The last seen Global State ID
+ map<string, int64> nameserviceStateIds = 9; // Last seen state IDs for multiple nameservices.
Review Comment:
But it is not supported in my Mac. My proto compiler version as below:
```
% protoc --version
libprotoc 2.5.0
```
In github release,I found it is supported after v3.0.0-alpha-2. See link: https://github.com/protocolbuffers/protobuf/releases?page=10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r924008090
##########
hadoop-common-project/hadoop-common/src/main/proto/RpcHeader.proto:
##########
@@ -91,6 +91,7 @@ message RpcRequestHeaderProto { // the header for the RpcRequest
optional RPCTraceInfoProto traceInfo = 6; // tracing info
optional RPCCallerContextProto callerContext = 7; // call context
optional int64 stateId = 8; // The last seen Global State ID
+ map<string, int64> nameserviceStateIds = 9; // Last seen state IDs for multiple nameservices.
Review Comment:
Interesting. I'm also on mac with the same version. Anyhow, I can convert to the backward compatible syntax.
`
message MapFieldEntry {
key_type key = 1;
value_type value = 2;
}
repeated MapFieldEntry map_field = N;
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r923272813
##########
hadoop-common-project/hadoop-common/src/main/proto/RpcHeader.proto:
##########
@@ -91,6 +91,7 @@ message RpcRequestHeaderProto { // the header for the RpcRequest
optional RPCTraceInfoProto traceInfo = 6; // tracing info
optional RPCCallerContextProto callerContext = 7; // call context
optional int64 stateId = 8; // The last seen Global State ID
+ map<string, int64> nameserviceStateIds = 9; // Last seen state IDs for multiple nameservices.
Review Comment:
Should we use map in proto?
It is not compatibility to old hadoop version which is based on protobuf2.5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] LiuGuH commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by "LiuGuH (via GitHub)" <gi...@apache.org>.
LiuGuH commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1715110949
The design doc https://issues.apache.org/jira/secure/attachment/13047193/observer_reads_in_rbf_proposal_simbadzina_v2.pdf is very helpful .
But the Third party communication(pdf) in the observer_reads_in_rbf_proposal_simbadzina_v2.pdf is too blurry to read , and the pdf link https://drive.google.com/file/d/1851oaVeayiKKbNFSa93G5CS-bXXZY7BM/view?usp=sharing is does not exist.
Can anyone put it againe？ Thanks very much
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1128981618
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 46s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 56s | | trunk passed |
| +1 :green_heart: | compile | 23m 6s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 35s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 35s | | trunk passed |
| -1 :x: | javadoc | 1m 50s | [/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/3/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-common in trunk failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| +1 :green_heart: | javadoc | 7m 44s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 33s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 8s | | the patch passed |
| +1 :green_heart: | compile | 22m 32s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 22m 32s | | the patch passed |
| +1 :green_heart: | compile | 21m 45s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 21m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 42s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/3/artifact/out/results-checkstyle-root.txt) | root: The patch generated 2 new + 199 unchanged - 1 fixed = 201 total (was 200) |
| +1 :green_heart: | mvnsite | 7m 24s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML file. |
| -1 :x: | javadoc | 1m 30s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/3/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-common in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| +1 :green_heart: | javadoc | 7m 14s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 13m 3s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 43s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 32s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 15s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 252m 28s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 23m 33s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 46s | | The patch does not generate ASF License warnings. |
| | | 559m 24s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux a8fa86f61127 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4286ba88c975cf46ccb02504daa67464aed93ea4 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/3/testReport/ |
| Max. process+thread count | 3276 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1160660530
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 51s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 54m 43s | | trunk passed |
| +1 :green_heart: | compile | 22m 55s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 36s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 18s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 48s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 30s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 7s | | the patch passed |
| +1 :green_heart: | compile | 22m 4s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 22m 4s | | the patch passed |
| +1 :green_heart: | compile | 20m 23s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 23s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/9/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 21s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/9/artifact/out/results-checkstyle-root.txt) | root: The patch generated 3 new + 198 unchanged - 1 fixed = 201 total (was 199) |
| +1 :green_heart: | mvnsite | 7m 42s | | the patch passed |
| +1 :green_heart: | javadoc | 6m 22s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 35s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 59s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 58s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 23s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 18s | | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 243m 45s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/9/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 25m 15s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 40s | | The patch does not generate ASF License warnings. |
| | | 579m 42s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.cli.TestHDFSCLI |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 1693df608f95 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 38ee8b13c7e6cdfe04dac1de90e67d852df8140e |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/9/testReport/ |
| Max. process+thread count | 3465 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/9/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1186182868
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 3 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 32s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 6s | | trunk passed |
| +1 :green_heart: | compile | 23m 9s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 32s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 35s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 56s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 49s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 10s | | the patch passed |
| +1 :green_heart: | compile | 22m 12s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | cc | 22m 12s | | the patch passed |
| +1 :green_heart: | javac | 22m 12s | | the patch passed |
| +1 :green_heart: | compile | 20m 55s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 20m 56s | | the patch passed |
| +1 :green_heart: | javac | 20m 55s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 43s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/15/artifact/out/results-checkstyle-root.txt) | root: The patch generated 3 new + 177 unchanged - 1 fixed = 180 total (was 178) |
| +1 :green_heart: | mvnsite | 7m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 57s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 41s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 13m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 50s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 44s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 15s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 235m 42s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 23m 43s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 48s | | The patch does not generate ASF License warnings. |
| | | 541m 31s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/15/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat |
| uname | Linux ebe93ecdbdce 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b71606e227c27826dfd4dfd23d55e8aca7082393 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/15/testReport/ |
| Max. process+thread count | 3200 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/15/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
goiri commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1185950784
Could we have a HADOOP JIRA just for the changes in commons?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1188525110
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 42s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 3 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 12s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 16s | | trunk passed |
| +1 :green_heart: | compile | 26m 53s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 22m 51s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 5m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 8s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 0s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 40s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 12s | | the patch passed |
| +1 :green_heart: | compile | 23m 14s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | cc | 23m 14s | | the patch passed |
| +1 :green_heart: | javac | 23m 14s | | the patch passed |
| +1 :green_heart: | compile | 21m 51s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 21m 51s | | the patch passed |
| +1 :green_heart: | javac | 21m 51s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 9s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/16/artifact/out/results-checkstyle-root.txt) | root: The patch generated 3 new + 177 unchanged - 1 fixed = 180 total (was 178) |
| +1 :green_heart: | mvnsite | 7m 7s | | the patch passed |
| +1 :green_heart: | javadoc | 6m 7s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 29s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 32s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 36s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 43s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 17s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 240m 7s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 24m 10s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 48s | | The patch does not generate ASF License warnings. |
| | | 555m 47s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/16/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat |
| uname | Linux 1a94b98381d3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 40aed50f2cd637bc6934204b72b90b2e80c14053 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/16/testReport/ |
| Max. process+thread count | 3306 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/16/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] omalley commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
omalley commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r940558557
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java:
##########
@@ -37,14 +37,27 @@
@InterfaceStability.Evolving
public class ClientGSIContext implements AlignmentContext {
- private final LongAccumulator lastSeenStateId =
- new LongAccumulator(Math::max, Long.MIN_VALUE);
+ private final NamespaceStateId lastSeenStateId;
+ private ByteString routerFederatedState;
+
+ public ClientGSIContext() {
+ this(new NamespaceStateId());
+ }
+
+ public ClientGSIContext(NamespaceStateId lastSeenStateId) {
+ this.lastSeenStateId = lastSeenStateId;
+ routerFederatedState = null;
+ }
@Override
public long getLastSeenStateId() {
return lastSeenStateId.get();
}
+ public void updateLastSeenStateID(Long stateId) {
Review Comment:
I don't see anyone calling this.
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java:
##########
@@ -66,15 +79,23 @@ public void updateResponseState(RpcResponseHeaderProto.Builder header) {
*/
@Override
public void receiveResponseState(RpcResponseHeaderProto header) {
- lastSeenStateId.accumulate(header.getStateId());
+ lastSeenStateId.update(header.getStateId());
Review Comment:
I think we should branch on the hasRouterFederatedState and update one or the other. This method should be synchronized because you are updating state in shared variables.
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RouterFederatedStateProto;
+import org.apache.hadoop.thirdparty.protobuf.InvalidProtocolBufferException;
+import org.apache.hadoop.thirdparty.protobuf.ByteString;
+
+
+/** Collection of last-seen namespace state Ids for a set of namespaces. */
+public class FederatedNamespaceIds {
Review Comment:
HashMap isn't synchronized, so you need explicit synchronization.
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RouterFederatedStateProto;
+import org.apache.hadoop.thirdparty.protobuf.InvalidProtocolBufferException;
+import org.apache.hadoop.thirdparty.protobuf.ByteString;
+
+
+/** Collection of last-seen namespace state Ids for a set of namespaces. */
+public class FederatedNamespaceIds {
+ private final Map<String, NamespaceStateId> namespaceIdMap = new ConcurrentHashMap<>();
+
+ public void updateStateUsingRequestHeader(RpcHeaderProtos.RpcRequestHeaderProto header) {
+ if (header.hasRouterFederatedState()) {
+ RouterFederatedStateProto federatedState = null;
+ try {
+ federatedState = RouterFederatedStateProto.parseFrom(header.getRouterFederatedState());
+ } catch (InvalidProtocolBufferException e) {
+ throw new RuntimeException(e);
+ }
+ federatedState.getNamespaceStateIdsMap().forEach(this::updateNamespaceState);
+ }
+ }
+
+ public void updateNamespaceState(String nsId, Long stateId) {
+ if (!namespaceIdMap.containsKey(nsId)) {
+ namespaceIdMap.putIfAbsent(nsId, new NamespaceStateId());
+ }
+ namespaceIdMap.get(nsId).update(stateId);
+ }
+
+ public void setResponseHeaderState(RpcHeaderProtos.RpcResponseHeaderProto.Builder headerBuilder) {
+ RouterFederatedStateProto.Builder federatedStateBuilder = RouterFederatedStateProto.newBuilder();
+ namespaceIdMap
Review Comment:
This loop also needs a lock around it.
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RouterFederatedStateProto;
+import org.apache.hadoop.thirdparty.protobuf.InvalidProtocolBufferException;
+import org.apache.hadoop.thirdparty.protobuf.ByteString;
+
+
+/** Collection of last-seen namespace state Ids for a set of namespaces. */
+public class FederatedNamespaceIds {
+ private final Map<String, NamespaceStateId> namespaceIdMap = new ConcurrentHashMap<>();
+
+ public void updateStateUsingRequestHeader(RpcHeaderProtos.RpcRequestHeaderProto header) {
+ if (header.hasRouterFederatedState()) {
+ RouterFederatedStateProto federatedState = null;
+ try {
+ federatedState = RouterFederatedStateProto.parseFrom(header.getRouterFederatedState());
+ } catch (InvalidProtocolBufferException e) {
+ throw new RuntimeException(e);
+ }
+ federatedState.getNamespaceStateIdsMap().forEach(this::updateNamespaceState);
+ }
+ }
+
+ public void updateNamespaceState(String nsId, Long stateId) {
+ if (!namespaceIdMap.containsKey(nsId)) {
+ namespaceIdMap.putIfAbsent(nsId, new NamespaceStateId());
+ }
+ namespaceIdMap.get(nsId).update(stateId);
+ }
+
+ public void setResponseHeaderState(RpcHeaderProtos.RpcResponseHeaderProto.Builder headerBuilder) {
+ RouterFederatedStateProto.Builder federatedStateBuilder = RouterFederatedStateProto.newBuilder();
+ namespaceIdMap
+ .forEach((k, v) -> federatedStateBuilder.putNamespaceStateIds(k, v.get()));
+ headerBuilder.setRouterFederatedState(federatedStateBuilder.build().toByteString());
+ }
+
+ public NamespaceStateId getNamespaceId(String nsId) {
Review Comment:
And this method should be synchronized.
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java:
##########
@@ -37,14 +37,27 @@
@InterfaceStability.Evolving
public class ClientGSIContext implements AlignmentContext {
- private final LongAccumulator lastSeenStateId =
- new LongAccumulator(Math::max, Long.MIN_VALUE);
+ private final NamespaceStateId lastSeenStateId;
+ private ByteString routerFederatedState;
+
+ public ClientGSIContext() {
+ this(new NamespaceStateId());
+ }
+
+ public ClientGSIContext(NamespaceStateId lastSeenStateId) {
+ this.lastSeenStateId = lastSeenStateId;
+ routerFederatedState = null;
+ }
@Override
public long getLastSeenStateId() {
return lastSeenStateId.get();
}
+ public void updateLastSeenStateID(Long stateId) {
Review Comment:
I don't see anyone calling this. This method seems problematic in that it only updates the lastSeenStateId, which shouldn't be changed if this a router connection.
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,85 @@
+/**
Review Comment:
This class should be moved to RBF rather than down in the HDFS client.
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RouterFederatedStateProto;
+import org.apache.hadoop.thirdparty.protobuf.InvalidProtocolBufferException;
+import org.apache.hadoop.thirdparty.protobuf.ByteString;
+
+
+/** Collection of last-seen namespace state Ids for a set of namespaces. */
Review Comment:
What level of sharing is there between these? Is a single one used for a single user? or is it per an incoming connection to the router? I assume they shared across out going connections to the different NameNodes. A comment saying how they are shared would be very useful.
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RouterFederatedStateProto;
+import org.apache.hadoop.thirdparty.protobuf.InvalidProtocolBufferException;
+import org.apache.hadoop.thirdparty.protobuf.ByteString;
+
+
+/** Collection of last-seen namespace state Ids for a set of namespaces. */
+public class FederatedNamespaceIds {
+ private final Map<String, NamespaceStateId> namespaceIdMap = new ConcurrentHashMap<>();
+
+ public void updateStateUsingRequestHeader(RpcHeaderProtos.RpcRequestHeaderProto header) {
+ if (header.hasRouterFederatedState()) {
+ RouterFederatedStateProto federatedState = null;
+ try {
+ federatedState = RouterFederatedStateProto.parseFrom(header.getRouterFederatedState());
+ } catch (InvalidProtocolBufferException e) {
+ throw new RuntimeException(e);
+ }
+ federatedState.getNamespaceStateIdsMap().forEach(this::updateNamespaceState);
Review Comment:
This loop needs a lock around it.
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RouterFederatedStateProto;
+import org.apache.hadoop.thirdparty.protobuf.InvalidProtocolBufferException;
+import org.apache.hadoop.thirdparty.protobuf.ByteString;
+
+
+/** Collection of last-seen namespace state Ids for a set of namespaces. */
+public class FederatedNamespaceIds {
+ private final Map<String, NamespaceStateId> namespaceIdMap = new ConcurrentHashMap<>();
+
+ public void updateStateUsingRequestHeader(RpcHeaderProtos.RpcRequestHeaderProto header) {
+ if (header.hasRouterFederatedState()) {
+ RouterFederatedStateProto federatedState = null;
+ try {
+ federatedState = RouterFederatedStateProto.parseFrom(header.getRouterFederatedState());
+ } catch (InvalidProtocolBufferException e) {
+ throw new RuntimeException(e);
+ }
+ federatedState.getNamespaceStateIdsMap().forEach(this::updateNamespaceState);
+ }
+ }
+
+ public void updateNamespaceState(String nsId, Long stateId) {
Review Comment:
Make this method synchronized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1225307012
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 29s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 31s | | trunk passed |
| +1 :green_heart: | compile | 23m 15s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 40s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 35s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 46s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 27s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 2s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 12s | | the patch passed |
| +1 :green_heart: | compile | 22m 30s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 22m 30s | | the patch passed |
| +1 :green_heart: | compile | 20m 50s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 20m 50s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/24/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 2643 unchanged - 1 fixed = 2644 total (was 2644) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 18s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/24/artifact/out/results-checkstyle-root.txt) | root: The patch generated 4 new + 236 unchanged - 9 fixed = 240 total (was 245) |
| +1 :green_heart: | mvnsite | 7m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 6m 15s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 47s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 54s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 51s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 41s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 17s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 241m 20s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 23m 3s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 33s | | The patch does not generate ASF License warnings. |
| | | 548m 13s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/24/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 69a4b5a7bb43 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7ef4074719633503d20b6c0b2e79a9c99bf227e8 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/24/testReport/ |
| Max. process+thread count | 3482 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/24/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1198065242
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 3 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 24s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 34s | | trunk passed |
| +1 :green_heart: | compile | 23m 33s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 47s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 33s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 49s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 21s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 13s | | the patch passed |
| +1 :green_heart: | compile | 22m 34s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | cc | 22m 34s | | the patch passed |
| +1 :green_heart: | javac | 22m 34s | | the patch passed |
| +1 :green_heart: | compile | 20m 57s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 20m 57s | | the patch passed |
| +1 :green_heart: | javac | 20m 57s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 20s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/19/artifact/out/results-checkstyle-root.txt) | root: The patch generated 4 new + 177 unchanged - 1 fixed = 181 total (was 178) |
| +1 :green_heart: | mvnsite | 7m 45s | | the patch passed |
| +1 :green_heart: | javadoc | 6m 20s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 45s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 55s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 49s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 49s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 18s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 242m 58s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 23m 15s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 58s | | The patch does not generate ASF License warnings. |
| | | 551m 16s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/19/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat |
| uname | Linux db762eab4506 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 01b4b0558a3d6d03403100282772a84403fe9b7a |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/19/testReport/ |
| Max. process+thread count | 2927 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/19/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1241680737
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 48s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 45s | | trunk passed |
| +1 :green_heart: | compile | 23m 12s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 20m 39s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 19s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 48s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 11m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 5s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 4s | | the patch passed |
| +1 :green_heart: | compile | 22m 26s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 22m 26s | | the patch passed |
| +1 :green_heart: | compile | 20m 39s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 39s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 57s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/29/artifact/out/results-checkstyle-root.txt) | root: The patch generated 4 new + 175 unchanged - 8 fixed = 179 total (was 183) |
| +1 :green_heart: | mvnsite | 6m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 11s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 51s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 0s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 30s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 42s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 5s | | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 238m 52s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/29/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 23m 47s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 38s | | The patch does not generate ASF License warnings. |
| | | 535m 16s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
| | hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/29/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux f49301d82fea 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8b59b8077787cd383edf3b25012ba9765556daff |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/29/testReport/ |
| Max. process+thread count | 3152 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/29/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] omalley commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
omalley commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r965216043
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcConstants.java:
##########
@@ -37,7 +37,9 @@ private RpcConstants() {
public static final int INVALID_RETRY_COUNT = -1;
-
+ // Special state ID value to indicate client request header has routerFederatedState set.
Review Comment:
This is left over from the previous version of the patch, right?
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java:
##########
@@ -349,6 +349,9 @@ public static ClientProtocol createProxyWithAlignmentContext(
boolean withRetries, AtomicBoolean fallbackToSimpleAuth,
AlignmentContext alignmentContext)
throws IOException {
+ if (alignmentContext == null) {
Review Comment:
Why do we need to override the null value? The null value means that they don't want to track alignment. Which call path lead to getting null here when it shouldn't?
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterStateIdContext.java:
##########
@@ -0,0 +1,105 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.lang.reflect.Method;
+import java.util.HashSet;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.protocol.ClientProtocol;
+import org.apache.hadoop.hdfs.server.namenode.ha.ReadOnly;
+import org.apache.hadoop.ipc.AlignmentContext;
+import org.apache.hadoop.ipc.RetriableException;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RpcRequestHeaderProto;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RpcResponseHeaderProto;
+
+
+/**
+ * This is the router implementation to hold the state Ids for all
+ * namespaces. This object is only updated by responses from NameNodes.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Evolving
+class RouterStateIdContext implements AlignmentContext {
+
+ private final HashSet<String> coordinatedMethods;
+ private final FederatedNamespaceIds federatedNamespaceIds;
+ /**
+ * Size limit for the map of state Ids to send to clients.
+ */
+ private final int maxSizeOfFederatedStateToPropagate;
+
+ RouterStateIdContext(Configuration conf, FederatedNamespaceIds federatedNamespaceIds) {
+ this.federatedNamespaceIds = federatedNamespaceIds;
+ this.coordinatedMethods = new HashSet<>();
+ // For now, only ClientProtocol methods can be coordinated, so only checking
+ // against ClientProtocol.
+ for (Method method : ClientProtocol.class.getDeclaredMethods()) {
+ if (method.isAnnotationPresent(ReadOnly.class)
+ && method.getAnnotationsByType(ReadOnly.class)[0].isCoordinated()) {
+ coordinatedMethods.add(method.getName());
+ }
+ }
+
+ maxSizeOfFederatedStateToPropagate =
+ conf.getInt(RBFConfigKeys.DFS_ROUTER_OBSERVER_FEDERATED_STATE_PROPAGATION_MAXSIZE,
+ RBFConfigKeys.DFS_ROUTER_OBSERVER_FEDERATED_STATE_PROPAGATION_MAXSIZE_DEFAULT);
+ }
+
+ @Override
+ public void updateResponseState(RpcResponseHeaderProto.Builder header) {
Review Comment:
We should change the behavior instead of a global msync, that any requests that come in without the federated state ids are sent to the active nn. That can be done in a different jira/pr though.
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcClient.java:
##########
@@ -368,8 +370,20 @@ private ConnectionContext getConnection(UserGroupInformation ugi, String nsId,
connUGI = UserGroupInformation.createProxyUser(
ugi.getUserName(), routerUser);
}
+
Review Comment:
Couldn't we move this code inside of ConnectionManager.getConnection and avoid changing the API to add the clientStateId?
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/FederatedNamespaceIds.java:
##########
@@ -0,0 +1,78 @@
+/**
Review Comment:
I think this functionality should be merged into RouterStateIdContext, which serves the same purpose and wraps the instance of this class in the Router.
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/PoolAlignmentContext.java:
##########
@@ -0,0 +1,89 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.io.IOException;
+import org.apache.hadoop.hdfs.NamespaceStateId;
+import org.apache.hadoop.ipc.AlignmentContext;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+
+
Review Comment:
Please add a javadoc that describes how this class is used.
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java:
##########
@@ -252,21 +252,21 @@ public class RouterRpcServer extends AbstractService implements ClientProtocol,
/**
* Construct a router RPC server.
*
- * @param configuration HDFS Configuration.
+ * @param conf HDFS Configuration.
* @param router A router using this RPC server.
* @param nnResolver The NN resolver instance to determine active NNs in HA.
- * @param fileResolver File resolver to resolve file paths to subclusters.
+ * @param fResolver File resolver to resolve file paths to subclusters.
* @throws IOException If the RPC server could not be created.
*/
- public RouterRpcServer(Configuration configuration, Router router,
- ActiveNamenodeResolver nnResolver, FileSubclusterResolver fileResolver)
+ public RouterRpcServer(Configuration conf, Router router,
+ ActiveNamenodeResolver nnResolver, FileSubclusterResolver fResolver)
Review Comment:
Why did you change the name? fileResolver is easier to read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #4311: HDFS-13522: IPC changes to support observer reads through routers.
Posted by GitBox <gi...@apache.org>.
simbadzina commented on code in PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#discussion_r966375731
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/PoolAlignmentContext.java:
##########
@@ -0,0 +1,89 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.io.IOException;
+import org.apache.hadoop.hdfs.NamespaceStateId;
+import org.apache.hadoop.ipc.AlignmentContext;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos;
+
+
Review Comment:
Added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org