You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/11/14 12:39:01 UTC
[GitHub] [hadoop] zhengchenyu opened a new pull request, #5132: HDFS-16830. [SBN READ] dfsrouter transmit state id according to clie…
zhengchenyu opened a new pull request, #5132:
URL: https://github.com/apache/hadoop/pull/5132
### Description of PR
For now, dfsrouter transmit the state of all nameservice. In this PR, dfsrouter only transmit state id according to client's demand.
### How was this patch tested?
unit test.
### For code changes:
Key code is PoolAlignmentContext::updateRequestState, here we record used namenode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu closed pull request #5132: HDFS-16830. [SBN READ] dfsrouter transmit state id according to clie…
Posted by GitBox <gi...@apache.org>.
zhengchenyu closed pull request #5132: HDFS-16830. [SBN READ] dfsrouter transmit state id according to clie…
URL: https://github.com/apache/hadoop/pull/5132
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5132: HDFS-16830. [SBN READ] dfsrouter transmit state id according to clie…
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5132:
URL: https://github.com/apache/hadoop/pull/5132#issuecomment-1320197295
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 23s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 3 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 49s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 0m 30s | [/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/branch-mvninstall-root.txt) | root in trunk failed. |
| +1 :green_heart: | compile | 40m 54s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 22m 23s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 5m 49s | | trunk passed |
| -1 :x: | mvnsite | 1m 17s | [/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt) | hadoop-common in trunk failed. |
| -1 :x: | mvnsite | 0m 48s | [/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in trunk failed. |
| +1 :green_heart: | javadoc | 4m 56s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 11s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| -1 :x: | spotbugs | 0m 42s | [/branch-spotbugs-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common.txt) | hadoop-common in trunk failed. |
| -1 :x: | spotbugs | 0m 41s | [/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in trunk failed. |
| +1 :green_heart: | shadedclient | 25m 50s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 14s | [/patch-mvninstall-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| -1 :x: | mvninstall | 0m 41s | [/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-client.txt) | hadoop-hdfs-client in the patch failed. |
| -1 :x: | mvninstall | 0m 14s | [/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch failed. |
| -1 :x: | mvninstall | 0m 28s | [/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) | hadoop-hdfs-rbf in the patch failed. |
| +1 :green_heart: | compile | 24m 39s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 24m 39s | | the patch passed |
| +1 :green_heart: | javac | 24m 39s | | the patch passed |
| +1 :green_heart: | compile | 24m 15s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 24m 15s | | the patch passed |
| +1 :green_heart: | javac | 24m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 5m 4s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/results-checkstyle-root.txt) | root: The patch generated 3 new + 163 unchanged - 1 fixed = 166 total (was 164) |
| -1 :x: | mvnsite | 1m 1s | [/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| -1 :x: | mvnsite | 1m 34s | [/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-client.txt) | hadoop-hdfs-client in the patch failed. |
| -1 :x: | mvnsite | 1m 6s | [/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch failed. |
| -1 :x: | mvnsite | 1m 16s | [/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) | hadoop-hdfs-rbf in the patch failed. |
| -1 :x: | javadoc | 1m 19s | [/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt) | hadoop-common-project_hadoop-common-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | javadoc | 5m 34s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| -1 :x: | spotbugs | 0m 48s | [/patch-spotbugs-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/patch-spotbugs-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| -1 :x: | spotbugs | 1m 18s | [/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt) | hadoop-hdfs-client in the patch failed. |
| -1 :x: | spotbugs | 0m 43s | [/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch failed. |
| -1 :x: | spotbugs | 0m 56s | [/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) | hadoop-hdfs-rbf in the patch failed. |
| +1 :green_heart: | shadedclient | 56m 46s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 0m 42s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| -1 :x: | unit | 1m 9s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt) | hadoop-hdfs-client in the patch failed. |
| -1 :x: | unit | 0m 42s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch failed. |
| -1 :x: | unit | 0m 41s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) | hadoop-hdfs-rbf in the patch failed. |
| +1 :green_heart: | asflicense | 1m 8s | | The patch does not generate ASF License warnings. |
| | | 229m 10s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5132 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat |
| uname | Linux 3ec06b769e5a 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c432a3ce3cba2cf215aafdc79b3f5ba434c052c0 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/testReport/ |
| Max. process+thread count | 598 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5132/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on pull request #5132: HDFS-16830. [SBN READ] dfsrouter transmit state id according to clie…
Posted by GitBox <gi...@apache.org>.
simbadzina commented on PR #5132:
URL: https://github.com/apache/hadoop/pull/5132#issuecomment-1319544438
> @simbadzina @omalley @ZanderXu Can you please review this PR? It is unnecessary to transmit all state id.
Thanks for the patch @zhengchenyu.
When thinking of a way to not send all state IDs I ran into the following problem.
1) Client makes call and has **ns0** only in it's header.
2) Router connects to **ns1** to serve the clients call.
How do you ensure **ns1** is included in the response to the client?
Could you also please rebase the patch on the current trunk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on pull request #5132: HDFS-16830. [SBN READ] dfsrouter transmit state id according to clie…
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on PR #5132:
URL: https://github.com/apache/hadoop/pull/5132#issuecomment-1319904398
## 1 Background
After HDFS-13522, observer read on RBF is supported. For now, client and router transmit state id of all nameservice, this may has two disadvantage:
* may carry lots of unnecessary data.
* msync must invokeConcurrent on all nameservice.
> Note: In my experience, the call which invokeConcurrent on all nameservice will harm to the stability of dfsrouter. For example , in our production, after apply HDFS-16283, avoid renewlease invoke on all nameservice, the callqueuelength of router have keep very low.
## 2 How to transmit state id according to client's demand?
The problem is that client can't know the downstreaming nameservice.
But in PoolAlignmentContext::updateRequestState, we will know the nameservice, we could record it in the call.
Here is an example:
* Initial state: There are no state id in client. Client make call1 without any federated state id.
* Router will find the downstreaming nameservice, then invoke PoolAlignmentContext::updateRequestState. Here we can record the nsid and stateid in call1.
* When Router will return the response to client (RouterStateIdContext::updateResponseState). we can get the nsid and stateid from call1, then return to client.
* When client recieve the call from client, client will store the state id of ns1.
* When client reconnect to dfsrouter, client will make call2 with ns1. This means we transmit the state id according to client's demand.
## 3 How to msync?
In RouterClientProtocol::msync, we can get the nameservice from the current call. Then we can invoke msync to downstreaming namenode according to client's demand.
## 4 How to handle the first msync?
There is a problem which I did not mention in section 2. How to handle the first mysnc?
In initial state, there are no federated state ids in client side, dfsrouter will not invoke mysnc to any namenode.
In next call, the first call to every downstreaming namenode except msync, RouterStateIdContext.getClientStateIdFromCurrentCall will return Long.MIN_VALUE, will connect to active. Then the following read call to every downstreaming namenode , will connect to observer namenode.
> Note: In fact, I prefer that do msync in router side, do not msync in client side, just like https://issues.apache.org/jira/secure/attachment/13011000/HDFS-13522_WIP.patch. Then there is no problem about first msync.
## 5 What about the unit test?
/ns0 is mount on ns0, /ns1 is mount on ns1. fs0 can only transmit ns0 state id. fs1 can only transmit ns1 state id.
Set DFS_ROUTER_OBSERVER_FEDERATED_STATE_PROPAGATION_MAXSIZE to 1, then not allow transmit more than one state ids.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on pull request #5132: HDFS-16830. [SBN READ] dfsrouter transmit state id according to clie…
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on PR #5132:
URL: https://github.com/apache/hadoop/pull/5132#issuecomment-1313633460
@simbadzina @omalley @ZanderXu Can you please review this PR? It is unnecessary to transmit all state id.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org