You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/10/14 08:41:22 UTC
[GitHub] [hadoop] slfan1989 opened a new pull request, #5032: YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode.
slfan1989 opened a new pull request, #5032:
URL: https://github.com/apache/hadoop/pull/5032
JIRA: YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #5032: YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #5032:
URL: https://github.com/apache/hadoop/pull/5032#issuecomment-1286226835
@goiri Thank you very much for helping to review the code!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5032: YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #5032:
URL: https://github.com/apache/hadoop/pull/5032#discussion_r998847570
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##########
@@ -558,4 +572,35 @@ protected void finalize() {
public Map<String, RequestInterceptorChainWrapper> getUserPipelineMap() {
return userPipelineMap;
}
+
+ /**
+ * Create RouterRMDelegationTokenSecretManager.
+ * In the YARN federation, the Router will replace the RM to
+ * manage the RMDelegationToken (generate, update, cancel),
+ * so the relevant configuration parameters still obtain the configuration parameters of the RM.
+ *
+ * @param conf Configuration
+ * @return RouterDelegationTokenSecretManager.
+ */
+ protected RouterDelegationTokenSecretManager createRouterRMDelegationTokenSecretManager(
+ Configuration conf) {
+
+ long secretKeyInterval = conf.getLong(YarnConfiguration.RM_DELEGATION_KEY_UPDATE_INTERVAL_KEY,
+ YarnConfiguration.RM_DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT);
+ long tokenMaxLifetime = conf.getLong(YarnConfiguration.RM_DELEGATION_TOKEN_MAX_LIFETIME_KEY,
+ YarnConfiguration.RM_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT);
+ long tokenRenewInterval = conf.getLong(YarnConfiguration.RM_DELEGATION_TOKEN_RENEW_INTERVAL_KEY,
+ YarnConfiguration.RM_DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT);
+ long removeScanInterval =
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5032: YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #5032:
URL: https://github.com/apache/hadoop/pull/5032#discussion_r998847383
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##########
@@ -508,6 +516,12 @@ private RequestInterceptorChainWrapper initializePipeline(String user) {
ClientRequestInterceptor interceptorChain =
this.createRequestInterceptorChain();
interceptorChain.init(user);
+
+ //
Review Comment:
Thanks for your help reviewing the code, this line should be commented and I'll fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5032: YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5032:
URL: https://github.com/apache/hadoop/pull/5032#issuecomment-1283348668
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 25s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 14s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 0m 56s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 46s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 5m 22s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not generate ASF License warnings. |
| | | 104m 12s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5032/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5032 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux c9375095785c 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 47758eee0dc0fc4bd5368e7f53d90c95975cd0af |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5032/3/testReport/ |
| Max. process+thread count | 765 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5032/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5032: YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5032:
URL: https://github.com/apache/hadoop/pull/5032#issuecomment-1280393881
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 16s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 44s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 51s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 24s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 24s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 0s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 32s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 5m 16s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not generate ASF License warnings. |
| | | 98m 29s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5032/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5032 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 4125af2f457c 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ba3958b26caa7f2520eef33ebc1e53fb9f33100b |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5032/2/testReport/ |
| Max. process+thread count | 752 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5032/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #5032: YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #5032:
URL: https://github.com/apache/hadoop/pull/5032#issuecomment-1282522830
@goiri Can you help review this pr? Thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri merged pull request #5032: YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode.
Posted by GitBox <gi...@apache.org>.
goiri merged PR #5032:
URL: https://github.com/apache/hadoop/pull/5032
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5032: YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5032:
URL: https://github.com/apache/hadoop/pull/5032#issuecomment-1284243118
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 8s | | trunk passed |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 39s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 39s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 24s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 0m 59s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 2s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 5m 23s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not generate ASF License warnings. |
| | | 101m 18s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5032/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5032 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e2cfb114f8da 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5cb35bcd6b2fe9f289625dfa0ad631c61789240f |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5032/4/testReport/ |
| Max. process+thread count | 748 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5032/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #5032: YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode.
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #5032:
URL: https://github.com/apache/hadoop/pull/5032#discussion_r999706050
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/security/RouterDelegationTokenSecretManager.java:
##########
@@ -0,0 +1,252 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.router.security;
+
+import org.apache.hadoop.classification.InterfaceAudience.Public;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager;
+import org.apache.hadoop.security.token.delegation.DelegationKey;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier;
+import org.apache.hadoop.yarn.security.client.YARNDelegationTokenIdentifier;
+import org.apache.hadoop.yarn.server.federation.store.records.RouterMasterKey;
+import org.apache.hadoop.yarn.server.federation.store.records.RouterMasterKeyResponse;
+import org.apache.hadoop.yarn.server.federation.store.records.RouterRMTokenResponse;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * A Router specific delegation token secret manager.
+ * The secret manager is responsible for generating and accepting the password
+ * for each token.
+ */
+public class RouterDelegationTokenSecretManager
+ extends AbstractDelegationTokenSecretManager<RMDelegationTokenIdentifier> {
+
+ private static final Logger LOG = LoggerFactory
+ .getLogger(RouterDelegationTokenSecretManager.class);
+
+ private FederationStateStoreFacade federationFacade;
+
+ /**
+ * Create a Router Secret manager.
+ *
+ * @param delegationKeyUpdateInterval the number of milliseconds for rolling
+ * new secret keys.
+ * @param delegationTokenMaxLifetime the maximum lifetime of the delegation
+ * tokens in milliseconds
+ * @param delegationTokenRenewInterval how often the tokens must be renewed
+ * in milliseconds
+ * @param delegationTokenRemoverScanInterval how often the tokens are scanned
+ */
+ public RouterDelegationTokenSecretManager(long delegationKeyUpdateInterval,
+ long delegationTokenMaxLifetime, long delegationTokenRenewInterval,
+ long delegationTokenRemoverScanInterval) {
+ super(delegationKeyUpdateInterval, delegationTokenMaxLifetime,
+ delegationTokenRenewInterval, delegationTokenRemoverScanInterval);
+ this.federationFacade = FederationStateStoreFacade.getInstance();
+ }
+
+ @Override
+ public RMDelegationTokenIdentifier createIdentifier() {
+ return new RMDelegationTokenIdentifier();
+ }
+
+ private boolean shouldIgnoreException(Exception e) {
+ return !running && e.getCause() instanceof InterruptedException;
+ }
+
+ /**
+ * The Router Supports Store the New Master Key.
+ * During this Process, Facade will call the specific StateStore to store the MasterKey.
+ *
+ * @param newKey DelegationKey
+ */
+ @Override
+ public void storeNewMasterKey(DelegationKey newKey) {
+ try {
+ federationFacade.storeNewMasterKey(newKey);
+ } catch (Exception e) {
+ if (!shouldIgnoreException(e)) {
+ LOG.error("Error in storing master key with KeyID: {}.", newKey.getKeyId());
+ ExitUtil.terminate(1, e);
+ }
+ }
+ }
+
+ /**
+ * The Router Supports Remove the master key.
+ * During this Process, Facade will call the specific StateStore to remove the MasterKey.
+ *
+ * @param delegationKey DelegationKey
+ */
+ @Override
+ public void removeStoredMasterKey(DelegationKey delegationKey) {
+ try {
+ federationFacade.removeStoredMasterKey(delegationKey);
+ } catch (Exception e) {
+ if (!shouldIgnoreException(e)) {
+ LOG.error("Error in removing master key with KeyID: {}.", delegationKey.getKeyId());
+ ExitUtil.terminate(1, e);
+ }
+ }
+ }
+
+ /**
+ * The Router Supports Store new Token.
+ *
+ * @param identifier RMDelegationToken
+ * @param renewDate renewDate
+ * @throws IOException IO exception occurred.
+ */
+ @Override
+ public void storeNewToken(RMDelegationTokenIdentifier identifier,
+ long renewDate) throws IOException {
+ try {
+ federationFacade.storeNewToken(identifier, renewDate);
+ } catch (Exception e) {
+ if (!shouldIgnoreException(e)) {
+ LOG.error("Error in storing RMDelegationToken with sequence number: {}.",
+ identifier.getSequenceNumber());
+ ExitUtil.terminate(1, e);
+ }
+ }
+ }
+
+ /**
+ * The Router Supports Update Token.
+ *
+ * @param id RMDelegationToken
+ * @param renewDate renewDate
+ * @throws IOException IO exception occurred
+ */
+ @Override
+ public void updateStoredToken(RMDelegationTokenIdentifier id, long renewDate) throws IOException {
+ try {
+ federationFacade.updateStoredToken(id, renewDate);
+ } catch (Exception e) {
+ if (!shouldIgnoreException(e)) {
+ LOG.error("Error in updating persisted RMDelegationToken with sequence number: {}.",
+ id.getSequenceNumber());
+ ExitUtil.terminate(1, e);
+ }
+ }
+ }
+
+ /**
+ * The Router Supports Remove Token.
+ *
+ * @param identifier Delegation Token
+ * @throws IOException IO exception occurred.
+ */
+ @Override
+ public void removeStoredToken(RMDelegationTokenIdentifier identifier) throws IOException {
+ try {
+ federationFacade.removeStoredToken(identifier);
+ } catch (Exception e) {
+ if (!shouldIgnoreException(e)) {
+ LOG.error("Error in removing RMDelegationToken with sequence number: {}",
+ identifier.getSequenceNumber());
+ ExitUtil.terminate(1, e);
+ }
+ }
+ }
+
+ /**
+ * The Router supports obtaining the DelegationKey stored in the Router StateStote
+ * according to the DelegationKey.
+ *
+ * @param key Param DelegationKey
+ * @return Delegation Token
+ * @throws YarnException An internal conversion error occurred when getting the Token
+ * @throws IOException IO exception occurred
+ */
+ public DelegationKey getMasterKeyByDelegationKey(DelegationKey key)
+ throws YarnException, IOException {
+ try {
+ RouterMasterKeyResponse response = federationFacade.getMasterKeyByDelegationKey(key);
+ RouterMasterKey masterKey = response.getRouterMasterKey();
+ ByteBuffer keyByteBuf = masterKey.getKeyBytes();
+ byte[] keyBytes = new byte[keyByteBuf.remaining()];
+ keyByteBuf.get(keyBytes);
+ DelegationKey delegationKey =
+ new DelegationKey(masterKey.getKeyId(), masterKey.getExpiryDate(), keyBytes);
+ return delegationKey;
+ } catch (IOException ex) {
+ throw new IOException(ex);
+ } catch (YarnException ex) {
+ throw new YarnException(ex);
+ }
+ }
+
+ /**
+ * Get RMDelegationTokenIdentifier according to RouterStoreToken.
+ *
+ * @param identifier RMDelegationTokenIdentifier
+ * @return RMDelegationTokenIdentifier
+ * @throws YarnException An internal conversion error occurred when getting the Token
+ * @throws IOException IO exception occurred
+ */
+ public RMDelegationTokenIdentifier getTokenByRouterStoreToken(
+ RMDelegationTokenIdentifier identifier) throws YarnException, IOException {
+ try {
+ RouterRMTokenResponse response = federationFacade.getTokenByRouterStoreToken(identifier);
+ YARNDelegationTokenIdentifier responseIdentifier =
+ response.getRouterStoreToken().getTokenIdentifier();
+ return (RMDelegationTokenIdentifier) responseIdentifier;
+ } catch (Exception ex) {
+ throw new YarnException(ex);
+ }
+ }
+
+ public void setFederationFacade(FederationStateStoreFacade federationFacade) {
+ this.federationFacade = federationFacade;
+ }
+
+ @Public
+ @VisibleForTesting
+ public int getLatestDTSequenceNumber() {
+ return delegationTokenSequenceNumber;
+ }
+
+ @Public
+ @VisibleForTesting
+ public synchronized Set<DelegationKey> getAllMasterKeys() {
+ return new HashSet<>(allKeys.values());
+ }
+
+ @Public
+ @VisibleForTesting
+ public synchronized Map<RMDelegationTokenIdentifier, Long> getAllTokens() {
+ Map<RMDelegationTokenIdentifier, Long> allTokens = new HashMap<>();
+ for (Map.Entry<RMDelegationTokenIdentifier,
+ DelegationTokenInformation> entry : currentTokens.entrySet()) {
+ allTokens.put(entry.getKey(), entry.getValue().getRenewDate());
Review Comment:
It would be better to extract this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5032: YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5032:
URL: https://github.com/apache/hadoop/pull/5032#issuecomment-1284858290
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 59s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 16s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 46s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 0s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 13s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 5m 26s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not generate ASF License warnings. |
| | | 104m 13s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5032/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5032 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 6ef39e54c4d6 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 48a8557465acbb26cc238d999ce33035bf290ed6 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5032/6/testReport/ |
| Max. process+thread count | 858 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5032/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #5032: YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode.
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #5032:
URL: https://github.com/apache/hadoop/pull/5032#discussion_r998791385
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##########
@@ -558,4 +572,35 @@ protected void finalize() {
public Map<String, RequestInterceptorChainWrapper> getUserPipelineMap() {
return userPipelineMap;
}
+
+ /**
+ * Create RouterRMDelegationTokenSecretManager.
+ * In the YARN federation, the Router will replace the RM to
+ * manage the RMDelegationToken (generate, update, cancel),
+ * so the relevant configuration parameters still obtain the configuration parameters of the RM.
+ *
+ * @param conf Configuration
+ * @return RouterDelegationTokenSecretManager.
+ */
+ protected RouterDelegationTokenSecretManager createRouterRMDelegationTokenSecretManager(
+ Configuration conf) {
+
+ long secretKeyInterval = conf.getLong(YarnConfiguration.RM_DELEGATION_KEY_UPDATE_INTERVAL_KEY,
+ YarnConfiguration.RM_DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT);
+ long tokenMaxLifetime = conf.getLong(YarnConfiguration.RM_DELEGATION_TOKEN_MAX_LIFETIME_KEY,
+ YarnConfiguration.RM_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT);
+ long tokenRenewInterval = conf.getLong(YarnConfiguration.RM_DELEGATION_TOKEN_RENEW_INTERVAL_KEY,
+ YarnConfiguration.RM_DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT);
+ long removeScanInterval =
Review Comment:
Make it
```
long removeScanInterval = conf.getTimeDuration(
YarnConfiguration.RM_DELEGATION_TOKEN_REMOVE_SCAN_INTERVAL_KEY,
````
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/security/RouterDelegationTokenSecretManager.java:
##########
@@ -0,0 +1,254 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.router.security;
+
+import org.apache.hadoop.classification.InterfaceAudience.Public;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager;
+import org.apache.hadoop.security.token.delegation.DelegationKey;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier;
+import org.apache.hadoop.yarn.security.client.YARNDelegationTokenIdentifier;
+import org.apache.hadoop.yarn.server.federation.store.records.RouterMasterKey;
+import org.apache.hadoop.yarn.server.federation.store.records.RouterMasterKeyResponse;
+import org.apache.hadoop.yarn.server.federation.store.records.RouterRMTokenResponse;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * A Router specific delegation token secret manager.
+ * The secret manager is responsible for generating and accepting the password
+ * for each token.
+ */
+public class RouterDelegationTokenSecretManager
+ extends AbstractDelegationTokenSecretManager<RMDelegationTokenIdentifier> {
+
+ private static final Logger LOG = LoggerFactory
+ .getLogger(RouterDelegationTokenSecretManager.class);
+
+ private FederationStateStoreFacade federationFacade;
+
+ /**
+ * Create a Router Secret manager.
+ *
+ * @param delegationKeyUpdateInterval the number of milliseconds for rolling
+ * new secret keys.
+ * @param delegationTokenMaxLifetime the maximum lifetime of the delegation
+ * tokens in milliseconds
+ * @param delegationTokenRenewInterval how often the tokens must be renewed
+ * in milliseconds
+ * @param delegationTokenRemoverScanInterval how often the tokens are scanned
+ */
+ public RouterDelegationTokenSecretManager(long delegationKeyUpdateInterval,
+ long delegationTokenMaxLifetime, long delegationTokenRenewInterval,
+ long delegationTokenRemoverScanInterval) {
+ super(delegationKeyUpdateInterval, delegationTokenMaxLifetime,
+ delegationTokenRenewInterval, delegationTokenRemoverScanInterval);
+ this.federationFacade = FederationStateStoreFacade.getInstance();
+ }
+
+ @Override
+ public RMDelegationTokenIdentifier createIdentifier() {
+ return new RMDelegationTokenIdentifier();
+ }
+
+ private boolean shouldIgnoreException(Exception e) {
+ return !running && e.getCause() instanceof InterruptedException;
+ }
+
+ /**
+ * The Router Supports Store the New Master Key.
+ * During this Process, Facade will call the specific StateStore to store the MasterKey.
+ *
+ * @param newKey DelegationKey
+ */
+ @Override
+ public void storeNewMasterKey(DelegationKey newKey) {
+ try {
+ federationFacade.storeNewMasterKey(newKey);
+ } catch (Exception e) {
+ if (!shouldIgnoreException(e)) {
+ LOG.error("Error in storing master key with KeyID: {}.", newKey.getKeyId());
+ ExitUtil.terminate(1, e);
+ }
+ }
+ }
+
+ /**
+ * The Router Supports Remove the master key.
+ * During this Process, Facade will call the specific StateStore to remove the MasterKey.
+ *
+ * @param delegationKey DelegationKey
+ */
+ @Override
+ public void removeStoredMasterKey(DelegationKey delegationKey) {
+ try {
+ federationFacade.removeStoredMasterKey(delegationKey);
+ } catch (Exception e) {
+ if (!shouldIgnoreException(e)) {
+ LOG.error("Error in removing master key with KeyID: {}.", delegationKey.getKeyId());
+ ExitUtil.terminate(1, e);
+ }
+ }
+ }
+
+ /**
+ * The Router Supports Store new Token.
+ *
+ * @param identifier RMDelegationToken
+ * @param renewDate renewDate
+ * @throws IOException IO exception occurred.
+ */
+ @Override
+ public void storeNewToken(RMDelegationTokenIdentifier identifier,
+ long renewDate) throws IOException {
+ try {
+ federationFacade.storeNewToken(identifier, renewDate);
+ } catch (Exception e) {
+ if (!shouldIgnoreException(e)) {
+ LOG.error("Error in storing RMDelegationToken with sequence number: {}.",
+ identifier.getSequenceNumber());
+ ExitUtil.terminate(1, e);
+ }
+ }
+ }
+
+ /**
+ * The Router Supports Update Token.
+ *
+ * @param id RMDelegationToken
+ * @param renewDate renewDate
+ * @throws IOException IO exception occurred
+ */
+ @Override
+ public void updateStoredToken(RMDelegationTokenIdentifier id, long renewDate) throws IOException {
+ try {
+ federationFacade.updateStoredToken(id, renewDate);
+ } catch (Exception e) {
+ if (!shouldIgnoreException(e)) {
+ LOG.error("Error in updating persisted RMDelegationToken with sequence number: {}.",
+ id.getSequenceNumber());
+ ExitUtil.terminate(1, e);
+ }
+ }
+ }
+
+ /**
+ * The Router Supports Remove Token.
+ *
+ * @param identifier Delegation Token
+ * @throws IOException IO exception occurred.
+ */
+ @Override
+ public void removeStoredToken(RMDelegationTokenIdentifier identifier) throws IOException {
+ try {
+ federationFacade.removeStoredToken(identifier);
+ } catch (Exception e) {
+ if (!shouldIgnoreException(e)) {
+ LOG.error("Error in removing RMDelegationToken with sequence number: {}",
+ identifier.getSequenceNumber());
+ ExitUtil.terminate(1, e);
+ }
+ }
+ }
+
+ /**
+ * The Router supports obtaining the DelegationKey stored in the Router StateStote
+ * according to the DelegationKey.
+ *
+ * @param key Param DelegationKey
+ * @return Delegation Token
+ * @throws YarnException An internal conversion error occurred when getting the Token
+ * @throws IOException IO exception occurred
+ */
+ public DelegationKey getMasterKeyByDelegationKey(DelegationKey key)
+ throws YarnException, IOException {
+ try {
+ RouterMasterKeyResponse response = federationFacade.getMasterKeyByDelegationKey(key);
+ RouterMasterKey masterKey = response.getRouterMasterKey();
+ ByteBuffer keyByteBuf = masterKey.getKeyBytes();
+ byte[] keyBytes = new byte[keyByteBuf.remaining()];
+ keyByteBuf.get(keyBytes);
+ DelegationKey delegationKey =
+ new DelegationKey(masterKey.getKeyId(), masterKey.getExpiryDate(), keyBytes);
+ return delegationKey;
+ } catch (IOException ex) {
+ throw new IOException(ex);
+ } catch (YarnException ex) {
+ throw new YarnException(ex);
+ }
+ }
+
+ /**
+ * Get RMDelegationTokenIdentifier according to RouterStoreToken.
+ *
+ * @param identifier RMDelegationTokenIdentifier
+ * @return RMDelegationTokenIdentifier
+ * @throws YarnException An internal conversion error occurred when getting the Token
+ * @throws IOException IO exception occurred
+ */
+ public RMDelegationTokenIdentifier getTokenByRouterStoreToken(
+ RMDelegationTokenIdentifier identifier) throws YarnException, IOException {
+ try {
+ RouterRMTokenResponse response = federationFacade.getTokenByRouterStoreToken(identifier);
+ YARNDelegationTokenIdentifier responseIdentifier =
+ response.getRouterStoreToken().getTokenIdentifier();
+ return (RMDelegationTokenIdentifier) responseIdentifier;
+ } catch (Exception ex) {
+ throw new YarnException(ex);
+ }
+ }
+
+ public void setFederationFacade(FederationStateStoreFacade federationFacade) {
+ this.federationFacade = federationFacade;
+ }
+
+ @Public
+ @VisibleForTesting
+ public int getLatestDTSequenceNumber() {
+ return delegationTokenSequenceNumber;
+ }
+
+ @Public
+ @VisibleForTesting
+ public synchronized Set<DelegationKey> getAllMasterKeys() {
+ HashSet<DelegationKey> keySet = new HashSet<>();
Review Comment:
return new HashSet<>(allKeys.values());
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##########
@@ -508,6 +516,12 @@ private RequestInterceptorChainWrapper initializePipeline(String user) {
ClientRequestInterceptor interceptorChain =
this.createRequestInterceptorChain();
interceptorChain.init(user);
+
+ //
Review Comment:
Empty?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5032: YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #5032:
URL: https://github.com/apache/hadoop/pull/5032#discussion_r998852009
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/security/RouterDelegationTokenSecretManager.java:
##########
@@ -0,0 +1,254 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.router.security;
+
+import org.apache.hadoop.classification.InterfaceAudience.Public;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager;
+import org.apache.hadoop.security.token.delegation.DelegationKey;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier;
+import org.apache.hadoop.yarn.security.client.YARNDelegationTokenIdentifier;
+import org.apache.hadoop.yarn.server.federation.store.records.RouterMasterKey;
+import org.apache.hadoop.yarn.server.federation.store.records.RouterMasterKeyResponse;
+import org.apache.hadoop.yarn.server.federation.store.records.RouterRMTokenResponse;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * A Router specific delegation token secret manager.
+ * The secret manager is responsible for generating and accepting the password
+ * for each token.
+ */
+public class RouterDelegationTokenSecretManager
+ extends AbstractDelegationTokenSecretManager<RMDelegationTokenIdentifier> {
+
+ private static final Logger LOG = LoggerFactory
+ .getLogger(RouterDelegationTokenSecretManager.class);
+
+ private FederationStateStoreFacade federationFacade;
+
+ /**
+ * Create a Router Secret manager.
+ *
+ * @param delegationKeyUpdateInterval the number of milliseconds for rolling
+ * new secret keys.
+ * @param delegationTokenMaxLifetime the maximum lifetime of the delegation
+ * tokens in milliseconds
+ * @param delegationTokenRenewInterval how often the tokens must be renewed
+ * in milliseconds
+ * @param delegationTokenRemoverScanInterval how often the tokens are scanned
+ */
+ public RouterDelegationTokenSecretManager(long delegationKeyUpdateInterval,
+ long delegationTokenMaxLifetime, long delegationTokenRenewInterval,
+ long delegationTokenRemoverScanInterval) {
+ super(delegationKeyUpdateInterval, delegationTokenMaxLifetime,
+ delegationTokenRenewInterval, delegationTokenRemoverScanInterval);
+ this.federationFacade = FederationStateStoreFacade.getInstance();
+ }
+
+ @Override
+ public RMDelegationTokenIdentifier createIdentifier() {
+ return new RMDelegationTokenIdentifier();
+ }
+
+ private boolean shouldIgnoreException(Exception e) {
+ return !running && e.getCause() instanceof InterruptedException;
+ }
+
+ /**
+ * The Router Supports Store the New Master Key.
+ * During this Process, Facade will call the specific StateStore to store the MasterKey.
+ *
+ * @param newKey DelegationKey
+ */
+ @Override
+ public void storeNewMasterKey(DelegationKey newKey) {
+ try {
+ federationFacade.storeNewMasterKey(newKey);
+ } catch (Exception e) {
+ if (!shouldIgnoreException(e)) {
+ LOG.error("Error in storing master key with KeyID: {}.", newKey.getKeyId());
+ ExitUtil.terminate(1, e);
+ }
+ }
+ }
+
+ /**
+ * The Router Supports Remove the master key.
+ * During this Process, Facade will call the specific StateStore to remove the MasterKey.
+ *
+ * @param delegationKey DelegationKey
+ */
+ @Override
+ public void removeStoredMasterKey(DelegationKey delegationKey) {
+ try {
+ federationFacade.removeStoredMasterKey(delegationKey);
+ } catch (Exception e) {
+ if (!shouldIgnoreException(e)) {
+ LOG.error("Error in removing master key with KeyID: {}.", delegationKey.getKeyId());
+ ExitUtil.terminate(1, e);
+ }
+ }
+ }
+
+ /**
+ * The Router Supports Store new Token.
+ *
+ * @param identifier RMDelegationToken
+ * @param renewDate renewDate
+ * @throws IOException IO exception occurred.
+ */
+ @Override
+ public void storeNewToken(RMDelegationTokenIdentifier identifier,
+ long renewDate) throws IOException {
+ try {
+ federationFacade.storeNewToken(identifier, renewDate);
+ } catch (Exception e) {
+ if (!shouldIgnoreException(e)) {
+ LOG.error("Error in storing RMDelegationToken with sequence number: {}.",
+ identifier.getSequenceNumber());
+ ExitUtil.terminate(1, e);
+ }
+ }
+ }
+
+ /**
+ * The Router Supports Update Token.
+ *
+ * @param id RMDelegationToken
+ * @param renewDate renewDate
+ * @throws IOException IO exception occurred
+ */
+ @Override
+ public void updateStoredToken(RMDelegationTokenIdentifier id, long renewDate) throws IOException {
+ try {
+ federationFacade.updateStoredToken(id, renewDate);
+ } catch (Exception e) {
+ if (!shouldIgnoreException(e)) {
+ LOG.error("Error in updating persisted RMDelegationToken with sequence number: {}.",
+ id.getSequenceNumber());
+ ExitUtil.terminate(1, e);
+ }
+ }
+ }
+
+ /**
+ * The Router Supports Remove Token.
+ *
+ * @param identifier Delegation Token
+ * @throws IOException IO exception occurred.
+ */
+ @Override
+ public void removeStoredToken(RMDelegationTokenIdentifier identifier) throws IOException {
+ try {
+ federationFacade.removeStoredToken(identifier);
+ } catch (Exception e) {
+ if (!shouldIgnoreException(e)) {
+ LOG.error("Error in removing RMDelegationToken with sequence number: {}",
+ identifier.getSequenceNumber());
+ ExitUtil.terminate(1, e);
+ }
+ }
+ }
+
+ /**
+ * The Router supports obtaining the DelegationKey stored in the Router StateStote
+ * according to the DelegationKey.
+ *
+ * @param key Param DelegationKey
+ * @return Delegation Token
+ * @throws YarnException An internal conversion error occurred when getting the Token
+ * @throws IOException IO exception occurred
+ */
+ public DelegationKey getMasterKeyByDelegationKey(DelegationKey key)
+ throws YarnException, IOException {
+ try {
+ RouterMasterKeyResponse response = federationFacade.getMasterKeyByDelegationKey(key);
+ RouterMasterKey masterKey = response.getRouterMasterKey();
+ ByteBuffer keyByteBuf = masterKey.getKeyBytes();
+ byte[] keyBytes = new byte[keyByteBuf.remaining()];
+ keyByteBuf.get(keyBytes);
+ DelegationKey delegationKey =
+ new DelegationKey(masterKey.getKeyId(), masterKey.getExpiryDate(), keyBytes);
+ return delegationKey;
+ } catch (IOException ex) {
+ throw new IOException(ex);
+ } catch (YarnException ex) {
+ throw new YarnException(ex);
+ }
+ }
+
+ /**
+ * Get RMDelegationTokenIdentifier according to RouterStoreToken.
+ *
+ * @param identifier RMDelegationTokenIdentifier
+ * @return RMDelegationTokenIdentifier
+ * @throws YarnException An internal conversion error occurred when getting the Token
+ * @throws IOException IO exception occurred
+ */
+ public RMDelegationTokenIdentifier getTokenByRouterStoreToken(
+ RMDelegationTokenIdentifier identifier) throws YarnException, IOException {
+ try {
+ RouterRMTokenResponse response = federationFacade.getTokenByRouterStoreToken(identifier);
+ YARNDelegationTokenIdentifier responseIdentifier =
+ response.getRouterStoreToken().getTokenIdentifier();
+ return (RMDelegationTokenIdentifier) responseIdentifier;
+ } catch (Exception ex) {
+ throw new YarnException(ex);
+ }
+ }
+
+ public void setFederationFacade(FederationStateStoreFacade federationFacade) {
+ this.federationFacade = federationFacade;
+ }
+
+ @Public
+ @VisibleForTesting
+ public int getLatestDTSequenceNumber() {
+ return delegationTokenSequenceNumber;
+ }
+
+ @Public
+ @VisibleForTesting
+ public synchronized Set<DelegationKey> getAllMasterKeys() {
+ HashSet<DelegationKey> keySet = new HashSet<>();
Review Comment:
Thanks for your suggestion, I will modify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5032: YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5032:
URL: https://github.com/apache/hadoop/pull/5032#issuecomment-1278807192
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 4s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 49s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 22s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 0m 59s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 13s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 5m 20s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not generate ASF License warnings. |
| | | 98m 32s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5032/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5032 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 33c63b34d102 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d77f175bb367ad306939c89235cd5319a6122e07 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5032/1/testReport/ |
| Max. process+thread count | 829 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5032/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #5032: YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #5032:
URL: https://github.com/apache/hadoop/pull/5032#issuecomment-1279165570
@goiri Can you help review this pr? Thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5032: YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #5032:
URL: https://github.com/apache/hadoop/pull/5032#discussion_r999929174
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/security/RouterDelegationTokenSecretManager.java:
##########
@@ -0,0 +1,252 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.router.security;
+
+import org.apache.hadoop.classification.InterfaceAudience.Public;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager;
+import org.apache.hadoop.security.token.delegation.DelegationKey;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier;
+import org.apache.hadoop.yarn.security.client.YARNDelegationTokenIdentifier;
+import org.apache.hadoop.yarn.server.federation.store.records.RouterMasterKey;
+import org.apache.hadoop.yarn.server.federation.store.records.RouterMasterKeyResponse;
+import org.apache.hadoop.yarn.server.federation.store.records.RouterRMTokenResponse;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * A Router specific delegation token secret manager.
+ * The secret manager is responsible for generating and accepting the password
+ * for each token.
+ */
+public class RouterDelegationTokenSecretManager
+ extends AbstractDelegationTokenSecretManager<RMDelegationTokenIdentifier> {
+
+ private static final Logger LOG = LoggerFactory
+ .getLogger(RouterDelegationTokenSecretManager.class);
+
+ private FederationStateStoreFacade federationFacade;
+
+ /**
+ * Create a Router Secret manager.
+ *
+ * @param delegationKeyUpdateInterval the number of milliseconds for rolling
+ * new secret keys.
+ * @param delegationTokenMaxLifetime the maximum lifetime of the delegation
+ * tokens in milliseconds
+ * @param delegationTokenRenewInterval how often the tokens must be renewed
+ * in milliseconds
+ * @param delegationTokenRemoverScanInterval how often the tokens are scanned
+ */
+ public RouterDelegationTokenSecretManager(long delegationKeyUpdateInterval,
+ long delegationTokenMaxLifetime, long delegationTokenRenewInterval,
+ long delegationTokenRemoverScanInterval) {
+ super(delegationKeyUpdateInterval, delegationTokenMaxLifetime,
+ delegationTokenRenewInterval, delegationTokenRemoverScanInterval);
+ this.federationFacade = FederationStateStoreFacade.getInstance();
+ }
+
+ @Override
+ public RMDelegationTokenIdentifier createIdentifier() {
+ return new RMDelegationTokenIdentifier();
+ }
+
+ private boolean shouldIgnoreException(Exception e) {
+ return !running && e.getCause() instanceof InterruptedException;
+ }
+
+ /**
+ * The Router Supports Store the New Master Key.
+ * During this Process, Facade will call the specific StateStore to store the MasterKey.
+ *
+ * @param newKey DelegationKey
+ */
+ @Override
+ public void storeNewMasterKey(DelegationKey newKey) {
+ try {
+ federationFacade.storeNewMasterKey(newKey);
+ } catch (Exception e) {
+ if (!shouldIgnoreException(e)) {
+ LOG.error("Error in storing master key with KeyID: {}.", newKey.getKeyId());
+ ExitUtil.terminate(1, e);
+ }
+ }
+ }
+
+ /**
+ * The Router Supports Remove the master key.
+ * During this Process, Facade will call the specific StateStore to remove the MasterKey.
+ *
+ * @param delegationKey DelegationKey
+ */
+ @Override
+ public void removeStoredMasterKey(DelegationKey delegationKey) {
+ try {
+ federationFacade.removeStoredMasterKey(delegationKey);
+ } catch (Exception e) {
+ if (!shouldIgnoreException(e)) {
+ LOG.error("Error in removing master key with KeyID: {}.", delegationKey.getKeyId());
+ ExitUtil.terminate(1, e);
+ }
+ }
+ }
+
+ /**
+ * The Router Supports Store new Token.
+ *
+ * @param identifier RMDelegationToken
+ * @param renewDate renewDate
+ * @throws IOException IO exception occurred.
+ */
+ @Override
+ public void storeNewToken(RMDelegationTokenIdentifier identifier,
+ long renewDate) throws IOException {
+ try {
+ federationFacade.storeNewToken(identifier, renewDate);
+ } catch (Exception e) {
+ if (!shouldIgnoreException(e)) {
+ LOG.error("Error in storing RMDelegationToken with sequence number: {}.",
+ identifier.getSequenceNumber());
+ ExitUtil.terminate(1, e);
+ }
+ }
+ }
+
+ /**
+ * The Router Supports Update Token.
+ *
+ * @param id RMDelegationToken
+ * @param renewDate renewDate
+ * @throws IOException IO exception occurred
+ */
+ @Override
+ public void updateStoredToken(RMDelegationTokenIdentifier id, long renewDate) throws IOException {
+ try {
+ federationFacade.updateStoredToken(id, renewDate);
+ } catch (Exception e) {
+ if (!shouldIgnoreException(e)) {
+ LOG.error("Error in updating persisted RMDelegationToken with sequence number: {}.",
+ id.getSequenceNumber());
+ ExitUtil.terminate(1, e);
+ }
+ }
+ }
+
+ /**
+ * The Router Supports Remove Token.
+ *
+ * @param identifier Delegation Token
+ * @throws IOException IO exception occurred.
+ */
+ @Override
+ public void removeStoredToken(RMDelegationTokenIdentifier identifier) throws IOException {
+ try {
+ federationFacade.removeStoredToken(identifier);
+ } catch (Exception e) {
+ if (!shouldIgnoreException(e)) {
+ LOG.error("Error in removing RMDelegationToken with sequence number: {}",
+ identifier.getSequenceNumber());
+ ExitUtil.terminate(1, e);
+ }
+ }
+ }
+
+ /**
+ * The Router supports obtaining the DelegationKey stored in the Router StateStote
+ * according to the DelegationKey.
+ *
+ * @param key Param DelegationKey
+ * @return Delegation Token
+ * @throws YarnException An internal conversion error occurred when getting the Token
+ * @throws IOException IO exception occurred
+ */
+ public DelegationKey getMasterKeyByDelegationKey(DelegationKey key)
+ throws YarnException, IOException {
+ try {
+ RouterMasterKeyResponse response = federationFacade.getMasterKeyByDelegationKey(key);
+ RouterMasterKey masterKey = response.getRouterMasterKey();
+ ByteBuffer keyByteBuf = masterKey.getKeyBytes();
+ byte[] keyBytes = new byte[keyByteBuf.remaining()];
+ keyByteBuf.get(keyBytes);
+ DelegationKey delegationKey =
+ new DelegationKey(masterKey.getKeyId(), masterKey.getExpiryDate(), keyBytes);
+ return delegationKey;
+ } catch (IOException ex) {
+ throw new IOException(ex);
+ } catch (YarnException ex) {
+ throw new YarnException(ex);
+ }
+ }
+
+ /**
+ * Get RMDelegationTokenIdentifier according to RouterStoreToken.
+ *
+ * @param identifier RMDelegationTokenIdentifier
+ * @return RMDelegationTokenIdentifier
+ * @throws YarnException An internal conversion error occurred when getting the Token
+ * @throws IOException IO exception occurred
+ */
+ public RMDelegationTokenIdentifier getTokenByRouterStoreToken(
+ RMDelegationTokenIdentifier identifier) throws YarnException, IOException {
+ try {
+ RouterRMTokenResponse response = federationFacade.getTokenByRouterStoreToken(identifier);
+ YARNDelegationTokenIdentifier responseIdentifier =
+ response.getRouterStoreToken().getTokenIdentifier();
+ return (RMDelegationTokenIdentifier) responseIdentifier;
+ } catch (Exception ex) {
+ throw new YarnException(ex);
+ }
+ }
+
+ public void setFederationFacade(FederationStateStoreFacade federationFacade) {
+ this.federationFacade = federationFacade;
+ }
+
+ @Public
+ @VisibleForTesting
+ public int getLatestDTSequenceNumber() {
+ return delegationTokenSequenceNumber;
+ }
+
+ @Public
+ @VisibleForTesting
+ public synchronized Set<DelegationKey> getAllMasterKeys() {
+ return new HashSet<>(allKeys.values());
+ }
+
+ @Public
+ @VisibleForTesting
+ public synchronized Map<RMDelegationTokenIdentifier, Long> getAllTokens() {
+ Map<RMDelegationTokenIdentifier, Long> allTokens = new HashMap<>();
+ for (Map.Entry<RMDelegationTokenIdentifier,
+ DelegationTokenInformation> entry : currentTokens.entrySet()) {
+ allTokens.put(entry.getKey(), entry.getValue().getRenewDate());
Review Comment:
I will refactor this part of the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5032: YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5032:
URL: https://github.com/apache/hadoop/pull/5032#issuecomment-1284256005
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 14s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 52s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 38s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 44s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 0m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 49s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 5m 13s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not generate ASF License warnings. |
| | | 110m 18s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5032/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5032 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 036f6e94acc9 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5cb35bcd6b2fe9f289625dfa0ad631c61789240f |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5032/5/testReport/ |
| Max. process+thread count | 818 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5032/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org