You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/06/30 08:02:12 UTC
[GitHub] [hadoop] fengnanli opened a new pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
fengnanli opened a new pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110
## NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#issuecomment-654618574
Hmm @fengnanli it seems not all comments were addressed in the [latest commit](https://github.com/apache/hadoop/pull/2110/commits/eee3bf835215916aaf1632920523d612bee5429f)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on a change in pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#discussion_r447965794
##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RBFConfigKeys.java
##########
@@ -79,6 +79,10 @@
public static final Class<? extends RouterRpcMonitor>
DFS_ROUTER_METRICS_CLASS_DEFAULT =
FederationRPCPerformanceMonitor.class;
+ public static final String DFS_ROUTER_METRICS_TOP_NUM_TOKEN_OWNERS_KEY =
Review comment:
not sure if there is a metrics.md page for RBF - if so we should add there too.
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -726,4 +732,41 @@ public TokenIdent decodeTokenIdentifier(Token<TokenIdent> token) throws IOExcept
return token.decodeIdentifier();
}
+ /**
+ * Return top token real owners list as well as the tokens count.
+ *
+ * @param n top number of users
+ * @return map of owners to counts
+ */
+ public List<NameValuePair> getTopTokenRealOwners(int n) {
Review comment:
will this get pretty expensive if there are lots of tokens stored? as every metrics pull needs to iterate through all tokens.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#issuecomment-651720321
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 19s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 9s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 16s | trunk passed |
| +1 :green_heart: | compile | 19m 23s | trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 |
| +1 :green_heart: | compile | 16m 35s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | checkstyle | 2m 40s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 17s | trunk passed |
| +1 :green_heart: | shadedclient | 19m 33s | branch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 44s | hadoop-common in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| -1 :x: | javadoc | 0m 41s | hadoop-hdfs-rbf in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| +1 :green_heart: | javadoc | 1m 51s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +0 :ok: | spotbugs | 1m 23s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 31s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 20s | the patch passed |
| +1 :green_heart: | compile | 18m 35s | the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 |
| +1 :green_heart: | javac | 18m 35s | the patch passed |
| +1 :green_heart: | compile | 16m 39s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | javac | 16m 39s | the patch passed |
| -0 :warning: | checkstyle | 2m 38s | root: The patch generated 1 new + 46 unchanged - 0 fixed = 47 total (was 46) |
| +1 :green_heart: | mvnsite | 2m 18s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 58s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 44s | hadoop-common in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| -1 :x: | javadoc | 0m 40s | hadoop-hdfs-rbf in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| +1 :green_heart: | javadoc | 1m 48s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | findbugs | 3m 49s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 27s | hadoop-common in the patch passed. |
| -1 :x: | unit | 8m 15s | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 55s | The patch does not generate ASF License warnings. |
| | | 171m 3s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.federation.router.TestRBFConfigFields |
| | hadoop.hdfs.server.federation.security.TestRouterSecurityManager |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2110 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 8a06221d6636 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / cd188ea9f0e |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/1/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/1/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/1/artifact/out/diff-checkstyle-root.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/1/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/1/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/1/testReport/ |
| Max. process+thread count | 3141 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] fengnanli edited a comment on pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
fengnanli edited a comment on pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#issuecomment-653946194
Thanks very much @sunchao @goiri @Hexiaoqiao for the detailed review. I have addressed all comments and please give it another look.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] fengnanli commented on a change in pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
fengnanli commented on a change in pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#discussion_r451226040
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -726,4 +741,86 @@ public TokenIdent decodeTokenIdentifier(Token<TokenIdent> token) throws IOExcept
return token.decodeIdentifier();
}
+ /**
+ * Return top token real owners list as well as the tokens count.
+ *
+ * @param n top number of users
+ * @return map of owners to counts
+ */
+ public List<NameValuePair> getTopTokenRealOwners(int n) {
Review comment:
I will keep it as it is.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] fengnanli commented on a change in pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
fengnanli commented on a change in pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#discussion_r447994083
##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/security/TestRouterSecurityManager.java
##########
@@ -124,6 +126,71 @@ public void testDelegationTokens() throws IOException {
securityManager.renewDelegationToken(token);
}
+ @Test
+ public void testDelgationTokenTopOwners() throws Exception {
+ List<NameValuePair> topOwners;
+
+ UserGroupInformation user = UserGroupInformation
+ .createUserForTesting("abc", new String[]{"router_group"});
+ UserGroupInformation.setLoginUser(user);
+ Token dt = securityManager.getDelegationToken(new Text("abc"));
+ topOwners = securityManager.getSecretManager().getTopTokenRealOwners(2);
+ assertEquals(1, topOwners.size());
+ assertEquals("abc", topOwners.get(0).getName());
+ assertEquals(1, topOwners.get(0).getValue());
+
+ securityManager.renewDelegationToken(dt);
+ topOwners = securityManager.getSecretManager().getTopTokenRealOwners(2);
+ assertEquals(1, topOwners.size());
+ assertEquals("abc", topOwners.get(0).getName());
+ assertEquals(1, topOwners.get(0).getValue());
+
+ securityManager.cancelDelegationToken(dt);
+ topOwners = securityManager.getSecretManager().getTopTokenRealOwners(2);
+ assertEquals(0, topOwners.size());
+
+
+ // Use proxy user - the code should use the proxy user as the real owner
+ UserGroupInformation routerUser =
+ UserGroupInformation.createRemoteUser("router");
+ UserGroupInformation proxyUser = UserGroupInformation
+ .createProxyUserForTesting("abc",
+ routerUser,
+ new String[]{"router_group"});
+ UserGroupInformation.setLoginUser(proxyUser);
+
+ Token proxyDT = securityManager.getDelegationToken(new Text("router"));
+ topOwners = securityManager.getSecretManager().getTopTokenRealOwners(2);
+ assertEquals(1, topOwners.size());
+ assertEquals("router", topOwners.get(0).getName());
+ assertEquals(1, topOwners.get(0).getValue());
+
+ // router to renew tokens
+ UserGroupInformation.setLoginUser(routerUser);
+ securityManager.renewDelegationToken(proxyDT);
+ topOwners = securityManager.getSecretManager().getTopTokenRealOwners(2);
+ assertEquals(1, topOwners.size());
+ assertEquals("router", topOwners.get(0).getName());
+ assertEquals(1, topOwners.get(0).getValue());
+
+ securityManager.cancelDelegationToken(proxyDT);
+ topOwners = securityManager.getSecretManager().getTopTokenRealOwners(2);
+ assertEquals(0, topOwners.size());
+
Review comment:
will remove
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] fengnanli commented on pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
fengnanli commented on pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#issuecomment-654625072
I didn't know I have to click `resolve conversation` to publish the reply. Just resolved all of the comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#discussion_r448004743
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -34,6 +38,8 @@
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
import org.apache.hadoop.io.Text;
+import static org.apache.hadoop.metrics2.util.Metrics2Util.NameValuePair;
Review comment:
If the method/class is very obvious like assertTrue(), it usually makes sense to do a static import.
In this case, I guess is fine either way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#issuecomment-652182483
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 29s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 7s | trunk passed |
| +1 :green_heart: | compile | 23m 8s | trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 |
| +1 :green_heart: | compile | 19m 20s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | checkstyle | 2m 49s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 4s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 20s | branch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 36s | hadoop-common in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| -1 :x: | javadoc | 0m 33s | hadoop-hdfs-rbf in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| +1 :green_heart: | javadoc | 1m 42s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +0 :ok: | spotbugs | 1m 29s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 48s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 37s | the patch passed |
| +1 :green_heart: | compile | 23m 56s | the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 |
| +1 :green_heart: | javac | 23m 56s | the patch passed |
| +1 :green_heart: | compile | 20m 7s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | javac | 20m 7s | the patch passed |
| -0 :warning: | checkstyle | 3m 2s | root: The patch generated 2 new + 60 unchanged - 2 fixed = 62 total (was 62) |
| -1 :x: | mvnsite | 0m 36s | hadoop-hdfs-rbf in the patch failed. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 52s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 40s | hadoop-common in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| -1 :x: | javadoc | 0m 34s | hadoop-hdfs-rbf in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| +1 :green_heart: | javadoc | 1m 37s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| -1 :x: | findbugs | 0m 38s | hadoop-hdfs-rbf in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 11m 27s | hadoop-common in the patch passed. |
| -1 :x: | unit | 0m 38s | hadoop-hdfs-rbf in the patch failed. |
| +1 :green_heart: | asflicense | 0m 45s | The patch does not generate ASF License warnings. |
| | | 186m 43s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.ipc.TestIPC |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2110 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 895a36dcf8fc 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / e8dc862d385 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/3/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/3/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/3/artifact/out/diff-checkstyle-root.txt |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/3/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-rbf.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/3/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/3/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/3/artifact/out/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/3/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/3/testReport/ |
| Max. process+thread count | 1378 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/3/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] fengnanli commented on pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
fengnanli commented on pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#issuecomment-653946194
Thanks very much @sunchao @goiri for the detailed review. I have addressed all comments and please give it another look.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] fengnanli commented on pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
fengnanli commented on pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#issuecomment-656979333
Thanks @sunchao @Hexiaoqiao for the review!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on a change in pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#discussion_r448757877
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -726,4 +741,86 @@ public TokenIdent decodeTokenIdentifier(Token<TokenIdent> token) throws IOExcept
return token.decodeIdentifier();
}
+ /**
+ * Return top token real owners list as well as the tokens count.
+ *
+ * @param n top number of users
+ * @return map of owners to counts
+ */
+ public List<NameValuePair> getTopTokenRealOwners(int n) {
Review comment:
It's slightly unfortunate that we need to expose `NameValuePair` in a public method here as it is scope = private, but I don't know an easy way to avoid this.
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -726,4 +741,86 @@ public TokenIdent decodeTokenIdentifier(Token<TokenIdent> token) throws IOExcept
return token.decodeIdentifier();
}
+ /**
+ * Return top token real owners list as well as the tokens count.
+ *
+ * @param n top number of users
+ * @return map of owners to counts
+ */
+ public List<NameValuePair> getTopTokenRealOwners(int n) {
+ n = Math.min(n, tokenOwnerStats.size());
+ if (n == 0) {
+ return new ArrayList<>();
+ }
+
+ TopN topN = new TopN(n);
+ for (Map.Entry<String, Long> entry : tokenOwnerStats.entrySet()) {
+ topN.offer(new NameValuePair(
+ entry.getKey(), entry.getValue()));
+ }
+
+ List<NameValuePair> list = new ArrayList<>();
+ while (!topN.isEmpty()) {
+ list.add(topN.poll());
+ }
+ Collections.reverse(list);
+ return list;
+ }
+
+ /**
+ * Return the real owner for a token. If this is a token from a proxy user,
+ * the real/effective user will be returned.
+ *
+ * @param id
+ * @return real owner
+ */
+ public String getTokenRealOwner(TokenIdent id) {
+ String realUser;
+ if (id.getRealUser() != null && !id.getRealUser().toString().isEmpty()) {
+ realUser = id.getRealUser().toString();
+ } else {
+ // if there is no real user -> this is a non proxy user
+ // the user itself is the real owner
+ realUser = id.getUser().getUserName();
+ }
+ return realUser;
+ }
+
+ /**
+ * Add token stats to the owner to token count mapping.
+ *
+ * @param id
+ */
+ public void addTokenForOwnerStats(TokenIdent id) {
+ String realOwner = getTokenRealOwner(id);
+ tokenOwnerStats.put(realOwner,
+ tokenOwnerStats.getOrDefault(realOwner, 0l)+1);
Review comment:
nit: use capital `0L` to make IDE happy
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -578,6 +591,7 @@ public synchronized TokenIdent cancelToken(Token<TokenIdent> token,
if (info == null) {
throw new InvalidToken("Token not found " + formatTokenId(id));
}
+ removeTokenForOwnerStats(id);
Review comment:
same here - swap order and update stats as the last step?
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -64,7 +69,13 @@ private String formatTokenId(TokenIdent id) {
*/
protected final Map<TokenIdent, DelegationTokenInformation> currentTokens
= new ConcurrentHashMap<>();
-
+
+ /**
+ * Map of token real owners to its token count. This is used to generate
+ * top users by owned tokens.
+ */
+ protected final Map<String, Long> tokenOwnerStats = new ConcurrentHashMap<>();
Review comment:
Seems this won't be updated if a standby NN updates its own token info by pulling from edit log?
##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/resources/hdfs-rbf-default.xml
##########
@@ -657,4 +657,12 @@
</description>
</property>
+ <property>
+ <name>dfs.federation.router.top.num.token.realowners</name>
+ <value>10</value>
+ <description>
+ The number of top real owners by tokens count to report in the JMX metrics.
Review comment:
nit: might explain a little bit what "real owners" mean here
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -292,6 +303,7 @@ protected DelegationTokenInformation getTokenInfo(TokenIdent ident) {
protected void storeToken(TokenIdent ident,
DelegationTokenInformation tokenInfo) throws IOException {
currentTokens.put(ident, tokenInfo);
+ addTokenForOwnerStats(ident);
Review comment:
can we swap the order and update the stats as last step?
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -64,7 +69,13 @@ private String formatTokenId(TokenIdent id) {
*/
protected final Map<TokenIdent, DelegationTokenInformation> currentTokens
= new ConcurrentHashMap<>();
-
+
+ /**
+ * Map of token real owners to its token count. This is used to generate
+ * top users by owned tokens.
Review comment:
`top users` -> `top user metrics`?
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -726,4 +741,86 @@ public TokenIdent decodeTokenIdentifier(Token<TokenIdent> token) throws IOExcept
return token.decodeIdentifier();
}
+ /**
+ * Return top token real owners list as well as the tokens count.
+ *
+ * @param n top number of users
+ * @return map of owners to counts
+ */
+ public List<NameValuePair> getTopTokenRealOwners(int n) {
+ n = Math.min(n, tokenOwnerStats.size());
+ if (n == 0) {
+ return new ArrayList<>();
+ }
+
+ TopN topN = new TopN(n);
+ for (Map.Entry<String, Long> entry : tokenOwnerStats.entrySet()) {
+ topN.offer(new NameValuePair(
+ entry.getKey(), entry.getValue()));
+ }
+
+ List<NameValuePair> list = new ArrayList<>();
+ while (!topN.isEmpty()) {
+ list.add(topN.poll());
+ }
+ Collections.reverse(list);
+ return list;
+ }
+
+ /**
+ * Return the real owner for a token. If this is a token from a proxy user,
+ * the real/effective user will be returned.
+ *
+ * @param id
+ * @return real owner
+ */
+ public String getTokenRealOwner(TokenIdent id) {
+ String realUser;
+ if (id.getRealUser() != null && !id.getRealUser().toString().isEmpty()) {
+ realUser = id.getRealUser().toString();
+ } else {
+ // if there is no real user -> this is a non proxy user
+ // the user itself is the real owner
+ realUser = id.getUser().getUserName();
+ }
+ return realUser;
+ }
+
+ /**
+ * Add token stats to the owner to token count mapping.
+ *
+ * @param id
+ */
+ public void addTokenForOwnerStats(TokenIdent id) {
+ String realOwner = getTokenRealOwner(id);
+ tokenOwnerStats.put(realOwner,
+ tokenOwnerStats.getOrDefault(realOwner, 0l)+1);
+ }
+
+ /**
+ * Remove token stats to the owner to token count mapping.
+ *
+ * @param id
+ */
+ public void removeTokenForOwnerStats(TokenIdent id) {
+ String realOwner = getTokenRealOwner(id);
+ if (tokenOwnerStats.containsKey(realOwner)) {
+ // unlikely to be less than 1 but in case
+ if (tokenOwnerStats.get(realOwner) <= 1) {
Review comment:
Even though `tokenOwnerStats` is a concurrent map, you may run into race conditions if multiple threads operate on this method at the same time. We can potentially make the method synchronized to avoid that. Not sure we should care much though since this is only for metrics.
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -726,4 +741,86 @@ public TokenIdent decodeTokenIdentifier(Token<TokenIdent> token) throws IOExcept
return token.decodeIdentifier();
}
+ /**
+ * Return top token real owners list as well as the tokens count.
+ *
+ * @param n top number of users
+ * @return map of owners to counts
+ */
+ public List<NameValuePair> getTopTokenRealOwners(int n) {
+ n = Math.min(n, tokenOwnerStats.size());
+ if (n == 0) {
+ return new ArrayList<>();
+ }
+
+ TopN topN = new TopN(n);
+ for (Map.Entry<String, Long> entry : tokenOwnerStats.entrySet()) {
+ topN.offer(new NameValuePair(
+ entry.getKey(), entry.getValue()));
+ }
+
+ List<NameValuePair> list = new ArrayList<>();
+ while (!topN.isEmpty()) {
+ list.add(topN.poll());
+ }
+ Collections.reverse(list);
+ return list;
+ }
+
+ /**
+ * Return the real owner for a token. If this is a token from a proxy user,
+ * the real/effective user will be returned.
+ *
+ * @param id
+ * @return real owner
+ */
+ public String getTokenRealOwner(TokenIdent id) {
+ String realUser;
+ if (id.getRealUser() != null && !id.getRealUser().toString().isEmpty()) {
+ realUser = id.getRealUser().toString();
+ } else {
+ // if there is no real user -> this is a non proxy user
+ // the user itself is the real owner
+ realUser = id.getUser().getUserName();
+ }
+ return realUser;
+ }
+
+ /**
+ * Add token stats to the owner to token count mapping.
+ *
+ * @param id
+ */
+ public void addTokenForOwnerStats(TokenIdent id) {
+ String realOwner = getTokenRealOwner(id);
+ tokenOwnerStats.put(realOwner,
+ tokenOwnerStats.getOrDefault(realOwner, 0l)+1);
+ }
+
+ /**
+ * Remove token stats to the owner to token count mapping.
+ *
+ * @param id
+ */
+ public void removeTokenForOwnerStats(TokenIdent id) {
Review comment:
This and a few others do not need to be public?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] fengnanli commented on a change in pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
fengnanli commented on a change in pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#discussion_r447986241
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -34,6 +38,8 @@
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
import org.apache.hadoop.io.Text;
+import static org.apache.hadoop.metrics2.util.Metrics2Util.NameValuePair;
Review comment:
Inigo told me once about it and I think it can have the usage of member variables easier and cleaner.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] fengnanli commented on a change in pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
fengnanli commented on a change in pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#discussion_r451227291
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -578,6 +591,7 @@ public synchronized TokenIdent cancelToken(Token<TokenIdent> token,
if (info == null) {
throw new InvalidToken("Token not found " + formatTokenId(id));
}
+ removeTokenForOwnerStats(id);
Review comment:
The reason i left the order as this is that the metric is a in-memory reflection `currentTokens` so it can be calculated once the memory data structure is changed.
The procedure after is for persistent storage.
##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/resources/hdfs-rbf-default.xml
##########
@@ -657,4 +657,12 @@
</description>
</property>
+ <property>
+ <name>dfs.federation.router.top.num.token.realowners</name>
+ <value>10</value>
+ <description>
+ The number of top real owners by tokens count to report in the JMX metrics.
Review comment:
fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao merged pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
sunchao merged pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#issuecomment-653976649
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 21m 42s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 11s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 13s | trunk passed |
| +1 :green_heart: | compile | 19m 11s | trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 |
| +1 :green_heart: | compile | 16m 35s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | checkstyle | 3m 17s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 21s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 16s | branch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 45s | hadoop-common in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| -1 :x: | javadoc | 0m 41s | hadoop-hdfs-rbf in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| +1 :green_heart: | javadoc | 1m 48s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +0 :ok: | spotbugs | 1m 21s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 26s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 20s | the patch passed |
| +1 :green_heart: | compile | 18m 25s | the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 |
| +1 :green_heart: | javac | 18m 25s | the patch passed |
| +1 :green_heart: | compile | 16m 36s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | javac | 16m 36s | the patch passed |
| -0 :warning: | checkstyle | 2m 41s | root: The patch generated 1 new + 60 unchanged - 2 fixed = 61 total (was 62) |
| +1 :green_heart: | mvnsite | 2m 17s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 8s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 44s | hadoop-common in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| -1 :x: | javadoc | 0m 41s | hadoop-hdfs-rbf in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| +1 :green_heart: | javadoc | 1m 48s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | findbugs | 3m 42s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 29s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 8m 15s | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 55s | The patch does not generate ASF License warnings. |
| | | 191m 42s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2110 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux a23c7d24a34e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 55a2ae80dc9 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/4/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/4/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/4/artifact/out/diff-checkstyle-root.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/4/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/4/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/4/testReport/ |
| Max. process+thread count | 3337 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/4/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] fengnanli commented on a change in pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
fengnanli commented on a change in pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#discussion_r448036448
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -726,4 +732,41 @@ public TokenIdent decodeTokenIdentifier(Token<TokenIdent> token) throws IOExcept
return token.decodeIdentifier();
}
+ /**
+ * Return top token real owners list as well as the tokens count.
+ *
+ * @param n top number of users
+ * @return map of owners to counts
+ */
+ public List<NameValuePair> getTopTokenRealOwners(int n) {
Review comment:
I am fine with it. We also need to add an initialization step to make sure this structure has the initial information from currentTokens.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] fengnanli commented on a change in pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
fengnanli commented on a change in pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#discussion_r451227069
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -726,4 +741,86 @@ public TokenIdent decodeTokenIdentifier(Token<TokenIdent> token) throws IOExcept
return token.decodeIdentifier();
}
+ /**
+ * Return top token real owners list as well as the tokens count.
+ *
+ * @param n top number of users
+ * @return map of owners to counts
+ */
+ public List<NameValuePair> getTopTokenRealOwners(int n) {
+ n = Math.min(n, tokenOwnerStats.size());
+ if (n == 0) {
+ return new ArrayList<>();
+ }
+
+ TopN topN = new TopN(n);
+ for (Map.Entry<String, Long> entry : tokenOwnerStats.entrySet()) {
+ topN.offer(new NameValuePair(
+ entry.getKey(), entry.getValue()));
+ }
+
+ List<NameValuePair> list = new ArrayList<>();
+ while (!topN.isEmpty()) {
+ list.add(topN.poll());
+ }
+ Collections.reverse(list);
+ return list;
+ }
+
+ /**
+ * Return the real owner for a token. If this is a token from a proxy user,
+ * the real/effective user will be returned.
+ *
+ * @param id
+ * @return real owner
+ */
+ public String getTokenRealOwner(TokenIdent id) {
+ String realUser;
+ if (id.getRealUser() != null && !id.getRealUser().toString().isEmpty()) {
+ realUser = id.getRealUser().toString();
+ } else {
+ // if there is no real user -> this is a non proxy user
+ // the user itself is the real owner
+ realUser = id.getUser().getUserName();
+ }
+ return realUser;
+ }
+
+ /**
+ * Add token stats to the owner to token count mapping.
+ *
+ * @param id
+ */
+ public void addTokenForOwnerStats(TokenIdent id) {
+ String realOwner = getTokenRealOwner(id);
+ tokenOwnerStats.put(realOwner,
+ tokenOwnerStats.getOrDefault(realOwner, 0l)+1);
+ }
+
+ /**
+ * Remove token stats to the owner to token count mapping.
+ *
+ * @param id
+ */
+ public void removeTokenForOwnerStats(TokenIdent id) {
Review comment:
this was fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#issuecomment-654735425
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 21m 37s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 2s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 18m 59s | trunk passed |
| +1 :green_heart: | compile | 20m 8s | trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 |
| +1 :green_heart: | compile | 17m 22s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | checkstyle | 2m 37s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 9s | trunk passed |
| +1 :green_heart: | shadedclient | 19m 15s | branch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 45s | hadoop-common in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| -1 :x: | javadoc | 0m 42s | hadoop-hdfs-rbf in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| +1 :green_heart: | javadoc | 1m 49s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +0 :ok: | spotbugs | 1m 21s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 28s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 21s | the patch passed |
| +1 :green_heart: | compile | 18m 30s | the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 |
| +1 :green_heart: | javac | 18m 30s | the patch passed |
| +1 :green_heart: | compile | 16m 34s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | javac | 16m 34s | the patch passed |
| -0 :warning: | checkstyle | 2m 45s | root: The patch generated 1 new + 60 unchanged - 2 fixed = 61 total (was 62) |
| +1 :green_heart: | mvnsite | 2m 21s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 33s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 39s | hadoop-common in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| -1 :x: | javadoc | 0m 37s | hadoop-hdfs-rbf in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| +1 :green_heart: | javadoc | 1m 35s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | findbugs | 3m 52s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 16s | hadoop-common in the patch passed. |
| -1 :x: | unit | 8m 15s | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 51s | The patch does not generate ASF License warnings. |
| | | 191m 47s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2110 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 89c2dced6705 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / f77bbc2123e |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/5/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/5/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/5/artifact/out/diff-checkstyle-root.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/5/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/5/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/5/testReport/ |
| Max. process+thread count | 3171 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/5/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#discussion_r448069115
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -34,6 +38,8 @@
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
import org.apache.hadoop.io.Text;
+import static org.apache.hadoop.metrics2.util.Metrics2Util.NameValuePair;
Review comment:
Not always the case but I'd say this is fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] fengnanli commented on a change in pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
fengnanli commented on a change in pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#discussion_r451227559
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -726,4 +741,86 @@ public TokenIdent decodeTokenIdentifier(Token<TokenIdent> token) throws IOExcept
return token.decodeIdentifier();
}
+ /**
+ * Return top token real owners list as well as the tokens count.
+ *
+ * @param n top number of users
+ * @return map of owners to counts
+ */
+ public List<NameValuePair> getTopTokenRealOwners(int n) {
+ n = Math.min(n, tokenOwnerStats.size());
+ if (n == 0) {
+ return new ArrayList<>();
+ }
+
+ TopN topN = new TopN(n);
+ for (Map.Entry<String, Long> entry : tokenOwnerStats.entrySet()) {
+ topN.offer(new NameValuePair(
+ entry.getKey(), entry.getValue()));
+ }
+
+ List<NameValuePair> list = new ArrayList<>();
+ while (!topN.isEmpty()) {
+ list.add(topN.poll());
+ }
+ Collections.reverse(list);
+ return list;
+ }
+
+ /**
+ * Return the real owner for a token. If this is a token from a proxy user,
+ * the real/effective user will be returned.
+ *
+ * @param id
+ * @return real owner
+ */
+ public String getTokenRealOwner(TokenIdent id) {
+ String realUser;
+ if (id.getRealUser() != null && !id.getRealUser().toString().isEmpty()) {
+ realUser = id.getRealUser().toString();
+ } else {
+ // if there is no real user -> this is a non proxy user
+ // the user itself is the real owner
+ realUser = id.getUser().getUserName();
+ }
+ return realUser;
+ }
+
+ /**
+ * Add token stats to the owner to token count mapping.
+ *
+ * @param id
+ */
+ public void addTokenForOwnerStats(TokenIdent id) {
+ String realOwner = getTokenRealOwner(id);
+ tokenOwnerStats.put(realOwner,
+ tokenOwnerStats.getOrDefault(realOwner, 0l)+1);
+ }
+
+ /**
+ * Remove token stats to the owner to token count mapping.
+ *
+ * @param id
+ */
+ public void removeTokenForOwnerStats(TokenIdent id) {
+ String realOwner = getTokenRealOwner(id);
+ if (tokenOwnerStats.containsKey(realOwner)) {
+ // unlikely to be less than 1 but in case
+ if (tokenOwnerStats.get(realOwner) <= 1) {
Review comment:
The function is called from `createPassword` and `cancelToken` which are all synchronized so it is safe here. Similarly `currentTokens` is used in the pattern.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on a change in pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#discussion_r451729071
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -64,7 +69,13 @@ private String formatTokenId(TokenIdent id) {
*/
protected final Map<TokenIdent, DelegationTokenInformation> currentTokens
= new ConcurrentHashMap<>();
-
+
+ /**
+ * Map of token real owners to its token count. This is used to generate
+ * top users by owned tokens.
+ */
+ protected final Map<String, Long> tokenOwnerStats = new ConcurrentHashMap<>();
Review comment:
OK. Can you add a comment for this though indicating that this only support RBF for now?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] fengnanli commented on a change in pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
fengnanli commented on a change in pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#discussion_r447993331
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -726,4 +732,41 @@ public TokenIdent decodeTokenIdentifier(Token<TokenIdent> token) throws IOExcept
return token.decodeIdentifier();
}
+ /**
+ * Return top token real owners list as well as the tokens count.
+ *
+ * @param n top number of users
+ * @return map of owners to counts
+ */
+ public List<NameValuePair> getTopTokenRealOwners(int n) {
Review comment:
I had similar thoughts as well but didn't come up with a better way. In namenode TopN metrics, it is doing so as well just at a lower frequency like every 5/15/25 minutes. We can potentially do that by reducing the metric reporting frequency.
I also checked modern CPU for looping over 1M, which takes about 1ms-10ms.
Another one would be to maintain a data structure to dynamically maintain the ordering of users and edit ordering per getdelegationtoken and per canceldelegationtoken like stream processing. I am not sure about the cost overall since in reality we generally have < 1 Million tokens.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
Hexiaoqiao commented on pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#issuecomment-651850748
Please check the failed unit tests and checkstyle report by yetus.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on a change in pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#discussion_r447965934
##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/security/TestRouterSecurityManager.java
##########
@@ -124,6 +126,71 @@ public void testDelegationTokens() throws IOException {
securityManager.renewDelegationToken(token);
}
+ @Test
+ public void testDelgationTokenTopOwners() throws Exception {
+ List<NameValuePair> topOwners;
+
+ UserGroupInformation user = UserGroupInformation
+ .createUserForTesting("abc", new String[]{"router_group"});
+ UserGroupInformation.setLoginUser(user);
+ Token dt = securityManager.getDelegationToken(new Text("abc"));
+ topOwners = securityManager.getSecretManager().getTopTokenRealOwners(2);
+ assertEquals(1, topOwners.size());
+ assertEquals("abc", topOwners.get(0).getName());
+ assertEquals(1, topOwners.get(0).getValue());
+
+ securityManager.renewDelegationToken(dt);
+ topOwners = securityManager.getSecretManager().getTopTokenRealOwners(2);
+ assertEquals(1, topOwners.size());
+ assertEquals("abc", topOwners.get(0).getName());
+ assertEquals(1, topOwners.get(0).getValue());
+
+ securityManager.cancelDelegationToken(dt);
+ topOwners = securityManager.getSecretManager().getTopTokenRealOwners(2);
+ assertEquals(0, topOwners.size());
+
+
+ // Use proxy user - the code should use the proxy user as the real owner
+ UserGroupInformation routerUser =
+ UserGroupInformation.createRemoteUser("router");
+ UserGroupInformation proxyUser = UserGroupInformation
+ .createProxyUserForTesting("abc",
+ routerUser,
+ new String[]{"router_group"});
+ UserGroupInformation.setLoginUser(proxyUser);
+
+ Token proxyDT = securityManager.getDelegationToken(new Text("router"));
+ topOwners = securityManager.getSecretManager().getTopTokenRealOwners(2);
+ assertEquals(1, topOwners.size());
+ assertEquals("router", topOwners.get(0).getName());
+ assertEquals(1, topOwners.get(0).getValue());
+
+ // router to renew tokens
+ UserGroupInformation.setLoginUser(routerUser);
+ securityManager.renewDelegationToken(proxyDT);
+ topOwners = securityManager.getSecretManager().getTopTokenRealOwners(2);
+ assertEquals(1, topOwners.size());
+ assertEquals("router", topOwners.get(0).getName());
+ assertEquals(1, topOwners.get(0).getValue());
+
+ securityManager.cancelDelegationToken(proxyDT);
+ topOwners = securityManager.getSecretManager().getTopTokenRealOwners(2);
+ assertEquals(0, topOwners.size());
+
Review comment:
nit: extra blank line
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -726,4 +732,41 @@ public TokenIdent decodeTokenIdentifier(Token<TokenIdent> token) throws IOExcept
return token.decodeIdentifier();
}
+ /**
+ * Return top token real owners list as well as the tokens count.
+ *
+ * @param n top number of users
+ * @return map of owners to counts
+ */
+ public List<NameValuePair> getTopTokenRealOwners(int n) {
+ Map<String, Integer> tokenOwnerMap = new HashMap<>();
+ for (TokenIdent id : currentTokens.keySet()) {
+ String realUser;
+ if (id.getRealUser() != null && !id.getRealUser().toString().isEmpty()) {
+ realUser = id.getRealUser().toString();
+ } else {
+ // if there is no real user -> this is a non proxy user
+ // the user itself is the real owner
+ realUser = id.getUser().getUserName();
+ }
+ tokenOwnerMap.put(realUser, tokenOwnerMap.getOrDefault(realUser, 0)+1);
+ }
+ n = Math.min(n, tokenOwnerMap.size());
+ if (n == 0) {
+ return new LinkedList<>();
+ }
+
+ TopN topN = new TopN(n);
+ for (Map.Entry<String, Integer> entry : tokenOwnerMap.entrySet()) {
+ topN.offer(new NameValuePair(
+ entry.getKey(), entry.getValue()));
+ }
+
+ List<NameValuePair> list = new LinkedList<>();
Review comment:
any reason to use `LinkedList` instead of `ArrayList`? the latter is usually more performant.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#issuecomment-657184620
Tests also passed locally. Merged. Thanks @fengnanli .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] fengnanli commented on a change in pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
fengnanli commented on a change in pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#discussion_r447993824
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -726,4 +732,41 @@ public TokenIdent decodeTokenIdentifier(Token<TokenIdent> token) throws IOExcept
return token.decodeIdentifier();
}
+ /**
+ * Return top token real owners list as well as the tokens count.
+ *
+ * @param n top number of users
+ * @return map of owners to counts
+ */
+ public List<NameValuePair> getTopTokenRealOwners(int n) {
+ Map<String, Integer> tokenOwnerMap = new HashMap<>();
+ for (TokenIdent id : currentTokens.keySet()) {
+ String realUser;
+ if (id.getRealUser() != null && !id.getRealUser().toString().isEmpty()) {
+ realUser = id.getRealUser().toString();
+ } else {
+ // if there is no real user -> this is a non proxy user
+ // the user itself is the real owner
+ realUser = id.getUser().getUserName();
+ }
+ tokenOwnerMap.put(realUser, tokenOwnerMap.getOrDefault(realUser, 0)+1);
+ }
+ n = Math.min(n, tokenOwnerMap.size());
+ if (n == 0) {
+ return new LinkedList<>();
+ }
+
+ TopN topN = new TopN(n);
+ for (Map.Entry<String, Integer> entry : tokenOwnerMap.entrySet()) {
+ topN.offer(new NameValuePair(
+ entry.getKey(), entry.getValue()));
+ }
+
+ List<NameValuePair> list = new LinkedList<>();
Review comment:
There is a reverse op and I think reverse linked list is faster without additional space.
Not sure how java implement the reverse array list, but I think it will introduce copy and reassign.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] fengnanli commented on pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
fengnanli commented on pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#issuecomment-655221130
github screwed up a lot of my comments and I will reply again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] fengnanli commented on a change in pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
fengnanli commented on a change in pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#discussion_r447988563
##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/security/TestRouterSecurityManager.java
##########
@@ -39,6 +39,7 @@
import org.junit.Rule;
import org.junit.Test;
+import static org.apache.hadoop.metrics2.util.Metrics2Util.*;
Review comment:
Sure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on a change in pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#discussion_r448022518
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -726,4 +732,41 @@ public TokenIdent decodeTokenIdentifier(Token<TokenIdent> token) throws IOExcept
return token.decodeIdentifier();
}
+ /**
+ * Return top token real owners list as well as the tokens count.
+ *
+ * @param n top number of users
+ * @return map of owners to counts
+ */
+ public List<NameValuePair> getTopTokenRealOwners(int n) {
+ Map<String, Integer> tokenOwnerMap = new HashMap<>();
+ for (TokenIdent id : currentTokens.keySet()) {
+ String realUser;
+ if (id.getRealUser() != null && !id.getRealUser().toString().isEmpty()) {
+ realUser = id.getRealUser().toString();
+ } else {
+ // if there is no real user -> this is a non proxy user
+ // the user itself is the real owner
+ realUser = id.getUser().getUserName();
+ }
+ tokenOwnerMap.put(realUser, tokenOwnerMap.getOrDefault(realUser, 0)+1);
+ }
+ n = Math.min(n, tokenOwnerMap.size());
+ if (n == 0) {
+ return new LinkedList<>();
+ }
+
+ TopN topN = new TopN(n);
+ for (Map.Entry<String, Integer> entry : tokenOwnerMap.entrySet()) {
+ topN.offer(new NameValuePair(
+ entry.getKey(), entry.getValue()));
+ }
+
+ List<NameValuePair> list = new LinkedList<>();
Review comment:
Reverse shouldn't need extra space - it uses two indexes from begin and end of the array and swaps elements. I don't see real difference between the two for the reverse.
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -726,4 +732,41 @@ public TokenIdent decodeTokenIdentifier(Token<TokenIdent> token) throws IOExcept
return token.decodeIdentifier();
}
+ /**
+ * Return top token real owners list as well as the tokens count.
+ *
+ * @param n top number of users
+ * @return map of owners to counts
+ */
+ public List<NameValuePair> getTopTokenRealOwners(int n) {
Review comment:
Can we update the `TopN` queue when creating/deleting tokens? we are just paying an extra constant cost for updating that which I think is fine. Even though it is using concurrent hashmap, I'm not sure how much performance impact will be if one thread is iterating over the key set while others want to updating the map.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on a change in pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
Hexiaoqiao commented on a change in pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#discussion_r447745987
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -34,6 +38,8 @@
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
import org.apache.hadoop.io.Text;
+import static org.apache.hadoop.metrics2.util.Metrics2Util.NameValuePair;
Review comment:
is it necessary to import static?
##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RBFConfigKeys.java
##########
@@ -79,6 +79,10 @@
public static final Class<? extends RouterRpcMonitor>
DFS_ROUTER_METRICS_CLASS_DEFAULT =
FederationRPCPerformanceMonitor.class;
+ public static final String DFS_ROUTER_METRICS_TOP_NUM_TOKEN_OWNERS_KEY =
Review comment:
we need to define this new configure item at hdfs-rbf-default.xml.
##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/security/TestRouterSecurityManager.java
##########
@@ -39,6 +39,7 @@
import org.junit.Rule;
import org.junit.Test;
+import static org.apache.hadoop.metrics2.util.Metrics2Util.*;
Review comment:
just suggest to replace with single class imports.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#issuecomment-652173646
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 28m 59s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 10s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 1s | trunk passed |
| +1 :green_heart: | compile | 27m 52s | trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 |
| +1 :green_heart: | compile | 18m 8s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | checkstyle | 2m 54s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 11s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 7s | branch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 36s | hadoop-common in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| -1 :x: | javadoc | 0m 35s | hadoop-hdfs-rbf in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| +1 :green_heart: | javadoc | 1m 33s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +0 :ok: | spotbugs | 1m 20s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 29s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 32s | the patch passed |
| +1 :green_heart: | compile | 20m 36s | the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 |
| +1 :green_heart: | javac | 20m 36s | the patch passed |
| +1 :green_heart: | compile | 19m 57s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | javac | 19m 57s | the patch passed |
| -0 :warning: | checkstyle | 3m 0s | root: The patch generated 3 new + 60 unchanged - 2 fixed = 63 total (was 62) |
| -1 :x: | mvnsite | 0m 42s | hadoop-hdfs-rbf in the patch failed. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 31s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 36s | hadoop-common in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| -1 :x: | javadoc | 0m 33s | hadoop-hdfs-rbf in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| +1 :green_heart: | javadoc | 1m 39s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| -1 :x: | findbugs | 0m 39s | hadoop-hdfs-rbf in the patch failed. |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 55s | hadoop-common in the patch passed. |
| -1 :x: | unit | 0m 36s | hadoop-hdfs-rbf in the patch failed. |
| +1 :green_heart: | asflicense | 0m 47s | The patch does not generate ASF License warnings. |
| | | 212m 38s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2110 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 8a17ef9069ae 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / e8dc862d385 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/2/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/2/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/2/artifact/out/diff-checkstyle-root.txt |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/2/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-rbf.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/2/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/2/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/2/artifact/out/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/2/testReport/ |
| Max. process+thread count | 1458 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-2110/2/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] fengnanli commented on pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
fengnanli commented on pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#issuecomment-656994988
The test failure is relevant and it passed locally in my mac.
> [INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination
[INFO] Tests run: 49, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.51 s - in org.apache.hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 49, Failures: 0, Errors: 0, Skipped: 0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] fengnanli commented on a change in pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
fengnanli commented on a change in pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#discussion_r451227025
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##########
@@ -64,7 +69,13 @@ private String formatTokenId(TokenIdent id) {
*/
protected final Map<TokenIdent, DelegationTokenInformation> currentTokens
= new ConcurrentHashMap<>();
-
+
+ /**
+ * Map of token real owners to its token count. This is used to generate
+ * top users by owned tokens.
+ */
+ protected final Map<String, Long> tokenOwnerStats = new ConcurrentHashMap<>();
Review comment:
It depends on individual secret manager to initialize the `currentTokens`. In namenode, it is loading from edit log. In router, it is loading from ZK.
I would have a separate ticket for namenode.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] fengnanli commented on a change in pull request #2110: HDFS-15447 RBF: Add top real owners metrics for delegation tokens
Posted by GitBox <gi...@apache.org>.
fengnanli commented on a change in pull request #2110:
URL: https://github.com/apache/hadoop/pull/2110#discussion_r447988483
##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RBFConfigKeys.java
##########
@@ -79,6 +79,10 @@
public static final Class<? extends RouterRpcMonitor>
DFS_ROUTER_METRICS_CLASS_DEFAULT =
FederationRPCPerformanceMonitor.class;
+ public static final String DFS_ROUTER_METRICS_TOP_NUM_TOKEN_OWNERS_KEY =
Review comment:
Sure. I will add it here: https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/resources/hdfs-rbf-default.xml
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org