You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Wei-Chiu Chuang (JIRA)" <ji...@apache.org> on 2019/02/21 13:36:00 UTC
[jira] [Updated] (HADOOP-16130) Support delegation token operations
in KMS Benchmark
[ https://issues.apache.org/jira/browse/HADOOP-16130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wei-Chiu Chuang updated HADOOP-16130:
-------------------------------------
Description:
At the last Hadoop Contributors Meetup, [~daryn] mentioned another KMS throughput bottleneck is ZooKeeper -- KMS uses ZK to store delegation tokens. ZK would be brought to a halt when expired delegation tokens are purged. That sounds critical especially given that in most deployments KMS share the same ZK quorum as HDFS, it would cause NameNode failover.
The current KMS benchmark does not support delegation token operations (addDelegationTokens, cancelDelegationToken, renewDelegationToken) so it's hard to understand how bad it is, and hard to quantify the improvement of a fix.
File this jira to support those operations before we move on to the fix for the ZK issue.
was:
At the last Hadoop Contributors Meetup, [~daryn] mentioned another KMS throughput bottleneck is ZooKeeper -- KMS uses ZK to store delegation tokens. ZK would be brought to a halt under heavy KMS dt operations. That sounds critical especially given that in most deployments KMS share the same ZK quorum as HDFS, it would cause NameNode failover.
The current KMS benchmark does not support delegation token operations (addDelegationTokens, cancelDelegationToken, renewDelegationToken) so it's hard to understand how bad it is, and hard to quantify the improvement of a fix.
File this jira to support those operations before we move on to the fix for the ZK issue.
> Support delegation token operations in KMS Benchmark
> ----------------------------------------------------
>
> Key: HADOOP-16130
> URL: https://issues.apache.org/jira/browse/HADOOP-16130
> Project: Hadoop Common
> Issue Type: Improvement
> Affects Versions: 3.3.0
> Reporter: Wei-Chiu Chuang
> Priority: Major
>
> At the last Hadoop Contributors Meetup, [~daryn] mentioned another KMS throughput bottleneck is ZooKeeper -- KMS uses ZK to store delegation tokens. ZK would be brought to a halt when expired delegation tokens are purged. That sounds critical especially given that in most deployments KMS share the same ZK quorum as HDFS, it would cause NameNode failover.
> The current KMS benchmark does not support delegation token operations (addDelegationTokens, cancelDelegationToken, renewDelegationToken) so it's hard to understand how bad it is, and hard to quantify the improvement of a fix.
> File this jira to support those operations before we move on to the fix for the ZK issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org