You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Manikandan R (JIRA)" <ji...@apache.org> on 2017/09/03 13:36:00 UTC
[jira] [Commented] (YARN-6523) RM requires large memory in sending
out security tokens as part of Node Heartbeat in large cluster
[ https://issues.apache.org/jira/browse/YARN-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16151817#comment-16151817 ]
Manikandan R commented on YARN-6523:
------------------------------------
[~Naganarasimha]
I made an attempt to understand the discussion in this JIRA and come up with below steps to address this. Please review and share your thoughts. Can I work on this?
A) Sequence No flow:
1. Introduce a atomic long variable in RMContextImpl to hold this sequence no
2. Make sure the above sequence no current value passed to Nodes as part of node registration process through response
3. Increment above sequence no as and when there is any update in delegation tokens, specifically in DelegationTokenRenewer#requestNewHdfsDelegationTokenAsProxyUser
4. ResourceTrackerService#nodeHeartbeat would use the above sequence number to decide whether to update SystemCredentialsForApps in NodeHeartbeatResponse or not by comparing it with the number received as part of NodeHeartbeatRequest.
5. NodeHeartbeatRequest will start having sequence no as part of the request. This requires a change in corresponding proto class.
6. NodeHeartbeatResponse will start having sequence no as part of the response. This requires a change in corresponding proto class.
B) Caching system credentials objects
Will go through the code and share my proposal to achieve the same.
> RM requires large memory in sending out security tokens as part of Node Heartbeat in large cluster
> --------------------------------------------------------------------------------------------------
>
> Key: YARN-6523
> URL: https://issues.apache.org/jira/browse/YARN-6523
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: RM
> Affects Versions: 2.8.0, 2.7.3
> Reporter: Naganarasimha G R
> Assignee: Naganarasimha G R
> Priority: Critical
>
> Currently as part of heartbeat response RM sets all application's tokens though all applications might not be active on the node. On top of it NodeHeartbeatResponsePBImpl converts tokens for each app into SystemCredentialsForAppsProto. Hence for each node and each heartbeat too many SystemCredentialsForAppsProto objects were getting created.
> We hit a OOM while testing for 2000 concurrent apps on 500 nodes cluster with 8GB RAM configured for RM
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org