You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Eric Yang (JIRA)" <ji...@apache.org> on 2018/08/03 18:09:01 UTC
[jira] [Commented] (YARN-8402) Yarn Service Destroy: Delete service
entries from Zookeeper in the ServiceMaster instead of ServiceClient in the
RM
[ https://issues.apache.org/jira/browse/YARN-8402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568556#comment-16568556 ]
Eric Yang commented on YARN-8402:
---------------------------------
ServiceMaster runs in application master. ServiceMaster might not be running, if application is stopped. App destroy command needs to work even if application master is not running. This is the reason that the ZooKeeper clean up operation is in ServiceClient instead of ServiceMaster. CuratorClient is fairly optimized to use a persisted client in ServiceClient. This avoids to spend time on ZooKeeper session creation, and send as many of the delete commands to ZooKeeper as possible. The delay may come from else where other than ZooKeeper interaction, i.e. deleting yarnfile from HDFS with 1000 file with hdfs IO requests. I think benchmark is required to understand the real cause of delay.
> Yarn Service Destroy: Delete service entries from Zookeeper in the ServiceMaster instead of ServiceClient in the RM
> -------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-8402
> URL: https://issues.apache.org/jira/browse/YARN-8402
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Chandni Singh
> Assignee: Chandni Singh
> Priority: Major
>
> RM slows down considerably when multiple services are destroyed simultaneously.
> 1. Started approx 1000 services
> 2. Destroyed all the 1000 services.
> Observed considerable slowness in RM after this.
> The {{ServiceClient}} in RM uses the {{CuratorClient}} to delete zookeeper entries.
> The zookeeper client is the bottleneck and this could be avoided if the zookeeper entry can be deleted from the AM and then the {{ServiceClient}} can kill the app.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org