You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Weiwei Yang (JIRA)" <ji...@apache.org> on 2018/03/27 12:21:00 UTC
[jira] [Comment Edited] (YARN-7497) Add HDFSSchedulerConfigurationStore for RM HA

    [ https://issues.apache.org/jira/browse/YARN-7497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415500#comment-16415500 ] 

Weiwei Yang edited comment on YARN-7497 at 3/27/18 12:20 PM:
-------------------------------------------------------------

Hi [~yangjiandan]

Thanks for the patch, this is a big alternative of ZK implementation when dealing with a lot of queues. Some comments about the patch:
 # My major concern over the patch is that it doesn't use WAL fashion to prevent potential inconsistent in-memory/persisted conf. I suggest to implement the log/retrieve logic to make sure it works that way. Something like {{ZKConfigurationStore}}.
 # I feel it is better to call {{FSSchedulerConfigurationStore }}instead of {{HDFSSchedulerConfigurationStore}} as this is should work as long as on any HFS.

Some other misc comments:

*Conf*
 # Need to add some doc in {{yarn-site.xml}}, property {{yarn.scheduler.configuration.store.class}}; also the added configuration files, make sure {{TestYarnConfiguration}} passes.

*TestMutableCSConfigurationProvider*
 # writeConf: please use try-finally to make sure both fileSystem and outputStream are closed

*HDFSSchedulerConfigurationStore*
 # line 52: typo HFDS
 # line 63, 74: pendingId not used
 # line 68: typo configFilePathFileter -> configFilePathFilter
 # line 78: better to add a null check before using path
 # line 86: I don't think we need to throw a RuntimeException here, can we replace it with IOException? So CapacityScheduler can handle this during  initScheduler
 # line 102, 145: {{configInputStream}} is not closed
 # line 205: move {{outputStream}} to finally statement

Hope it makes sense. Thanks

 

 


was (Author: cheersyang):
Hi [~yangjiandan]

Thanks for the patch, this is a big alternative of ZK implementation when dealing with a lot of queues. Some comments about the patch:
 # My major concern over the patch is that it doesn't use WAL fashion to prevent potential inconsistent in-memory/persisted conf. I suggest to implement the log/retrieve logic to make sure it works that way. Something like \{{ZKConfigurationStore}}.
 # I feel it is better to call {{FSSchedulerConfigurationStore }}instead of \{{HDFSSchedulerConfigurationStore}} as this is should work as long as on any HFS.

Some other misc comments:

*Conf*
 # Need to add some doc in \{{yarn-site.xml}}, property \{{yarn.scheduler.configuration.store.class}}; also the added configuration files, make sure \{{TestYarnConfiguration}} passes.

*TestMutableCSConfigurationProvider*
 # writeConf: please use try-finally to make sure both fileSystem and outputStream are closed

*HDFSSchedulerConfigurationStore*
 # line 52: typo HFDS
 # line 63, 74: pendingId not used
 # line 68: typo configFilePathFileter -> configFilePathFilter
 # line 78: better to add a null check before using \{{path}}
 # line 86: I don't think we need to throw a RuntimeException here, can we replace it with IOException? So CapacityScheduler can handle this during \{{initScheduler}}
 # line 102, 145: {{configInputStream}} is not closed
 # line 205: move \{{outputStream}} to finally statement

Hope it makes sense. Thanks

 

 

> Add HDFSSchedulerConfigurationStore for RM HA
> ---------------------------------------------
>
>                 Key: YARN-7497
>                 URL: https://issues.apache.org/jira/browse/YARN-7497
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: yarn
>            Reporter: Jiandan Yang 
>            Assignee: Jiandan Yang 
>            Priority: Major
>         Attachments: YARN-7497.001.patch, YARN-7497.002.patch, YARN-7497.003.patch, YARN-7497.004.patch, YARN-7497.005.patch, YARN-7497.006.patch, YARN-7497.007.patch, YARN-7497.008.patch
>
>
> YARN-5947 add LeveldbConfigurationStore using Leveldb as backing store, but it does not support Yarn RM HA. 
> YARN-6840 supports RM HA, but too many scheduler configurations may exceed znode limit, for example 10 thousand queues.
> HDFSSchedulerConfigurationStore store conf file in HDFS, when RM failover, new active RM can load scheduler configuration from HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org