You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ambari.apache.org by "Swapan Shridhar (JIRA)" <ji...@apache.org> on 2017/12/14 09:01:02 UTC
[jira] [Comment Edited] (AMBARI-22649) Library for querying
cluster_settings and stack_settings in command*.json.
[ https://issues.apache.org/jira/browse/AMBARI-22649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290572#comment-16290572 ]
Swapan Shridhar edited comment on AMBARI-22649 at 12/14/17 9:00 AM:
--------------------------------------------------------------------
*Testing:*
Tested on live cluster
*=================*
*clusterSettings:*
*=================*
*A.* get_cluster_setting_entries():
*------------------------------------*
- 1. Retrieve *single* setting : 'recovery_enabled'
-- In get_cluster_setting_entries(). Passed-in setting(s) : set(['recovery_enabled'])
*o/p*: {'recovery_enabled': True}
- 2. Retrieve *two* settings : 'recovery_enabled', 'sysprep_skip_setup_jce'
-- In get_cluster_setting_entries(). Passed-in setting(s) : set(['recovery_enabled', 'sysprep_skip_setup_jce'])
*o/p*: {'recovery_enabled': True, 'sysprep_skip_setup_jce': False}
- 3. Retrieve settings where passed in empty set -> Expected nothing returned
-- In get_cluster_setting_entries(). Passed-in setting(s) : set([])
*o/p*: None
- 4. Retrieve *three* settings : 'smokeuser', 'recovery_enabled', 'sysprep_skip_setup_jce'
-- In get_cluster_setting_entries(). Passed-in setting(s) : set(['smokeuser', 'recovery_enabled', 'sysprep_skip_setup_jce'])
*o/p*: {'recovery_enabled': True, 'sysprep_skip_setup_jce': False, 'smokeuser': 'ambari-qa'}
- 5. Retrieve *three* settings where *middle setting is non-existent*
-- In get_cluster_setting_entries(). Passed-in setting(s) : set(['recovery_enabled', 'abc', 'sysprep_skip_setup_jce'])
*o/p* : {'recovery_enabled': True, 'sysprep_skip_setup_jce': False}
- 6. Retrieve *three* settings where *1st setting is non-existent*
-- In get_cluster_setting_entries(). Passed-in setting(s) : set(['abc', 'recovery_enabled', 'sysprep_skip_setup_jce'])
*o/p* : {'recovery_enabled': True, 'sysprep_skip_setup_jce': False}
- 7. Retrieve *three* settings where *last setting is non-existent*
-- In get_cluster_setting_entries(). Passed-in setting(s) : set(['recovery_enabled', 'sysprep_skip_setup_jce', 'abc'])
*o/p*: {'recovery_enabled': True, 'sysprep_skip_setup_jce': False}
- 8. Retrieve passed in setting which is *non-existent*
-- In get_cluster_setting_entries(). Passed-in setting(s) : set(['non-existent1'])
*o/p* : None
- 9. Retrieve *two* passed in settings and both are non-existent
-- In get_cluster_setting_entries(). Passed-in setting(s) : set(['non-existent1', 'non-existent2'])
*o/p* : None
- 10. Retrieve settings where set passed in is None. -> *returns all settings.*
-- In get_cluster_setting_entries(). Passed-in setting(s) : None
*o/p*: {'security_enabled': 'false', 'namenode_rolling_restart_timeout': '4200', 'enable_external_ranger': 'false', 'override_uid': 'true', 'kerberos_domain': 'EXAMPLE.COM', 'one_dir_per_partition': 'false', 'agent_mounts_ignore_list': '', 'repo_ubuntu_template': '{{package_type}} {{base_url}} {{components}}', 'ignore_groupsusers_create': 'false', 'alerts_repeat_tolerance': '1', 'hide_yarn_memory_widget': 'false', 'fetch_nonlocal_groups': 'true', 'manage_dirs_on_root': 'true', 'recovery_lifetime_max_count': '1024', 'recovery_type': 'AUTO_START', 'ignore_bad_mounts': 'false', 'recovery_window_in_minutes': '60', 'sysprep_skip_copy_tarballs_hdfs': 'false', 'user_group': 'hadoop', 'namenode_rolling_restart_safemode_exit_timeout': '3600', 'recovery_retry_interval': '5', 'sysprep_skip_copy_oozie_share_lib_to_hdfs': 'false', 'sysprep_skip_setup_jce': 'false', 'manage_hive_fsroot': 'true', 'service_check_type': 'full', 'recovery_enabled': 'true', 'recovery_max_count': '6', 'sysprep_skip_create_users_and_groups': 'false', 'smokeuser_keytab': '/etc/security/keytabs/smokeuser.headless.keytab', 'managed_hdfs_resource_property_names': 'false', 'smokeuser': 'ambari-qa', 'sysprep_skip_copy_fast_jar_hdfs': 'false'}
*B.* get_cluster_setting_value():
*------------------------------------*
- 1. Retrieve cluster_setting : 'hide_yarn_memory_widget'
-- In get_cluster_setting_value(). Passed-in setting : hide_yarn_memory_widget
*o/p* : false
- 2. Retrieve cluster_setting : 'recovery_max_count'
-- In get_cluster_setting_value(). Passed-in setting : recovery_max_count
*o/p* : 6
- 3. Retrieve cluster_setting where passed in setting is 'non_existing' --> Empty string
-- In get_cluster_setting_value(). Passed-in setting : non_existing
*o/p* : None
- 4. Retrieve cluster_setting setting passed in is "None"
-- In get_cluster_setting_value(). Passed-in setting : None
*o/p* : None
*C*. is_security_enabled():
*------------------------------------*
- 1. Retrieve security state of cluster by calling 'is_security_enabled()' which in turn calls *get_cluster_setting_value()*
-- In get_cluster_setting_value(). Passed-in setting : security_enabled
*o/p* : false
*=================*
*stackSettings*:
*=================*
*A.* Tests for *get_stack_setting_entries()*
*------------------------------------*
- 1. Retrieve : *stack_name*
-- In get_stack_setting_entries(). Passed-in setting(s) : set(['stack_name'])
o/p: {'stack_name': 'HDP'}
- 2. Retrieve : *stack_root*
-- In get_stack_setting_entries(). Passed-in setting(s) : set(['stack_root'])
*o/p*: {'stack_root': '{"HDP":"/usr/hdp"}'}
- 3. Retrieve *three* stack settings: *'stack_name', 'stack_root', 'stack_tools'*
-- In get_stack_setting_entries(). Passed-in setting(s) : set(['stack_name', 'stack_root', 'stack_tools'])
*o/p* : {'stack_name': 'HDP', 'stack_root': '{"HDP":"/usr/hdp"}', 'stack_tools': '{\n "HDP": {\n "stack_selector": [\n "hdp-select",\n "/usr/bin/hdp-select",\n "hdp-select"\n ],\n "conf_selector": [\n "conf-select",\n "/usr/bin/conf-select",\n "conf-select"\n ]\n }\n}'}
- 4. Retrieve *four* stack settings : *'BAD_VALUE', 'stack_name', 'stack_root', 'stack_tools'* ---> (*BAD_VALUE* should get ignored)
-- In get_stack_setting_entries(). Passed-in setting(s) : set(['stack_name', 'stack_root', 'stack_tools'])
*o/p* : {'stack_name': 'HDP', 'stack_root': '{"HDP":"/usr/hdp"}', 'stack_tools': '{\n "HDP": {\n "stack_selector": [\n "hdp-select",\n "/usr/bin/hdp-select",\n "hdp-select"\n ],\n "conf_selector": [\n "conf-select",\n "/usr/bin/conf-select",\n "conf-select"\n ]\n }\n}'}
*B.* Tests for existing and modified : *stack_select.get_packages()* which internally now calls *stack_settings.get_stack_setting_value()*
*------------------------------------------------------------------------------------------------------------------------------------------------------------*
- 1. Retrieve *specific component package* with call -> stack_select.get_packages(orchestration, "HDFS", "DATANODE")
-- In get_stack_setting_value(). Passed-in setting : stack_packages
*o/p* : ['hadoop-hdfs-datanode']
- 2. Retrieve *specific component package* with call -> stack_select.get_packages(orchestration, "ZOOKEEPER", "ZOOKEEPER_CLIENT")
-- In get_stack_setting_value(). Passed-in setting : stack_packages
*o/p* : ['zookeeper-client']
- 3. Retrieve *non-existing component package* with call -> stack_select.get_packages(orchestration, "ZOOKEEPER", "NON_EXISTING")
-- In get_stack_setting_value(). Passed-in setting : stack_packages
Skipping stack-select on ABC because it does not exist in the stack-select package structure.
*o/p* : None
*C*. Tests for existing and modified : *stack_features.check_stack_feature()* which internally now calls *stack_settings.get_stack_setting_value()*
*------------------------------------------------------------------------------------------------------------------------------------------------------------*
- 1. Retrieve *specific component stack feature* with call -> stack_features.check_stack_feature("hive_server_interactive", "2.6")
-- In get_stack_setting_value(). Passed-in setting : stack_features
*o/p* : True
- 2. Retrieve *specific component stack feature* with call -> stack_features.check_stack_feature("hive_server_interactive", "2.2")
-- In get_stack_setting_value(). Passed-in setting : stack_features
*o/p* : False
*D*. Test for existing and modified : *stack_tools.get_stack_tool()* which internally now calls *stack_settings.get_stack_setting_value()*
*------------------------------------------------------------------------------------------------------------------------------------------------------------*
- 1. Retrieve *stack relevant tools* with call -> stack_tools.get_stack_tool("stack_selector")
-- In get_stack_setting_value(). Passed-in setting : stack_tools
*o/p*: ('hdp-select', '/usr/bin/hdp-select', 'hdp-select')
was (Author: swapanshridhar):
*Testing:*
Tested on live cluster
*=================*
*clusterSettings:*
*=================*
*A.* get_cluster_setting_entries():
*------------------------------------*
- 1. Retrieve *single* setting : 'recovery_enabled'
-- In get_cluster_setting_entries(). Passed-in setting(s) : set(['recovery_enabled'])
*o/p*: {'recovery_enabled': True}
- 2. Retrieve *two* settings : 'recovery_enabled', 'sysprep_skip_setup_jce'
-- In get_cluster_setting_entries(). Passed-in setting(s) : set(['recovery_enabled', 'sysprep_skip_setup_jce'])
*o/p*: {'recovery_enabled': True, 'sysprep_skip_setup_jce': False}
- 3. Retrieve settings where passed in empty set -> Expected nothing returned
-- In get_cluster_setting_entries(). Passed-in setting(s) : set([])
*o/p*: None
- 4. Retrieve *three* settings : 'smokeuser', 'recovery_enabled', 'sysprep_skip_setup_jce'
-- In get_cluster_setting_entries(). Passed-in setting(s) : set(['smokeuser', 'recovery_enabled', 'sysprep_skip_setup_jce'])
*o/p*: {'recovery_enabled': True, 'sysprep_skip_setup_jce': False, 'smokeuser': 'ambari-qa'}
- 5. Retrieve *three* settings where *middle setting is non-existent*
-- In get_cluster_setting_entries(). Passed-in setting(s) : set(['recovery_enabled', 'abc', 'sysprep_skip_setup_jce'])
*o/p* : {'recovery_enabled': True, 'sysprep_skip_setup_jce': False}
- 6. Retrieve *three* settings where *1st setting is non-existent*
-- In get_cluster_setting_entries(). Passed-in setting(s) : set(['abc', 'recovery_enabled', 'sysprep_skip_setup_jce'])
*o/p* : {'recovery_enabled': True, 'sysprep_skip_setup_jce': False}
- 7. Retrieve *three* settings where *last setting is non-existent*
-- In get_cluster_setting_entries(). Passed-in setting(s) : set(['recovery_enabled', 'sysprep_skip_setup_jce', 'abc'])
*o/p*: {'recovery_enabled': True, 'sysprep_skip_setup_jce': False}
- 8. Retrieve passed in setting which is *non-existent*
-- In get_cluster_setting_entries(). Passed-in setting(s) : set(['non-existent1'])
*o/p* : None
- 9. Retrieve *two* passed in settings and both are non-existent
-- In get_cluster_setting_entries(). Passed-in setting(s) : set(['non-existent1', 'non-existent2'])
*o/p* : None
- 10. Retrieve settings where set passed in is None. -> *returns all settings.*
-- In get_cluster_setting_entries(). Passed-in setting(s) : None
*o/p*: {'security_enabled': 'false', 'namenode_rolling_restart_timeout': '4200', 'enable_external_ranger': 'false', 'override_uid': 'true', 'kerberos_domain': 'EXAMPLE.COM', 'one_dir_per_partition': 'false', 'agent_mounts_ignore_list': '', 'repo_ubuntu_template': '{{package_type}} {{base_url}} {{components}}', 'ignore_groupsusers_create': 'false', 'alerts_repeat_tolerance': '1', 'hide_yarn_memory_widget': 'false', 'fetch_nonlocal_groups': 'true', 'manage_dirs_on_root': 'true', 'recovery_lifetime_max_count': '1024', 'recovery_type': 'AUTO_START', 'ignore_bad_mounts': 'false', 'recovery_window_in_minutes': '60', 'sysprep_skip_copy_tarballs_hdfs': 'false', 'user_group': 'hadoop', 'namenode_rolling_restart_safemode_exit_timeout': '3600', 'recovery_retry_interval': '5', 'sysprep_skip_copy_oozie_share_lib_to_hdfs': 'false', 'sysprep_skip_setup_jce': 'false', 'manage_hive_fsroot': 'true', 'service_check_type': 'full', 'recovery_enabled': 'true', 'recovery_max_count': '6', 'sysprep_skip_create_users_and_groups': 'false', 'smokeuser_keytab': '/etc/security/keytabs/smokeuser.headless.keytab', 'managed_hdfs_resource_property_names': 'false', 'smokeuser': 'ambari-qa', 'sysprep_skip_copy_fast_jar_hdfs': 'false'}
*B.* get_cluster_setting_value():
*------------------------------------*
- 1. Retrieve cluster_setting : 'hide_yarn_memory_widget'
-- In get_cluster_setting_value(). Passed-in setting : hide_yarn_memory_widget
*o/p* : false
- 2. Retrieve cluster_setting : 'recovery_max_count'
-- In get_cluster_setting_value(). Passed-in setting : recovery_max_count
*o/p* : 6
- 3. Retrieve cluster_setting where passed in setting is 'non_existing' --> Empty string
-- In get_cluster_setting_value(). Passed-in setting : non_existing
*o/p* : None
- 4. Retrieve cluster_setting setting passed in is "None"
-- In get_cluster_setting_value(). Passed-in setting : None
*o/p* : None
*C*. is_security_enabled():
*------------------------------------*
- 1. Retrieve security state of cluster by calling 'is_security_enabled()' which in turn calls *get_cluster_setting_value()*
-- In get_cluster_setting_value(). Passed-in setting : security_enabled
*o/p* : false
*=================*
*stackSettings*:
*=================*
*A.* Tests for *get_stack_setting_entries()*
*------------------------------------*
- 1. Retrieve : *stack_name*
-- In get_stack_setting_entries(). Passed-in setting(s) : set(['stack_name'])
o/p: {'stack_name': 'HDP'}
- 2. Retrieve : *stack_root*
-- In get_stack_setting_entries(). Passed-in setting(s) : set(['stack_root'])
*o/p*: {'stack_root': '{"HDP":"/usr/hdp"}'}
- 3. Retrieve *three* stack settings: *'stack_name', 'stack_root', 'stack_tools'*
-- In get_stack_setting_entries(). Passed-in setting(s) : set(['stack_name', 'stack_root', 'stack_tools'])
*o/p* : {'stack_name': 'HDP', 'stack_root': '{"HDP":"/usr/hdp"}', 'stack_tools': '{\n "HDP": {\n "stack_selector": [\n "hdp-select",\n "/usr/bin/hdp-select",\n "hdp-select"\n ],\n "conf_selector": [\n "conf-select",\n "/usr/bin/conf-select",\n "conf-select"\n ]\n }\n}'}
- 4. Retrieve *four* stack settings : *'BAD_VALUE', 'stack_name', 'stack_root', 'stack_tools'* ---> (*BAD_VALUE* should get ignored)
-- In get_stack_setting_entries(). Passed-in setting(s) : set(['stack_name', 'stack_root', 'stack_tools'])
*o/p* : {'stack_name': 'HDP', 'stack_root': '{"HDP":"/usr/hdp"}', 'stack_tools': '{\n "HDP": {\n "stack_selector": [\n "hdp-select",\n "/usr/bin/hdp-select",\n "hdp-select"\n ],\n "conf_selector": [\n "conf-select",\n "/usr/bin/conf-select",\n "conf-select"\n ]\n }\n}'}
*B.* Tests for existing and modified : *stack_select.get_packages()* which internally now calls *stack_settings.get_stack_setting_value()*
*------------------------------------------------------------------------------------------------------------------------------------------------------------*
- 1. Retrieve *specific component package* with call -> stack_select.get_packages(orchestration, "HDFS", "DATANODE")
-- In get_stack_setting_value(). Passed-in setting : stack_packages
*o/p* : ['hadoop-hdfs-datanode']
- 2. Retrieve *specific component package* with call -> stack_select.get_packages(orchestration, "ZOOKEEPER", "ZOOKEEPER_CLIENT")
-- In get_stack_setting_value(). Passed-in setting : stack_packages
*o/p* : ['zookeeper-client']
- 3. Retrieve *non-existing component package* with call -> stack_select.get_packages(orchestration, "ZOOKEEPER", "NON_EXISTING")
-- In get_stack_setting_value(). Passed-in setting : stack_packages
Skipping stack-select on ABC because it does not exist in the stack-select package structure.
*o/p* : None
*C*. Tests for existing and modified : *stack_features.check_stack_feature()* which internally now calls *stack_settings.get_stack_setting_value()*
*------------------------------------------------------------------------------------------------------------------------------------------------------------*
- 1. Retrieve *specific component stack feature* with call -> stack_features.check_stack_feature("hive_server_interactive", "2.6")
-- In get_stack_setting_value(). Passed-in setting : stack_features
*o/p* : True
- 2. Retrieve *specific component stack feature* with call -> stack_features.check_stack_feature("hive_server_interactive", "2.2")
-- In get_stack_setting_value(). Passed-in setting : stack_features
*o/p* : False
*D*. Test for existing and modified : *stack_tools.get_stack_tool()* which internally now calls *stack_settings.get_stack_setting_value()*
*------------------------------------------------------------------------------------------------------------------------------------------------------------*
- 1. Retrieve *stack relevant tools* with call -> stack_tools.get_stack_tool("stack_selector")
-- In get_stack_setting_value(). Passed-in setting : stack_tools
*o/p*: ('hdp-select', '/usr/bin/hdp-select', 'hdp-select')
CC [~jluniya] | [~bsari] | [~mradhakrishnan]
> Library for querying cluster_settings and stack_settings in command*.json.
> --------------------------------------------------------------------------
>
> Key: AMBARI-22649
> URL: https://issues.apache.org/jira/browse/AMBARI-22649
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
> Reporter: Swapan Shridhar
> Assignee: Swapan Shridhar
> Fix For: 3.0.0
>
> Attachments: AMBARI-22649.patch
>
>
> Background : AMBARI-22198 added "stack settings", and AMBARI-22196 introduced "cluster settings" in Ambari.
> *=========================================================================*
> *Library for querying _clusterSettings_ and _stackSettings_ for its contents in command\*.json.*
> *=========================================================================*
> One should be able to query for a given *clusterSettings* or *stackSettings*:
> - by passing in the setting name(one or more) in order to get it back as key-value map, or
> - just get the value back for a passed-in setting.
> *Functions for clusterSettings:*
> *-----------------------------------------*
> - get_cluster_setting_entries(setting_names) :
> -- Retrieves the passed-in cluster setting entr(y/ies) and their values as a map.
> If 'setting_names' is passed-in as None : all the settings names and their corresponding values will be returned as map.
> If 'setting_names' is passed-in as empty set : None will be returned.
> - get_cluster_setting_value(setting_name) :
> -- Retrieves the passed-in cluster setting entry's value.
> - is_security_enabled() :
> -- Retrieves the cluster's security status.
> *Functions for stackSettings:*
> *-----------------------------------------*
> Stack settings as of now has 5 settings : stack_name, stack_root, stack_features, stack_tools, stack_packages. stack_name, stack_root have string as values, whereas stack_features, stack_tools, stack_packages have values as JSON. Further there already exists python functions in files : *stack_features.py*, *stack_tools.py* and *stack_select.py*.
> - get_stack_setting_entries(setting_names) :
> -- Retrieves the passed-in stack setting entr(y/ies) and their values as a map.
> If 'setting_names' is passed-in as None, all the settings names and their corresponding values will be returned as map.
> If 'setting_names' is passed-in as empty set : None will be returned.
> - get_stack_setting_value(setting_name):
> -- Retrieves the passed-in stack setting entry's value.
> - get_stack_name():
> -- Retrieves the stack name.
> - get_stack_root():
> -- Retrieves the stack root.
>
> *Modifications in _stack_features.py, stack_tools.py and stack_select.py_ files:*
> *--------------------------------------------------------------------------------------------------------------*
> - Given that these already exist and as of now they read the relevant stack setting from *configurations/cluster_env*.
> - Thus, code has been added to try reading from /stackSettings first by calling the new fn.() get_stack_setting_value(). if setting not found, go for the fall back *configurations/cluster_env* (which would be removed soon, when we remove cluster_env).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)