You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Jayush Luniya <jl...@hortonworks.com> on 2016/03/15 07:18:26 UTC
Review Request 44835: AMBARI-15420: Refactor resource_management
library
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/
-----------------------------------------------------------
Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
Bugs: AMBARI-15420
https://issues.apache.org/jira/browse/AMBARI-15420
Repository: ambari
Description
-------
Refactor resource_management library, to generalize the library and remove HDP-specific hardcodings
Specifically,
1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding hdp-select and conf-select
2. Make stack-root stack-driven instead of hardcoding /usr/hdp
3. Make copy_tarball mappings stack-driven
4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of hardcoding the "/usr/hdp"
In addition, also added a feature in the stack processing engine to support properties values to be defined a external property file (See tarball_map.json, stack_tools.json in patch)
Remaining HDP-specific logic in resource_management library
1. conf_select::_valid()
2. conf_select::get_hadoop_conf_dir()
3. list_ambari_managed_repos::repository_names
4. version_select_util::get_component_version()
5. script::get_stack_version()
6. script::should_expose_component_version()
7. get_lzo_packages::get_lzo_packages()
Refactoring this HDP-specific logic would require "Stack Featurization" (AMBARI-13364) to be in place.
Diffs
-----
ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py ee7db0a
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 59c717b
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 647b8b6
ambari-common/src/main/python/resource_management/libraries/functions/default.py 23383dc
ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py a20b03c
ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py f2e6567
ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py f1cd9cb
ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py d585dea
ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py 4d9d8a4
ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py 2ccc0c6
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py c94d956
ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 95c5cba
ambari-common/src/main/python/resource_management/libraries/script/script.py a8098a0
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 4be4049
ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java 7f21aaa
ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 0c7faea
ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java c739211
ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 7d934bb
ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java e7c9c27
ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java 3f7f756
ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java PRE-CREATION
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py 12ca388
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py 45dbb24
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py d000846
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py 2c06c40
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py 535c821
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 85e7012
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py c3d15e5
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py c5e61e6
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py d0924b9
ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 30b9a41
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 81a227e
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py 030fb2d
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py 27e2766
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/status_params.py 954bb80
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py e5b54cd
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 906b198
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py ec37243
ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/status_params.py 984a4ba
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py eb80ad6
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_windows.py dd732f5
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 8b2aec5
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py e05ed60
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 08bdcc3
ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py e69a2e4
ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py b8bf176
ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 96dc104
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py 07faae4
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/tarball_map.json PRE-CREATION
ambari-server/src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java 92c1200
ambari-server/src/test/java/org/apache/ambari/server/utils/TestJsonUtils.java PRE-CREATION
ambari-server/src/test/python/custom_actions/test_ru_set_all.py 3090f6b
ambari-server/src/test/python/host_scripts/TestAlertDiskSpace.py 2608050
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b5b43d6
ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py efe6038
ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 4601092
ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py daee726
ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py ab08776
Diff: https://reviews.apache.org/r/44835/diff/
Testing
-------
Thanks,
Jayush Luniya
Re: Review Request 44835: AMBARI-15420: Refactor resource_management
library
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/#review123708
-----------------------------------------------------------
Ship it!
Ship It!
- Nate Cole
On March 15, 2016, 2:32 a.m., Jayush Luniya wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44835/
> -----------------------------------------------------------
>
> (Updated March 15, 2016, 2:32 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-15420
> https://issues.apache.org/jira/browse/AMBARI-15420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Refactor resource_management library, to generalize the library and remove HDP-specific hardcodings
>
> Specifically,
> 1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding hdp-select and conf-select
> 2. Make stack-root stack-driven instead of hardcoding /usr/hdp
> 3. Make copy_tarball mappings stack-driven
> 4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of hardcoding the "/usr/hdp"
>
> In addition, also added a feature in the stack processing engine to support properties values to be defined a external property file (See tarball_map.json, stack_tools.json in patch)
>
> Three config properties are added
> 1. cluster-env/stack_root
> 2. cluster-env/stack_tools
> 3. cluster-env/tarball_map
> Corresponding helper functions get_stack_root(), get_stack_tool(), get_tarball_map() are added in script.py, which will set the defaults if these config properties are not defined (ambari-server upgrade scenario needs to be addressed to add these config properties on upgrade). These helper functions are used to remove hardcodings in resource_management library.
>
>
> Remaining HDP-specific logic in resource_management library
>
> 1. conf_select::_valid()
> 2. conf_select::get_hadoop_conf_dir()
> 3. list_ambari_managed_repos::repository_names
> 4. version_select_util::get_component_version()
> 5. script::get_stack_version()
> 6. script::should_expose_component_version()
> 7. get_lzo_packages::get_lzo_packages()
>
> Refactoring this HDP-specific logic would require "Stack Featurization" (AMBARI-13364) to be in place.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py ee7db0a
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 59c717b
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 647b8b6
> ambari-common/src/main/python/resource_management/libraries/functions/default.py 23383dc
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py a20b03c
> ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py f2e6567
> ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py f1cd9cb
> ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py d585dea
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py 4d9d8a4
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py 2ccc0c6
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py c94d956
> ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 95c5cba
> ambari-common/src/main/python/resource_management/libraries/script/script.py a8098a0
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 4be4049
> ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java 7f21aaa
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 0c7faea
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java c739211
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 7d934bb
> ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java e7c9c27
> ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java 3f7f756
> ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java PRE-CREATION
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py 12ca388
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py 45dbb24
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py d000846
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py 2c06c40
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py 535c821
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 85e7012
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py c3d15e5
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py c5e61e6
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py d0924b9
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 30b9a41
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 81a227e
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py 030fb2d
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py 27e2766
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/status_params.py 954bb80
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py e5b54cd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 906b198
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py ec37243
> ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/status_params.py 984a4ba
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py eb80ad6
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_windows.py dd732f5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 8b2aec5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py e05ed60
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 08bdcc3
> ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py e69a2e4
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py b8bf176
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 96dc104
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py 07faae4
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/tarball_map.json PRE-CREATION
> ambari-server/src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java 92c1200
> ambari-server/src/test/java/org/apache/ambari/server/utils/TestJsonUtils.java PRE-CREATION
> ambari-server/src/test/python/custom_actions/test_ru_set_all.py 3090f6b
> ambari-server/src/test/python/host_scripts/TestAlertDiskSpace.py 2608050
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b5b43d6
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py efe6038
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 4601092
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py daee726
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py ab08776
>
> Diff: https://reviews.apache.org/r/44835/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jayush Luniya
>
>
Re: Review Request 44835: AMBARI-15420: Refactor resource_management
library
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/#review123657
-----------------------------------------------------------
By the logic in this review, does "current" belong with STACK_ROOT_PATTERN? The amount of changes here are significant - is there a smaller piece we can do first to get the pattern down first? Also, how do we account for the case where there are two possible roots:
/usr/hdp/current/hadoop-client which is the symlink to /usr/hdp/<version>/hadoop. There are times where we need the former, and other cases for the latter. How is that done?
And, what if a stack has no selector or tools? Are we just going to error out?
It's fine to generalize how stacks represent how to do these things, but you're just shuffling around how those are getting determined in this review.
Did you consider refactoring instead the common-services stuff to consult the stack for python-sourced modules? That would be the cleaner approach over trying to make up command paths on the fly based on what's on the stack and making things nearly impossible to debug. We can then keep HDP-specific in the HDP stack directories.
- Nate Cole
On March 15, 2016, 2:32 a.m., Jayush Luniya wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44835/
> -----------------------------------------------------------
>
> (Updated March 15, 2016, 2:32 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-15420
> https://issues.apache.org/jira/browse/AMBARI-15420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Refactor resource_management library, to generalize the library and remove HDP-specific hardcodings
>
> Specifically,
> 1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding hdp-select and conf-select
> 2. Make stack-root stack-driven instead of hardcoding /usr/hdp
> 3. Make copy_tarball mappings stack-driven
> 4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of hardcoding the "/usr/hdp"
>
> In addition, also added a feature in the stack processing engine to support properties values to be defined a external property file (See tarball_map.json, stack_tools.json in patch)
>
> Three config properties are added
> 1. cluster-env/stack_root
> 2. cluster-env/stack_tools
> 3. cluster-env/tarball_map
> Corresponding helper functions get_stack_root(), get_stack_tool(), get_tarball_map() are added in script.py, which will set the defaults if these config properties are not defined (ambari-server upgrade scenario needs to be addressed to add these config properties on upgrade). These helper functions are used to remove hardcodings in resource_management library.
>
>
> Remaining HDP-specific logic in resource_management library
>
> 1. conf_select::_valid()
> 2. conf_select::get_hadoop_conf_dir()
> 3. list_ambari_managed_repos::repository_names
> 4. version_select_util::get_component_version()
> 5. script::get_stack_version()
> 6. script::should_expose_component_version()
> 7. get_lzo_packages::get_lzo_packages()
>
> Refactoring this HDP-specific logic would require "Stack Featurization" (AMBARI-13364) to be in place.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py ee7db0a
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 59c717b
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 647b8b6
> ambari-common/src/main/python/resource_management/libraries/functions/default.py 23383dc
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py a20b03c
> ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py f2e6567
> ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py f1cd9cb
> ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py d585dea
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py 4d9d8a4
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py 2ccc0c6
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py c94d956
> ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 95c5cba
> ambari-common/src/main/python/resource_management/libraries/script/script.py a8098a0
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 4be4049
> ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java 7f21aaa
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 0c7faea
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java c739211
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 7d934bb
> ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java e7c9c27
> ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java 3f7f756
> ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java PRE-CREATION
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py 12ca388
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py 45dbb24
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py d000846
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py 2c06c40
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py 535c821
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 85e7012
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py c3d15e5
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py c5e61e6
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py d0924b9
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 30b9a41
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 81a227e
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py 030fb2d
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py 27e2766
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/status_params.py 954bb80
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py e5b54cd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 906b198
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py ec37243
> ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/status_params.py 984a4ba
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py eb80ad6
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_windows.py dd732f5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 8b2aec5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py e05ed60
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 08bdcc3
> ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py e69a2e4
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py b8bf176
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 96dc104
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py 07faae4
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/tarball_map.json PRE-CREATION
> ambari-server/src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java 92c1200
> ambari-server/src/test/java/org/apache/ambari/server/utils/TestJsonUtils.java PRE-CREATION
> ambari-server/src/test/python/custom_actions/test_ru_set_all.py 3090f6b
> ambari-server/src/test/python/host_scripts/TestAlertDiskSpace.py 2608050
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b5b43d6
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py efe6038
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 4601092
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py daee726
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py ab08776
>
> Diff: https://reviews.apache.org/r/44835/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jayush Luniya
>
>
Re: Review Request 44835: AMBARI-15420: Refactor resource_management
library
Posted by Jayush Luniya <jl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/#review123683
-----------------------------------------------------------
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py (line 42)
<https://reviews.apache.org/r/44835/#comment185915>
Will fix
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py (line 46)
<https://reviews.apache.org/r/44835/#comment185916>
At present HDP, PHD and HDF stacks have the same structure except for the stack root being /usr/hdp, /usr/phd.
Yes in the long run additional refactoring would be required, but that is a long haul work and we dont have a use case yet for it.
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py (lines 201 - 202)
<https://reviews.apache.org/r/44835/#comment185923>
The conf-selectors and stack-selectors used by the stacks are pretty much the same in usage.
Yes, in the future we might have a conf-selector, stack-selector which is entirely different, but I am not going with refactoring to that level right now.
- Jayush Luniya
On March 15, 2016, 6:32 a.m., Jayush Luniya wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44835/
> -----------------------------------------------------------
>
> (Updated March 15, 2016, 6:32 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-15420
> https://issues.apache.org/jira/browse/AMBARI-15420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Refactor resource_management library, to generalize the library and remove HDP-specific hardcodings
>
> Specifically,
> 1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding hdp-select and conf-select
> 2. Make stack-root stack-driven instead of hardcoding /usr/hdp
> 3. Make copy_tarball mappings stack-driven
> 4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of hardcoding the "/usr/hdp"
>
> In addition, also added a feature in the stack processing engine to support properties values to be defined a external property file (See tarball_map.json, stack_tools.json in patch)
>
> Three config properties are added
> 1. cluster-env/stack_root
> 2. cluster-env/stack_tools
> 3. cluster-env/tarball_map
> Corresponding helper functions get_stack_root(), get_stack_tool(), get_tarball_map() are added in script.py, which will set the defaults if these config properties are not defined (ambari-server upgrade scenario needs to be addressed to add these config properties on upgrade). These helper functions are used to remove hardcodings in resource_management library.
>
>
> Remaining HDP-specific logic in resource_management library
>
> 1. conf_select::_valid()
> 2. conf_select::get_hadoop_conf_dir()
> 3. list_ambari_managed_repos::repository_names
> 4. version_select_util::get_component_version()
> 5. script::get_stack_version()
> 6. script::should_expose_component_version()
> 7. get_lzo_packages::get_lzo_packages()
>
> Refactoring this HDP-specific logic would require "Stack Featurization" (AMBARI-13364) to be in place.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py ee7db0a
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 59c717b
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 647b8b6
> ambari-common/src/main/python/resource_management/libraries/functions/default.py 23383dc
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py a20b03c
> ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py f2e6567
> ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py f1cd9cb
> ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py d585dea
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py 4d9d8a4
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py 2ccc0c6
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py c94d956
> ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 95c5cba
> ambari-common/src/main/python/resource_management/libraries/script/script.py a8098a0
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 4be4049
> ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java 7f21aaa
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 0c7faea
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java c739211
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 7d934bb
> ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java e7c9c27
> ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java 3f7f756
> ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java PRE-CREATION
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py 12ca388
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py 45dbb24
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py d000846
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py 2c06c40
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py 535c821
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 85e7012
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py c3d15e5
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py c5e61e6
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py d0924b9
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 30b9a41
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 81a227e
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py 030fb2d
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py 27e2766
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/status_params.py 954bb80
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py e5b54cd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 906b198
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py ec37243
> ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/status_params.py 984a4ba
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py eb80ad6
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_windows.py dd732f5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 8b2aec5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py e05ed60
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 08bdcc3
> ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py e69a2e4
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py b8bf176
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 96dc104
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py 07faae4
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/tarball_map.json PRE-CREATION
> ambari-server/src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java 92c1200
> ambari-server/src/test/java/org/apache/ambari/server/utils/TestJsonUtils.java PRE-CREATION
> ambari-server/src/test/python/custom_actions/test_ru_set_all.py 3090f6b
> ambari-server/src/test/python/host_scripts/TestAlertDiskSpace.py 2608050
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b5b43d6
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py efe6038
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 4601092
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py daee726
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py ab08776
>
> Diff: https://reviews.apache.org/r/44835/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jayush Luniya
>
>
Re: Review Request 44835: AMBARI-15420: Refactor resource_management
library
Posted by Jayush Luniya <jl...@hortonworks.com>.
> On March 15, 2016, 2:44 p.m., Jonathan Hurley wrote:
> > ambari-common/src/main/python/resource_management/libraries/script/script.py, line 375
> > <https://reviews.apache.org/r/44835/diff/1/?file=1299208#file1299208line375>
> >
> > I don't really understand this abstraction. You're saying that the stack tools can be parameterized. But that all still depends on the custom stack tools fucntioning in 100% the exact same way as hdp-select and conf-select.
Yes that is correct. The changes are not going far enough to generalize everything. In common-services scripts there are mutiple places where we are using /usr/hdp/current, /usr/hdp/{version} hardcodings and in a way assume the way stack tools work. The intention of the changes is not to do a complete overhaul, but to parameterize so that variants of stacks similar to HDP can work.
- Jayush
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/#review123658
-----------------------------------------------------------
On March 15, 2016, 6:32 a.m., Jayush Luniya wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44835/
> -----------------------------------------------------------
>
> (Updated March 15, 2016, 6:32 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-15420
> https://issues.apache.org/jira/browse/AMBARI-15420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Refactor resource_management library, to generalize the library and remove HDP-specific hardcodings
>
> Specifically,
> 1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding hdp-select and conf-select
> 2. Make stack-root stack-driven instead of hardcoding /usr/hdp
> 3. Make copy_tarball mappings stack-driven
> 4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of hardcoding the "/usr/hdp"
>
> In addition, also added a feature in the stack processing engine to support properties values to be defined a external property file (See tarball_map.json, stack_tools.json in patch)
>
> Three config properties are added
> 1. cluster-env/stack_root
> 2. cluster-env/stack_tools
> 3. cluster-env/tarball_map
> Corresponding helper functions get_stack_root(), get_stack_tool(), get_tarball_map() are added in script.py, which will set the defaults if these config properties are not defined (ambari-server upgrade scenario needs to be addressed to add these config properties on upgrade). These helper functions are used to remove hardcodings in resource_management library.
>
>
> Remaining HDP-specific logic in resource_management library
>
> 1. conf_select::_valid()
> 2. conf_select::get_hadoop_conf_dir()
> 3. list_ambari_managed_repos::repository_names
> 4. version_select_util::get_component_version()
> 5. script::get_stack_version()
> 6. script::should_expose_component_version()
> 7. get_lzo_packages::get_lzo_packages()
>
> Refactoring this HDP-specific logic would require "Stack Featurization" (AMBARI-13364) to be in place.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py ee7db0a
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 59c717b
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 647b8b6
> ambari-common/src/main/python/resource_management/libraries/functions/default.py 23383dc
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py a20b03c
> ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py f2e6567
> ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py f1cd9cb
> ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py d585dea
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py 4d9d8a4
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py 2ccc0c6
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py c94d956
> ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 95c5cba
> ambari-common/src/main/python/resource_management/libraries/script/script.py a8098a0
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 4be4049
> ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java 7f21aaa
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 0c7faea
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java c739211
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 7d934bb
> ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java e7c9c27
> ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java 3f7f756
> ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java PRE-CREATION
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py 12ca388
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py 45dbb24
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py d000846
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py 2c06c40
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py 535c821
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 85e7012
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py c3d15e5
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py c5e61e6
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py d0924b9
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 30b9a41
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 81a227e
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py 030fb2d
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py 27e2766
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/status_params.py 954bb80
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py e5b54cd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 906b198
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py ec37243
> ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/status_params.py 984a4ba
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py eb80ad6
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_windows.py dd732f5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 8b2aec5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py e05ed60
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 08bdcc3
> ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py e69a2e4
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py b8bf176
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 96dc104
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py 07faae4
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/tarball_map.json PRE-CREATION
> ambari-server/src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java 92c1200
> ambari-server/src/test/java/org/apache/ambari/server/utils/TestJsonUtils.java PRE-CREATION
> ambari-server/src/test/python/custom_actions/test_ru_set_all.py 3090f6b
> ambari-server/src/test/python/host_scripts/TestAlertDiskSpace.py 2608050
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b5b43d6
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py efe6038
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 4601092
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py daee726
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py ab08776
>
> Diff: https://reviews.apache.org/r/44835/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jayush Luniya
>
>
Re: Review Request 44835: AMBARI-15420: Refactor resource_management
library
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On March 15, 2016, 10:44 a.m., Jonathan Hurley wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py, line 46
> > <https://reviews.apache.org/r/44835/diff/1/?file=1299197#file1299197line46>
> >
> > I don't think these changes have been thought through all of the way. You're trying to abstract out the "stack root" of /usr/hdp, but you're still leaving "current" as a hard-coded item.
> >
> > I fail to see in what context this is useful. The stack scripts all still use /usr/hdp, so why are we adding more complex logic to abstract it out.
> >
> > Additionally, you're assuming that another stack will come along and use a different root but the exact same "current" convention. That's not very likely.
>
> Jayush Luniya wrote:
> At present HDP, PHD and HDF stacks have the same structure except for the stack root being /usr/hdp, /usr/phd.
>
> Yes in the long run additional refactoring would be required, but that is a long haul work and we dont have a use case yet for it.
OK, so we're acknowledging this limitation.
> On March 15, 2016, 10:44 a.m., Jonathan Hurley wrote:
> > ambari-common/src/main/python/resource_management/libraries/script/script.py, line 375
> > <https://reviews.apache.org/r/44835/diff/1/?file=1299208#file1299208line375>
> >
> > I don't really understand this abstraction. You're saying that the stack tools can be parameterized. But that all still depends on the custom stack tools fucntioning in 100% the exact same way as hdp-select and conf-select.
>
> Jayush Luniya wrote:
> Yes that is correct. The changes are not going far enough to generalize everything. In common-services scripts there are mutiple places where we are using /usr/hdp/current, /usr/hdp/{version} hardcodings and in a way assume the way stack tools work. The intention of the changes is not to do a complete overhaul, but to parameterize so that variants of stacks similar to HDP can work.
Right, so as long as we're are admitting, upfront, that these changes rely 100% on stack tools outside of our controlling remaining consistent, then these changes make sense. I know it's a first step; it's just a step that seems very fragile since we can't predict how tools outside of our control can change.
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/#review123658
-----------------------------------------------------------
On March 15, 2016, 2:32 a.m., Jayush Luniya wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44835/
> -----------------------------------------------------------
>
> (Updated March 15, 2016, 2:32 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-15420
> https://issues.apache.org/jira/browse/AMBARI-15420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Refactor resource_management library, to generalize the library and remove HDP-specific hardcodings
>
> Specifically,
> 1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding hdp-select and conf-select
> 2. Make stack-root stack-driven instead of hardcoding /usr/hdp
> 3. Make copy_tarball mappings stack-driven
> 4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of hardcoding the "/usr/hdp"
>
> In addition, also added a feature in the stack processing engine to support properties values to be defined a external property file (See tarball_map.json, stack_tools.json in patch)
>
> Three config properties are added
> 1. cluster-env/stack_root
> 2. cluster-env/stack_tools
> 3. cluster-env/tarball_map
> Corresponding helper functions get_stack_root(), get_stack_tool(), get_tarball_map() are added in script.py, which will set the defaults if these config properties are not defined (ambari-server upgrade scenario needs to be addressed to add these config properties on upgrade). These helper functions are used to remove hardcodings in resource_management library.
>
>
> Remaining HDP-specific logic in resource_management library
>
> 1. conf_select::_valid()
> 2. conf_select::get_hadoop_conf_dir()
> 3. list_ambari_managed_repos::repository_names
> 4. version_select_util::get_component_version()
> 5. script::get_stack_version()
> 6. script::should_expose_component_version()
> 7. get_lzo_packages::get_lzo_packages()
>
> Refactoring this HDP-specific logic would require "Stack Featurization" (AMBARI-13364) to be in place.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py ee7db0a
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 59c717b
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 647b8b6
> ambari-common/src/main/python/resource_management/libraries/functions/default.py 23383dc
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py a20b03c
> ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py f2e6567
> ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py f1cd9cb
> ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py d585dea
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py 4d9d8a4
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py 2ccc0c6
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py c94d956
> ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 95c5cba
> ambari-common/src/main/python/resource_management/libraries/script/script.py a8098a0
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 4be4049
> ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java 7f21aaa
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 0c7faea
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java c739211
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 7d934bb
> ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java e7c9c27
> ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java 3f7f756
> ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java PRE-CREATION
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py 12ca388
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py 45dbb24
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py d000846
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py 2c06c40
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py 535c821
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 85e7012
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py c3d15e5
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py c5e61e6
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py d0924b9
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 30b9a41
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 81a227e
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py 030fb2d
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py 27e2766
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/status_params.py 954bb80
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py e5b54cd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 906b198
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py ec37243
> ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/status_params.py 984a4ba
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py eb80ad6
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_windows.py dd732f5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 8b2aec5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py e05ed60
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 08bdcc3
> ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py e69a2e4
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py b8bf176
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 96dc104
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py 07faae4
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/tarball_map.json PRE-CREATION
> ambari-server/src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java 92c1200
> ambari-server/src/test/java/org/apache/ambari/server/utils/TestJsonUtils.java PRE-CREATION
> ambari-server/src/test/python/custom_actions/test_ru_set_all.py 3090f6b
> ambari-server/src/test/python/host_scripts/TestAlertDiskSpace.py 2608050
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b5b43d6
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py efe6038
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 4601092
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py daee726
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py ab08776
>
> Diff: https://reviews.apache.org/r/44835/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jayush Luniya
>
>
Re: Review Request 44835: AMBARI-15420: Refactor resource_management
library
Posted by Jayush Luniya <jl...@hortonworks.com>.
> On March 15, 2016, 2:44 p.m., Jonathan Hurley wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py, line 42
> > <https://reviews.apache.org/r/44835/diff/1/?file=1299197#file1299197line42>
> >
> > This should no longer be accessible outside of this module since the get_package_dirs below is now required in order to access this data.
Will fix
> On March 15, 2016, 2:44 p.m., Jonathan Hurley wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py, lines 201-202
> > <https://reviews.apache.org/r/44835/diff/1/?file=1299197#file1299197line201>
> >
> > Once again, why is this being done? This module is a wrapper around a specific tool; the conf-select tool.
> >
> > Abstracting away the path is just odd since it's not changing.
> >
> > If you're assuming that conf-select can be implemented by other stacks, then you're putting a lot of faith into a custom-conf-select working in basically the exact same way as the version provided in hdp.
The conf-selectors and stack-selectors used by the stacks are pretty much the same in usage except for the tool name etc.
Yes, in the future we might have a conf-selector, stack-selector which is entirely different, but I am not going with refactoring to that level right now.
> On March 15, 2016, 2:44 p.m., Jonathan Hurley wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py, line 46
> > <https://reviews.apache.org/r/44835/diff/1/?file=1299197#file1299197line46>
> >
> > I don't think these changes have been thought through all of the way. You're trying to abstract out the "stack root" of /usr/hdp, but you're still leaving "current" as a hard-coded item.
> >
> > I fail to see in what context this is useful. The stack scripts all still use /usr/hdp, so why are we adding more complex logic to abstract it out.
> >
> > Additionally, you're assuming that another stack will come along and use a different root but the exact same "current" convention. That's not very likely.
At present HDP, PHD and HDF stacks have the same structure except for the stack root being /usr/hdp, /usr/phd.
Yes in the long run additional refactoring would be required, but that is a long haul work and we dont have a use case yet for it.
- Jayush
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/#review123658
-----------------------------------------------------------
On March 15, 2016, 6:32 a.m., Jayush Luniya wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44835/
> -----------------------------------------------------------
>
> (Updated March 15, 2016, 6:32 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-15420
> https://issues.apache.org/jira/browse/AMBARI-15420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Refactor resource_management library, to generalize the library and remove HDP-specific hardcodings
>
> Specifically,
> 1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding hdp-select and conf-select
> 2. Make stack-root stack-driven instead of hardcoding /usr/hdp
> 3. Make copy_tarball mappings stack-driven
> 4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of hardcoding the "/usr/hdp"
>
> In addition, also added a feature in the stack processing engine to support properties values to be defined a external property file (See tarball_map.json, stack_tools.json in patch)
>
> Three config properties are added
> 1. cluster-env/stack_root
> 2. cluster-env/stack_tools
> 3. cluster-env/tarball_map
> Corresponding helper functions get_stack_root(), get_stack_tool(), get_tarball_map() are added in script.py, which will set the defaults if these config properties are not defined (ambari-server upgrade scenario needs to be addressed to add these config properties on upgrade). These helper functions are used to remove hardcodings in resource_management library.
>
>
> Remaining HDP-specific logic in resource_management library
>
> 1. conf_select::_valid()
> 2. conf_select::get_hadoop_conf_dir()
> 3. list_ambari_managed_repos::repository_names
> 4. version_select_util::get_component_version()
> 5. script::get_stack_version()
> 6. script::should_expose_component_version()
> 7. get_lzo_packages::get_lzo_packages()
>
> Refactoring this HDP-specific logic would require "Stack Featurization" (AMBARI-13364) to be in place.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py ee7db0a
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 59c717b
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 647b8b6
> ambari-common/src/main/python/resource_management/libraries/functions/default.py 23383dc
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py a20b03c
> ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py f2e6567
> ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py f1cd9cb
> ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py d585dea
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py 4d9d8a4
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py 2ccc0c6
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py c94d956
> ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 95c5cba
> ambari-common/src/main/python/resource_management/libraries/script/script.py a8098a0
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 4be4049
> ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java 7f21aaa
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 0c7faea
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java c739211
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 7d934bb
> ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java e7c9c27
> ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java 3f7f756
> ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java PRE-CREATION
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py 12ca388
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py 45dbb24
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py d000846
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py 2c06c40
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py 535c821
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 85e7012
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py c3d15e5
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py c5e61e6
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py d0924b9
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 30b9a41
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 81a227e
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py 030fb2d
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py 27e2766
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/status_params.py 954bb80
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py e5b54cd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 906b198
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py ec37243
> ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/status_params.py 984a4ba
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py eb80ad6
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_windows.py dd732f5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 8b2aec5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py e05ed60
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 08bdcc3
> ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py e69a2e4
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py b8bf176
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 96dc104
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py 07faae4
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/tarball_map.json PRE-CREATION
> ambari-server/src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java 92c1200
> ambari-server/src/test/java/org/apache/ambari/server/utils/TestJsonUtils.java PRE-CREATION
> ambari-server/src/test/python/custom_actions/test_ru_set_all.py 3090f6b
> ambari-server/src/test/python/host_scripts/TestAlertDiskSpace.py 2608050
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b5b43d6
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py efe6038
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 4601092
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py daee726
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py ab08776
>
> Diff: https://reviews.apache.org/r/44835/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jayush Luniya
>
>
Re: Review Request 44835: AMBARI-15420: Refactor resource_management
library
Posted by Jayush Luniya <jl...@hortonworks.com>.
> On March 15, 2016, 2:44 p.m., Jonathan Hurley wrote:
> > ambari-common/src/main/python/resource_management/libraries/script/script.py, lines 76-88
> > <https://reviews.apache.org/r/44835/diff/1/?file=1299208#file1299208line76>
> >
> > Why is the Script class controlling these now? That seems like coupling two areas which should not be coupled.
As we discussed, will address it
- Jayush
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/#review123658
-----------------------------------------------------------
On March 15, 2016, 6:32 a.m., Jayush Luniya wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44835/
> -----------------------------------------------------------
>
> (Updated March 15, 2016, 6:32 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-15420
> https://issues.apache.org/jira/browse/AMBARI-15420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Refactor resource_management library, to generalize the library and remove HDP-specific hardcodings
>
> Specifically,
> 1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding hdp-select and conf-select
> 2. Make stack-root stack-driven instead of hardcoding /usr/hdp
> 3. Make copy_tarball mappings stack-driven
> 4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of hardcoding the "/usr/hdp"
>
> In addition, also added a feature in the stack processing engine to support properties values to be defined a external property file (See tarball_map.json, stack_tools.json in patch)
>
> Three config properties are added
> 1. cluster-env/stack_root
> 2. cluster-env/stack_tools
> 3. cluster-env/tarball_map
> Corresponding helper functions get_stack_root(), get_stack_tool(), get_tarball_map() are added in script.py, which will set the defaults if these config properties are not defined (ambari-server upgrade scenario needs to be addressed to add these config properties on upgrade). These helper functions are used to remove hardcodings in resource_management library.
>
>
> Remaining HDP-specific logic in resource_management library
>
> 1. conf_select::_valid()
> 2. conf_select::get_hadoop_conf_dir()
> 3. list_ambari_managed_repos::repository_names
> 4. version_select_util::get_component_version()
> 5. script::get_stack_version()
> 6. script::should_expose_component_version()
> 7. get_lzo_packages::get_lzo_packages()
>
> Refactoring this HDP-specific logic would require "Stack Featurization" (AMBARI-13364) to be in place.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py ee7db0a
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 59c717b
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 647b8b6
> ambari-common/src/main/python/resource_management/libraries/functions/default.py 23383dc
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py a20b03c
> ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py f2e6567
> ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py f1cd9cb
> ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py d585dea
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py 4d9d8a4
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py 2ccc0c6
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py c94d956
> ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 95c5cba
> ambari-common/src/main/python/resource_management/libraries/script/script.py a8098a0
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 4be4049
> ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java 7f21aaa
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 0c7faea
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java c739211
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 7d934bb
> ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java e7c9c27
> ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java 3f7f756
> ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java PRE-CREATION
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py 12ca388
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py 45dbb24
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py d000846
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py 2c06c40
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py 535c821
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 85e7012
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py c3d15e5
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py c5e61e6
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py d0924b9
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 30b9a41
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 81a227e
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py 030fb2d
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py 27e2766
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/status_params.py 954bb80
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py e5b54cd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 906b198
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py ec37243
> ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/status_params.py 984a4ba
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py eb80ad6
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_windows.py dd732f5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 8b2aec5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py e05ed60
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 08bdcc3
> ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py e69a2e4
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py b8bf176
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 96dc104
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py 07faae4
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/tarball_map.json PRE-CREATION
> ambari-server/src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java 92c1200
> ambari-server/src/test/java/org/apache/ambari/server/utils/TestJsonUtils.java PRE-CREATION
> ambari-server/src/test/python/custom_actions/test_ru_set_all.py 3090f6b
> ambari-server/src/test/python/host_scripts/TestAlertDiskSpace.py 2608050
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b5b43d6
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py efe6038
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 4601092
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py daee726
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py ab08776
>
> Diff: https://reviews.apache.org/r/44835/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jayush Luniya
>
>
Re: Review Request 44835: AMBARI-15420: Refactor resource_management
library
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/#review123658
-----------------------------------------------------------
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py (line 42)
<https://reviews.apache.org/r/44835/#comment185882>
This should no longer be accessible outside of this module since the get_package_dirs below is now required in order to access this data.
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py (line 46)
<https://reviews.apache.org/r/44835/#comment185883>
I don't think these changes have been thought through all of the way. You're trying to abstract out the "stack root" of /usr/hdp, but you're still leaving "current" as a hard-coded item.
I fail to see in what context this is useful. The stack scripts all still use /usr/hdp, so why are we adding more complex logic to abstract it out.
Additionally, you're assuming that another stack will come along and use a different root but the exact same "current" convention. That's not very likely.
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py (lines 201 - 202)
<https://reviews.apache.org/r/44835/#comment185884>
Once again, why is this being done? This module is a wrapper around a specific tool; the conf-select tool.
Abstracting away the path is just odd since it's not changing.
If you're assuming that conf-select can be implemented by other stacks, then you're putting a lot of faith into a custom-conf-select working in basically the exact same way as the version provided in hdp.
ambari-common/src/main/python/resource_management/libraries/script/script.py (lines 75 - 87)
<https://reviews.apache.org/r/44835/#comment185885>
Why is the Script class controlling these now? That seems like coupling two areas which should not be coupled.
ambari-common/src/main/python/resource_management/libraries/script/script.py (line 374)
<https://reviews.apache.org/r/44835/#comment185886>
I don't really understand this abstraction. You're saying that the stack tools can be parameterized. But that all still depends on the custom stack tools fucntioning in 100% the exact same way as hdp-select and conf-select.
- Jonathan Hurley
On March 15, 2016, 2:32 a.m., Jayush Luniya wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44835/
> -----------------------------------------------------------
>
> (Updated March 15, 2016, 2:32 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-15420
> https://issues.apache.org/jira/browse/AMBARI-15420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Refactor resource_management library, to generalize the library and remove HDP-specific hardcodings
>
> Specifically,
> 1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding hdp-select and conf-select
> 2. Make stack-root stack-driven instead of hardcoding /usr/hdp
> 3. Make copy_tarball mappings stack-driven
> 4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of hardcoding the "/usr/hdp"
>
> In addition, also added a feature in the stack processing engine to support properties values to be defined a external property file (See tarball_map.json, stack_tools.json in patch)
>
> Three config properties are added
> 1. cluster-env/stack_root
> 2. cluster-env/stack_tools
> 3. cluster-env/tarball_map
> Corresponding helper functions get_stack_root(), get_stack_tool(), get_tarball_map() are added in script.py, which will set the defaults if these config properties are not defined (ambari-server upgrade scenario needs to be addressed to add these config properties on upgrade). These helper functions are used to remove hardcodings in resource_management library.
>
>
> Remaining HDP-specific logic in resource_management library
>
> 1. conf_select::_valid()
> 2. conf_select::get_hadoop_conf_dir()
> 3. list_ambari_managed_repos::repository_names
> 4. version_select_util::get_component_version()
> 5. script::get_stack_version()
> 6. script::should_expose_component_version()
> 7. get_lzo_packages::get_lzo_packages()
>
> Refactoring this HDP-specific logic would require "Stack Featurization" (AMBARI-13364) to be in place.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py ee7db0a
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 59c717b
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 647b8b6
> ambari-common/src/main/python/resource_management/libraries/functions/default.py 23383dc
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py a20b03c
> ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py f2e6567
> ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py f1cd9cb
> ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py d585dea
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py 4d9d8a4
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py 2ccc0c6
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py c94d956
> ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 95c5cba
> ambari-common/src/main/python/resource_management/libraries/script/script.py a8098a0
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 4be4049
> ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java 7f21aaa
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 0c7faea
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java c739211
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 7d934bb
> ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java e7c9c27
> ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java 3f7f756
> ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java PRE-CREATION
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py 12ca388
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py 45dbb24
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py d000846
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py 2c06c40
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py 535c821
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 85e7012
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py c3d15e5
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py c5e61e6
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py d0924b9
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 30b9a41
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 81a227e
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py 030fb2d
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py 27e2766
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/status_params.py 954bb80
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py e5b54cd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 906b198
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py ec37243
> ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/status_params.py 984a4ba
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py eb80ad6
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_windows.py dd732f5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 8b2aec5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py e05ed60
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 08bdcc3
> ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py e69a2e4
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py b8bf176
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 96dc104
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py 07faae4
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/tarball_map.json PRE-CREATION
> ambari-server/src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java 92c1200
> ambari-server/src/test/java/org/apache/ambari/server/utils/TestJsonUtils.java PRE-CREATION
> ambari-server/src/test/python/custom_actions/test_ru_set_all.py 3090f6b
> ambari-server/src/test/python/host_scripts/TestAlertDiskSpace.py 2608050
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b5b43d6
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py efe6038
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 4601092
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py daee726
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py ab08776
>
> Diff: https://reviews.apache.org/r/44835/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jayush Luniya
>
>
Re: Review Request 44835: AMBARI-15420: Refactor resource_management
library
Posted by Jayush Luniya <jl...@hortonworks.com>.
> On March 15, 2016, 6:54 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java, line 35
> > <https://reviews.apache.org/r/44835/diff/1/?file=1299216#file1299216line35>
> >
> > Constructing a new JsonParser is heavy; We should inject our singleton gson instance.
Made jsonparser static
- Jayush
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/#review123719
-----------------------------------------------------------
On March 18, 2016, 7:26 p.m., Jayush Luniya wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44835/
> -----------------------------------------------------------
>
> (Updated March 18, 2016, 7:26 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-15420
> https://issues.apache.org/jira/browse/AMBARI-15420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Refactor resource_management library, to generalize the library and remove HDP-specific hardcodings
>
> Specifically,
> 1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding hdp-select and conf-select
> 2. Make stack-root stack-driven instead of hardcoding /usr/hdp
> 3. Make copy_tarball mappings stack-driven
> 4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of hardcoding the "/usr/hdp"
>
> In addition, also added a feature in the stack processing engine to support properties values to be defined a external property file (See tarball_map.json, stack_tools.json in patch)
>
> Three config properties are added
> 1. cluster-env/stack_root
> 2. cluster-env/stack_tools
> 3. cluster-env/tarball_map
> Corresponding helper functions get_stack_root(), get_stack_tool(), get_tarball_map() are added in script.py, which will set the defaults if these config properties are not defined (ambari-server upgrade scenario needs to be addressed to add these config properties on upgrade). These helper functions are used to remove hardcodings in resource_management library.
>
>
> Remaining HDP-specific logic in resource_management library
>
> 1. conf_select::_valid()
> 2. conf_select::get_hadoop_conf_dir()
> 3. list_ambari_managed_repos::repository_names
> 4. version_select_util::get_component_version()
> 5. script::get_stack_version()
> 6. script::should_expose_component_version()
> 7. get_lzo_packages::get_lzo_packages()
>
> Refactoring this HDP-specific logic would require "Stack Featurization" (AMBARI-13364) to be in place.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py ee7db0a
> ambari-common/src/main/python/resource_management/core/utils.py 71d0008
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 770595f
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 647b8b6
> ambari-common/src/main/python/resource_management/libraries/functions/default.py 23383dc
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py a20b03c
> ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py f2e6567
> ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py f1cd9cb
> ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py d585dea
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py 4d9d8a4
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py 2ccc0c6
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 4a8eeb9
> ambari-common/src/main/python/resource_management/libraries/functions/stack_tools.py PRE-CREATION
> ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 95c5cba
> ambari-common/src/main/python/resource_management/libraries/script/script.py 1b0116e
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 4be4049
> ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java 7f21aaa
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 0c7faea
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java c739211
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 7d934bb
> ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java e7c9c27
> ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java 3f7f756
> ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java PRE-CREATION
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py 12ca388
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py 45dbb24
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py d000846
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py 2c06c40
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py 535c821
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 85e7012
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py c3d15e5
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py 215e012
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py 7aeac24
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 30b9a41
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 81a227e
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py 030fb2d
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py 27e2766
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/status_params.py 954bb80
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py e5b54cd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 906b198
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py e110c0c
> ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/status_params.py 984a4ba
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py eb80ad6
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_windows.py dd732f5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py db4f32e
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py e05ed60
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py e0f3752
> ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py e69a2e4
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py b8bf176
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 6197d87
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 96dc104
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py 07faae4
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/tarball_map.json PRE-CREATION
> ambari-server/src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java 92c1200
> ambari-server/src/test/java/org/apache/ambari/server/utils/TestJsonUtils.java PRE-CREATION
> ambari-server/src/test/python/custom_actions/test_ru_set_all.py 3090f6b
> ambari-server/src/test/python/host_scripts/TestAlertDiskSpace.py 2608050
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b5b43d6
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py efe6038
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 4601092
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py 31dac56
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py ab08776
>
> Diff: https://reviews.apache.org/r/44835/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jayush Luniya
>
>
Re: Review Request 44835: AMBARI-15420: Refactor resource_management
library
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/#review123719
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java (line 35)
<https://reviews.apache.org/r/44835/#comment185965>
Constructing a new JsonParser is heavy; We should inject our singleton gson instance.
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py (line 23)
<https://reviews.apache.org/r/44835/#comment185966>
Can we get rid of the import *
- Jonathan Hurley
On March 15, 2016, 2:32 a.m., Jayush Luniya wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44835/
> -----------------------------------------------------------
>
> (Updated March 15, 2016, 2:32 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-15420
> https://issues.apache.org/jira/browse/AMBARI-15420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Refactor resource_management library, to generalize the library and remove HDP-specific hardcodings
>
> Specifically,
> 1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding hdp-select and conf-select
> 2. Make stack-root stack-driven instead of hardcoding /usr/hdp
> 3. Make copy_tarball mappings stack-driven
> 4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of hardcoding the "/usr/hdp"
>
> In addition, also added a feature in the stack processing engine to support properties values to be defined a external property file (See tarball_map.json, stack_tools.json in patch)
>
> Three config properties are added
> 1. cluster-env/stack_root
> 2. cluster-env/stack_tools
> 3. cluster-env/tarball_map
> Corresponding helper functions get_stack_root(), get_stack_tool(), get_tarball_map() are added in script.py, which will set the defaults if these config properties are not defined (ambari-server upgrade scenario needs to be addressed to add these config properties on upgrade). These helper functions are used to remove hardcodings in resource_management library.
>
>
> Remaining HDP-specific logic in resource_management library
>
> 1. conf_select::_valid()
> 2. conf_select::get_hadoop_conf_dir()
> 3. list_ambari_managed_repos::repository_names
> 4. version_select_util::get_component_version()
> 5. script::get_stack_version()
> 6. script::should_expose_component_version()
> 7. get_lzo_packages::get_lzo_packages()
>
> Refactoring this HDP-specific logic would require "Stack Featurization" (AMBARI-13364) to be in place.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py ee7db0a
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 59c717b
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 647b8b6
> ambari-common/src/main/python/resource_management/libraries/functions/default.py 23383dc
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py a20b03c
> ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py f2e6567
> ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py f1cd9cb
> ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py d585dea
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py 4d9d8a4
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py 2ccc0c6
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py c94d956
> ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 95c5cba
> ambari-common/src/main/python/resource_management/libraries/script/script.py a8098a0
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 4be4049
> ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java 7f21aaa
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 0c7faea
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java c739211
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 7d934bb
> ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java e7c9c27
> ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java 3f7f756
> ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java PRE-CREATION
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py 12ca388
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py 45dbb24
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py d000846
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py 2c06c40
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py 535c821
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 85e7012
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py c3d15e5
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py c5e61e6
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py d0924b9
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 30b9a41
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 81a227e
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py 030fb2d
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py 27e2766
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/status_params.py 954bb80
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py e5b54cd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 906b198
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py ec37243
> ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/status_params.py 984a4ba
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py eb80ad6
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_windows.py dd732f5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 8b2aec5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py e05ed60
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 08bdcc3
> ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py e69a2e4
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py b8bf176
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 96dc104
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py 07faae4
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/tarball_map.json PRE-CREATION
> ambari-server/src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java 92c1200
> ambari-server/src/test/java/org/apache/ambari/server/utils/TestJsonUtils.java PRE-CREATION
> ambari-server/src/test/python/custom_actions/test_ru_set_all.py 3090f6b
> ambari-server/src/test/python/host_scripts/TestAlertDiskSpace.py 2608050
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b5b43d6
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py efe6038
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 4601092
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py daee726
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py ab08776
>
> Diff: https://reviews.apache.org/r/44835/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jayush Luniya
>
>
Re: Review Request 44835: AMBARI-15420: Refactor resource_management
library
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/#review124302
-----------------------------------------------------------
Ship it!
Ship It!
- Alejandro Fernandez
On March 18, 2016, 8:57 p.m., Jayush Luniya wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44835/
> -----------------------------------------------------------
>
> (Updated March 18, 2016, 8:57 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-15420
> https://issues.apache.org/jira/browse/AMBARI-15420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Refactor resource_management library, to generalize the library and remove HDP-specific hardcodings
>
> Specifically,
> 1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding hdp-select and conf-select
> 2. Make stack-root stack-driven instead of hardcoding /usr/hdp
> 3. Make copy_tarball mappings stack-driven
> 4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of hardcoding the "/usr/hdp"
>
> In addition, also added a feature in the stack processing engine to support properties values to be defined a external property file (See tarball_map.json, stack_tools.json in patch)
>
> Three config properties are added
> 1. cluster-env/stack_root
> 2. cluster-env/stack_tools
> 3. cluster-env/tarball_map
> Corresponding helper functions get_stack_root(), get_stack_tool(), get_tarball_map() are added in script.py, which will set the defaults if these config properties are not defined (ambari-server upgrade scenario needs to be addressed to add these config properties on upgrade). These helper functions are used to remove hardcodings in resource_management library.
>
>
> Remaining HDP-specific logic in resource_management library
>
> 1. conf_select::_valid()
> 2. conf_select::get_hadoop_conf_dir()
> 3. list_ambari_managed_repos::repository_names
> 4. version_select_util::get_component_version()
> 5. script::get_stack_version()
> 6. script::should_expose_component_version()
> 7. get_lzo_packages::get_lzo_packages()
>
> Refactoring this HDP-specific logic would require "Stack Featurization" (AMBARI-13364) to be in place.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py ee7db0a
> ambari-common/src/main/python/resource_management/core/utils.py 71d0008
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 770595f
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 647b8b6
> ambari-common/src/main/python/resource_management/libraries/functions/default.py 23383dc
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py a20b03c
> ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py f2e6567
> ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py f1cd9cb
> ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py d585dea
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py 4d9d8a4
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py 2ccc0c6
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 4a8eeb9
> ambari-common/src/main/python/resource_management/libraries/functions/stack_tools.py PRE-CREATION
> ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 95c5cba
> ambari-common/src/main/python/resource_management/libraries/script/script.py 1b0116e
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 4be4049
> ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java 7f21aaa
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 0c7faea
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java c739211
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 7d934bb
> ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java e7c9c27
> ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java 3f7f756
> ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java PRE-CREATION
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py 12ca388
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py 45dbb24
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py d000846
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py 2c06c40
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py 535c821
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 85e7012
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py c3d15e5
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py 215e012
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py 7aeac24
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 30b9a41
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 81a227e
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py 030fb2d
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py 27e2766
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/status_params.py 954bb80
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py e5b54cd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 906b198
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py e110c0c
> ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/status_params.py 984a4ba
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py eb80ad6
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_windows.py dd732f5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py db4f32e
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py e05ed60
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py e0f3752
> ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py e69a2e4
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py b8bf176
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 6197d87
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 96dc104
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py 07faae4
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/tarball_map.json PRE-CREATION
> ambari-server/src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java 92c1200
> ambari-server/src/test/java/org/apache/ambari/server/utils/TestJsonUtils.java PRE-CREATION
> ambari-server/src/test/python/custom_actions/test_ru_set_all.py 3090f6b
> ambari-server/src/test/python/host_scripts/TestAlertDiskSpace.py 2608050
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b5b43d6
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py efe6038
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 4601092
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py 31dac56
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py ab08776
>
> Diff: https://reviews.apache.org/r/44835/diff/
>
>
> Testing
> -------
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ....................................... SUCCESS [3.906s]
> [INFO] Apache Ambari Project POM ......................... SUCCESS [0.029s]
> [INFO] Ambari Web ........................................ SUCCESS [50.786s]
> [INFO] Ambari Views ...................................... SUCCESS [3.417s]
> [INFO] Ambari Admin View ................................. SUCCESS [6.596s]
> [INFO] ambari-metrics .................................... SUCCESS [0.215s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [3.073s]
> [INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [5.126s]
> [INFO] Ambari Metrics Flume Sink ......................... SUCCESS [3.080s]
> [INFO] Ambari Metrics Kafka Sink ......................... SUCCESS [4.413s]
> [INFO] Ambari Metrics Storm Sink ......................... SUCCESS [1.547s]
> [INFO] Ambari Metrics Collector .......................... SUCCESS [9:53.524s]
> [INFO] Ambari Metrics Monitor ............................ SUCCESS [1.886s]
> [INFO] Ambari Metrics Grafana ............................ SUCCESS [39.843s]
> [INFO] Ambari Metrics Assembly ........................... SUCCESS [3:25.123s]
> [INFO] Ambari Server ..................................... SUCCESS [1:07:30.219s]
> [INFO] Ambari Functional Tests ........................... SUCCESS [3.081s]
> [INFO] Ambari Agent ...................................... SUCCESS [14.506s]
> [INFO] Ambari Client ..................................... SUCCESS [0.063s]
> [INFO] Ambari Python Client .............................. SUCCESS [0.381s]
> [INFO] Ambari Groovy Client .............................. SUCCESS [13.112s]
> [INFO] Ambari Shell ...................................... SUCCESS [0.034s]
> [INFO] Ambari Python Shell ............................... SUCCESS [0.044s]
> [INFO] Ambari Groovy Shell ............................... SUCCESS [9.936s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 1:23:34.755s
> [INFO] Finished at: Fri Mar 18 13:49:52 PDT 2016
> [INFO] Final Memory: 103M/1003M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jayush Luniya
>
>
Re: Review Request 44835: AMBARI-15420: Refactor resource_management
library
Posted by Jayush Luniya <jl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/
-----------------------------------------------------------
(Updated March 18, 2016, 8:57 p.m.)
Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
Bugs: AMBARI-15420
https://issues.apache.org/jira/browse/AMBARI-15420
Repository: ambari
Description
-------
Refactor resource_management library, to generalize the library and remove HDP-specific hardcodings
Specifically,
1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding hdp-select and conf-select
2. Make stack-root stack-driven instead of hardcoding /usr/hdp
3. Make copy_tarball mappings stack-driven
4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of hardcoding the "/usr/hdp"
In addition, also added a feature in the stack processing engine to support properties values to be defined a external property file (See tarball_map.json, stack_tools.json in patch)
Three config properties are added
1. cluster-env/stack_root
2. cluster-env/stack_tools
3. cluster-env/tarball_map
Corresponding helper functions get_stack_root(), get_stack_tool(), get_tarball_map() are added in script.py, which will set the defaults if these config properties are not defined (ambari-server upgrade scenario needs to be addressed to add these config properties on upgrade). These helper functions are used to remove hardcodings in resource_management library.
Remaining HDP-specific logic in resource_management library
1. conf_select::_valid()
2. conf_select::get_hadoop_conf_dir()
3. list_ambari_managed_repos::repository_names
4. version_select_util::get_component_version()
5. script::get_stack_version()
6. script::should_expose_component_version()
7. get_lzo_packages::get_lzo_packages()
Refactoring this HDP-specific logic would require "Stack Featurization" (AMBARI-13364) to be in place.
Diffs
-----
ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py ee7db0a
ambari-common/src/main/python/resource_management/core/utils.py 71d0008
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 770595f
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 647b8b6
ambari-common/src/main/python/resource_management/libraries/functions/default.py 23383dc
ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py a20b03c
ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py f2e6567
ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py f1cd9cb
ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py d585dea
ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py 4d9d8a4
ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py 2ccc0c6
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 4a8eeb9
ambari-common/src/main/python/resource_management/libraries/functions/stack_tools.py PRE-CREATION
ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 95c5cba
ambari-common/src/main/python/resource_management/libraries/script/script.py 1b0116e
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 4be4049
ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java 7f21aaa
ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 0c7faea
ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java c739211
ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 7d934bb
ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java e7c9c27
ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java 3f7f756
ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java PRE-CREATION
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py 12ca388
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py 45dbb24
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py d000846
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py 2c06c40
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py 535c821
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 85e7012
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py c3d15e5
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py 215e012
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py 7aeac24
ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 30b9a41
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 81a227e
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py 030fb2d
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py 27e2766
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/status_params.py 954bb80
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py e5b54cd
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 906b198
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py e110c0c
ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/status_params.py 984a4ba
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py eb80ad6
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_windows.py dd732f5
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py db4f32e
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py e05ed60
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py e0f3752
ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py e69a2e4
ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py b8bf176
ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 6197d87
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 96dc104
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py 07faae4
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/tarball_map.json PRE-CREATION
ambari-server/src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java 92c1200
ambari-server/src/test/java/org/apache/ambari/server/utils/TestJsonUtils.java PRE-CREATION
ambari-server/src/test/python/custom_actions/test_ru_set_all.py 3090f6b
ambari-server/src/test/python/host_scripts/TestAlertDiskSpace.py 2608050
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b5b43d6
ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py efe6038
ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 4601092
ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py 31dac56
ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py ab08776
Diff: https://reviews.apache.org/r/44835/diff/
Testing (updated)
-------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ....................................... SUCCESS [3.906s]
[INFO] Apache Ambari Project POM ......................... SUCCESS [0.029s]
[INFO] Ambari Web ........................................ SUCCESS [50.786s]
[INFO] Ambari Views ...................................... SUCCESS [3.417s]
[INFO] Ambari Admin View ................................. SUCCESS [6.596s]
[INFO] ambari-metrics .................................... SUCCESS [0.215s]
[INFO] Ambari Metrics Common ............................. SUCCESS [3.073s]
[INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [5.126s]
[INFO] Ambari Metrics Flume Sink ......................... SUCCESS [3.080s]
[INFO] Ambari Metrics Kafka Sink ......................... SUCCESS [4.413s]
[INFO] Ambari Metrics Storm Sink ......................... SUCCESS [1.547s]
[INFO] Ambari Metrics Collector .......................... SUCCESS [9:53.524s]
[INFO] Ambari Metrics Monitor ............................ SUCCESS [1.886s]
[INFO] Ambari Metrics Grafana ............................ SUCCESS [39.843s]
[INFO] Ambari Metrics Assembly ........................... SUCCESS [3:25.123s]
[INFO] Ambari Server ..................................... SUCCESS [1:07:30.219s]
[INFO] Ambari Functional Tests ........................... SUCCESS [3.081s]
[INFO] Ambari Agent ...................................... SUCCESS [14.506s]
[INFO] Ambari Client ..................................... SUCCESS [0.063s]
[INFO] Ambari Python Client .............................. SUCCESS [0.381s]
[INFO] Ambari Groovy Client .............................. SUCCESS [13.112s]
[INFO] Ambari Shell ...................................... SUCCESS [0.034s]
[INFO] Ambari Python Shell ............................... SUCCESS [0.044s]
[INFO] Ambari Groovy Shell ............................... SUCCESS [9.936s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:23:34.755s
[INFO] Finished at: Fri Mar 18 13:49:52 PDT 2016
[INFO] Final Memory: 103M/1003M
[INFO] ------------------------------------------------------------------------
Thanks,
Jayush Luniya
Re: Review Request 44835: AMBARI-15420: Refactor resource_management
library
Posted by Jayush Luniya <jl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/
-----------------------------------------------------------
(Updated March 18, 2016, 7:26 p.m.)
Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
Bugs: AMBARI-15420
https://issues.apache.org/jira/browse/AMBARI-15420
Repository: ambari
Description
-------
Refactor resource_management library, to generalize the library and remove HDP-specific hardcodings
Specifically,
1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding hdp-select and conf-select
2. Make stack-root stack-driven instead of hardcoding /usr/hdp
3. Make copy_tarball mappings stack-driven
4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of hardcoding the "/usr/hdp"
In addition, also added a feature in the stack processing engine to support properties values to be defined a external property file (See tarball_map.json, stack_tools.json in patch)
Three config properties are added
1. cluster-env/stack_root
2. cluster-env/stack_tools
3. cluster-env/tarball_map
Corresponding helper functions get_stack_root(), get_stack_tool(), get_tarball_map() are added in script.py, which will set the defaults if these config properties are not defined (ambari-server upgrade scenario needs to be addressed to add these config properties on upgrade). These helper functions are used to remove hardcodings in resource_management library.
Remaining HDP-specific logic in resource_management library
1. conf_select::_valid()
2. conf_select::get_hadoop_conf_dir()
3. list_ambari_managed_repos::repository_names
4. version_select_util::get_component_version()
5. script::get_stack_version()
6. script::should_expose_component_version()
7. get_lzo_packages::get_lzo_packages()
Refactoring this HDP-specific logic would require "Stack Featurization" (AMBARI-13364) to be in place.
Diffs (updated)
-----
ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py ee7db0a
ambari-common/src/main/python/resource_management/core/utils.py 71d0008
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 770595f
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 647b8b6
ambari-common/src/main/python/resource_management/libraries/functions/default.py 23383dc
ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py a20b03c
ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py f2e6567
ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py f1cd9cb
ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py d585dea
ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py 4d9d8a4
ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py 2ccc0c6
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 4a8eeb9
ambari-common/src/main/python/resource_management/libraries/functions/stack_tools.py PRE-CREATION
ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 95c5cba
ambari-common/src/main/python/resource_management/libraries/script/script.py 1b0116e
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 4be4049
ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java 7f21aaa
ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 0c7faea
ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java c739211
ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 7d934bb
ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java e7c9c27
ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java 3f7f756
ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java PRE-CREATION
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py 12ca388
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py 45dbb24
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py d000846
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py 2c06c40
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py 535c821
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 85e7012
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py c3d15e5
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py 215e012
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py 7aeac24
ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 30b9a41
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 81a227e
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py 030fb2d
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py 27e2766
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/status_params.py 954bb80
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py e5b54cd
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 906b198
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py e110c0c
ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/status_params.py 984a4ba
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py eb80ad6
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_windows.py dd732f5
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py db4f32e
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py e05ed60
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py e0f3752
ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py e69a2e4
ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py b8bf176
ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 6197d87
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 96dc104
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py 07faae4
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/tarball_map.json PRE-CREATION
ambari-server/src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java 92c1200
ambari-server/src/test/java/org/apache/ambari/server/utils/TestJsonUtils.java PRE-CREATION
ambari-server/src/test/python/custom_actions/test_ru_set_all.py 3090f6b
ambari-server/src/test/python/host_scripts/TestAlertDiskSpace.py 2608050
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b5b43d6
ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py efe6038
ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 4601092
ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py 31dac56
ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py ab08776
Diff: https://reviews.apache.org/r/44835/diff/
Testing
-------
Thanks,
Jayush Luniya
Re: Review Request 44835: AMBARI-15420: Refactor resource_management
library
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/#review123879
-----------------------------------------------------------
Ship it!
Ship It!
- Nate Cole
On March 15, 2016, 4:23 p.m., Jayush Luniya wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44835/
> -----------------------------------------------------------
>
> (Updated March 15, 2016, 4:23 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-15420
> https://issues.apache.org/jira/browse/AMBARI-15420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Refactor resource_management library, to generalize the library and remove HDP-specific hardcodings
>
> Specifically,
> 1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding hdp-select and conf-select
> 2. Make stack-root stack-driven instead of hardcoding /usr/hdp
> 3. Make copy_tarball mappings stack-driven
> 4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of hardcoding the "/usr/hdp"
>
> In addition, also added a feature in the stack processing engine to support properties values to be defined a external property file (See tarball_map.json, stack_tools.json in patch)
>
> Three config properties are added
> 1. cluster-env/stack_root
> 2. cluster-env/stack_tools
> 3. cluster-env/tarball_map
> Corresponding helper functions get_stack_root(), get_stack_tool(), get_tarball_map() are added in script.py, which will set the defaults if these config properties are not defined (ambari-server upgrade scenario needs to be addressed to add these config properties on upgrade). These helper functions are used to remove hardcodings in resource_management library.
>
>
> Remaining HDP-specific logic in resource_management library
>
> 1. conf_select::_valid()
> 2. conf_select::get_hadoop_conf_dir()
> 3. list_ambari_managed_repos::repository_names
> 4. version_select_util::get_component_version()
> 5. script::get_stack_version()
> 6. script::should_expose_component_version()
> 7. get_lzo_packages::get_lzo_packages()
>
> Refactoring this HDP-specific logic would require "Stack Featurization" (AMBARI-13364) to be in place.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py ee7db0a
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 770595f
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 647b8b6
> ambari-common/src/main/python/resource_management/libraries/functions/default.py 23383dc
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py a20b03c
> ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py f2e6567
> ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py f1cd9cb
> ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py d585dea
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py 4d9d8a4
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py 2ccc0c6
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 4a8eeb9
> ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 95c5cba
> ambari-common/src/main/python/resource_management/libraries/script/script.py a8098a0
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 4be4049
> ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java 7f21aaa
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 0c7faea
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java c739211
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 7d934bb
> ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java e7c9c27
> ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java 3f7f756
> ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java PRE-CREATION
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py 12ca388
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py 45dbb24
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py d000846
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py 2c06c40
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py 535c821
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 85e7012
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py c3d15e5
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py 215e012
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py 7aeac24
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 30b9a41
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 81a227e
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py 030fb2d
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py 27e2766
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/status_params.py 954bb80
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py e5b54cd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 906b198
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py ec37243
> ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/status_params.py 984a4ba
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py eb80ad6
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_windows.py dd732f5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 8b2aec5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py e05ed60
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py e0f3752
> ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py e69a2e4
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py b8bf176
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 96dc104
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py 07faae4
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/tarball_map.json PRE-CREATION
> ambari-server/src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java 92c1200
> ambari-server/src/test/java/org/apache/ambari/server/utils/TestJsonUtils.java PRE-CREATION
> ambari-server/src/test/python/custom_actions/test_ru_set_all.py 3090f6b
> ambari-server/src/test/python/host_scripts/TestAlertDiskSpace.py 2608050
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b5b43d6
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py efe6038
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 4601092
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py 31dac56
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py ab08776
>
> Diff: https://reviews.apache.org/r/44835/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jayush Luniya
>
>
Re: Review Request 44835: AMBARI-15420: Refactor resource_management
library
Posted by Jayush Luniya <jl...@hortonworks.com>.
> On March 16, 2016, 5:53 p.m., Alejandro Fernandez wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py, line 106
> > <https://reviews.apache.org/r/44835/diff/3/?file=1300170#file1300170line106>
> >
> > This may not necessarily be a rolling upgrade.
fixed
> On March 16, 2016, 5:53 p.m., Alejandro Fernandez wrote:
> > ambari-common/src/main/python/resource_management/libraries/script/script.py, line 384
> > <https://reviews.apache.org/r/44835/diff/3/?file=1300180#file1300180line384>
> >
> > May want to ensure this is also a 3-tuple
padded the result to always return 3-element tuple
- Jayush
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/#review123892
-----------------------------------------------------------
On March 18, 2016, 7:26 p.m., Jayush Luniya wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44835/
> -----------------------------------------------------------
>
> (Updated March 18, 2016, 7:26 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-15420
> https://issues.apache.org/jira/browse/AMBARI-15420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Refactor resource_management library, to generalize the library and remove HDP-specific hardcodings
>
> Specifically,
> 1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding hdp-select and conf-select
> 2. Make stack-root stack-driven instead of hardcoding /usr/hdp
> 3. Make copy_tarball mappings stack-driven
> 4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of hardcoding the "/usr/hdp"
>
> In addition, also added a feature in the stack processing engine to support properties values to be defined a external property file (See tarball_map.json, stack_tools.json in patch)
>
> Three config properties are added
> 1. cluster-env/stack_root
> 2. cluster-env/stack_tools
> 3. cluster-env/tarball_map
> Corresponding helper functions get_stack_root(), get_stack_tool(), get_tarball_map() are added in script.py, which will set the defaults if these config properties are not defined (ambari-server upgrade scenario needs to be addressed to add these config properties on upgrade). These helper functions are used to remove hardcodings in resource_management library.
>
>
> Remaining HDP-specific logic in resource_management library
>
> 1. conf_select::_valid()
> 2. conf_select::get_hadoop_conf_dir()
> 3. list_ambari_managed_repos::repository_names
> 4. version_select_util::get_component_version()
> 5. script::get_stack_version()
> 6. script::should_expose_component_version()
> 7. get_lzo_packages::get_lzo_packages()
>
> Refactoring this HDP-specific logic would require "Stack Featurization" (AMBARI-13364) to be in place.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py ee7db0a
> ambari-common/src/main/python/resource_management/core/utils.py 71d0008
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 770595f
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 647b8b6
> ambari-common/src/main/python/resource_management/libraries/functions/default.py 23383dc
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py a20b03c
> ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py f2e6567
> ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py f1cd9cb
> ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py d585dea
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py 4d9d8a4
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py 2ccc0c6
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 4a8eeb9
> ambari-common/src/main/python/resource_management/libraries/functions/stack_tools.py PRE-CREATION
> ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 95c5cba
> ambari-common/src/main/python/resource_management/libraries/script/script.py 1b0116e
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 4be4049
> ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java 7f21aaa
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 0c7faea
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java c739211
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 7d934bb
> ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java e7c9c27
> ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java 3f7f756
> ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java PRE-CREATION
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py 12ca388
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py 45dbb24
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py d000846
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py 2c06c40
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py 535c821
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 85e7012
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py c3d15e5
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py 215e012
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py 7aeac24
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 30b9a41
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 81a227e
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py 030fb2d
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py 27e2766
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/status_params.py 954bb80
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py e5b54cd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 906b198
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py e110c0c
> ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/status_params.py 984a4ba
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py eb80ad6
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_windows.py dd732f5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py db4f32e
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py e05ed60
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py e0f3752
> ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py e69a2e4
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py b8bf176
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 6197d87
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 96dc104
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py 07faae4
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/tarball_map.json PRE-CREATION
> ambari-server/src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java 92c1200
> ambari-server/src/test/java/org/apache/ambari/server/utils/TestJsonUtils.java PRE-CREATION
> ambari-server/src/test/python/custom_actions/test_ru_set_all.py 3090f6b
> ambari-server/src/test/python/host_scripts/TestAlertDiskSpace.py 2608050
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b5b43d6
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py efe6038
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 4601092
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py 31dac56
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py ab08776
>
> Diff: https://reviews.apache.org/r/44835/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jayush Luniya
>
>
Re: Review Request 44835: AMBARI-15420: Refactor resource_management
library
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/#review123892
-----------------------------------------------------------
Ship it!
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py (line 106)
<https://reviews.apache.org/r/44835/#comment186184>
This may not necessarily be a rolling upgrade.
ambari-common/src/main/python/resource_management/libraries/script/script.py (line 383)
<https://reviews.apache.org/r/44835/#comment186185>
May want to ensure this is also a 3-tuple
- Alejandro Fernandez
On March 15, 2016, 8:23 p.m., Jayush Luniya wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44835/
> -----------------------------------------------------------
>
> (Updated March 15, 2016, 8:23 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-15420
> https://issues.apache.org/jira/browse/AMBARI-15420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Refactor resource_management library, to generalize the library and remove HDP-specific hardcodings
>
> Specifically,
> 1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding hdp-select and conf-select
> 2. Make stack-root stack-driven instead of hardcoding /usr/hdp
> 3. Make copy_tarball mappings stack-driven
> 4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of hardcoding the "/usr/hdp"
>
> In addition, also added a feature in the stack processing engine to support properties values to be defined a external property file (See tarball_map.json, stack_tools.json in patch)
>
> Three config properties are added
> 1. cluster-env/stack_root
> 2. cluster-env/stack_tools
> 3. cluster-env/tarball_map
> Corresponding helper functions get_stack_root(), get_stack_tool(), get_tarball_map() are added in script.py, which will set the defaults if these config properties are not defined (ambari-server upgrade scenario needs to be addressed to add these config properties on upgrade). These helper functions are used to remove hardcodings in resource_management library.
>
>
> Remaining HDP-specific logic in resource_management library
>
> 1. conf_select::_valid()
> 2. conf_select::get_hadoop_conf_dir()
> 3. list_ambari_managed_repos::repository_names
> 4. version_select_util::get_component_version()
> 5. script::get_stack_version()
> 6. script::should_expose_component_version()
> 7. get_lzo_packages::get_lzo_packages()
>
> Refactoring this HDP-specific logic would require "Stack Featurization" (AMBARI-13364) to be in place.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py ee7db0a
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 770595f
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 647b8b6
> ambari-common/src/main/python/resource_management/libraries/functions/default.py 23383dc
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py a20b03c
> ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py f2e6567
> ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py f1cd9cb
> ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py d585dea
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py 4d9d8a4
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py 2ccc0c6
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 4a8eeb9
> ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 95c5cba
> ambari-common/src/main/python/resource_management/libraries/script/script.py a8098a0
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 4be4049
> ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java 7f21aaa
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 0c7faea
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java c739211
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 7d934bb
> ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java e7c9c27
> ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java 3f7f756
> ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java PRE-CREATION
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py 12ca388
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py 45dbb24
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py d000846
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py 2c06c40
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py 535c821
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 85e7012
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py c3d15e5
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py 215e012
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py 7aeac24
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 30b9a41
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 81a227e
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py 030fb2d
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py 27e2766
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/status_params.py 954bb80
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py e5b54cd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 906b198
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py ec37243
> ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/status_params.py 984a4ba
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py eb80ad6
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_windows.py dd732f5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 8b2aec5
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py e05ed60
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py e0f3752
> ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py e69a2e4
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py b8bf176
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 96dc104
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py 07faae4
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/tarball_map.json PRE-CREATION
> ambari-server/src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java 92c1200
> ambari-server/src/test/java/org/apache/ambari/server/utils/TestJsonUtils.java PRE-CREATION
> ambari-server/src/test/python/custom_actions/test_ru_set_all.py 3090f6b
> ambari-server/src/test/python/host_scripts/TestAlertDiskSpace.py 2608050
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b5b43d6
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py efe6038
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 4601092
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py 31dac56
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py ab08776
>
> Diff: https://reviews.apache.org/r/44835/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jayush Luniya
>
>
Re: Review Request 44835: AMBARI-15420: Refactor resource_management
library
Posted by Jayush Luniya <jl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/
-----------------------------------------------------------
(Updated March 15, 2016, 8:23 p.m.)
Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
Bugs: AMBARI-15420
https://issues.apache.org/jira/browse/AMBARI-15420
Repository: ambari
Description
-------
Refactor resource_management library, to generalize the library and remove HDP-specific hardcodings
Specifically,
1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding hdp-select and conf-select
2. Make stack-root stack-driven instead of hardcoding /usr/hdp
3. Make copy_tarball mappings stack-driven
4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of hardcoding the "/usr/hdp"
In addition, also added a feature in the stack processing engine to support properties values to be defined a external property file (See tarball_map.json, stack_tools.json in patch)
Three config properties are added
1. cluster-env/stack_root
2. cluster-env/stack_tools
3. cluster-env/tarball_map
Corresponding helper functions get_stack_root(), get_stack_tool(), get_tarball_map() are added in script.py, which will set the defaults if these config properties are not defined (ambari-server upgrade scenario needs to be addressed to add these config properties on upgrade). These helper functions are used to remove hardcodings in resource_management library.
Remaining HDP-specific logic in resource_management library
1. conf_select::_valid()
2. conf_select::get_hadoop_conf_dir()
3. list_ambari_managed_repos::repository_names
4. version_select_util::get_component_version()
5. script::get_stack_version()
6. script::should_expose_component_version()
7. get_lzo_packages::get_lzo_packages()
Refactoring this HDP-specific logic would require "Stack Featurization" (AMBARI-13364) to be in place.
Diffs (updated)
-----
ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py ee7db0a
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 770595f
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 647b8b6
ambari-common/src/main/python/resource_management/libraries/functions/default.py 23383dc
ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py a20b03c
ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py f2e6567
ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py f1cd9cb
ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py d585dea
ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py 4d9d8a4
ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py 2ccc0c6
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 4a8eeb9
ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 95c5cba
ambari-common/src/main/python/resource_management/libraries/script/script.py a8098a0
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 4be4049
ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java 7f21aaa
ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 0c7faea
ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java c739211
ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 7d934bb
ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java e7c9c27
ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java 3f7f756
ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java PRE-CREATION
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py 12ca388
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py 45dbb24
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py d000846
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py 2c06c40
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py 535c821
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 85e7012
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py c3d15e5
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py 215e012
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py 7aeac24
ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 30b9a41
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 81a227e
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py 030fb2d
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py 27e2766
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/status_params.py 954bb80
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py e5b54cd
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 906b198
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py ec37243
ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/status_params.py 984a4ba
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py eb80ad6
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_windows.py dd732f5
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 8b2aec5
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py e05ed60
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py e0f3752
ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py e69a2e4
ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py b8bf176
ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 96dc104
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py 07faae4
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/tarball_map.json PRE-CREATION
ambari-server/src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java 92c1200
ambari-server/src/test/java/org/apache/ambari/server/utils/TestJsonUtils.java PRE-CREATION
ambari-server/src/test/python/custom_actions/test_ru_set_all.py 3090f6b
ambari-server/src/test/python/host_scripts/TestAlertDiskSpace.py 2608050
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b5b43d6
ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py efe6038
ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 4601092
ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py 31dac56
ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py ab08776
Diff: https://reviews.apache.org/r/44835/diff/
Testing
-------
Thanks,
Jayush Luniya
Re: Review Request 44835: AMBARI-15420: Refactor resource_management
library
Posted by Jayush Luniya <jl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/
-----------------------------------------------------------
(Updated March 15, 2016, 8:17 p.m.)
Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
Changes
-------
Revised patch against latest in trunk
Bugs: AMBARI-15420
https://issues.apache.org/jira/browse/AMBARI-15420
Repository: ambari
Description
-------
Refactor resource_management library, to generalize the library and remove HDP-specific hardcodings
Specifically,
1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding hdp-select and conf-select
2. Make stack-root stack-driven instead of hardcoding /usr/hdp
3. Make copy_tarball mappings stack-driven
4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of hardcoding the "/usr/hdp"
In addition, also added a feature in the stack processing engine to support properties values to be defined a external property file (See tarball_map.json, stack_tools.json in patch)
Three config properties are added
1. cluster-env/stack_root
2. cluster-env/stack_tools
3. cluster-env/tarball_map
Corresponding helper functions get_stack_root(), get_stack_tool(), get_tarball_map() are added in script.py, which will set the defaults if these config properties are not defined (ambari-server upgrade scenario needs to be addressed to add these config properties on upgrade). These helper functions are used to remove hardcodings in resource_management library.
Remaining HDP-specific logic in resource_management library
1. conf_select::_valid()
2. conf_select::get_hadoop_conf_dir()
3. list_ambari_managed_repos::repository_names
4. version_select_util::get_component_version()
5. script::get_stack_version()
6. script::should_expose_component_version()
7. get_lzo_packages::get_lzo_packages()
Refactoring this HDP-specific logic would require "Stack Featurization" (AMBARI-13364) to be in place.
Diffs (updated)
-----
ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py ee7db0a
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 770595f
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 647b8b6
ambari-common/src/main/python/resource_management/libraries/functions/default.py 23383dc
ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py a20b03c
ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py f2e6567
ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py f1cd9cb
ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py d585dea
ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py 4d9d8a4
ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py 2ccc0c6
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 4a8eeb9
ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 95c5cba
ambari-common/src/main/python/resource_management/libraries/script/script.py a8098a0
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 4be4049
ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java 7f21aaa
ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 0c7faea
ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java c739211
ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 7d934bb
ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java e7c9c27
ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java 3f7f756
ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java PRE-CREATION
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py 12ca388
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py 45dbb24
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py d000846
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py 2c06c40
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py 535c821
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 85e7012
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py c3d15e5
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py 215e012
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py 7aeac24
ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 30b9a41
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 81a227e
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py 030fb2d
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py 27e2766
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/status_params.py 954bb80
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py e5b54cd
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 906b198
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py ec37243
ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/status_params.py 984a4ba
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py eb80ad6
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_windows.py dd732f5
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 8b2aec5
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py e05ed60
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py e0f3752
ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py e69a2e4
ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py b8bf176
ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 96dc104
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py 07faae4
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/tarball_map.json PRE-CREATION
ambari-server/src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java 92c1200
ambari-server/src/test/java/org/apache/ambari/server/utils/TestJsonUtils.java PRE-CREATION
ambari-server/src/test/python/custom_actions/test_ru_set_all.py 3090f6b
ambari-server/src/test/python/host_scripts/TestAlertDiskSpace.py 2608050
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b5b43d6
ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py efe6038
ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 4601092
ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py 31dac56
ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py ab08776
Diff: https://reviews.apache.org/r/44835/diff/
Testing
-------
Thanks,
Jayush Luniya
Re: Review Request 44835: AMBARI-15420: Refactor resource_management
library
Posted by Jayush Luniya <jl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/
-----------------------------------------------------------
(Updated March 15, 2016, 6:32 a.m.)
Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
Bugs: AMBARI-15420
https://issues.apache.org/jira/browse/AMBARI-15420
Repository: ambari
Description (updated)
-------
Refactor resource_management library, to generalize the library and remove HDP-specific hardcodings
Specifically,
1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding hdp-select and conf-select
2. Make stack-root stack-driven instead of hardcoding /usr/hdp
3. Make copy_tarball mappings stack-driven
4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of hardcoding the "/usr/hdp"
In addition, also added a feature in the stack processing engine to support properties values to be defined a external property file (See tarball_map.json, stack_tools.json in patch)
Three config properties are added
1. cluster-env/stack_root
2. cluster-env/stack_tools
3. cluster-env/tarball_map
Corresponding helper functions get_stack_root(), get_stack_tool(), get_tarball_map() are added in script.py, which will set the defaults if these config properties are not defined (ambari-server upgrade scenario needs to be addressed to add these config properties on upgrade). These helper functions are used to remove hardcodings in resource_management library.
Remaining HDP-specific logic in resource_management library
1. conf_select::_valid()
2. conf_select::get_hadoop_conf_dir()
3. list_ambari_managed_repos::repository_names
4. version_select_util::get_component_version()
5. script::get_stack_version()
6. script::should_expose_component_version()
7. get_lzo_packages::get_lzo_packages()
Refactoring this HDP-specific logic would require "Stack Featurization" (AMBARI-13364) to be in place.
Diffs
-----
ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py ee7db0a
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 59c717b
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 647b8b6
ambari-common/src/main/python/resource_management/libraries/functions/default.py 23383dc
ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py a20b03c
ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py f2e6567
ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py f1cd9cb
ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py d585dea
ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py 4d9d8a4
ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py 2ccc0c6
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py c94d956
ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 95c5cba
ambari-common/src/main/python/resource_management/libraries/script/script.py a8098a0
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 4be4049
ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java 7f21aaa
ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 0c7faea
ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java c739211
ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 7d934bb
ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java e7c9c27
ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java 3f7f756
ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java PRE-CREATION
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py 12ca388
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py 45dbb24
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py d000846
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py 2c06c40
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py 535c821
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 85e7012
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py c3d15e5
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py c5e61e6
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py d0924b9
ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 30b9a41
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 81a227e
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py 030fb2d
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py 27e2766
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/status_params.py 954bb80
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py e5b54cd
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 906b198
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py ec37243
ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/status_params.py 984a4ba
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py eb80ad6
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_windows.py dd732f5
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 8b2aec5
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py e05ed60
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 08bdcc3
ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py e69a2e4
ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py b8bf176
ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 96dc104
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py 07faae4
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/tarball_map.json PRE-CREATION
ambari-server/src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java 92c1200
ambari-server/src/test/java/org/apache/ambari/server/utils/TestJsonUtils.java PRE-CREATION
ambari-server/src/test/python/custom_actions/test_ru_set_all.py 3090f6b
ambari-server/src/test/python/host_scripts/TestAlertDiskSpace.py 2608050
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b5b43d6
ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py efe6038
ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 4601092
ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py daee726
ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py ab08776
Diff: https://reviews.apache.org/r/44835/diff/
Testing
-------
Thanks,
Jayush Luniya