You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@ambari.apache.org by Aravindan Vijayan <av...@hortonworks.com> on 2015/12/22 22:59:33 UTC

Review Request 41660: AMBARI-14445 Switch to cluster Zookeeper for AMS in distributed mode deployment

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41660/
-----------------------------------------------------------

Review request for Ambari, Dmytro Sen, Jaimin Jetly, Sumit Mohanty, and Sid Wagle.

Bugs: AMBARI-14445
https://issues.apache.org/jira/browse/AMBARI-14445

Repository: ambari

Description
-------

This patch involves changing of AMS in distributed mode to use cluster zookeeper instead of local zookeeper.

Summary of Changes in different areas.

AMS service side
1. Added new config amshbase.zookeeper.quorum in ams-site
2. Moved the zookeeper.znode.parent config item from ams-hbase-security-site to ams-hbase-site
3. Changed the default value of zookeeper.znode.parent in configs and Java code to /ams-hbase-unsecure to make sure there is no conflict with HBase service.
4. Remove -Dzookeeper.sasl.client.username={{zk_servicename}} in ams-env and ams-hbase-env content.

Stack Advisor
1. Recommend cluster zookeeper quorum to amshbase.zookeeper.quorum in ams-site.
2. Recommend and validate correct ports for hbase.zookeeper.property.clientPort (cluster zookeeper port for distributed and default 61181 for embedded)
3. Minor refactoring of util method in stack_advisor to make sure we have 1 method each for getZookeeperHost and getZookeeperHostsWithPort.

AMS start up scripts
1. Do not start hbase zookeeper in distributed mode.
2. hbase_zookeeper_quorum should resolve to cluster zookeeper quorum and localhost in distributed and embedded modes respectively.
3. If embedded mode, delete core-site.xml from collector and ams-hbase conf directories to make sure embedded mode works in secure cluster.

UI
1. Add amshbase.zookeeper.quorum in ams-site to the list of "watchers" in the config initialization to make sure cluster zookeeper quorum is captured in the config.
2. Add the same config in the UI warning messages when a Zookeeper server is being added / deleted.

Upgrade
1. Move zookeeper.znode.parent from ams-hbase-security-site to ams-hbase-site.
2. Add amshbase.zookeeper.quorum in ams-site.
3. Update ams-env and ams-hbase-env to remove -Dzookeeper.sasl.client.username={{zk_servicename}}
4. Fix minor bug in UpgradeCatalog220.

Security
1. Change in kerberos.json to move zookeeper.znode.parent config item from ams-hbase-security-site to ams-hbase-site.
2. Similar changes in ranger upgrade JSON files.

Others
1. Remove AMS alert that checks zookeeper state.

Diffs
-----

ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/DefaultPhoenixDataSource.java 562049b
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java e1d256d
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog220.java 85b142d
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog221.java a27a2b2
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/alerts.json 4015590
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-env.xml 96e2bb3
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-env.xml a061006
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-security-site.xml 5e7bc518
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-site.xml e97cfee
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml beb0fa9
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/kerberos.json dac60f3
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams_service.py 0726802
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/hbase.py 556380e
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py 87fd476
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py a723f75
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog220Test.java 896011a
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog221Test.java 7cf386e
ambari-server/src/test/python/stacks/2.0.6/AMBARI_METRICS/test_metrics_collector.py ab4d006
ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 1ee3e7a
ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json 61a7cdc
ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json bba7542
ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_2_1_3.json 3b4dff4
ambari-web/app/assets/data/stacks/HDP-2.1/service_components.json ca9ac3c
ambari-web/app/messages.js c0e1e16
ambari-web/app/utils/configs/add_component_config_initializer.js 6fc505c

Diff: https://reviews.apache.org/r/41660/diff/

Testing
-------

Manual Testing on embedded & distributed mode in secure & unsecure environments.

Ambari server and agent python unit tests pass.

Java Unit tests passed before last set of changes. Currently being run again.

Patch submitted through apache.

Added unit tests for config changes to UpgradeCatalog221.

Thanks,

Aravindan Vijayan

Re: Review Request 41660: AMBARI-14445 Switch to cluster Zookeeper for AMS in distributed mode deployment

Posted by Sid Wagle <sw...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41660/#review111691
-----------------------------------------------------------

Ship it!


Ship It!

- Sid Wagle


On Dec. 22, 2015, 11:48 p.m., Aravindan Vijayan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41660/
> -----------------------------------------------------------
> 
> (Updated Dec. 22, 2015, 11:48 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Jaimin Jetly, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-14445
>     https://issues.apache.org/jira/browse/AMBARI-14445
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> This patch involves changing of AMS in distributed mode to use cluster zookeeper instead of local zookeeper.
> 
> Summary of Changes in different areas.
> 
> AMS service side
> 1. Added new config amshbase.zookeeper.quorum in ams-site
> 2. Moved the zookeeper.znode.parent config item from ams-hbase-security-site to ams-hbase-site
> 3. Changed the default value of zookeeper.znode.parent in configs and Java code to /ams-hbase-unsecure to make sure there is no conflict with HBase service.
> 4. Remove -Dzookeeper.sasl.client.username={{zk_servicename}} in ams-env and ams-hbase-env content.
> 
> Stack Advisor
> 1. Recommend cluster zookeeper quorum to amshbase.zookeeper.quorum in ams-site.
> 2. Recommend and validate correct ports for hbase.zookeeper.property.clientPort (cluster zookeeper port for distributed and default 61181 for embedded)
> 3. Minor refactoring of util method in stack_advisor to make sure we have 1 method each for getZookeeperHost and getZookeeperHostsWithPort.
> 
> AMS start up scripts
> 1. Do not start hbase zookeeper in distributed mode.
> 2. hbase_zookeeper_quorum should resolve to cluster zookeeper quorum and localhost in distributed and embedded modes respectively.
> 3. If embedded mode, delete core-site.xml from collector and ams-hbase conf directories to make sure embedded mode works in secure cluster.
> 
> UI
> 1. Add amshbase.zookeeper.quorum in ams-site to the list of "watchers" in the config initialization to make sure cluster zookeeper quorum is captured in the config.
> 2. Add the same config in the UI warning messages when a Zookeeper server is being added / deleted.
> 
> Upgrade
> 1. Move zookeeper.znode.parent from ams-hbase-security-site to ams-hbase-site.
> 2. Add amshbase.zookeeper.quorum in ams-site.
> 3. Update ams-env and ams-hbase-env to remove -Dzookeeper.sasl.client.username={{zk_servicename}}
> 4. Fix minor bug in UpgradeCatalog220.
> 
> Security
> 1. Change in kerberos.json to move zookeeper.znode.parent config item from ams-hbase-security-site to ams-hbase-site.
> 2. Similar changes in ranger upgrade JSON files.
> 
> Others
> 1. Remove AMS alert that checks zookeeper state.
> 
> 
> Diffs
> -----
> 
>   ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/DefaultPhoenixDataSource.java 562049b 
>   ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java e1d256d 
>   ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog220.java 85b142d 
>   ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog221.java a27a2b2 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/alerts.json 4015590 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-env.xml 96e2bb3 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-env.xml a061006 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-security-site.xml 5e7bc518 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-site.xml e97cfee 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/kerberos.json dac60f3 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams_service.py 0726802 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/hbase.py 556380e 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py 87fd476 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py a723f75 
>   ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog220Test.java 896011a 
>   ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog221Test.java 7cf386e 
>   ambari-server/src/test/python/stacks/2.0.6/AMBARI_METRICS/test_metrics_collector.py ab4d006 
>   ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 1ee3e7a 
>   ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json 61a7cdc 
>   ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json bba7542 
>   ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_2_1_3.json 3b4dff4 
>   ambari-web/app/assets/data/stacks/HDP-2.1/service_components.json ca9ac3c 
> 
> Diff: https://reviews.apache.org/r/41660/diff/
> 
> 
> Testing
> -------
> 
> Manual Testing on embedded & distributed mode in secure & unsecure environments.
> 
> Ambari server and agent python unit tests pass.
> 
> Java Unit tests passed before last set of changes. Currently being run again.
> 
> Patch submitted through apache.
> 
> Added unit tests for config changes to UpgradeCatalog221.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Re: Review Request 41660: AMBARI-14445 Switch to cluster Zookeeper for AMS in distributed mode deployment

Posted by Dmytro Sen <ds...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41660/#review111784
-----------------------------------------------------------

Ship it!


Ship It!

- Dmytro Sen


On Дек. 23, 2015, 12:53 д.п., Aravindan Vijayan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41660/
> -----------------------------------------------------------
> 
> (Updated Дек. 23, 2015, 12:53 д.п.)
> 
> 
> Review request for Ambari, Dmytro Sen, Jaimin Jetly, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-14445
>     https://issues.apache.org/jira/browse/AMBARI-14445
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> This patch involves changing of AMS in distributed mode to use cluster zookeeper instead of local zookeeper.
> 
> Summary of Changes in different areas.
> 
> AMS service side
> 1. Added new config amshbase.zookeeper.quorum in ams-site
> 2. Moved the zookeeper.znode.parent config item from ams-hbase-security-site to ams-hbase-site
> 3. Changed the default value of zookeeper.znode.parent in configs and Java code to /ams-hbase-unsecure to make sure there is no conflict with HBase service.
> 4. Remove -Dzookeeper.sasl.client.username={{zk_servicename}} in ams-env and ams-hbase-env content.
> 
> Stack Advisor
> 1. Recommend cluster zookeeper quorum to amshbase.zookeeper.quorum in ams-site.
> 2. Recommend and validate correct ports for hbase.zookeeper.property.clientPort (cluster zookeeper port for distributed and default 61181 for embedded)
> 3. Minor refactoring of util method in stack_advisor to make sure we have 1 method each for getZookeeperHost and getZookeeperHostsWithPort.
> 
> AMS start up scripts
> 1. Do not start hbase zookeeper in distributed mode.
> 2. hbase_zookeeper_quorum should resolve to cluster zookeeper quorum and localhost in distributed and embedded modes respectively.
> 3. If embedded mode, delete core-site.xml from collector and ams-hbase conf directories to make sure embedded mode works in secure cluster.
> 
> UI
> 1. Add amshbase.zookeeper.quorum in ams-site to the list of "watchers" in the config initialization to make sure cluster zookeeper quorum is captured in the config.
> 2. Add the same config in the UI warning messages when a Zookeeper server is being added / deleted.
> 
> Upgrade
> 1. Move zookeeper.znode.parent from ams-hbase-security-site to ams-hbase-site.
> 2. Add amshbase.zookeeper.quorum in ams-site.
> 3. Update ams-env and ams-hbase-env to remove -Dzookeeper.sasl.client.username={{zk_servicename}}
> 4. Fix minor bug in UpgradeCatalog220.
> 
> Security
> 1. Change in kerberos.json to move zookeeper.znode.parent config item from ams-hbase-security-site to ams-hbase-site.
> 2. Similar changes in ranger upgrade JSON files.
> 
> Others
> 1. Remove AMS alert that checks zookeeper state.
> 
> 
> Diffs
> -----
> 
>   ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/DefaultPhoenixDataSource.java 562049b 
>   ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java e1d256d 
>   ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog220.java 1e39143 
>   ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog221.java 21f601e 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/alerts.json 4015590 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-env.xml 96e2bb3 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-env.xml a061006 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-security-site.xml 5e7bc518 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-site.xml e97cfee 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/kerberos.json dac60f3 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams_service.py 0726802 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/hbase.py 556380e 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py 87fd476 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py a723f75 
>   ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog220Test.java f0b4501 
>   ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog221Test.java 5eb3c14 
>   ambari-server/src/test/python/stacks/2.0.6/AMBARI_METRICS/test_metrics_collector.py ab4d006 
>   ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 1ee3e7a 
>   ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json 61a7cdc 
>   ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json bba7542 
>   ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_2_1_3.json 3b4dff4 
>   ambari-web/app/assets/data/stacks/HDP-2.1/service_components.json ca9ac3c 
> 
> Diff: https://reviews.apache.org/r/41660/diff/
> 
> 
> Testing
> -------
> 
> Manual Testing on embedded & distributed mode in secure & unsecure environments.
> 
> Ambari server and agent python unit tests pass.
> 
> Java Unit tests passed before last set of changes. Currently being run again.
> 
> Patch submitted through apache.
> 
> Added unit tests for config changes to UpgradeCatalog221.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Re: Review Request 41660: AMBARI-14445 Switch to cluster Zookeeper for AMS in distributed mode deployment

Posted by Aravindan Vijayan <av...@hortonworks.com>.

(Updated Dec. 23, 2015, 12:53 a.m.)

Review request for Ambari, Dmytro Sen, Jaimin Jetly, Sumit Mohanty, and Sid Wagle.

Changes
-------

Fixed bug discovered through manual testing of changes.

Bugs: AMBARI-14445
https://issues.apache.org/jira/browse/AMBARI-14445

Repository: ambari

Description
-------

This patch involves changing of AMS in distributed mode to use cluster zookeeper instead of local zookeeper.

Summary of Changes in different areas.

Security
1. Change in kerberos.json to move zookeeper.znode.parent config item from ams-hbase-security-site to ams-hbase-site.
2. Similar changes in ranger upgrade JSON files.

Others
1. Remove AMS alert that checks zookeeper state.

Diffs (updated)
-----

ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/DefaultPhoenixDataSource.java 562049b
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java e1d256d
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog220.java 1e39143
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog221.java 21f601e
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/alerts.json 4015590
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-env.xml 96e2bb3
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-env.xml a061006
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-security-site.xml 5e7bc518
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-site.xml e97cfee
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/kerberos.json dac60f3
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams_service.py 0726802
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/hbase.py 556380e
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py 87fd476
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py a723f75
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog220Test.java f0b4501
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog221Test.java 5eb3c14
ambari-server/src/test/python/stacks/2.0.6/AMBARI_METRICS/test_metrics_collector.py ab4d006
ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 1ee3e7a
ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json 61a7cdc
ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json bba7542
ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_2_1_3.json 3b4dff4
ambari-web/app/assets/data/stacks/HDP-2.1/service_components.json ca9ac3c

Diff: https://reviews.apache.org/r/41660/diff/

Testing
-------

Manual Testing on embedded & distributed mode in secure & unsecure environments.

Ambari server and agent python unit tests pass.

Java Unit tests passed before last set of changes. Currently being run again.

Patch submitted through apache.

Added unit tests for config changes to UpgradeCatalog221.

Thanks,

Aravindan Vijayan

Re: Review Request 41660: AMBARI-14445 Switch to cluster Zookeeper for AMS in distributed mode deployment

Posted by Aravindan Vijayan <av...@hortonworks.com>.

(Updated Dec. 22, 2015, 11:48 p.m.)

Review request for Ambari, Dmytro Sen, Jaimin Jetly, Sumit Mohanty, and Sid Wagle.

Changes
-------

Fixed issues.

Bugs: AMBARI-14445
https://issues.apache.org/jira/browse/AMBARI-14445

Repository: ambari

Description
-------

This patch involves changing of AMS in distributed mode to use cluster zookeeper instead of local zookeeper.

Summary of Changes in different areas.

Security
1. Change in kerberos.json to move zookeeper.znode.parent config item from ams-hbase-security-site to ams-hbase-site.
2. Similar changes in ranger upgrade JSON files.

Others
1. Remove AMS alert that checks zookeeper state.

Diffs (updated)
-----

ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/DefaultPhoenixDataSource.java 562049b
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java e1d256d
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog220.java 85b142d
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog221.java a27a2b2
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/alerts.json 4015590
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-env.xml 96e2bb3
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-env.xml a061006
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-security-site.xml 5e7bc518
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-site.xml e97cfee
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/kerberos.json dac60f3
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams_service.py 0726802
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/hbase.py 556380e
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py 87fd476
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py a723f75
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog220Test.java 896011a
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog221Test.java 7cf386e
ambari-server/src/test/python/stacks/2.0.6/AMBARI_METRICS/test_metrics_collector.py ab4d006
ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 1ee3e7a
ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json 61a7cdc
ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json bba7542
ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_2_1_3.json 3b4dff4
ambari-web/app/assets/data/stacks/HDP-2.1/service_components.json ca9ac3c

Diff: https://reviews.apache.org/r/41660/diff/

Testing
-------

Manual Testing on embedded & distributed mode in secure & unsecure environments.

Ambari server and agent python unit tests pass.

Java Unit tests passed before last set of changes. Currently being run again.

Patch submitted through apache.

Added unit tests for config changes to UpgradeCatalog221.

Thanks,

Aravindan Vijayan

Re: Review Request 41660: AMBARI-14445 Switch to cluster Zookeeper for AMS in distributed mode deployment

Posted by Aravindan Vijayan <av...@hortonworks.com>.


> On Dec. 22, 2015, 10:30 p.m., Sid Wagle wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog221.java, line 241
> > <https://reviews.apache.org/r/41660/diff/1/?file=1174747#file1174747line241>
> >
> >     Lets check existing value and if it is defualt "/hbase", change it unconditionally to /ams-hbase-(security-classfier).
> >     
> >     However, this needs check of Security State.

Made the change


- Aravindan


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41660/#review111672
-----------------------------------------------------------


On Dec. 22, 2015, 11:48 p.m., Aravindan Vijayan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41660/
> -----------------------------------------------------------
> 
> (Updated Dec. 22, 2015, 11:48 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Jaimin Jetly, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-14445
>     https://issues.apache.org/jira/browse/AMBARI-14445
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> This patch involves changing of AMS in distributed mode to use cluster zookeeper instead of local zookeeper.
> 
> Summary of Changes in different areas.
> 
> AMS service side
> 1. Added new config amshbase.zookeeper.quorum in ams-site
> 2. Moved the zookeeper.znode.parent config item from ams-hbase-security-site to ams-hbase-site
> 3. Changed the default value of zookeeper.znode.parent in configs and Java code to /ams-hbase-unsecure to make sure there is no conflict with HBase service.
> 4. Remove -Dzookeeper.sasl.client.username={{zk_servicename}} in ams-env and ams-hbase-env content.
> 
> Stack Advisor
> 1. Recommend cluster zookeeper quorum to amshbase.zookeeper.quorum in ams-site.
> 2. Recommend and validate correct ports for hbase.zookeeper.property.clientPort (cluster zookeeper port for distributed and default 61181 for embedded)
> 3. Minor refactoring of util method in stack_advisor to make sure we have 1 method each for getZookeeperHost and getZookeeperHostsWithPort.
> 
> AMS start up scripts
> 1. Do not start hbase zookeeper in distributed mode.
> 2. hbase_zookeeper_quorum should resolve to cluster zookeeper quorum and localhost in distributed and embedded modes respectively.
> 3. If embedded mode, delete core-site.xml from collector and ams-hbase conf directories to make sure embedded mode works in secure cluster.
> 
> UI
> 1. Add amshbase.zookeeper.quorum in ams-site to the list of "watchers" in the config initialization to make sure cluster zookeeper quorum is captured in the config.
> 2. Add the same config in the UI warning messages when a Zookeeper server is being added / deleted.
> 
> Upgrade
> 1. Move zookeeper.znode.parent from ams-hbase-security-site to ams-hbase-site.
> 2. Add amshbase.zookeeper.quorum in ams-site.
> 3. Update ams-env and ams-hbase-env to remove -Dzookeeper.sasl.client.username={{zk_servicename}}
> 4. Fix minor bug in UpgradeCatalog220.
> 
> Security
> 1. Change in kerberos.json to move zookeeper.znode.parent config item from ams-hbase-security-site to ams-hbase-site.
> 2. Similar changes in ranger upgrade JSON files.
> 
> Others
> 1. Remove AMS alert that checks zookeeper state.
> 
> 
> Diffs
> -----
> 
>   ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/DefaultPhoenixDataSource.java 562049b 
>   ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java e1d256d 
>   ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog220.java 85b142d 
>   ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog221.java a27a2b2 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/alerts.json 4015590 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-env.xml 96e2bb3 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-env.xml a061006 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-security-site.xml 5e7bc518 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-site.xml e97cfee 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/kerberos.json dac60f3 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams_service.py 0726802 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/hbase.py 556380e 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py 87fd476 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py a723f75 
>   ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog220Test.java 896011a 
>   ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog221Test.java 7cf386e 
>   ambari-server/src/test/python/stacks/2.0.6/AMBARI_METRICS/test_metrics_collector.py ab4d006 
>   ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 1ee3e7a 
>   ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json 61a7cdc 
>   ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json bba7542 
>   ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_2_1_3.json 3b4dff4 
>   ambari-web/app/assets/data/stacks/HDP-2.1/service_components.json ca9ac3c 
> 
> Diff: https://reviews.apache.org/r/41660/diff/
> 
> 
> Testing
> -------
> 
> Manual Testing on embedded & distributed mode in secure & unsecure environments.
> 
> Ambari server and agent python unit tests pass.
> 
> Java Unit tests passed before last set of changes. Currently being run again.
> 
> Patch submitted through apache.
> 
> Added unit tests for config changes to UpgradeCatalog221.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Re: Review Request 41660: AMBARI-14445 Switch to cluster Zookeeper for AMS in distributed mode deployment

Posted by Sid Wagle <sw...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41660/#review111672
-----------------------------------------------------------



ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog221.java (line 59)
<https://reviews.apache.org/r/41660/#comment171886>

    We should use the zookeeper quorum property asis from ams-hbase-site.



ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog221.java (line 205)
<https://reviews.apache.org/r/41660/#comment171887>

    This should not be required.



ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog221.java (line 241)
<https://reviews.apache.org/r/41660/#comment171891>

    Lets check existing value and if it is defualt "/hbase", change it unconditionally to /ams-hbase-(security-classfier).
    
    However, this needs check of Security State.



ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml (line 530)
<https://reviews.apache.org/r/41660/#comment171892>

    Delete.



ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams_service.py (line 61)
<https://reviews.apache.org/r/41660/#comment171894>

    We should do the same for hdfs-site.



ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py (line 150)
<https://reviews.apache.org/r/41660/#comment171895>

    Change.



ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py (line 459)
<https://reviews.apache.org/r/41660/#comment171896>

    Two calls registered for "ams-site"



ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py (line 476)
<https://reviews.apache.org/r/41660/#comment171898>

    Change.


- Sid Wagle


On Dec. 22, 2015, 9:59 p.m., Aravindan Vijayan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41660/
> -----------------------------------------------------------
> 
> (Updated Dec. 22, 2015, 9:59 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Jaimin Jetly, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-14445
>     https://issues.apache.org/jira/browse/AMBARI-14445
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> This patch involves changing of AMS in distributed mode to use cluster zookeeper instead of local zookeeper.
> 
> Summary of Changes in different areas.
> 
> AMS service side
> 1. Added new config amshbase.zookeeper.quorum in ams-site
> 2. Moved the zookeeper.znode.parent config item from ams-hbase-security-site to ams-hbase-site
> 3. Changed the default value of zookeeper.znode.parent in configs and Java code to /ams-hbase-unsecure to make sure there is no conflict with HBase service.
> 4. Remove -Dzookeeper.sasl.client.username={{zk_servicename}} in ams-env and ams-hbase-env content.
> 
> Stack Advisor
> 1. Recommend cluster zookeeper quorum to amshbase.zookeeper.quorum in ams-site.
> 2. Recommend and validate correct ports for hbase.zookeeper.property.clientPort (cluster zookeeper port for distributed and default 61181 for embedded)
> 3. Minor refactoring of util method in stack_advisor to make sure we have 1 method each for getZookeeperHost and getZookeeperHostsWithPort.
> 
> AMS start up scripts
> 1. Do not start hbase zookeeper in distributed mode.
> 2. hbase_zookeeper_quorum should resolve to cluster zookeeper quorum and localhost in distributed and embedded modes respectively.
> 3. If embedded mode, delete core-site.xml from collector and ams-hbase conf directories to make sure embedded mode works in secure cluster.
> 
> UI
> 1. Add amshbase.zookeeper.quorum in ams-site to the list of "watchers" in the config initialization to make sure cluster zookeeper quorum is captured in the config.
> 2. Add the same config in the UI warning messages when a Zookeeper server is being added / deleted.
> 
> Upgrade
> 1. Move zookeeper.znode.parent from ams-hbase-security-site to ams-hbase-site.
> 2. Add amshbase.zookeeper.quorum in ams-site.
> 3. Update ams-env and ams-hbase-env to remove -Dzookeeper.sasl.client.username={{zk_servicename}}
> 4. Fix minor bug in UpgradeCatalog220.
> 
> Security
> 1. Change in kerberos.json to move zookeeper.znode.parent config item from ams-hbase-security-site to ams-hbase-site.
> 2. Similar changes in ranger upgrade JSON files.
> 
> Others
> 1. Remove AMS alert that checks zookeeper state.
> 
> 
> Diffs
> -----
> 
>   ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/DefaultPhoenixDataSource.java 562049b 
>   ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java e1d256d 
>   ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog220.java 85b142d 
>   ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog221.java a27a2b2 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/alerts.json 4015590 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-env.xml 96e2bb3 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-env.xml a061006 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-security-site.xml 5e7bc518 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-site.xml e97cfee 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml beb0fa9 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/kerberos.json dac60f3 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams_service.py 0726802 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/hbase.py 556380e 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py 87fd476 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py a723f75 
>   ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog220Test.java 896011a 
>   ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog221Test.java 7cf386e 
>   ambari-server/src/test/python/stacks/2.0.6/AMBARI_METRICS/test_metrics_collector.py ab4d006 
>   ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 1ee3e7a 
>   ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json 61a7cdc 
>   ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json bba7542 
>   ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_2_1_3.json 3b4dff4 
>   ambari-web/app/assets/data/stacks/HDP-2.1/service_components.json ca9ac3c 
>   ambari-web/app/messages.js c0e1e16 
>   ambari-web/app/utils/configs/add_component_config_initializer.js 6fc505c 
> 
> Diff: https://reviews.apache.org/r/41660/diff/
> 
> 
> Testing
> -------
> 
> Manual Testing on embedded & distributed mode in secure & unsecure environments.
> 
> Ambari server and agent python unit tests pass.
> 
> Java Unit tests passed before last set of changes. Currently being run again.
> 
> Patch submitted through apache.
> 
> Added unit tests for config changes to UpgradeCatalog221.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>