You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@ambari.apache.org by Dmitro Lisnichenko <dl...@hortonworks.com> on 2017/01/13 19:03:59 UTC

Review Request 55515: Perf: Create Rolling and Express Upgrade Packs

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55515/
-----------------------------------------------------------

Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Nate Cole, and Vitalyi Brodetskyi.


Bugs: AMBARI-18739
    https://issues.apache.org/jira/browse/AMBARI-18739


Repository: ambari


Description
-------

Also includes code for AMBARI-18738. Perf: Allow components to report their version using distro-select and change configs using conf-select

In order to run Rolling/Express Upgrade, will need to create the Rolling/NonRolling Upgrade Packs and potentially config packs if we decide to do it across major versions.

This means each service will need to support start commands with the upgrade_type option, have symlinks already in distro-select, etc.

At first, this can be a handful of services.
The goal is to simulate Rolling/Express Upgrade on the PERF stack.


Diffs
-----

  ambari-agent/src/main/python/ambari_agent/PythonExecutor.py d75fc0cef2 
  ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 513ceac8dd 
  ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 615a0cdcbc 
  ambari-common/src/main/python/resource_management/libraries/script/dummy.py 3dcece501d 
  ambari-server/src/main/resources/custom_actions/scripts/ru_execute_tasks.py 4dec16f372 
  ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py d70e87fb1e 
  ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/distro-select.py PRE-CREATION 
  ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
  ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
  ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
  ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
  ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
  ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
  ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
  ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
  ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/GRUMPY/metainfo.xml 15c5cfddd2 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HAPPY/metainfo.xml 53cc11b7eb 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/metainfo.xml f1f4b7d1ac 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/metainfo.xml 42edc8d493 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/datanode.py 6fc338b1da 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/journalnode.py 96be63060a 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/namenode.py c3488e8bcc 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/SLEEPY/metainfo.xml 5b4dbd4a2d 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/SNOW/metainfo.xml 21e685ad80 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/metainfo.xml bb2191b267 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/metainfo.xml 57667bb888 
  ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/nonrolling-upgrade-2.0.xml PRE-CREATION 
  ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/upgrade-2.0.xml PRE-CREATION 
  ambari-server/src/main/resources/stacks/PERF/install_packages.py PRE-CREATION 
  contrib/utils/perf/deploy-gce-perf-cluster.py 8ba7e42e0b 

Diff: https://reviews.apache.org/r/55515/diff/


Testing
-------

Tested on my local 3-node cluster (both upgrade and downgrade). 
Script for creating Perf cluster downloads Ambari code from Apache git, so did not manage to test on gce, but script should work too (added required code).

What works:
- deploy of Perf 1.0 cluster
- RU to 2.0 (upgrade and downgrade)
- EU to 2.0 (upgrade and downgrade)

Issues:
- Clean deploy of perf 2.0 stack (dummy component install scripts are hardcoded with 1.1.1.1 version)


Thanks,

Dmitro Lisnichenko

Re: Review Request 55515: Perf: Create Rolling and Express Upgrade Packs

Posted by Nate Cole <nc...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55515/#review162323
-----------------------------------------------------------


Ship it!




Ship It!

- Nate Cole


On Jan. 13, 2017, 2:03 p.m., Dmitro Lisnichenko wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55515/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2017, 2:03 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Nate Cole, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-18739
>     https://issues.apache.org/jira/browse/AMBARI-18739
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Also includes code for AMBARI-18738. Perf: Allow components to report their version using distro-select and change configs using conf-select
> 
> In order to run Rolling/Express Upgrade, will need to create the Rolling/NonRolling Upgrade Packs and potentially config packs if we decide to do it across major versions.
> 
> This means each service will need to support start commands with the upgrade_type option, have symlinks already in distro-select, etc.
> 
> At first, this can be a handful of services.
> The goal is to simulate Rolling/Express Upgrade on the PERF stack.
> 
> 
> Diffs
> -----
> 
>   ambari-agent/src/main/python/ambari_agent/PythonExecutor.py d75fc0cef2 
>   ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 513ceac8dd 
>   ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 615a0cdcbc 
>   ambari-common/src/main/python/resource_management/libraries/script/dummy.py 3dcece501d 
>   ambari-server/src/main/resources/custom_actions/scripts/ru_execute_tasks.py 4dec16f372 
>   ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py d70e87fb1e 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/distro-select.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
>   ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/GRUMPY/metainfo.xml 15c5cfddd2 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HAPPY/metainfo.xml 53cc11b7eb 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/metainfo.xml f1f4b7d1ac 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/metainfo.xml 42edc8d493 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/datanode.py 6fc338b1da 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/journalnode.py 96be63060a 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/namenode.py c3488e8bcc 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/SLEEPY/metainfo.xml 5b4dbd4a2d 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/SNOW/metainfo.xml 21e685ad80 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/metainfo.xml bb2191b267 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/metainfo.xml 57667bb888 
>   ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/nonrolling-upgrade-2.0.xml PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/upgrade-2.0.xml PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/install_packages.py PRE-CREATION 
>   contrib/utils/perf/deploy-gce-perf-cluster.py 8ba7e42e0b 
> 
> Diff: https://reviews.apache.org/r/55515/diff/
> 
> 
> Testing
> -------
> 
> Tested on my local 3-node cluster (both upgrade and downgrade). 
> Script for creating Perf cluster downloads Ambari code from Apache git, so did not manage to test on gce, but script should work too (added required code).
> 
> What works:
> - deploy of Perf 1.0 cluster
> - RU to 2.0 (upgrade and downgrade)
> - EU to 2.0 (upgrade and downgrade)
> 
> Issues:
> - Clean deploy of perf 2.0 stack (dummy component install scripts are hardcoded with 1.1.1.1 version)
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 55515: Perf: Create Rolling and Express Upgrade Packs

Posted by Nate Cole <nc...@hortonworks.com>.


> On Jan. 16, 2017, 8:03 a.m., Jonathan Hurley wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py, line 132
> > <https://reviews.apache.org/r/55515/diff/1/?file=1604991#file1604991line132>
> >
> >     This seems like it could change the behavior of all upgrades - what are the implications of passing in the `os.environ` to Execute/shell?

Agreed - the custom script also has access to os.environ, so it seems like this is just adding more work when the perf stack can check it anyway.


- Nate


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55515/#review161725
-----------------------------------------------------------


On Jan. 13, 2017, 2:03 p.m., Dmitro Lisnichenko wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55515/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2017, 2:03 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Nate Cole, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-18739
>     https://issues.apache.org/jira/browse/AMBARI-18739
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Also includes code for AMBARI-18738. Perf: Allow components to report their version using distro-select and change configs using conf-select
> 
> In order to run Rolling/Express Upgrade, will need to create the Rolling/NonRolling Upgrade Packs and potentially config packs if we decide to do it across major versions.
> 
> This means each service will need to support start commands with the upgrade_type option, have symlinks already in distro-select, etc.
> 
> At first, this can be a handful of services.
> The goal is to simulate Rolling/Express Upgrade on the PERF stack.
> 
> 
> Diffs
> -----
> 
>   ambari-agent/src/main/python/ambari_agent/PythonExecutor.py d75fc0cef2 
>   ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 513ceac8dd 
>   ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 615a0cdcbc 
>   ambari-common/src/main/python/resource_management/libraries/script/dummy.py 3dcece501d 
>   ambari-server/src/main/resources/custom_actions/scripts/ru_execute_tasks.py 4dec16f372 
>   ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py d70e87fb1e 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/distro-select.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
>   ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/GRUMPY/metainfo.xml 15c5cfddd2 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HAPPY/metainfo.xml 53cc11b7eb 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/metainfo.xml f1f4b7d1ac 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/metainfo.xml 42edc8d493 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/datanode.py 6fc338b1da 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/journalnode.py 96be63060a 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/namenode.py c3488e8bcc 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/SLEEPY/metainfo.xml 5b4dbd4a2d 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/SNOW/metainfo.xml 21e685ad80 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/metainfo.xml bb2191b267 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/metainfo.xml 57667bb888 
>   ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/nonrolling-upgrade-2.0.xml PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/upgrade-2.0.xml PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/install_packages.py PRE-CREATION 
>   contrib/utils/perf/deploy-gce-perf-cluster.py 8ba7e42e0b 
> 
> Diff: https://reviews.apache.org/r/55515/diff/
> 
> 
> Testing
> -------
> 
> Tested on my local 3-node cluster (both upgrade and downgrade). 
> Script for creating Perf cluster downloads Ambari code from Apache git, so did not manage to test on gce, but script should work too (added required code).
> 
> What works:
> - deploy of Perf 1.0 cluster
> - RU to 2.0 (upgrade and downgrade)
> - EU to 2.0 (upgrade and downgrade)
> 
> Issues:
> - Clean deploy of perf 2.0 stack (dummy component install scripts are hardcoded with 1.1.1.1 version)
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 55515: Perf: Create Rolling and Express Upgrade Packs

Posted by Jonathan Hurley <jh...@hortonworks.com>.


> On Jan. 16, 2017, 8:03 a.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/metainfo.xml, line 29
> > <https://reviews.apache.org/r/55515/diff/1/?file=1605008#file1605008line29>
> >
> >     Any reason this 100% dummy stack would use actual hadoop names? It would cause ligitimate greps to return lots of "who cares" hits from this stack.
> 
> Dmitro Lisnichenko wrote:
>     I think Alejandro is planning to use PERF stack also for testing HA modes, because actual hadoop names were present at original PERF stack
> 
> Alejandro Fernandez wrote:
>     I'm ok if a future patch changes the name by appending "FAKE", but I still think a grep would should up.
>     We actually want to borrow a lot of the metainfo and configs from actual services.

Why is that? The "PERF" stack could be all Disney names for all we care - the point is to demonstrate that:

- Ambari has no hard codings for specific services/components
- That Ambari can scale

None of that requires real metainfo - it's just confusing.


- Jonathan


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55515/#review161725
-----------------------------------------------------------


On Jan. 13, 2017, 2:03 p.m., Dmitro Lisnichenko wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55515/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2017, 2:03 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Nate Cole, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-18739
>     https://issues.apache.org/jira/browse/AMBARI-18739
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Also includes code for AMBARI-18738. Perf: Allow components to report their version using distro-select and change configs using conf-select
> 
> In order to run Rolling/Express Upgrade, will need to create the Rolling/NonRolling Upgrade Packs and potentially config packs if we decide to do it across major versions.
> 
> This means each service will need to support start commands with the upgrade_type option, have symlinks already in distro-select, etc.
> 
> At first, this can be a handful of services.
> The goal is to simulate Rolling/Express Upgrade on the PERF stack.
> 
> 
> Diffs
> -----
> 
>   ambari-agent/src/main/python/ambari_agent/PythonExecutor.py d75fc0cef2 
>   ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 513ceac8dd 
>   ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 615a0cdcbc 
>   ambari-common/src/main/python/resource_management/libraries/script/dummy.py 3dcece501d 
>   ambari-server/src/main/resources/custom_actions/scripts/ru_execute_tasks.py 4dec16f372 
>   ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py d70e87fb1e 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/distro-select.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
>   ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/GRUMPY/metainfo.xml 15c5cfddd2 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HAPPY/metainfo.xml 53cc11b7eb 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/metainfo.xml f1f4b7d1ac 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/metainfo.xml 42edc8d493 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/datanode.py 6fc338b1da 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/journalnode.py 96be63060a 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/namenode.py c3488e8bcc 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/SLEEPY/metainfo.xml 5b4dbd4a2d 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/SNOW/metainfo.xml 21e685ad80 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/metainfo.xml bb2191b267 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/metainfo.xml 57667bb888 
>   ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/nonrolling-upgrade-2.0.xml PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/upgrade-2.0.xml PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/install_packages.py PRE-CREATION 
>   contrib/utils/perf/deploy-gce-perf-cluster.py 8ba7e42e0b 
> 
> Diff: https://reviews.apache.org/r/55515/diff/
> 
> 
> Testing
> -------
> 
> Tested on my local 3-node cluster (both upgrade and downgrade). 
> Script for creating Perf cluster downloads Ambari code from Apache git, so did not manage to test on gce, but script should work too (added required code).
> 
> What works:
> - deploy of Perf 1.0 cluster
> - RU to 2.0 (upgrade and downgrade)
> - EU to 2.0 (upgrade and downgrade)
> 
> Issues:
> - Clean deploy of perf 2.0 stack (dummy component install scripts are hardcoded with 1.1.1.1 version)
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 55515: Perf: Create Rolling and Express Upgrade Packs

Posted by Alejandro Fernandez <af...@hortonworks.com>.


> On Jan. 16, 2017, 1:03 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/metainfo.xml, line 29
> > <https://reviews.apache.org/r/55515/diff/1/?file=1605008#file1605008line29>
> >
> >     Any reason this 100% dummy stack would use actual hadoop names? It would cause ligitimate greps to return lots of "who cares" hits from this stack.
> 
> Dmitro Lisnichenko wrote:
>     I think Alejandro is planning to use PERF stack also for testing HA modes, because actual hadoop names were present at original PERF stack

I'm ok if a future patch changes the name by appending "FAKE", but I still think a grep would should up.
We actually want to borrow a lot of the metainfo and configs from actual services.


- Alejandro


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55515/#review161725
-----------------------------------------------------------


On Jan. 13, 2017, 7:03 p.m., Dmitro Lisnichenko wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55515/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2017, 7:03 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Nate Cole, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-18739
>     https://issues.apache.org/jira/browse/AMBARI-18739
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Also includes code for AMBARI-18738. Perf: Allow components to report their version using distro-select and change configs using conf-select
> 
> In order to run Rolling/Express Upgrade, will need to create the Rolling/NonRolling Upgrade Packs and potentially config packs if we decide to do it across major versions.
> 
> This means each service will need to support start commands with the upgrade_type option, have symlinks already in distro-select, etc.
> 
> At first, this can be a handful of services.
> The goal is to simulate Rolling/Express Upgrade on the PERF stack.
> 
> 
> Diffs
> -----
> 
>   ambari-agent/src/main/python/ambari_agent/PythonExecutor.py d75fc0cef2 
>   ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 513ceac8dd 
>   ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 615a0cdcbc 
>   ambari-common/src/main/python/resource_management/libraries/script/dummy.py 3dcece501d 
>   ambari-server/src/main/resources/custom_actions/scripts/ru_execute_tasks.py 4dec16f372 
>   ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py d70e87fb1e 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/distro-select.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
>   ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/GRUMPY/metainfo.xml 15c5cfddd2 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HAPPY/metainfo.xml 53cc11b7eb 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/metainfo.xml f1f4b7d1ac 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/metainfo.xml 42edc8d493 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/datanode.py 6fc338b1da 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/journalnode.py 96be63060a 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/namenode.py c3488e8bcc 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/SLEEPY/metainfo.xml 5b4dbd4a2d 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/SNOW/metainfo.xml 21e685ad80 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/metainfo.xml bb2191b267 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/metainfo.xml 57667bb888 
>   ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/nonrolling-upgrade-2.0.xml PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/upgrade-2.0.xml PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/install_packages.py PRE-CREATION 
>   contrib/utils/perf/deploy-gce-perf-cluster.py 8ba7e42e0b 
> 
> Diff: https://reviews.apache.org/r/55515/diff/
> 
> 
> Testing
> -------
> 
> Tested on my local 3-node cluster (both upgrade and downgrade). 
> Script for creating Perf cluster downloads Ambari code from Apache git, so did not manage to test on gce, but script should work too (added required code).
> 
> What works:
> - deploy of Perf 1.0 cluster
> - RU to 2.0 (upgrade and downgrade)
> - EU to 2.0 (upgrade and downgrade)
> 
> Issues:
> - Clean deploy of perf 2.0 stack (dummy component install scripts are hardcoded with 1.1.1.1 version)
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 55515: Perf: Create Rolling and Express Upgrade Packs

Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.


> On Jan. 16, 2017, 3:03 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/metainfo.xml, line 29
> > <https://reviews.apache.org/r/55515/diff/1/?file=1605008#file1605008line29>
> >
> >     Any reason this 100% dummy stack would use actual hadoop names? It would cause ligitimate greps to return lots of "who cares" hits from this stack.

I think Alejandro is planning to use PERF stack also for testing HA modes, because actual hadoop names were present at original PERF stack


> On Jan. 16, 2017, 3:03 p.m., Jonathan Hurley wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py, line 132
> > <https://reviews.apache.org/r/55515/diff/1/?file=1604991#file1604991line132>
> >
> >     This seems like it could change the behavior of all upgrades - what are the implications of passing in the `os.environ` to Execute/shell?
> 
> Nate Cole wrote:
>     Agreed - the custom script also has access to os.environ, so it seems like this is just adding more work when the perf stack can check it anyway.

The issue with environment is that by default Execute starts scripts with clean (like a new login shell) environment. That's why it's not possible to extract our env variables from a perf script. We use AGENT_CACHE_DIR/AMBARI_AGENT_HOST_DIR at distr-select tool to determine which agent started this script. Distr-select output may be specific for every agent. Otherwise, I don't see how I can archieve that given that all agents are running in a single process space, and not in different docker containers.

We may consider whitelisting AGENT_CACHE_DIR/AMBARI_AGENT_HOST_DIR variables at Execute and other RMF utils, in order to get them always passed into a child environment. Does that sound like a good approach?


- Dmitro


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55515/#review161725
-----------------------------------------------------------


On Jan. 13, 2017, 9:03 p.m., Dmitro Lisnichenko wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55515/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2017, 9:03 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Nate Cole, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-18739
>     https://issues.apache.org/jira/browse/AMBARI-18739
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Also includes code for AMBARI-18738. Perf: Allow components to report their version using distro-select and change configs using conf-select
> 
> In order to run Rolling/Express Upgrade, will need to create the Rolling/NonRolling Upgrade Packs and potentially config packs if we decide to do it across major versions.
> 
> This means each service will need to support start commands with the upgrade_type option, have symlinks already in distro-select, etc.
> 
> At first, this can be a handful of services.
> The goal is to simulate Rolling/Express Upgrade on the PERF stack.
> 
> 
> Diffs
> -----
> 
>   ambari-agent/src/main/python/ambari_agent/PythonExecutor.py d75fc0cef2 
>   ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 513ceac8dd 
>   ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 615a0cdcbc 
>   ambari-common/src/main/python/resource_management/libraries/script/dummy.py 3dcece501d 
>   ambari-server/src/main/resources/custom_actions/scripts/ru_execute_tasks.py 4dec16f372 
>   ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py d70e87fb1e 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/distro-select.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
>   ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/GRUMPY/metainfo.xml 15c5cfddd2 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HAPPY/metainfo.xml 53cc11b7eb 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/metainfo.xml f1f4b7d1ac 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/metainfo.xml 42edc8d493 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/datanode.py 6fc338b1da 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/journalnode.py 96be63060a 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/namenode.py c3488e8bcc 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/SLEEPY/metainfo.xml 5b4dbd4a2d 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/SNOW/metainfo.xml 21e685ad80 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/metainfo.xml bb2191b267 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/metainfo.xml 57667bb888 
>   ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/nonrolling-upgrade-2.0.xml PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/upgrade-2.0.xml PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/install_packages.py PRE-CREATION 
>   contrib/utils/perf/deploy-gce-perf-cluster.py 8ba7e42e0b 
> 
> Diff: https://reviews.apache.org/r/55515/diff/
> 
> 
> Testing
> -------
> 
> Tested on my local 3-node cluster (both upgrade and downgrade). 
> Script for creating Perf cluster downloads Ambari code from Apache git, so did not manage to test on gce, but script should work too (added required code).
> 
> What works:
> - deploy of Perf 1.0 cluster
> - RU to 2.0 (upgrade and downgrade)
> - EU to 2.0 (upgrade and downgrade)
> 
> Issues:
> - Clean deploy of perf 2.0 stack (dummy component install scripts are hardcoded with 1.1.1.1 version)
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 55515: Perf: Create Rolling and Express Upgrade Packs

Posted by Jonathan Hurley <jh...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55515/#review161725
-----------------------------------------------------------




ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py (line 132)
<https://reviews.apache.org/r/55515/#comment233031>

    This seems like it could change the behavior of all upgrades - what are the implications of passing in the `os.environ` to Execute/shell?



ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/metainfo.xml (line 29)
<https://reviews.apache.org/r/55515/#comment233030>

    Any reason this 100% dummy stack would use actual hadoop names? It would cause ligitimate greps to return lots of "who cares" hits from this stack.


- Jonathan Hurley


On Jan. 13, 2017, 2:03 p.m., Dmitro Lisnichenko wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55515/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2017, 2:03 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Nate Cole, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-18739
>     https://issues.apache.org/jira/browse/AMBARI-18739
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Also includes code for AMBARI-18738. Perf: Allow components to report their version using distro-select and change configs using conf-select
> 
> In order to run Rolling/Express Upgrade, will need to create the Rolling/NonRolling Upgrade Packs and potentially config packs if we decide to do it across major versions.
> 
> This means each service will need to support start commands with the upgrade_type option, have symlinks already in distro-select, etc.
> 
> At first, this can be a handful of services.
> The goal is to simulate Rolling/Express Upgrade on the PERF stack.
> 
> 
> Diffs
> -----
> 
>   ambari-agent/src/main/python/ambari_agent/PythonExecutor.py d75fc0cef2 
>   ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 513ceac8dd 
>   ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 615a0cdcbc 
>   ambari-common/src/main/python/resource_management/libraries/script/dummy.py 3dcece501d 
>   ambari-server/src/main/resources/custom_actions/scripts/ru_execute_tasks.py 4dec16f372 
>   ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py d70e87fb1e 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/distro-select.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
>   ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/GRUMPY/metainfo.xml 15c5cfddd2 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HAPPY/metainfo.xml 53cc11b7eb 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/metainfo.xml f1f4b7d1ac 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/metainfo.xml 42edc8d493 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/datanode.py 6fc338b1da 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/journalnode.py 96be63060a 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/namenode.py c3488e8bcc 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/SLEEPY/metainfo.xml 5b4dbd4a2d 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/SNOW/metainfo.xml 21e685ad80 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/metainfo.xml bb2191b267 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/metainfo.xml 57667bb888 
>   ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/nonrolling-upgrade-2.0.xml PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/upgrade-2.0.xml PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/install_packages.py PRE-CREATION 
>   contrib/utils/perf/deploy-gce-perf-cluster.py 8ba7e42e0b 
> 
> Diff: https://reviews.apache.org/r/55515/diff/
> 
> 
> Testing
> -------
> 
> Tested on my local 3-node cluster (both upgrade and downgrade). 
> Script for creating Perf cluster downloads Ambari code from Apache git, so did not manage to test on gce, but script should work too (added required code).
> 
> What works:
> - deploy of Perf 1.0 cluster
> - RU to 2.0 (upgrade and downgrade)
> - EU to 2.0 (upgrade and downgrade)
> 
> Issues:
> - Clean deploy of perf 2.0 stack (dummy component install scripts are hardcoded with 1.1.1.1 version)
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 55515: Perf: Create Rolling and Express Upgrade Packs

Posted by Vitalyi Brodetskyi <vb...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55515/#review162405
-----------------------------------------------------------


Ship it!




Ship It!

- Vitalyi Brodetskyi


On \u0421\u0456\u0447. 19, 2017, 6:26 \u043f\u0456\u0441\u043b\u044f \u043f\u043e\u043b\u0443\u0434\u043d\u044f, Dmitro Lisnichenko wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55515/
> -----------------------------------------------------------
> 
> (Updated \u0421\u0456\u0447. 19, 2017, 6:26 \u043f\u0456\u0441\u043b\u044f \u043f\u043e\u043b\u0443\u0434\u043d\u044f)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Nate Cole, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-18739
>     https://issues.apache.org/jira/browse/AMBARI-18739
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Also includes code for AMBARI-18738. Perf: Allow components to report their version using distro-select and change configs using conf-select
> 
> In order to run Rolling/Express Upgrade, will need to create the Rolling/NonRolling Upgrade Packs and potentially config packs if we decide to do it across major versions.
> 
> This means each service will need to support start commands with the upgrade_type option, have symlinks already in distro-select, etc.
> 
> At first, this can be a handful of services.
> The goal is to simulate Rolling/Express Upgrade on the PERF stack.
> 
> 
> Diffs
> -----
> 
>   ambari-agent/src/main/python/ambari_agent/PythonExecutor.py d75fc0cef2 
>   ambari-common/src/main/python/resource_management/libraries/script/dummy.py 3dcece501d 
>   ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py d70e87fb1e 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/distro-select.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
>   ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/GRUMPY/kerberos.json 3bf5fbd589 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/GRUMPY/metainfo.xml 15c5cfddd2 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HAPPY/kerberos.json ae84eaff8e 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HAPPY/metainfo.xml 53cc11b7eb 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/alerts.json 24141e8389 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-alert-config.xml  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-env.xml d3dda94fac 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-log4j.xml 880e8837cf 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-logsearch-conf.xml 7c481f29cf 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-policy.xml 8500ee02de 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-site.xml 058facdfe9 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/ranger-hbase-audit.xml e518ea0b10 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/ranger-hbase-policymgr-ssl.xml  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/ranger-hbase-security.xml  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/kerberos.json 9ed40ef7ed 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/metainfo.xml f1f4b7d1ac 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/metrics.json bf1fa6e7d9 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/alerts/hbase_master_process.py  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/alerts/hbase_regionserver_process.py  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/scripts/hbase_client.py c4943fed3c 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/scripts/hbase_master.py 3761fcf03a 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/scripts/hbase_regionserver.py 101c36fe56 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/scripts/phoenix_queryserver.py 76a49d9105 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/scripts/service_check.py  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/quicklinks/quicklinks.json 5568122034 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/themes/theme.json 7a5732bdac 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/widgets.json ae47833059 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/alerts.json 34cea4c2ed 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/core-site.xml 8375be17a7 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hadoop-env.xml 3be63be416 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hadoop-metrics2.properties.xml  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hadoop-policy.xml  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hdfs-alert-config.xml  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hdfs-log4j.xml a4eacc247a 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hdfs-logsearch-conf.xml 6540c865a4 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hdfs-site.xml 995632f339 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/ranger-hdfs-audit.xml 3dc46b3aea 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/ranger-hdfs-plugin-properties.xml deede1c1ba 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/ranger-hdfs-policymgr-ssl.xml  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/ranger-hdfs-security.xml  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/ssl-client.xml  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/ssl-server.xml  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/kerberos.json 974a69cbdc 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/metainfo.xml 42edc8d493 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/metrics.json c66387d3a1 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/alerts/alert_checkpoint_time.py  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/alerts/alert_datanode_unmounted_data_dir.py  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/alerts/alert_nfs_gateway_process.py  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/alerts/alert_snamenode_process.py  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/alerts/alert_upgrade_finalized.py  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/datanode.py 6fc338b1da 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/hdfs_client.py 4280c6cc98 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/journalnode.py 96be63060a 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/namenode.py c3488e8bcc 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/nfsgateway.py b750522220 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/service_check.py  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/snamenode.py 91ce7da9de 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/zkfc_slave.py b4310725a3 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/quicklinks/quicklinks.json a4216e35ac 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/themes/theme.json 6f2b79741b 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/widgets.json 4a645b01dc 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/SLEEPY/kerberos.json 45e33f5ec7 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/SLEEPY/metainfo.xml 5b4dbd4a2d 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/SNOW/kerberos.json 47a979a8c3 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/SNOW/metainfo.xml 21e685ad80 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/YARN_metrics.json a66bb34464 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/YARN_widgets.json 4b76a177b6 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/alerts.json 67cf881caa 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration-mapred/mapred-env.xml fbd056d2a2 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration-mapred/mapred-site.xml  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/capacity-scheduler.xml  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/ranger-yarn-audit.xml 0352424747 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/ranger-yarn-plugin-properties.xml 97867cc6f8 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/ranger-yarn-policymgr-ssl.xml  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/ranger-yarn-security.xml  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/yarn-alert-config.xml  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/yarn-env.xml f81080eadf 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/yarn-log4j.xml 5e9ac98a18 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/yarn-site.xml 2d56e38bd2 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/kerberos.json f33c07d042 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/metainfo.xml bb2191b267 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/alerts/alert_history_process.py  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/alerts/alert_nodemanager_health.py  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/alerts/alert_resourcemanager_process.py  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/alerts/alert_timeline_process.py  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/application_timeline_server.py d179b1f2ca 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/historyserver.py 0570987aa7 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/mapred_service_check.py 66571f8ca6 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/mapred_service_check.py  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/mapreduce2_client.py 9b1274d685 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/nodemanager.py 21db94fcbe 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/resourcemanager.py 5f6c535186 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/service_check.py  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/yarn_client.py 98a7c52b6f 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/quicklinks-mapred/quicklinks.json  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/quicklinks/quicklinks.json 0aca8e3b13 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/themes-mapred/theme.json  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/themes/theme.json 758cf0c969 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/alerts.json 4af7ae9447 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/configuration/zk-alert-config.xml  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/kerberos.json 0a64ea526e 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/metainfo.xml 57667bb888 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/package/alerts/alert_zk_server_process.py  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/package/scripts/service_check.py  
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/package/scripts/zookeeper_client.py 25b2409dcd 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/package/scripts/zookeeper_server.py 6ab88bb3c1 
>   ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/nonrolling-upgrade-2.0.xml PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/upgrade-2.0.xml PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/PythonExecutor.sed PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/install_packages.sed PRE-CREATION 
>   contrib/utils/perf/deploy-gce-perf-cluster.py 8ba7e42e0b 
> 
> Diff: https://reviews.apache.org/r/55515/diff/
> 
> 
> Testing
> -------
> 
> Tested on my local 3-node cluster (both upgrade and downgrade). 
> Script for creating Perf cluster downloads Ambari code from Apache git, so did not manage to test on gce, but script should work too (added required code).
> 
> What works:
> - deploy of Perf 1.0 cluster
> - RU to 2.0 (upgrade and downgrade)
> - EU to 2.0 (upgrade and downgrade)
> - Clean deploy of perf 2.0 stack (takes version and build number from base url)
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 55515: Perf: Create Rolling and Express Upgrade Packs

Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55515/
-----------------------------------------------------------

(Updated Jan. 19, 2017, 8:26 p.m.)


Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Nate Cole, and Vitalyi Brodetskyi.


Bugs: AMBARI-18739
    https://issues.apache.org/jira/browse/AMBARI-18739


Repository: ambari


Description
-------

Also includes code for AMBARI-18738. Perf: Allow components to report their version using distro-select and change configs using conf-select

In order to run Rolling/Express Upgrade, will need to create the Rolling/NonRolling Upgrade Packs and potentially config packs if we decide to do it across major versions.

This means each service will need to support start commands with the upgrade_type option, have symlinks already in distro-select, etc.

At first, this can be a handful of services.
The goal is to simulate Rolling/Express Upgrade on the PERF stack.


Diffs
-----

  ambari-agent/src/main/python/ambari_agent/PythonExecutor.py d75fc0cef2 
  ambari-common/src/main/python/resource_management/libraries/script/dummy.py 3dcece501d 
  ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py d70e87fb1e 
  ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/distro-select.py PRE-CREATION 
  ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
  ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
  ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
  ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
  ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
  ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
  ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
  ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/GRUMPY/kerberos.json 3bf5fbd589 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/GRUMPY/metainfo.xml 15c5cfddd2 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HAPPY/kerberos.json ae84eaff8e 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HAPPY/metainfo.xml 53cc11b7eb 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/alerts.json 24141e8389 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-alert-config.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-env.xml d3dda94fac 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-log4j.xml 880e8837cf 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-logsearch-conf.xml 7c481f29cf 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-policy.xml 8500ee02de 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-site.xml 058facdfe9 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/ranger-hbase-audit.xml e518ea0b10 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/ranger-hbase-policymgr-ssl.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/ranger-hbase-security.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/kerberos.json 9ed40ef7ed 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/metainfo.xml f1f4b7d1ac 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/metrics.json bf1fa6e7d9 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/alerts/hbase_master_process.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/alerts/hbase_regionserver_process.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/scripts/hbase_client.py c4943fed3c 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/scripts/hbase_master.py 3761fcf03a 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/scripts/hbase_regionserver.py 101c36fe56 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/scripts/phoenix_queryserver.py 76a49d9105 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/scripts/service_check.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/quicklinks/quicklinks.json 5568122034 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/themes/theme.json 7a5732bdac 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/widgets.json ae47833059 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/alerts.json 34cea4c2ed 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/core-site.xml 8375be17a7 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hadoop-env.xml 3be63be416 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hadoop-metrics2.properties.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hadoop-policy.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hdfs-alert-config.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hdfs-log4j.xml a4eacc247a 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hdfs-logsearch-conf.xml 6540c865a4 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hdfs-site.xml 995632f339 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/ranger-hdfs-audit.xml 3dc46b3aea 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/ranger-hdfs-plugin-properties.xml deede1c1ba 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/ranger-hdfs-policymgr-ssl.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/ranger-hdfs-security.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/ssl-client.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/ssl-server.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/kerberos.json 974a69cbdc 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/metainfo.xml 42edc8d493 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/metrics.json c66387d3a1 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/alerts/alert_checkpoint_time.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/alerts/alert_datanode_unmounted_data_dir.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/alerts/alert_nfs_gateway_process.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/alerts/alert_snamenode_process.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/alerts/alert_upgrade_finalized.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/datanode.py 6fc338b1da 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/hdfs_client.py 4280c6cc98 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/journalnode.py 96be63060a 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/namenode.py c3488e8bcc 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/nfsgateway.py b750522220 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/service_check.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/snamenode.py 91ce7da9de 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/zkfc_slave.py b4310725a3 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/quicklinks/quicklinks.json a4216e35ac 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/themes/theme.json 6f2b79741b 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/widgets.json 4a645b01dc 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/SLEEPY/kerberos.json 45e33f5ec7 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/SLEEPY/metainfo.xml 5b4dbd4a2d 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/SNOW/kerberos.json 47a979a8c3 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/SNOW/metainfo.xml 21e685ad80 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/YARN_metrics.json a66bb34464 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/YARN_widgets.json 4b76a177b6 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/alerts.json 67cf881caa 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration-mapred/mapred-env.xml fbd056d2a2 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration-mapred/mapred-site.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/capacity-scheduler.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/ranger-yarn-audit.xml 0352424747 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/ranger-yarn-plugin-properties.xml 97867cc6f8 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/ranger-yarn-policymgr-ssl.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/ranger-yarn-security.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/yarn-alert-config.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/yarn-env.xml f81080eadf 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/yarn-log4j.xml 5e9ac98a18 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/yarn-site.xml 2d56e38bd2 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/kerberos.json f33c07d042 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/metainfo.xml bb2191b267 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/alerts/alert_history_process.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/alerts/alert_nodemanager_health.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/alerts/alert_resourcemanager_process.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/alerts/alert_timeline_process.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/application_timeline_server.py d179b1f2ca 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/historyserver.py 0570987aa7 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/mapred_service_check.py 66571f8ca6 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/mapred_service_check.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/mapreduce2_client.py 9b1274d685 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/nodemanager.py 21db94fcbe 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/resourcemanager.py 5f6c535186 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/service_check.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/yarn_client.py 98a7c52b6f 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/quicklinks-mapred/quicklinks.json  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/quicklinks/quicklinks.json 0aca8e3b13 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/themes-mapred/theme.json  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/themes/theme.json 758cf0c969 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/alerts.json 4af7ae9447 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/configuration/zk-alert-config.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/kerberos.json 0a64ea526e 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/metainfo.xml 57667bb888 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/package/alerts/alert_zk_server_process.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/package/scripts/service_check.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/package/scripts/zookeeper_client.py 25b2409dcd 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/package/scripts/zookeeper_server.py 6ab88bb3c1 
  ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/nonrolling-upgrade-2.0.xml PRE-CREATION 
  ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/upgrade-2.0.xml PRE-CREATION 
  ambari-server/src/main/resources/stacks/PERF/PythonExecutor.sed PRE-CREATION 
  ambari-server/src/main/resources/stacks/PERF/install_packages.sed PRE-CREATION 
  contrib/utils/perf/deploy-gce-perf-cluster.py 8ba7e42e0b 

Diff: https://reviews.apache.org/r/55515/diff/


Testing (updated)
-------

Tested on my local 3-node cluster (both upgrade and downgrade). 
Script for creating Perf cluster downloads Ambari code from Apache git, so did not manage to test on gce, but script should work too (added required code).

What works:
- deploy of Perf 1.0 cluster
- RU to 2.0 (upgrade and downgrade)
- EU to 2.0 (upgrade and downgrade)
- Clean deploy of perf 2.0 stack (takes version and build number from base url)


Thanks,

Dmitro Lisnichenko

Re: Review Request 55515: Perf: Create Rolling and Express Upgrade Packs

Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55515/
-----------------------------------------------------------

(Updated Jan. 19, 2017, 8:25 p.m.)


Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Nate Cole, and Vitalyi Brodetskyi.


Changes
-------

Final patch
Addressed comments


Bugs: AMBARI-18739
    https://issues.apache.org/jira/browse/AMBARI-18739


Repository: ambari


Description
-------

Also includes code for AMBARI-18738. Perf: Allow components to report their version using distro-select and change configs using conf-select

In order to run Rolling/Express Upgrade, will need to create the Rolling/NonRolling Upgrade Packs and potentially config packs if we decide to do it across major versions.

This means each service will need to support start commands with the upgrade_type option, have symlinks already in distro-select, etc.

At first, this can be a handful of services.
The goal is to simulate Rolling/Express Upgrade on the PERF stack.


Diffs (updated)
-----

  ambari-agent/src/main/python/ambari_agent/PythonExecutor.py d75fc0cef2 
  ambari-common/src/main/python/resource_management/libraries/script/dummy.py 3dcece501d 
  ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py d70e87fb1e 
  ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/distro-select.py PRE-CREATION 
  ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
  ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
  ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
  ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
  ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
  ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
  ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
  ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/GRUMPY/kerberos.json 3bf5fbd589 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/GRUMPY/metainfo.xml 15c5cfddd2 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HAPPY/kerberos.json ae84eaff8e 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HAPPY/metainfo.xml 53cc11b7eb 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/alerts.json 24141e8389 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-alert-config.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-env.xml d3dda94fac 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-log4j.xml 880e8837cf 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-logsearch-conf.xml 7c481f29cf 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-policy.xml 8500ee02de 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-site.xml 058facdfe9 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/ranger-hbase-audit.xml e518ea0b10 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/ranger-hbase-policymgr-ssl.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/ranger-hbase-security.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/kerberos.json 9ed40ef7ed 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/metainfo.xml f1f4b7d1ac 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/metrics.json bf1fa6e7d9 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/alerts/hbase_master_process.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/alerts/hbase_regionserver_process.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/scripts/hbase_client.py c4943fed3c 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/scripts/hbase_master.py 3761fcf03a 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/scripts/hbase_regionserver.py 101c36fe56 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/scripts/phoenix_queryserver.py 76a49d9105 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/scripts/service_check.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/quicklinks/quicklinks.json 5568122034 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/themes/theme.json 7a5732bdac 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/widgets.json ae47833059 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/alerts.json 34cea4c2ed 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/core-site.xml 8375be17a7 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hadoop-env.xml 3be63be416 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hadoop-metrics2.properties.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hadoop-policy.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hdfs-alert-config.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hdfs-log4j.xml a4eacc247a 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hdfs-logsearch-conf.xml 6540c865a4 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hdfs-site.xml 995632f339 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/ranger-hdfs-audit.xml 3dc46b3aea 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/ranger-hdfs-plugin-properties.xml deede1c1ba 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/ranger-hdfs-policymgr-ssl.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/ranger-hdfs-security.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/ssl-client.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/ssl-server.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/kerberos.json 974a69cbdc 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/metainfo.xml 42edc8d493 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/metrics.json c66387d3a1 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/alerts/alert_checkpoint_time.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/alerts/alert_datanode_unmounted_data_dir.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/alerts/alert_nfs_gateway_process.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/alerts/alert_snamenode_process.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/alerts/alert_upgrade_finalized.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/datanode.py 6fc338b1da 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/hdfs_client.py 4280c6cc98 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/journalnode.py 96be63060a 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/namenode.py c3488e8bcc 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/nfsgateway.py b750522220 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/service_check.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/snamenode.py 91ce7da9de 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/zkfc_slave.py b4310725a3 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/quicklinks/quicklinks.json a4216e35ac 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/themes/theme.json 6f2b79741b 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/widgets.json 4a645b01dc 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/SLEEPY/kerberos.json 45e33f5ec7 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/SLEEPY/metainfo.xml 5b4dbd4a2d 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/SNOW/kerberos.json 47a979a8c3 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/SNOW/metainfo.xml 21e685ad80 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/YARN_metrics.json a66bb34464 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/YARN_widgets.json 4b76a177b6 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/alerts.json 67cf881caa 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration-mapred/mapred-env.xml fbd056d2a2 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration-mapred/mapred-site.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/capacity-scheduler.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/ranger-yarn-audit.xml 0352424747 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/ranger-yarn-plugin-properties.xml 97867cc6f8 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/ranger-yarn-policymgr-ssl.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/ranger-yarn-security.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/yarn-alert-config.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/yarn-env.xml f81080eadf 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/yarn-log4j.xml 5e9ac98a18 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/configuration/yarn-site.xml 2d56e38bd2 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/kerberos.json f33c07d042 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/metainfo.xml bb2191b267 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/alerts/alert_history_process.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/alerts/alert_nodemanager_health.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/alerts/alert_resourcemanager_process.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/alerts/alert_timeline_process.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/application_timeline_server.py d179b1f2ca 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/historyserver.py 0570987aa7 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/mapred_service_check.py 66571f8ca6 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/mapred_service_check.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/mapreduce2_client.py 9b1274d685 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/nodemanager.py 21db94fcbe 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/resourcemanager.py 5f6c535186 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/service_check.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/package/scripts/yarn_client.py 98a7c52b6f 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/quicklinks-mapred/quicklinks.json  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/quicklinks/quicklinks.json 0aca8e3b13 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/themes-mapred/theme.json  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/themes/theme.json 758cf0c969 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/alerts.json 4af7ae9447 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/configuration/zk-alert-config.xml  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/kerberos.json 0a64ea526e 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/metainfo.xml 57667bb888 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/package/alerts/alert_zk_server_process.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/package/scripts/service_check.py  
  ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/package/scripts/zookeeper_client.py 25b2409dcd 
  ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/package/scripts/zookeeper_server.py 6ab88bb3c1 
  ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/nonrolling-upgrade-2.0.xml PRE-CREATION 
  ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/upgrade-2.0.xml PRE-CREATION 
  ambari-server/src/main/resources/stacks/PERF/PythonExecutor.sed PRE-CREATION 
  ambari-server/src/main/resources/stacks/PERF/install_packages.sed PRE-CREATION 
  contrib/utils/perf/deploy-gce-perf-cluster.py 8ba7e42e0b 

Diff: https://reviews.apache.org/r/55515/diff/


Testing
-------

Tested on my local 3-node cluster (both upgrade and downgrade). 
Script for creating Perf cluster downloads Ambari code from Apache git, so did not manage to test on gce, but script should work too (added required code).

What works:
- deploy of Perf 1.0 cluster
- RU to 2.0 (upgrade and downgrade)
- EU to 2.0 (upgrade and downgrade)

Issues:
- Clean deploy of perf 2.0 stack (dummy component install scripts are hardcoded with 1.1.1.1 version)


Thanks,

Dmitro Lisnichenko

Re: Review Request 55515: Perf: Create Rolling and Express Upgrade Packs

Posted by Jonathan Hurley <jh...@hortonworks.com>.


> On Jan. 16, 2017, 1:40 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/stacks/PERF/install_packages.py, line 41
> > <https://reviews.apache.org/r/55515/diff/1/?file=1605019#file1605019line41>
> >
> >     How much of this was copied from the one in HDP?
> >     Can we derive both from a base class?

As the PERF stack is a fake stack used for specific tests, I don't think any production code should be derived from it. 

It's fine if PERF extends/overrides real stack code and alters behavior, but we should not have both test and production code trying to have shared logic. a


- Jonathan


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55515/#review161768
-----------------------------------------------------------


On Jan. 13, 2017, 2:03 p.m., Dmitro Lisnichenko wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55515/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2017, 2:03 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Nate Cole, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-18739
>     https://issues.apache.org/jira/browse/AMBARI-18739
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Also includes code for AMBARI-18738. Perf: Allow components to report their version using distro-select and change configs using conf-select
> 
> In order to run Rolling/Express Upgrade, will need to create the Rolling/NonRolling Upgrade Packs and potentially config packs if we decide to do it across major versions.
> 
> This means each service will need to support start commands with the upgrade_type option, have symlinks already in distro-select, etc.
> 
> At first, this can be a handful of services.
> The goal is to simulate Rolling/Express Upgrade on the PERF stack.
> 
> 
> Diffs
> -----
> 
>   ambari-agent/src/main/python/ambari_agent/PythonExecutor.py d75fc0cef2 
>   ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 513ceac8dd 
>   ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 615a0cdcbc 
>   ambari-common/src/main/python/resource_management/libraries/script/dummy.py 3dcece501d 
>   ambari-server/src/main/resources/custom_actions/scripts/ru_execute_tasks.py 4dec16f372 
>   ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py d70e87fb1e 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/distro-select.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
>   ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/GRUMPY/metainfo.xml 15c5cfddd2 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HAPPY/metainfo.xml 53cc11b7eb 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/metainfo.xml f1f4b7d1ac 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/metainfo.xml 42edc8d493 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/datanode.py 6fc338b1da 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/journalnode.py 96be63060a 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/namenode.py c3488e8bcc 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/SLEEPY/metainfo.xml 5b4dbd4a2d 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/SNOW/metainfo.xml 21e685ad80 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/metainfo.xml bb2191b267 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/metainfo.xml 57667bb888 
>   ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/nonrolling-upgrade-2.0.xml PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/upgrade-2.0.xml PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/install_packages.py PRE-CREATION 
>   contrib/utils/perf/deploy-gce-perf-cluster.py 8ba7e42e0b 
> 
> Diff: https://reviews.apache.org/r/55515/diff/
> 
> 
> Testing
> -------
> 
> Tested on my local 3-node cluster (both upgrade and downgrade). 
> Script for creating Perf cluster downloads Ambari code from Apache git, so did not manage to test on gce, but script should work too (added required code).
> 
> What works:
> - deploy of Perf 1.0 cluster
> - RU to 2.0 (upgrade and downgrade)
> - EU to 2.0 (upgrade and downgrade)
> 
> Issues:
> - Clean deploy of perf 2.0 stack (dummy component install scripts are hardcoded with 1.1.1.1 version)
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 55515: Perf: Create Rolling and Express Upgrade Packs

Posted by Alejandro Fernandez <af...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55515/#review161768
-----------------------------------------------------------


Fix it, then Ship it!





ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/distro-select.py (line 27)
<https://reviews.apache.org/r/55515/#comment233061>

    Nice, this is a nice template for other stacks.



ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py (line 30)
<https://reviews.apache.org/r/55515/#comment233062>

    let's make a variable here for the stack name and version so it's easier to find any hardcodings.



ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/nonrolling-upgrade-2.0.xml (line 22)
<https://reviews.apache.org/r/55515/#comment233064>

    Nice!



ambari-server/src/main/resources/stacks/PERF/install_packages.py (line 41)
<https://reviews.apache.org/r/55515/#comment233060>

    How much of this was copied from the one in HDP?
    Can we derive both from a base class?


- Alejandro Fernandez


On Jan. 13, 2017, 7:03 p.m., Dmitro Lisnichenko wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55515/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2017, 7:03 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Nate Cole, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-18739
>     https://issues.apache.org/jira/browse/AMBARI-18739
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Also includes code for AMBARI-18738. Perf: Allow components to report their version using distro-select and change configs using conf-select
> 
> In order to run Rolling/Express Upgrade, will need to create the Rolling/NonRolling Upgrade Packs and potentially config packs if we decide to do it across major versions.
> 
> This means each service will need to support start commands with the upgrade_type option, have symlinks already in distro-select, etc.
> 
> At first, this can be a handful of services.
> The goal is to simulate Rolling/Express Upgrade on the PERF stack.
> 
> 
> Diffs
> -----
> 
>   ambari-agent/src/main/python/ambari_agent/PythonExecutor.py d75fc0cef2 
>   ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 513ceac8dd 
>   ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 615a0cdcbc 
>   ambari-common/src/main/python/resource_management/libraries/script/dummy.py 3dcece501d 
>   ambari-server/src/main/resources/custom_actions/scripts/ru_execute_tasks.py 4dec16f372 
>   ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py d70e87fb1e 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/distro-select.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
>   ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/GRUMPY/metainfo.xml 15c5cfddd2 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HAPPY/metainfo.xml 53cc11b7eb 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/metainfo.xml f1f4b7d1ac 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/metainfo.xml 42edc8d493 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/datanode.py 6fc338b1da 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/journalnode.py 96be63060a 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/namenode.py c3488e8bcc 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/SLEEPY/metainfo.xml 5b4dbd4a2d 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/SNOW/metainfo.xml 21e685ad80 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/metainfo.xml bb2191b267 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/metainfo.xml 57667bb888 
>   ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/nonrolling-upgrade-2.0.xml PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/upgrade-2.0.xml PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/install_packages.py PRE-CREATION 
>   contrib/utils/perf/deploy-gce-perf-cluster.py 8ba7e42e0b 
> 
> Diff: https://reviews.apache.org/r/55515/diff/
> 
> 
> Testing
> -------
> 
> Tested on my local 3-node cluster (both upgrade and downgrade). 
> Script for creating Perf cluster downloads Ambari code from Apache git, so did not manage to test on gce, but script should work too (added required code).
> 
> What works:
> - deploy of Perf 1.0 cluster
> - RU to 2.0 (upgrade and downgrade)
> - EU to 2.0 (upgrade and downgrade)
> 
> Issues:
> - Clean deploy of perf 2.0 stack (dummy component install scripts are hardcoded with 1.1.1.1 version)
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 55515: Perf: Create Rolling and Express Upgrade Packs

Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55515/#review161746
-----------------------------------------------------------




ambari-agent/src/main/python/ambari_agent/PythonExecutor.py (line 178)
<https://reviews.apache.org/r/55515/#comment233048>

    By the way, we have always been passing full agent environment to python scripts. This environment was just dropped on first Execute()/shell.call() invocation and not passed to subscripts


- Dmitro Lisnichenko


On Jan. 13, 2017, 9:03 p.m., Dmitro Lisnichenko wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55515/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2017, 9:03 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Nate Cole, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-18739
>     https://issues.apache.org/jira/browse/AMBARI-18739
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Also includes code for AMBARI-18738. Perf: Allow components to report their version using distro-select and change configs using conf-select
> 
> In order to run Rolling/Express Upgrade, will need to create the Rolling/NonRolling Upgrade Packs and potentially config packs if we decide to do it across major versions.
> 
> This means each service will need to support start commands with the upgrade_type option, have symlinks already in distro-select, etc.
> 
> At first, this can be a handful of services.
> The goal is to simulate Rolling/Express Upgrade on the PERF stack.
> 
> 
> Diffs
> -----
> 
>   ambari-agent/src/main/python/ambari_agent/PythonExecutor.py d75fc0cef2 
>   ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 513ceac8dd 
>   ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py 615a0cdcbc 
>   ambari-common/src/main/python/resource_management/libraries/script/dummy.py 3dcece501d 
>   ambari-server/src/main/resources/custom_actions/scripts/ru_execute_tasks.py 4dec16f372 
>   ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py d70e87fb1e 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/distro-select.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-INSTALL/scripts/hook.py 9eca9ec724 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/metainfo.xml 45a63e5550 
>   ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
>   ambari-server/src/main/resources/stacks/PERF/1.0/repos/repoinfo.xml c5137fb133 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/GRUMPY/metainfo.xml 15c5cfddd2 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HAPPY/metainfo.xml 53cc11b7eb 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/metainfo.xml f1f4b7d1ac 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/metainfo.xml 42edc8d493 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/datanode.py 6fc338b1da 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/journalnode.py 96be63060a 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/package/scripts/namenode.py c3488e8bcc 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/SLEEPY/metainfo.xml 5b4dbd4a2d 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/SNOW/metainfo.xml 21e685ad80 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/YARN/metainfo.xml bb2191b267 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/ZOOKEEPER/metainfo.xml 57667bb888 
>   ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/nonrolling-upgrade-2.0.xml PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/1.0/upgrades/upgrade-2.0.xml PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/install_packages.py PRE-CREATION 
>   contrib/utils/perf/deploy-gce-perf-cluster.py 8ba7e42e0b 
> 
> Diff: https://reviews.apache.org/r/55515/diff/
> 
> 
> Testing
> -------
> 
> Tested on my local 3-node cluster (both upgrade and downgrade). 
> Script for creating Perf cluster downloads Ambari code from Apache git, so did not manage to test on gce, but script should work too (added required code).
> 
> What works:
> - deploy of Perf 1.0 cluster
> - RU to 2.0 (upgrade and downgrade)
> - EU to 2.0 (upgrade and downgrade)
> 
> Issues:
> - Clean deploy of perf 2.0 stack (dummy component install scripts are hardcoded with 1.1.1.1 version)
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>