You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Alejandro Fernandez <af...@hortonworks.com> on 2017/03/20 20:55:03 UTC
Review Request 57792: Fixes for Express Upgrade on large-scale
clusters:
batch execute-stage for hdp-select set all, alert for hdp-select versions,
EU from HDP 2.2 to 2.5
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/
-----------------------------------------------------------
Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
Bugs: AMBARI-20490
https://issues.apache.org/jira/browse/AMBARI-20490
Repository: ambari
Description
-------
On large-scale clusters over 1000 nodes, performance during Express Upgrade can be very slow and it takes more time to fix potential errors.
We can remedy these with the following fixes:
* "hdp-select set all" is currently ran in the ClusterGroup with a single stage instead of batching. This can cause Ambari Server to be the bottleneck when processing 1000+ requests. Allow the execute-stage inside a ClusterGroup to use batching.
* If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it will prevent the hosts from running "hdp-select versions". We can find these problematic hosts with an alert.
* There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no longer supported in Ambari 2.5).
Diffs
-----
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 20b4cbd
ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java 8556436
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java ca9ce07
ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java ee5451f
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java c45ec24
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java 8003223
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java 669d50f
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java 3cf3f8b
ambari-server/src/main/resources/alerts.json 2559b3a
ambari-server/src/main/resources/host_scripts/alert_version_select.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 28ca939
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 9aff1a9
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml adccac0
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 3c0104b
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml cd63651
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 39cb0dc
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml 57a5203
ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 035cb59
ambari-server/src/main/resources/upgrade-pack.xsd 1f11aa1
ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java f1af66f
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java bf6c41a
ambari-server/src/test/java/org/apache/ambari/server/metadata/AgentAlertDefinitionsTest.java cbc5e69
ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java 8e5ad0a
ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_batch.xml PRE-CREATION
ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_to_new_stack.xml fe387f3
Diff: https://reviews.apache.org/r/57792/diff/1/
Testing
-------
mvn clean test
Thanks,
Alejandro Fernandez
Re: Review Request 57792: Fixes for Express Upgrade on large-scale
clusters:
batch execute-stage for hdp-select set all, alert for hdp-select versions,
EU from HDP 2.2 to 2.5
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/#review170014
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java
Lines 511-514 (patched)
<https://reviews.apache.org/r/57792/#comment242747>
Still doesn't seem necessary. Original text vs "regular" text is still unclear. You shouldn't be putting hostname stuff on a StageWrapper.
- Nate Cole
On March 23, 2017, 9:29 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57792/
> -----------------------------------------------------------
>
> (Updated March 23, 2017, 9:29 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-20490
> https://issues.apache.org/jira/browse/AMBARI-20490
>
>
> Repository: ambari
>
>
> Description
> -------
>
> On large-scale clusters over 1000 nodes, performance during Express Upgrade can be very slow and it takes more time to fix potential errors.
> We can remedy these with the following fixes:
>
> * "hdp-select set all" is currently ran in the ClusterGroup with a single stage instead of batching. This can cause Ambari Server to be the bottleneck when processing 1000+ requests. Allow the execute-stage inside a ClusterGroup to use batching.
>
> * If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it will prevent the hosts from running "hdp-select versions". We can find these problematic hosts with an alert.
>
> * There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no longer supported in Ambari 2.5).
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 20b4cbd
> ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java 8556436
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java ca9ce07
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java ee5451f
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java c45ec24
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java 8003223
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ParallelScheduler.java 6004a12
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java 669d50f
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java 3cf3f8b
> ambari-server/src/main/resources/alerts.json 2559b3a
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-properties.xml ca14c1f
> ambari-server/src/main/resources/host_scripts/alert_version_select.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 28ca939
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 9aff1a9
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml adccac0
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 3c0104b
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml cd63651
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 6f7fbd0
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml 063b362
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 4a641ec
> ambari-server/src/main/resources/upgrade-pack.xsd 1f11aa1
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java f1af66f
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java bf6c41a
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java 3e70989
> ambari-server/src/test/java/org/apache/ambari/server/metadata/AgentAlertDefinitionsTest.java cbc5e69
> ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java 8e5ad0a
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_batch.xml PRE-CREATION
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_to_new_stack.xml fe387f3
>
>
> Diff: https://reviews.apache.org/r/57792/diff/8/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 57792: Fixes for Express Upgrade on large-scale
clusters:
batch execute-stage for hdp-select set all, alert for hdp-select versions,
EU from HDP 2.2 to 2.5
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On March 24, 2017, 10:08 a.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ParallelScheduler.java
> > Lines 56-57 (patched)
> > <https://reviews.apache.org/r/57792/diff/8/?file=1673723#file1673723line56>
> >
> > I thought I had flagged this before and you agreed it was a problem - doesn't this mean get a list of sets with each set containing only 1 item? The returned collection should essentially be a singleton list which contains a single set of all hosts.
Let me be clearer here:
- If the PS is omitted, then this code is correct - it should not be parallelized, so you should force it to 1
- If the PS is specified and the MaxDegreeOP is set to 0, then this should do what I suggested, which is run them all in parallel (singleton list of 1 set with all items in set)
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/#review170011
-----------------------------------------------------------
On March 23, 2017, 9:29 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57792/
> -----------------------------------------------------------
>
> (Updated March 23, 2017, 9:29 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-20490
> https://issues.apache.org/jira/browse/AMBARI-20490
>
>
> Repository: ambari
>
>
> Description
> -------
>
> On large-scale clusters over 1000 nodes, performance during Express Upgrade can be very slow and it takes more time to fix potential errors.
> We can remedy these with the following fixes:
>
> * "hdp-select set all" is currently ran in the ClusterGroup with a single stage instead of batching. This can cause Ambari Server to be the bottleneck when processing 1000+ requests. Allow the execute-stage inside a ClusterGroup to use batching.
>
> * If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it will prevent the hosts from running "hdp-select versions". We can find these problematic hosts with an alert.
>
> * There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no longer supported in Ambari 2.5).
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 20b4cbd
> ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java 8556436
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java ca9ce07
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java ee5451f
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java c45ec24
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java 8003223
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ParallelScheduler.java 6004a12
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java 669d50f
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java 3cf3f8b
> ambari-server/src/main/resources/alerts.json 2559b3a
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-properties.xml ca14c1f
> ambari-server/src/main/resources/host_scripts/alert_version_select.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 28ca939
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 9aff1a9
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml adccac0
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 3c0104b
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml cd63651
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 6f7fbd0
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml 063b362
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 4a641ec
> ambari-server/src/main/resources/upgrade-pack.xsd 1f11aa1
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java f1af66f
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java bf6c41a
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java 3e70989
> ambari-server/src/test/java/org/apache/ambari/server/metadata/AgentAlertDefinitionsTest.java cbc5e69
> ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java 8e5ad0a
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_batch.xml PRE-CREATION
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_to_new_stack.xml fe387f3
>
>
> Diff: https://reviews.apache.org/r/57792/diff/8/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 57792: Fixes for Express Upgrade on large-scale
clusters:
batch execute-stage for hdp-select set all, alert for hdp-select versions,
EU from HDP 2.2 to 2.5
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/#review170011
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
Line 194 (original), 204 (patched)
<https://reviews.apache.org/r/57792/#comment242743>
Log a warning here that we didn't catch the missed task type?
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ParallelScheduler.java
Lines 56-57 (patched)
<https://reviews.apache.org/r/57792/#comment242744>
I thought I had flagged this before and you agreed it was a problem - doesn't this mean get a list of sets with each set containing only 1 item? The returned collection should essentially be a singleton list which contains a single set of all hosts.
- Jonathan Hurley
On March 23, 2017, 9:29 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57792/
> -----------------------------------------------------------
>
> (Updated March 23, 2017, 9:29 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-20490
> https://issues.apache.org/jira/browse/AMBARI-20490
>
>
> Repository: ambari
>
>
> Description
> -------
>
> On large-scale clusters over 1000 nodes, performance during Express Upgrade can be very slow and it takes more time to fix potential errors.
> We can remedy these with the following fixes:
>
> * "hdp-select set all" is currently ran in the ClusterGroup with a single stage instead of batching. This can cause Ambari Server to be the bottleneck when processing 1000+ requests. Allow the execute-stage inside a ClusterGroup to use batching.
>
> * If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it will prevent the hosts from running "hdp-select versions". We can find these problematic hosts with an alert.
>
> * There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no longer supported in Ambari 2.5).
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 20b4cbd
> ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java 8556436
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java ca9ce07
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java ee5451f
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java c45ec24
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java 8003223
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ParallelScheduler.java 6004a12
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java 669d50f
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java 3cf3f8b
> ambari-server/src/main/resources/alerts.json 2559b3a
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-properties.xml ca14c1f
> ambari-server/src/main/resources/host_scripts/alert_version_select.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 28ca939
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 9aff1a9
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml adccac0
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 3c0104b
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml cd63651
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 6f7fbd0
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml 063b362
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 4a641ec
> ambari-server/src/main/resources/upgrade-pack.xsd 1f11aa1
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java f1af66f
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java bf6c41a
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java 3e70989
> ambari-server/src/test/java/org/apache/ambari/server/metadata/AgentAlertDefinitionsTest.java cbc5e69
> ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java 8e5ad0a
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_batch.xml PRE-CREATION
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_to_new_stack.xml fe387f3
>
>
> Diff: https://reviews.apache.org/r/57792/diff/8/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 57792: Fixes for Express Upgrade on large-scale
clusters:
batch execute-stage for hdp-select set all, alert for hdp-select versions,
EU from HDP 2.2 to 2.5
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/#review170013
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
Line 1357 (original), 1357 (patched)
<https://reviews.apache.org/r/57792/#comment242746>
I still don't think you need this. An HRC is scoped to a host by nature and the UI shows this host separate of this text.
Consider environments where the hostname is:
head-nn-masters-zyzfgd1405697.somelongcompanyname.internal.net
.. you don't want that showing up in the text.
- Jonathan Hurley
On March 23, 2017, 9:29 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57792/
> -----------------------------------------------------------
>
> (Updated March 23, 2017, 9:29 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-20490
> https://issues.apache.org/jira/browse/AMBARI-20490
>
>
> Repository: ambari
>
>
> Description
> -------
>
> On large-scale clusters over 1000 nodes, performance during Express Upgrade can be very slow and it takes more time to fix potential errors.
> We can remedy these with the following fixes:
>
> * "hdp-select set all" is currently ran in the ClusterGroup with a single stage instead of batching. This can cause Ambari Server to be the bottleneck when processing 1000+ requests. Allow the execute-stage inside a ClusterGroup to use batching.
>
> * If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it will prevent the hosts from running "hdp-select versions". We can find these problematic hosts with an alert.
>
> * There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no longer supported in Ambari 2.5).
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 20b4cbd
> ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java 8556436
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java ca9ce07
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java ee5451f
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java c45ec24
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java 8003223
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ParallelScheduler.java 6004a12
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java 669d50f
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java 3cf3f8b
> ambari-server/src/main/resources/alerts.json 2559b3a
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-properties.xml ca14c1f
> ambari-server/src/main/resources/host_scripts/alert_version_select.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 28ca939
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 9aff1a9
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml adccac0
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 3c0104b
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml cd63651
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 6f7fbd0
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml 063b362
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 4a641ec
> ambari-server/src/main/resources/upgrade-pack.xsd 1f11aa1
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java f1af66f
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java bf6c41a
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java 3e70989
> ambari-server/src/test/java/org/apache/ambari/server/metadata/AgentAlertDefinitionsTest.java cbc5e69
> ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java 8e5ad0a
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_batch.xml PRE-CREATION
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_to_new_stack.xml fe387f3
>
>
> Diff: https://reviews.apache.org/r/57792/diff/8/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 57792: Fixes for Express Upgrade on large-scale
clusters:
batch execute-stage for hdp-select set all, alert for hdp-select versions,
EU from HDP 2.2 to 2.5
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/
-----------------------------------------------------------
(Updated March 24, 2017, 1:29 a.m.)
Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
Bugs: AMBARI-20490
https://issues.apache.org/jira/browse/AMBARI-20490
Repository: ambari
Description
-------
On large-scale clusters over 1000 nodes, performance during Express Upgrade can be very slow and it takes more time to fix potential errors.
We can remedy these with the following fixes:
* "hdp-select set all" is currently ran in the ClusterGroup with a single stage instead of batching. This can cause Ambari Server to be the bottleneck when processing 1000+ requests. Allow the execute-stage inside a ClusterGroup to use batching.
* If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it will prevent the hosts from running "hdp-select versions". We can find these problematic hosts with an alert.
* There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no longer supported in Ambari 2.5).
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 20b4cbd
ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java 8556436
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java ca9ce07
ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java ee5451f
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java c45ec24
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java 8003223
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ParallelScheduler.java 6004a12
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java 669d50f
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java 3cf3f8b
ambari-server/src/main/resources/alerts.json 2559b3a
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-properties.xml ca14c1f
ambari-server/src/main/resources/host_scripts/alert_version_select.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 28ca939
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 9aff1a9
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml adccac0
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 3c0104b
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml cd63651
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 6f7fbd0
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml 063b362
ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 4a641ec
ambari-server/src/main/resources/upgrade-pack.xsd 1f11aa1
ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java f1af66f
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java bf6c41a
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java 3e70989
ambari-server/src/test/java/org/apache/ambari/server/metadata/AgentAlertDefinitionsTest.java cbc5e69
ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java 8e5ad0a
ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_batch.xml PRE-CREATION
ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_to_new_stack.xml fe387f3
Diff: https://reviews.apache.org/r/57792/diff/8/
Changes: https://reviews.apache.org/r/57792/diff/7-8/
Testing
-------
mvn clean test
Thanks,
Alejandro Fernandez
Re: Review Request 57792: Fixes for Express Upgrade on large-scale
clusters:
batch execute-stage for hdp-select set all, alert for hdp-select versions,
EU from HDP 2.2 to 2.5
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/#review169862
-----------------------------------------------------------
Let's have a chat about Original Text. I don't see why a stage needs specific "original" text to be set, making it non-original.
ambari-server/src/main/resources/upgrade-pack.xsd
Lines 139 (patched)
<https://reviews.apache.org/r/57792/#comment242573>
may as well make this an xs:int type while you're at it.
ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java
Lines 41 (patched)
<https://reviews.apache.org/r/57792/#comment242571>
should be org.junit.Assert
- Nate Cole
On March 22, 2017, 9:22 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57792/
> -----------------------------------------------------------
>
> (Updated March 22, 2017, 9:22 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-20490
> https://issues.apache.org/jira/browse/AMBARI-20490
>
>
> Repository: ambari
>
>
> Description
> -------
>
> On large-scale clusters over 1000 nodes, performance during Express Upgrade can be very slow and it takes more time to fix potential errors.
> We can remedy these with the following fixes:
>
> * "hdp-select set all" is currently ran in the ClusterGroup with a single stage instead of batching. This can cause Ambari Server to be the bottleneck when processing 1000+ requests. Allow the execute-stage inside a ClusterGroup to use batching.
>
> * If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it will prevent the hosts from running "hdp-select versions". We can find these problematic hosts with an alert.
>
> * There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no longer supported in Ambari 2.5).
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 20b4cbd
> ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java 8556436
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java ca9ce07
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java ee5451f
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java c45ec24
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java 8003223
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ParallelScheduler.java 6004a12
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java 669d50f
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java 3cf3f8b
> ambari-server/src/main/resources/alerts.json 2559b3a
> ambari-server/src/main/resources/host_scripts/alert_version_select.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 28ca939
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 9aff1a9
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml adccac0
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 3c0104b
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml cd63651
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 6f7fbd0
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml 063b362
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 4a641ec
> ambari-server/src/main/resources/upgrade-pack.xsd 1f11aa1
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java f1af66f
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java bf6c41a
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java 3e70989
> ambari-server/src/test/java/org/apache/ambari/server/metadata/AgentAlertDefinitionsTest.java cbc5e69
> ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java 8e5ad0a
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_batch.xml PRE-CREATION
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_to_new_stack.xml fe387f3
>
>
> Diff: https://reviews.apache.org/r/57792/diff/7/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 57792: Fixes for Express Upgrade on large-scale
clusters:
batch execute-stage for hdp-select set all, alert for hdp-select versions,
EU from HDP 2.2 to 2.5
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/
-----------------------------------------------------------
(Updated March 23, 2017, 1:22 a.m.)
Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
Changes
-------
Fixing more unit tests
Bugs: AMBARI-20490
https://issues.apache.org/jira/browse/AMBARI-20490
Repository: ambari
Description
-------
On large-scale clusters over 1000 nodes, performance during Express Upgrade can be very slow and it takes more time to fix potential errors.
We can remedy these with the following fixes:
* "hdp-select set all" is currently ran in the ClusterGroup with a single stage instead of batching. This can cause Ambari Server to be the bottleneck when processing 1000+ requests. Allow the execute-stage inside a ClusterGroup to use batching.
* If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it will prevent the hosts from running "hdp-select versions". We can find these problematic hosts with an alert.
* There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no longer supported in Ambari 2.5).
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 20b4cbd
ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java 8556436
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java ca9ce07
ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java ee5451f
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java c45ec24
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java 8003223
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ParallelScheduler.java 6004a12
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java 669d50f
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java 3cf3f8b
ambari-server/src/main/resources/alerts.json 2559b3a
ambari-server/src/main/resources/host_scripts/alert_version_select.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 28ca939
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 9aff1a9
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml adccac0
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 3c0104b
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml cd63651
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 6f7fbd0
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml 063b362
ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 4a641ec
ambari-server/src/main/resources/upgrade-pack.xsd 1f11aa1
ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java f1af66f
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java bf6c41a
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java 3e70989
ambari-server/src/test/java/org/apache/ambari/server/metadata/AgentAlertDefinitionsTest.java cbc5e69
ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java 8e5ad0a
ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_batch.xml PRE-CREATION
ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_to_new_stack.xml fe387f3
Diff: https://reviews.apache.org/r/57792/diff/7/
Changes: https://reviews.apache.org/r/57792/diff/6-7/
Testing
-------
mvn clean test
Thanks,
Alejandro Fernandez
Re: Review Request 57792: Fixes for Express Upgrade on large-scale
clusters:
batch execute-stage for hdp-select set all, alert for hdp-select versions,
EU from HDP 2.2 to 2.5
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/
-----------------------------------------------------------
(Updated March 23, 2017, 12:31 a.m.)
Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
Changes
-------
ReviewBoard isn't showing all of my changed files, so trying one more time.
Bugs: AMBARI-20490
https://issues.apache.org/jira/browse/AMBARI-20490
Repository: ambari
Description
-------
On large-scale clusters over 1000 nodes, performance during Express Upgrade can be very slow and it takes more time to fix potential errors.
We can remedy these with the following fixes:
* "hdp-select set all" is currently ran in the ClusterGroup with a single stage instead of batching. This can cause Ambari Server to be the bottleneck when processing 1000+ requests. Allow the execute-stage inside a ClusterGroup to use batching.
* If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it will prevent the hosts from running "hdp-select versions". We can find these problematic hosts with an alert.
* There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no longer supported in Ambari 2.5).
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 20b4cbd
ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java 8556436
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java ca9ce07
ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java ee5451f
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java c45ec24
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java 8003223
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ParallelScheduler.java 6004a12
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java 669d50f
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java 3cf3f8b
ambari-server/src/main/resources/alerts.json 2559b3a
ambari-server/src/main/resources/host_scripts/alert_version_select.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 28ca939
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 9aff1a9
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml adccac0
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 3c0104b
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml cd63651
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 6f7fbd0
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml 063b362
ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 4a641ec
ambari-server/src/main/resources/upgrade-pack.xsd 1f11aa1
ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java f1af66f
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java bf6c41a
ambari-server/src/test/java/org/apache/ambari/server/metadata/AgentAlertDefinitionsTest.java cbc5e69
ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java 8e5ad0a
ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_batch.xml PRE-CREATION
ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_to_new_stack.xml fe387f3
Diff: https://reviews.apache.org/r/57792/diff/6/
Changes: https://reviews.apache.org/r/57792/diff/5-6/
Testing
-------
mvn clean test
Thanks,
Alejandro Fernandez
Re: Review Request 57792: Fixes for Express Upgrade on large-scale
clusters:
batch execute-stage for hdp-select set all, alert for hdp-select versions,
EU from HDP 2.2 to 2.5
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/
-----------------------------------------------------------
(Updated March 23, 2017, 12:12 a.m.)
Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
Bugs: AMBARI-20490
https://issues.apache.org/jira/browse/AMBARI-20490
Repository: ambari
Description
-------
On large-scale clusters over 1000 nodes, performance during Express Upgrade can be very slow and it takes more time to fix potential errors.
We can remedy these with the following fixes:
* "hdp-select set all" is currently ran in the ClusterGroup with a single stage instead of batching. This can cause Ambari Server to be the bottleneck when processing 1000+ requests. Allow the execute-stage inside a ClusterGroup to use batching.
* If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it will prevent the hosts from running "hdp-select versions". We can find these problematic hosts with an alert.
* There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no longer supported in Ambari 2.5).
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java 8556436
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java ca9ce07
ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java ee5451f
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java c45ec24
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java 8003223
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ParallelScheduler.java 6004a12
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java 669d50f
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java 3cf3f8b
ambari-server/src/main/resources/alerts.json 2559b3a
ambari-server/src/main/resources/host_scripts/alert_version_select.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 28ca939
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 9aff1a9
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml adccac0
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 3c0104b
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml cd63651
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 39cb0dc
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml 57a5203
ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 4a641ec
ambari-server/src/main/resources/upgrade-pack.xsd 1f11aa1
ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java f1af66f
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java bf6c41a
ambari-server/src/test/java/org/apache/ambari/server/metadata/AgentAlertDefinitionsTest.java cbc5e69
ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java 8e5ad0a
ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_batch.xml PRE-CREATION
ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_to_new_stack.xml fe387f3
Diff: https://reviews.apache.org/r/57792/diff/5/
Changes: https://reviews.apache.org/r/57792/diff/4-5/
Testing
-------
mvn clean test
Thanks,
Alejandro Fernandez
Re: Review Request 57792: Fixes for Express Upgrade on large-scale
clusters:
batch execute-stage for hdp-select set all, alert for hdp-select versions,
EU from HDP 2.2 to 2.5
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/
-----------------------------------------------------------
(Updated March 23, 2017, 12:08 a.m.)
Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
Bugs: AMBARI-20490
https://issues.apache.org/jira/browse/AMBARI-20490
Repository: ambari
Description
-------
On large-scale clusters over 1000 nodes, performance during Express Upgrade can be very slow and it takes more time to fix potential errors.
We can remedy these with the following fixes:
* "hdp-select set all" is currently ran in the ClusterGroup with a single stage instead of batching. This can cause Ambari Server to be the bottleneck when processing 1000+ requests. Allow the execute-stage inside a ClusterGroup to use batching.
* If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it will prevent the hosts from running "hdp-select versions". We can find these problematic hosts with an alert.
* There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no longer supported in Ambari 2.5).
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 513ceac
ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java 8556436
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java d37e32b
ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java 3a2dc89
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java 80bb26c
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java be1f469
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ParallelScheduler.java 6004a12
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java 86a8f55
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java ec7279c
ambari-server/src/main/resources/alerts.json 9cffff5
ambari-server/src/main/resources/host_scripts/alert_version_select.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/config-upgrade.xml 0f5f48e
ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.2.xml 46f0fee
ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.3.xml a741722
ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.4.xml 933183b
ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.5.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 90f62dc
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml ecfc4d7
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml 68fb1d4
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 2e5c60e
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml 4e49097
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 3107bf3
ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java 538fa48
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java 1ea9407
ambari-server/src/test/java/org/apache/ambari/server/metadata/AgentAlertDefinitionsTest.java f5ce11e
ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_batch.xml PRE-CREATION
ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_to_new_stack.xml bfd994e
Diff: https://reviews.apache.org/r/57792/diff/4/
Changes: https://reviews.apache.org/r/57792/diff/3-4/
Testing
-------
mvn clean test
Thanks,
Alejandro Fernandez
Re: Review Request 57792: Fixes for Express Upgrade on large-scale
clusters:
batch execute-stage for hdp-select set all, alert for hdp-select versions,
EU from HDP 2.2 to 2.5
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/
-----------------------------------------------------------
(Updated March 22, 2017, 11:26 p.m.)
Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
Changes
-------
Addressed comments.
Bugs: AMBARI-20490
https://issues.apache.org/jira/browse/AMBARI-20490
Repository: ambari
Description
-------
On large-scale clusters over 1000 nodes, performance during Express Upgrade can be very slow and it takes more time to fix potential errors.
We can remedy these with the following fixes:
* "hdp-select set all" is currently ran in the ClusterGroup with a single stage instead of batching. This can cause Ambari Server to be the bottleneck when processing 1000+ requests. Allow the execute-stage inside a ClusterGroup to use batching.
* If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it will prevent the hosts from running "hdp-select versions". We can find these problematic hosts with an alert.
* There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no longer supported in Ambari 2.5).
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 20b4cbd
ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java 8556436
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java ca9ce07
ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java ee5451f
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java c45ec24
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java 8003223
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ParallelScheduler.java 6004a12
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java 669d50f
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java 3cf3f8b
ambari-server/src/main/resources/alerts.json 2559b3a
ambari-server/src/main/resources/host_scripts/alert_version_select.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 28ca939
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 9aff1a9
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml adccac0
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 3c0104b
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml cd63651
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 39cb0dc
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml 57a5203
ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 4a641ec
ambari-server/src/main/resources/upgrade-pack.xsd 1f11aa1
ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java f1af66f
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java bf6c41a
ambari-server/src/test/java/org/apache/ambari/server/metadata/AgentAlertDefinitionsTest.java cbc5e69
ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java 8e5ad0a
ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_batch.xml PRE-CREATION
ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_to_new_stack.xml fe387f3
Diff: https://reviews.apache.org/r/57792/diff/3/
Changes: https://reviews.apache.org/r/57792/diff/2-3/
Testing
-------
mvn clean test
Thanks,
Alejandro Fernandez
Re: Review Request 57792: Fixes for Express Upgrade on large-scale
clusters:
batch execute-stage for hdp-select set all, alert for hdp-select versions,
EU from HDP 2.2 to 2.5
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On March 22, 2017, 2:17 p.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java
> > Lines 511-514 (patched)
> > <https://reviews.apache.org/r/57792/diff/2/?file=1670561#file1670561line511>
> >
> > Not quite sure what your intent is here. Why change a stage's text like this?
Still need to do dynamic variable substitution, but not append things like "(Batch x out of y)"
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/#review169706
-----------------------------------------------------------
On March 22, 2017, 11:26 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57792/
> -----------------------------------------------------------
>
> (Updated March 22, 2017, 11:26 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-20490
> https://issues.apache.org/jira/browse/AMBARI-20490
>
>
> Repository: ambari
>
>
> Description
> -------
>
> On large-scale clusters over 1000 nodes, performance during Express Upgrade can be very slow and it takes more time to fix potential errors.
> We can remedy these with the following fixes:
>
> * "hdp-select set all" is currently ran in the ClusterGroup with a single stage instead of batching. This can cause Ambari Server to be the bottleneck when processing 1000+ requests. Allow the execute-stage inside a ClusterGroup to use batching.
>
> * If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it will prevent the hosts from running "hdp-select versions". We can find these problematic hosts with an alert.
>
> * There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no longer supported in Ambari 2.5).
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 20b4cbd
> ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java 8556436
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java ca9ce07
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java ee5451f
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java c45ec24
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java 8003223
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ParallelScheduler.java 6004a12
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java 669d50f
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java 3cf3f8b
> ambari-server/src/main/resources/alerts.json 2559b3a
> ambari-server/src/main/resources/host_scripts/alert_version_select.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 28ca939
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 9aff1a9
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml adccac0
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 3c0104b
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml cd63651
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 39cb0dc
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml 57a5203
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 4a641ec
> ambari-server/src/main/resources/upgrade-pack.xsd 1f11aa1
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java f1af66f
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java bf6c41a
> ambari-server/src/test/java/org/apache/ambari/server/metadata/AgentAlertDefinitionsTest.java cbc5e69
> ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java 8e5ad0a
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_batch.xml PRE-CREATION
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_to_new_stack.xml fe387f3
>
>
> Diff: https://reviews.apache.org/r/57792/diff/3/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 57792: Fixes for Express Upgrade on large-scale
clusters:
batch execute-stage for hdp-select set all, alert for hdp-select versions,
EU from HDP 2.2 to 2.5
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/#review169706
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java
Lines 511-514 (patched)
<https://reviews.apache.org/r/57792/#comment242226>
Not quite sure what your intent is here. Why change a stage's text like this?
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
Lines 311-314 (original), 290-312 (patched)
<https://reviews.apache.org/r/57792/#comment242225>
This code looks like a straight copy/paste out of Grouping.addTasksToStageInBatches(). It would be convenient to have a method in ParallelScheduler itself to do this work or abstract it some other way.
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java
Lines 40 (patched)
<https://reviews.apache.org/r/57792/#comment242230>
"...belong in this stage"
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java
Lines 41-42 (patched)
<https://reviews.apache.org/r/57792/#comment242232>
Can always formulate the HRC text based on the actual text, I don't see a need for the original text, since you're overwriting it anyway. It's not really the "original text" anymore.
ambari-server/src/main/resources/host_scripts/alert_version_select.py
Lines 72-73 (patched)
<https://reviews.apache.org/r/57792/#comment242222>
Nit: %s format is deprecated
ambari-server/src/main/resources/host_scripts/alert_version_select.py
Lines 82-83 (patched)
<https://reviews.apache.org/r/57792/#comment242221>
Nit: formatting
ambari-server/src/main/resources/upgrade-pack.xsd
Line 180 (original), 180-186 (patched)
<https://reviews.apache.org/r/57792/#comment242235>
This is a copy/paste. Make it it's own type and reference it in both places.
ambari-server/src/main/resources/upgrade-pack.xsd
Lines 187 (patched)
<https://reviews.apache.org/r/57792/#comment242236>
The code is one task per ExecuteStage, it's not a list. And it shouldn't be! There is more thought that has to be made before we make ExecuteStage support more than one task.
- Nate Cole
On March 20, 2017, 8:49 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57792/
> -----------------------------------------------------------
>
> (Updated March 20, 2017, 8:49 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-20490
> https://issues.apache.org/jira/browse/AMBARI-20490
>
>
> Repository: ambari
>
>
> Description
> -------
>
> On large-scale clusters over 1000 nodes, performance during Express Upgrade can be very slow and it takes more time to fix potential errors.
> We can remedy these with the following fixes:
>
> * "hdp-select set all" is currently ran in the ClusterGroup with a single stage instead of batching. This can cause Ambari Server to be the bottleneck when processing 1000+ requests. Allow the execute-stage inside a ClusterGroup to use batching.
>
> * If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it will prevent the hosts from running "hdp-select versions". We can find these problematic hosts with an alert.
>
> * There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no longer supported in Ambari 2.5).
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 20b4cbd
> ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java 8556436
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java ca9ce07
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java ee5451f
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java c45ec24
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java 8003223
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java 669d50f
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java 3cf3f8b
> ambari-server/src/main/resources/alerts.json 2559b3a
> ambari-server/src/main/resources/host_scripts/alert_version_select.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 28ca939
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 9aff1a9
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml adccac0
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 3c0104b
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml cd63651
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 39cb0dc
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml 57a5203
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 035cb59
> ambari-server/src/main/resources/upgrade-pack.xsd 1f11aa1
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java f1af66f
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java bf6c41a
> ambari-server/src/test/java/org/apache/ambari/server/metadata/AgentAlertDefinitionsTest.java cbc5e69
> ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java 8e5ad0a
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_batch.xml PRE-CREATION
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_to_new_stack.xml fe387f3
>
>
> Diff: https://reviews.apache.org/r/57792/diff/2/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 57792: Fixes for Express Upgrade on large-scale
clusters:
batch execute-stage for hdp-select set all, alert for hdp-select versions,
EU from HDP 2.2 to 2.5
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On March 22, 2017, 2:10 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
> > Lines 296 (patched)
> > <https://reviews.apache.org/r/57792/diff/2/?file=1670562#file1670562line342>
> >
> > Doesn't this mean split the original set into subsets of 1 each? When `numHostsPerBatch` is 0, then you should have a single set of all hosts, right? Why not just use a Collections.singleontList(hostNames)
You're right, should run in parallel by adding the existing hostnames set into the list.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/#review169707
-----------------------------------------------------------
On March 23, 2017, 12:08 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57792/
> -----------------------------------------------------------
>
> (Updated March 23, 2017, 12:08 a.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-20490
> https://issues.apache.org/jira/browse/AMBARI-20490
>
>
> Repository: ambari
>
>
> Description
> -------
>
> On large-scale clusters over 1000 nodes, performance during Express Upgrade can be very slow and it takes more time to fix potential errors.
> We can remedy these with the following fixes:
>
> * "hdp-select set all" is currently ran in the ClusterGroup with a single stage instead of batching. This can cause Ambari Server to be the bottleneck when processing 1000+ requests. Allow the execute-stage inside a ClusterGroup to use batching.
>
> * If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it will prevent the hosts from running "hdp-select versions". We can find these problematic hosts with an alert.
>
> * There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no longer supported in Ambari 2.5).
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 513ceac
> ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java 8556436
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java d37e32b
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java 3a2dc89
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java 80bb26c
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java be1f469
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ParallelScheduler.java 6004a12
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java 86a8f55
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java ec7279c
> ambari-server/src/main/resources/alerts.json 9cffff5
> ambari-server/src/main/resources/host_scripts/alert_version_select.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/config-upgrade.xml 0f5f48e
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.2.xml 46f0fee
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.3.xml a741722
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.4.xml 933183b
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.5.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 90f62dc
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml ecfc4d7
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml 68fb1d4
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 2e5c60e
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml 4e49097
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 3107bf3
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java 538fa48
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java 1ea9407
> ambari-server/src/test/java/org/apache/ambari/server/metadata/AgentAlertDefinitionsTest.java f5ce11e
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_batch.xml PRE-CREATION
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_to_new_stack.xml bfd994e
>
>
> Diff: https://reviews.apache.org/r/57792/diff/4/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 57792: Fixes for Express Upgrade on large-scale
clusters:
batch execute-stage for hdp-select set all, alert for hdp-select versions,
EU from HDP 2.2 to 2.5
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On March 22, 2017, 2:10 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml
> > Lines 307 (patched)
> > <https://reviews.apache.org/r/57792/diff/2/?file=1670568#file1670568line307>
> >
> > You're changing the behavior here to batch this by 100's ... is this what we want in an express upgrade?
Yes, this is what works for large clusters. Most users on less than 500 hosts should not be impacted.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/#review169707
-----------------------------------------------------------
On March 22, 2017, 11:26 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57792/
> -----------------------------------------------------------
>
> (Updated March 22, 2017, 11:26 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-20490
> https://issues.apache.org/jira/browse/AMBARI-20490
>
>
> Repository: ambari
>
>
> Description
> -------
>
> On large-scale clusters over 1000 nodes, performance during Express Upgrade can be very slow and it takes more time to fix potential errors.
> We can remedy these with the following fixes:
>
> * "hdp-select set all" is currently ran in the ClusterGroup with a single stage instead of batching. This can cause Ambari Server to be the bottleneck when processing 1000+ requests. Allow the execute-stage inside a ClusterGroup to use batching.
>
> * If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it will prevent the hosts from running "hdp-select versions". We can find these problematic hosts with an alert.
>
> * There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no longer supported in Ambari 2.5).
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 20b4cbd
> ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java 8556436
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java ca9ce07
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java ee5451f
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java c45ec24
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java 8003223
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ParallelScheduler.java 6004a12
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java 669d50f
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java 3cf3f8b
> ambari-server/src/main/resources/alerts.json 2559b3a
> ambari-server/src/main/resources/host_scripts/alert_version_select.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 28ca939
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 9aff1a9
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml adccac0
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 3c0104b
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml cd63651
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 39cb0dc
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml 57a5203
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 4a641ec
> ambari-server/src/main/resources/upgrade-pack.xsd 1f11aa1
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java f1af66f
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java bf6c41a
> ambari-server/src/test/java/org/apache/ambari/server/metadata/AgentAlertDefinitionsTest.java cbc5e69
> ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java 8e5ad0a
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_batch.xml PRE-CREATION
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_to_new_stack.xml fe387f3
>
>
> Diff: https://reviews.apache.org/r/57792/diff/3/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 57792: Fixes for Express Upgrade on large-scale
clusters:
batch execute-stage for hdp-select set all, alert for hdp-select versions,
EU from HDP 2.2 to 2.5
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/#review169707
-----------------------------------------------------------
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
Lines 296 (patched)
<https://reviews.apache.org/r/57792/#comment242227>
be -> by
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
Line 1357 (original), 1357 (patched)
<https://reviews.apache.org/r/57792/#comment242228>
Why do you want to include the host name on the command - the command is scoped to a host already.
ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java
Lines 511 (patched)
<https://reviews.apache.org/r/57792/#comment242229>
isBlank instead?
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
Lines 296 (patched)
<https://reviews.apache.org/r/57792/#comment242231>
Doesn't this mean split the original set into subsets of 1 each? When `numHostsPerBatch` is 0, then you should have a single set of all hosts, right? Why not just use a Collections.singleontList(hostNames)
ambari-server/src/main/resources/alerts.json
Lines 184 (patched)
<https://reviews.apache.org/r/57792/#comment242233>
prefix with ambari_agent to be consistent
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml
Lines 307 (patched)
<https://reviews.apache.org/r/57792/#comment242234>
You're changing the behavior here to batch this by 100's ... is this what we want in an express upgrade?
- Jonathan Hurley
On March 20, 2017, 8:49 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57792/
> -----------------------------------------------------------
>
> (Updated March 20, 2017, 8:49 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-20490
> https://issues.apache.org/jira/browse/AMBARI-20490
>
>
> Repository: ambari
>
>
> Description
> -------
>
> On large-scale clusters over 1000 nodes, performance during Express Upgrade can be very slow and it takes more time to fix potential errors.
> We can remedy these with the following fixes:
>
> * "hdp-select set all" is currently ran in the ClusterGroup with a single stage instead of batching. This can cause Ambari Server to be the bottleneck when processing 1000+ requests. Allow the execute-stage inside a ClusterGroup to use batching.
>
> * If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it will prevent the hosts from running "hdp-select versions". We can find these problematic hosts with an alert.
>
> * There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no longer supported in Ambari 2.5).
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 20b4cbd
> ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java 8556436
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java ca9ce07
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java ee5451f
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java c45ec24
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java 8003223
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java 669d50f
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java 3cf3f8b
> ambari-server/src/main/resources/alerts.json 2559b3a
> ambari-server/src/main/resources/host_scripts/alert_version_select.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 28ca939
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 9aff1a9
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml adccac0
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 3c0104b
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml cd63651
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 39cb0dc
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml 57a5203
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 035cb59
> ambari-server/src/main/resources/upgrade-pack.xsd 1f11aa1
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java f1af66f
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java bf6c41a
> ambari-server/src/test/java/org/apache/ambari/server/metadata/AgentAlertDefinitionsTest.java cbc5e69
> ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java 8e5ad0a
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_batch.xml PRE-CREATION
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_to_new_stack.xml fe387f3
>
>
> Diff: https://reviews.apache.org/r/57792/diff/2/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 57792: Fixes for Express Upgrade on large-scale
clusters:
batch execute-stage for hdp-select set all, alert for hdp-select versions,
EU from HDP 2.2 to 2.5
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/#review169784
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
Line 1357 (original), 1357 (patched)
<https://reviews.apache.org/r/57792/#comment242381>
For any tasks in the ClusterGroup, the current text is pretty generic and the user has to click the link in order to find out the hostname.
RestartGroup already has the hostname if scoped to 1 host, but not ClusterGroup.
ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java
Lines 511-514 (patched)
<https://reviews.apache.org/r/57792/#comment242383>
The text of a stage goes through 2 edits.
1. Replace dynamic variables like %direction
2. Append the batch number (Batch x of y)
When a stage has multiple hosts, I actually want to append "on host foo" on the original text + edit #1.
So I have to keep track of both strings.
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
Lines 311-314 (original), 290-312 (patched)
<https://reviews.apache.org/r/57792/#comment242387>
Good idea.
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java
Lines 41-42 (patched)
<https://reviews.apache.org/r/57792/#comment242388>
Only one override is happening to replace dynamic variables.
ambari-server/src/main/resources/host_scripts/alert_version_select.py
Lines 72-73 (patched)
<https://reviews.apache.org/r/57792/#comment242395>
Will fix.
ambari-server/src/main/resources/host_scripts/alert_version_select.py
Lines 82-83 (patched)
<https://reviews.apache.org/r/57792/#comment242396>
Will fix.
- Alejandro Fernandez
On March 21, 2017, 12:49 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57792/
> -----------------------------------------------------------
>
> (Updated March 21, 2017, 12:49 a.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-20490
> https://issues.apache.org/jira/browse/AMBARI-20490
>
>
> Repository: ambari
>
>
> Description
> -------
>
> On large-scale clusters over 1000 nodes, performance during Express Upgrade can be very slow and it takes more time to fix potential errors.
> We can remedy these with the following fixes:
>
> * "hdp-select set all" is currently ran in the ClusterGroup with a single stage instead of batching. This can cause Ambari Server to be the bottleneck when processing 1000+ requests. Allow the execute-stage inside a ClusterGroup to use batching.
>
> * If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it will prevent the hosts from running "hdp-select versions". We can find these problematic hosts with an alert.
>
> * There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no longer supported in Ambari 2.5).
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 20b4cbd
> ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java 8556436
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java ca9ce07
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java ee5451f
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java c45ec24
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java 8003223
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java 669d50f
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java 3cf3f8b
> ambari-server/src/main/resources/alerts.json 2559b3a
> ambari-server/src/main/resources/host_scripts/alert_version_select.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 28ca939
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 9aff1a9
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml adccac0
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 3c0104b
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml cd63651
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 39cb0dc
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml 57a5203
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 035cb59
> ambari-server/src/main/resources/upgrade-pack.xsd 1f11aa1
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java f1af66f
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java bf6c41a
> ambari-server/src/test/java/org/apache/ambari/server/metadata/AgentAlertDefinitionsTest.java cbc5e69
> ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java 8e5ad0a
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_batch.xml PRE-CREATION
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_to_new_stack.xml fe387f3
>
>
> Diff: https://reviews.apache.org/r/57792/diff/2/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 57792: Fixes for Express Upgrade on large-scale
clusters:
batch execute-stage for hdp-select set all, alert for hdp-select versions,
EU from HDP 2.2 to 2.5
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/
-----------------------------------------------------------
(Updated March 21, 2017, 12:49 a.m.)
Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
Changes
-------
Fixed unit tests
Bugs: AMBARI-20490
https://issues.apache.org/jira/browse/AMBARI-20490
Repository: ambari
Description
-------
On large-scale clusters over 1000 nodes, performance during Express Upgrade can be very slow and it takes more time to fix potential errors.
We can remedy these with the following fixes:
* "hdp-select set all" is currently ran in the ClusterGroup with a single stage instead of batching. This can cause Ambari Server to be the bottleneck when processing 1000+ requests. Allow the execute-stage inside a ClusterGroup to use batching.
* If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it will prevent the hosts from running "hdp-select versions". We can find these problematic hosts with an alert.
* There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no longer supported in Ambari 2.5).
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 20b4cbd
ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java 8556436
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java ca9ce07
ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java ee5451f
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java c45ec24
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java 8003223
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java 669d50f
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java 3cf3f8b
ambari-server/src/main/resources/alerts.json 2559b3a
ambari-server/src/main/resources/host_scripts/alert_version_select.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 28ca939
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 9aff1a9
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml adccac0
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 3c0104b
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml cd63651
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 39cb0dc
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml 57a5203
ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 035cb59
ambari-server/src/main/resources/upgrade-pack.xsd 1f11aa1
ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java f1af66f
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java bf6c41a
ambari-server/src/test/java/org/apache/ambari/server/metadata/AgentAlertDefinitionsTest.java cbc5e69
ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java 8e5ad0a
ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_batch.xml PRE-CREATION
ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_to_new_stack.xml fe387f3
Diff: https://reviews.apache.org/r/57792/diff/2/
Changes: https://reviews.apache.org/r/57792/diff/1-2/
Testing
-------
mvn clean test
Thanks,
Alejandro Fernandez
Re: Review Request 57792: Fixes for Express Upgrade on large-scale
clusters:
batch execute-stage for hdp-select set all, alert for hdp-select versions,
EU from HDP 2.2 to 2.5
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/#review169503
-----------------------------------------------------------
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
Lines 293 (patched)
<https://reviews.apache.org/r/57792/#comment241862>
This is needed so that the alert can find if "hdp-select versions" is broken.
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java
Line 222 (original)
<https://reviews.apache.org/r/57792/#comment241863>
This variable was unused.
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java
Lines 139 (patched)
<https://reviews.apache.org/r/57792/#comment241864>
Used so I can preserve the original text of the stage in order to create the name of the commands.
ambari-server/src/main/resources/alerts.json
Lines 184 (patched)
<https://reviews.apache.org/r/57792/#comment241865>
New alert definition.
ambari-server/src/main/resources/host_scripts/alert_version_select.py
Lines 89 (patched)
<https://reviews.apache.org/r/57792/#comment241866>
New alert to detect if "hdp-select versions" won't work because of incorrect symlinks/dirs in /usr/hdp.
- Alejandro Fernandez
On March 20, 2017, 8:55 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57792/
> -----------------------------------------------------------
>
> (Updated March 20, 2017, 8:55 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-20490
> https://issues.apache.org/jira/browse/AMBARI-20490
>
>
> Repository: ambari
>
>
> Description
> -------
>
> On large-scale clusters over 1000 nodes, performance during Express Upgrade can be very slow and it takes more time to fix potential errors.
> We can remedy these with the following fixes:
>
> * "hdp-select set all" is currently ran in the ClusterGroup with a single stage instead of batching. This can cause Ambari Server to be the bottleneck when processing 1000+ requests. Allow the execute-stage inside a ClusterGroup to use batching.
>
> * If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it will prevent the hosts from running "hdp-select versions". We can find these problematic hosts with an alert.
>
> * There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no longer supported in Ambari 2.5).
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py 20b4cbd
> ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java 8556436
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java ca9ce07
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java ee5451f
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java c45ec24
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java 8003223
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java 669d50f
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java 3cf3f8b
> ambari-server/src/main/resources/alerts.json 2559b3a
> ambari-server/src/main/resources/host_scripts/alert_version_select.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 28ca939
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 9aff1a9
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml adccac0
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 3c0104b
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml cd63651
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 39cb0dc
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml 57a5203
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 035cb59
> ambari-server/src/main/resources/upgrade-pack.xsd 1f11aa1
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java f1af66f
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java bf6c41a
> ambari-server/src/test/java/org/apache/ambari/server/metadata/AgentAlertDefinitionsTest.java cbc5e69
> ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java 8e5ad0a
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_batch.xml PRE-CREATION
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_to_new_stack.xml fe387f3
>
>
> Diff: https://reviews.apache.org/r/57792/diff/1/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Alejandro Fernandez
>
>