You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Robert Nettleton <rn...@hortonworks.com> on 2015/06/10 22:43:20 UTC
Review Request 35322: Blueprint export performance enhancements
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35322/
-----------------------------------------------------------
Review request for Ambari, John Speidel, Mahadev Konar, Robert Levas, Sumit Mohanty, and Yusaku Sako.
Bugs: AMBARI-11850
https://issues.apache.org/jira/browse/AMBARI-11850
Repository: ambari
Description
-------
This patch resolves the performance issues detailed in AMBARI-11850.
The Blueprint export process was taking a long time, due to the fact that the PropertyProvider instances associated with Ambari Metrics and Alerting always execute for the Cluster REST resource. In the case of a Blueprint export, the Cluster resource is accessed with a different format, and a different renderer that creates the Blueprint based on the Cluster resource's data. The Blueprint process does not require the Metrics or Alerting data, so the extra processing time is wasted for this particular case. For very large clusters, this process can take over 30 minutes or more.
The performance problems have been resolved by the following changes:
1. Added a new method to the Renderer interface, so that custom renderer implementations can indicate if the property provider support is needed.
2. The BaseRenderer's implementation of this method always returns true, to keep the behavior for the rest of the renderers as it currently is, with the exception of the ClusterBlueprintRenderer. In the future, other renderers may be modified to take advantage of this optimization, depending upon the functionality required.
3. The ClusterBlueprintRenderer's implementation of this method always returns false. This indicates that the Blueprint renderer does not require the additional data provided by AMS or Alerting during a Blueprint export. Previously, the processing of this data was happening regardless of the Renderer type.
4. Updated the QueryImpl class to skip the call to ClusterController.populateResources() if the Renderer does not require property provider supoprt. This check occurs for resources and any associated sub-resources.
5. Adds unit tests to verify this change.
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/api/query/QueryImpl.java 2319683
ambari-server/src/main/java/org/apache/ambari/server/api/query/render/BaseRenderer.java 7866aa4
ambari-server/src/main/java/org/apache/ambari/server/api/query/render/ClusterBlueprintRenderer.java cfc9bc0
ambari-server/src/main/java/org/apache/ambari/server/api/query/render/Renderer.java f353d53
ambari-server/src/test/java/org/apache/ambari/server/api/query/QueryImplTest.java 01361d2
ambari-server/src/test/java/org/apache/ambari/server/api/query/render/ClusterBlueprintRendererTest.java 96abb8c
ambari-server/src/test/java/org/apache/ambari/server/api/query/render/DefaultRendererTest.java 4981a67
Diff: https://reviews.apache.org/r/35322/diff/
Testing
-------
1. Ran the ambari-server unit tests (all passing).
2. Ran basic timing tests against a Blueprint export with this patch applied. Previously, the export for a 3-node cluster was taking between 7-10 seconds. With this new patch applied, the Blueprint export takes roughly 50 milliseconds or less on average. I used a timed "curl" command to obtain this data.
3. Took the Blueprint exported from the test in #2, and used this Blueprint to deploy a brand new cluster, to verify that the Blueprint was valid. The cluster deployment succeeded. I had to make a slight change to the Blueprint, to add command retry configuration, but this is un-related to the performance changes. The exported Blueprint will POST correctly without any manual changes. The cluster-env changes are just to active the new command retry feature for multi-node Blueprint deployments.
Thanks,
Robert Nettleton
Re: Review Request 35322: Blueprint export performance enhancements
Posted by Robert Levas <rl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35322/#review87469
-----------------------------------------------------------
Ship it!
Ship It!
- Robert Levas
On June 10, 2015, 4:43 p.m., Robert Nettleton wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35322/
> -----------------------------------------------------------
>
> (Updated June 10, 2015, 4:43 p.m.)
>
>
> Review request for Ambari, John Speidel, Mahadev Konar, Robert Levas, Sumit Mohanty, and Yusaku Sako.
>
>
> Bugs: AMBARI-11850
> https://issues.apache.org/jira/browse/AMBARI-11850
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This patch resolves the performance issues detailed in AMBARI-11850.
>
> The Blueprint export process was taking a long time, due to the fact that the PropertyProvider instances associated with Ambari Metrics and Alerting always execute for the Cluster REST resource. In the case of a Blueprint export, the Cluster resource is accessed with a different format, and a different renderer that creates the Blueprint based on the Cluster resource's data. The Blueprint process does not require the Metrics or Alerting data, so the extra processing time is wasted for this particular case. For very large clusters, this process can take over 30 minutes or more.
>
> The performance problems have been resolved by the following changes:
>
> 1. Added a new method to the Renderer interface, so that custom renderer implementations can indicate if the property provider support is needed.
> 2. The BaseRenderer's implementation of this method always returns true, to keep the behavior for the rest of the renderers as it currently is, with the exception of the ClusterBlueprintRenderer. In the future, other renderers may be modified to take advantage of this optimization, depending upon the functionality required.
> 3. The ClusterBlueprintRenderer's implementation of this method always returns false. This indicates that the Blueprint renderer does not require the additional data provided by AMS or Alerting during a Blueprint export. Previously, the processing of this data was happening regardless of the Renderer type.
> 4. Updated the QueryImpl class to skip the call to ClusterController.populateResources() if the Renderer does not require property provider supoprt. This check occurs for resources and any associated sub-resources.
> 5. Adds unit tests to verify this change.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/api/query/QueryImpl.java 2319683
> ambari-server/src/main/java/org/apache/ambari/server/api/query/render/BaseRenderer.java 7866aa4
> ambari-server/src/main/java/org/apache/ambari/server/api/query/render/ClusterBlueprintRenderer.java cfc9bc0
> ambari-server/src/main/java/org/apache/ambari/server/api/query/render/Renderer.java f353d53
> ambari-server/src/test/java/org/apache/ambari/server/api/query/QueryImplTest.java 01361d2
> ambari-server/src/test/java/org/apache/ambari/server/api/query/render/ClusterBlueprintRendererTest.java 96abb8c
> ambari-server/src/test/java/org/apache/ambari/server/api/query/render/DefaultRendererTest.java 4981a67
>
> Diff: https://reviews.apache.org/r/35322/diff/
>
>
> Testing
> -------
>
> 1. Ran the ambari-server unit tests (all passing).
> 2. Ran basic timing tests against a Blueprint export with this patch applied. Previously, the export for a 3-node cluster was taking between 7-10 seconds. With this new patch applied, the Blueprint export takes roughly 50 milliseconds or less on average. I used a timed "curl" command to obtain this data.
> 3. Took the Blueprint exported from the test in #2, and used this Blueprint to deploy a brand new cluster, to verify that the Blueprint was valid. The cluster deployment succeeded. I had to make a slight change to the Blueprint, to add command retry configuration, but this is un-related to the performance changes. The exported Blueprint will POST correctly without any manual changes. The cluster-env changes are just to active the new command retry feature for multi-node Blueprint deployments.
>
>
> Thanks,
>
> Robert Nettleton
>
>
Re: Review Request 35322: Blueprint export performance enhancements
Posted by John Speidel <js...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35322/#review87500
-----------------------------------------------------------
Ship it!
Ship It!
- John Speidel
On June 10, 2015, 8:43 p.m., Robert Nettleton wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35322/
> -----------------------------------------------------------
>
> (Updated June 10, 2015, 8:43 p.m.)
>
>
> Review request for Ambari, John Speidel, Mahadev Konar, Robert Levas, Sumit Mohanty, and Yusaku Sako.
>
>
> Bugs: AMBARI-11850
> https://issues.apache.org/jira/browse/AMBARI-11850
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This patch resolves the performance issues detailed in AMBARI-11850.
>
> The Blueprint export process was taking a long time, due to the fact that the PropertyProvider instances associated with Ambari Metrics and Alerting always execute for the Cluster REST resource. In the case of a Blueprint export, the Cluster resource is accessed with a different format, and a different renderer that creates the Blueprint based on the Cluster resource's data. The Blueprint process does not require the Metrics or Alerting data, so the extra processing time is wasted for this particular case. For very large clusters, this process can take over 30 minutes or more.
>
> The performance problems have been resolved by the following changes:
>
> 1. Added a new method to the Renderer interface, so that custom renderer implementations can indicate if the property provider support is needed.
> 2. The BaseRenderer's implementation of this method always returns true, to keep the behavior for the rest of the renderers as it currently is, with the exception of the ClusterBlueprintRenderer. In the future, other renderers may be modified to take advantage of this optimization, depending upon the functionality required.
> 3. The ClusterBlueprintRenderer's implementation of this method always returns false. This indicates that the Blueprint renderer does not require the additional data provided by AMS or Alerting during a Blueprint export. Previously, the processing of this data was happening regardless of the Renderer type.
> 4. Updated the QueryImpl class to skip the call to ClusterController.populateResources() if the Renderer does not require property provider supoprt. This check occurs for resources and any associated sub-resources.
> 5. Adds unit tests to verify this change.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/api/query/QueryImpl.java 2319683
> ambari-server/src/main/java/org/apache/ambari/server/api/query/render/BaseRenderer.java 7866aa4
> ambari-server/src/main/java/org/apache/ambari/server/api/query/render/ClusterBlueprintRenderer.java cfc9bc0
> ambari-server/src/main/java/org/apache/ambari/server/api/query/render/Renderer.java f353d53
> ambari-server/src/test/java/org/apache/ambari/server/api/query/QueryImplTest.java 01361d2
> ambari-server/src/test/java/org/apache/ambari/server/api/query/render/ClusterBlueprintRendererTest.java 96abb8c
> ambari-server/src/test/java/org/apache/ambari/server/api/query/render/DefaultRendererTest.java 4981a67
>
> Diff: https://reviews.apache.org/r/35322/diff/
>
>
> Testing
> -------
>
> 1. Ran the ambari-server unit tests (all passing).
> 2. Ran basic timing tests against a Blueprint export with this patch applied. Previously, the export for a 3-node cluster was taking between 7-10 seconds. With this new patch applied, the Blueprint export takes roughly 50 milliseconds or less on average. I used a timed "curl" command to obtain this data.
> 3. Took the Blueprint exported from the test in #2, and used this Blueprint to deploy a brand new cluster, to verify that the Blueprint was valid. The cluster deployment succeeded. I had to make a slight change to the Blueprint, to add command retry configuration, but this is un-related to the performance changes. The exported Blueprint will POST correctly without any manual changes. The cluster-env changes are just to active the new command retry feature for multi-node Blueprint deployments.
>
>
> Thanks,
>
> Robert Nettleton
>
>