You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Sebastian Toader <st...@hortonworks.com> on 2016/02/16 22:39:25 UTC
Review Request 43621: Decrease the load on ambari database after
cluster creation
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43621/
-----------------------------------------------------------
Review request for Ambari, Myroslav Papirkovskyy, Sumit Mohanty, and Sid Wagle.
Bugs: AMBARI-15011
https://issues.apache.org/jira/browse/AMBARI-15011
Repository: ambari
Description
-------
1. Created new index on host_role_command table on (status, role) fields as this table is being queries quite often by status and role
2. Removed uneccessary querying of ambari server actions by hostname
3. Cache HostRoleCommandStatusSummaryDTO computed objects using guava cache
4. The method returning service configs made one call to the database to get the active service config entities. Then retrieved from database all service config entities and while building the response it was using the first collection to determine which reponse item should be marked as active/inactive. This has been modified to make only one roundtrip to the database and determine the active flag using plain java code.
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java 91f2d30
ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java d2fe4fc
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java bd5fb5a
ambari-server/src/main/java/org/apache/ambari/server/serveraction/ServerActionExecutor.java 20cf5bb
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java d3fdc65
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java 88b3151
ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql eded221
ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 780e81a
ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql d4a1ddb
ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 64f2a4e
ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql c557cf6
ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql adda30c
ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java bc4d397
ambari-server/src/test/java/org/apache/ambari/server/agent/AgentResourceTest.java 510e1fb
ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java 1f90813
ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java 862776f
ambari-server/src/test/java/org/apache/ambari/server/state/ConfigHelperTest.java 0cdf50a
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog222Test.java aa4090e
ambari-server/src/test/java/org/apache/ambari/server/utils/StageUtilsTest.java 2d663d9
Diff: https://reviews.apache.org/r/43621/diff/
Testing
-------
Manual testing:
Created a cluster of 3 nodes waited 30 mins with UI running and collected postgres sql statement stats, than left the cluster running for another 30 mins wiht UI closed and getting new sql statement stats. Verified that the number of execution of queries mentioned in the JIRA has decreased.
Unit tests:
Total run:884
Total errors:0
Total failures:0
OK
Thanks,
Sebastian Toader
Re: Review Request 43621: Decrease the load on ambari database after
cluster creation
Posted by Sid Wagle <sw...@hortonworks.com>.
> On Feb. 16, 2016, 11:41 p.m., Sid Wagle wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java, line 2433
> > <https://reviews.apache.org/r/43621/diff/1/?file=1251923#file1251923line2433>
> >
> > Why not store state in the DB and query active only?
>
> Sebastian Toader wrote:
> I don't know the reason why the active state state was never stored in the db. I'm just guessing that since this active flag is not used by the server it just needed by the client (eg. UI) of the server this was never stored in the db. The state is in the db already in a way (version and create time stamp) and the most recent records are considered the active ones. Having a separate active column is kind of redundant.
It would certainly simplfy the query on already computed state.
- Sid
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43621/#review119375
-----------------------------------------------------------
On Feb. 17, 2016, 2:32 p.m., Sebastian Toader wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43621/
> -----------------------------------------------------------
>
> (Updated Feb. 17, 2016, 2:32 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-15011
> https://issues.apache.org/jira/browse/AMBARI-15011
>
>
> Repository: ambari
>
>
> Description
> -------
>
> 1. Created new index on host_role_command table on (status, role) fields as this table is being queries quite often by status and role
> 2. Removed uneccessary querying of ambari server actions by hostname
> 3. Cache HostRoleCommandStatusSummaryDTO computed objects using guava cache
> 4. The method returning service configs made one call to the database to get the active service config entities. Then retrieved from database all service config entities and while building the response it was using the first collection to determine which reponse item should be marked as active/inactive. This has been modified to make only one roundtrip to the database and determine the active flag using plain java code.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java 91f2d30
> ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java d2fe4fc
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java bd5fb5a
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/ServerActionExecutor.java 20cf5bb
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java d3fdc65
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java 88b3151
> ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql eded221
> ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 780e81a
> ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql d4a1ddb
> ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 64f2a4e
> ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql c557cf6
> ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql adda30c
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java bc4d397
> ambari-server/src/test/java/org/apache/ambari/server/agent/AgentResourceTest.java 510e1fb
> ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java 1f90813
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java 862776f
> ambari-server/src/test/java/org/apache/ambari/server/state/ConfigHelperTest.java 0cdf50a
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog222Test.java aa4090e
> ambari-server/src/test/java/org/apache/ambari/server/utils/StageUtilsTest.java 2d663d9
>
> Diff: https://reviews.apache.org/r/43621/diff/
>
>
> Testing
> -------
>
> Manual testing:
>
> Created a cluster of 3 nodes waited 30 mins with UI running and collected postgres sql statement stats, than left the cluster running for another 30 mins wiht UI closed and getting new sql statement stats. Verified that the number of execution of queries mentioned in the JIRA has decreased.
>
> Unit tests:
>
> Total run:884
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Sebastian Toader
>
>
Re: Review Request 43621: Decrease the load on ambari database after
cluster creation
Posted by Sebastian Toader <st...@hortonworks.com>.
> On Feb. 17, 2016, 12:41 a.m., Sid Wagle wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java, line 358
> > <https://reviews.apache.org/r/43621/diff/1/?file=1251919#file1251919line358>
> >
> > Could you provide some guidance of size of the DTO object as a *comment* here to help gugage the size of the cache relative to memory available.
The values in the cache are HostRoleCommandStatusSummaryDTO objects grouped by stage id. The approx size of an entry in the cache is: 480 * nr_of_stages_in_request + 4 * nr_of_tasks. (I used this http://java-performance.info/memory-consumption-of-java-data-types-2/ as reference to estimate the footprint of HashMaps)
Let's assume that there is 15000 node cluster that is being provisioned using Blueprint (this is one provision cluster request). That will create 15000 stages. Let's assume that we install all HDP services on each host (23 service * 5 avg components * 2 install_and_start tasks => roughly 200 tasks). I think we can consider the provision cluster request via Blueprints being the largest request that will yield the largest entry into the cache: 480 * 15000 + 4 * (200 * 15000) => 18 MB one entry in the cache. Most of the requests (like restart all components on a host, restart all components of a service etc) will result in a much smaller entry in the cache ~10 KB
Hope this makes sense and my estimation is correct.
> On Feb. 17, 2016, 12:41 a.m., Sid Wagle wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java, line 2433
> > <https://reviews.apache.org/r/43621/diff/1/?file=1251923#file1251923line2433>
> >
> > Why not store state in the DB and query active only?
I don't know the reason why the active state state was never stored in the db. I'm just guessing that since this active flag is not used by the server it just needed by the client (eg. UI) of the server this was never stored in the db. The state is in the db already in a way (version and create time stamp) and the most recent records are considered the active ones. Having a separate active column is kind of redundant.
> On Feb. 17, 2016, 12:41 a.m., Sid Wagle wrote:
> > ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql, line 656
> > <https://reviews.apache.org/r/43621/diff/1/?file=1251925#file1251925line656>
> >
> > Status and role are not unique enough for an index, what sort of gain does this buy us ?
The most often executed queries against host_role_command table are:
SELECT DISTINCT task_id FROM host_role_command WHERE ((role = @P0) AND (status = @P1)) ORDER BY task_id
SELECT COUNT(task_id) FROM host_role_command WHERE (status IN (@P0,@P1,@P2,@P3,@P4,@P5))
This index will cover these two queries. Also many of the other queries against this table filter by status and role so these two fields seems to be common accros the quries.
- Sebastian
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43621/#review119375
-----------------------------------------------------------
On Feb. 16, 2016, 10:39 p.m., Sebastian Toader wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43621/
> -----------------------------------------------------------
>
> (Updated Feb. 16, 2016, 10:39 p.m.)
>
>
> Review request for Ambari, Myroslav Papirkovskyy, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-15011
> https://issues.apache.org/jira/browse/AMBARI-15011
>
>
> Repository: ambari
>
>
> Description
> -------
>
> 1. Created new index on host_role_command table on (status, role) fields as this table is being queries quite often by status and role
> 2. Removed uneccessary querying of ambari server actions by hostname
> 3. Cache HostRoleCommandStatusSummaryDTO computed objects using guava cache
> 4. The method returning service configs made one call to the database to get the active service config entities. Then retrieved from database all service config entities and while building the response it was using the first collection to determine which reponse item should be marked as active/inactive. This has been modified to make only one roundtrip to the database and determine the active flag using plain java code.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java 91f2d30
> ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java d2fe4fc
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java bd5fb5a
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/ServerActionExecutor.java 20cf5bb
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java d3fdc65
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java 88b3151
> ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql eded221
> ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 780e81a
> ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql d4a1ddb
> ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 64f2a4e
> ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql c557cf6
> ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql adda30c
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java bc4d397
> ambari-server/src/test/java/org/apache/ambari/server/agent/AgentResourceTest.java 510e1fb
> ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java 1f90813
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java 862776f
> ambari-server/src/test/java/org/apache/ambari/server/state/ConfigHelperTest.java 0cdf50a
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog222Test.java aa4090e
> ambari-server/src/test/java/org/apache/ambari/server/utils/StageUtilsTest.java 2d663d9
>
> Diff: https://reviews.apache.org/r/43621/diff/
>
>
> Testing
> -------
>
> Manual testing:
>
> Created a cluster of 3 nodes waited 30 mins with UI running and collected postgres sql statement stats, than left the cluster running for another 30 mins wiht UI closed and getting new sql statement stats. Verified that the number of execution of queries mentioned in the JIRA has decreased.
>
> Unit tests:
>
> Total run:884
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Sebastian Toader
>
>
Re: Review Request 43621: Decrease the load on ambari database after
cluster creation
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43621/#review119375
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java (line 358)
<https://reviews.apache.org/r/43621/#comment180697>
Could you provide some guidance of size of the DTO object as a *comment* here to help gugage the size of the cache relative to memory available.
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java (line 2432)
<https://reviews.apache.org/r/43621/#comment180716>
Why not store state in the DB and query active only?
ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql (line 656)
<https://reviews.apache.org/r/43621/#comment180696>
Status and role are not unique enough for an index, what sort of gain does this buy us ?
- Sid Wagle
On Feb. 16, 2016, 9:39 p.m., Sebastian Toader wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43621/
> -----------------------------------------------------------
>
> (Updated Feb. 16, 2016, 9:39 p.m.)
>
>
> Review request for Ambari, Myroslav Papirkovskyy, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-15011
> https://issues.apache.org/jira/browse/AMBARI-15011
>
>
> Repository: ambari
>
>
> Description
> -------
>
> 1. Created new index on host_role_command table on (status, role) fields as this table is being queries quite often by status and role
> 2. Removed uneccessary querying of ambari server actions by hostname
> 3. Cache HostRoleCommandStatusSummaryDTO computed objects using guava cache
> 4. The method returning service configs made one call to the database to get the active service config entities. Then retrieved from database all service config entities and while building the response it was using the first collection to determine which reponse item should be marked as active/inactive. This has been modified to make only one roundtrip to the database and determine the active flag using plain java code.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java 91f2d30
> ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java d2fe4fc
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java bd5fb5a
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/ServerActionExecutor.java 20cf5bb
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java d3fdc65
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java 88b3151
> ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql eded221
> ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 780e81a
> ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql d4a1ddb
> ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 64f2a4e
> ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql c557cf6
> ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql adda30c
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java bc4d397
> ambari-server/src/test/java/org/apache/ambari/server/agent/AgentResourceTest.java 510e1fb
> ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java 1f90813
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java 862776f
> ambari-server/src/test/java/org/apache/ambari/server/state/ConfigHelperTest.java 0cdf50a
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog222Test.java aa4090e
> ambari-server/src/test/java/org/apache/ambari/server/utils/StageUtilsTest.java 2d663d9
>
> Diff: https://reviews.apache.org/r/43621/diff/
>
>
> Testing
> -------
>
> Manual testing:
>
> Created a cluster of 3 nodes waited 30 mins with UI running and collected postgres sql statement stats, than left the cluster running for another 30 mins wiht UI closed and getting new sql statement stats. Verified that the number of execution of queries mentioned in the JIRA has decreased.
>
> Unit tests:
>
> Total run:884
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Sebastian Toader
>
>