You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Dmitro Lisnichenko <dl...@hortonworks.com> on 2014/08/26 14:48:53 UTC
Review Request 25062: OutOfMemoryError during host checks on 2k nodes
cluster
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25062/
-----------------------------------------------------------
Review request for Ambari, Andrew Onischuk and Myroslav Papirkovskyy.
Bugs: AMBARI-6061
https://issues.apache.org/jira/browse/AMBARI-6061
Repository: ambari
Description
-------
Host checks are executed as custom actions. But each task contains list of all hosts in roleParams field of execution command.
On 2k node cluster this results in 170kB overhead for each task and 170kB more after deserialization, which seem to be enough to cause OutOfMemory error.
Workaround is to increase max heap size to 4096m.
For tasks generated by server similar trouble was resolved by moving clusterHostInfo to Stage level.
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java b9a67b7
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/Stage.java c4bbb46
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/StageFactory.java fab2e96
ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java fdf96df
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java 64f132e
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 339194f
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 3b80515
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java 245c9e9
ambari-server/src/main/java/org/apache/ambari/server/stageplanner/RoleGraph.java 913b878
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog170.java eeba932
ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java 7160bcf
ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 29fa041
ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 457404d
ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql b58fd21
ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 664bbe9
ambari-server/src/test/java/org/apache/ambari/server/actionmanager/ExecutionCommandWrapperTest.java 034ac5b
ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionDBAccessorImpl.java 2850897
ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionManager.java 39bc762
ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java a536bef
ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestStage.java ba6286f
ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java 5c4a4f1
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java 3aafb9a
ambari-server/src/test/java/org/apache/ambari/server/stageplanner/TestStagePlanner.java cf1e412
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog170Test.java 13cb231
ambari-server/src/test/java/org/apache/ambari/server/utils/TestStageUtils.java 6e587f1
ambari-web/app/controllers/wizard/step3_controller.js 88a328e
Diff: https://reviews.apache.org/r/25062/diff/
Testing
-------
[INFO]
[INFO] Ambari Views ...................................... SUCCESS [8.805s]
[INFO] Ambari Server ..................................... SUCCESS [24:48.010s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 24:59.801s
[INFO] Finished at: Tue Aug 26 15:28:41 FET 2014
[INFO] Final Memory: 23M/256M
Thanks,
Dmitro Lisnichenko
Re: Review Request 25062: OutOfMemoryError during host checks on 2k
nodes cluster
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25062/#review51523
-----------------------------------------------------------
Ship it!
Ship It!
- Andrew Onischuk
On Aug. 26, 2014, 12:48 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25062/
> -----------------------------------------------------------
>
> (Updated Aug. 26, 2014, 12:48 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk and Myroslav Papirkovskyy.
>
>
> Bugs: AMBARI-6061
> https://issues.apache.org/jira/browse/AMBARI-6061
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Host checks are executed as custom actions. But each task contains list of all hosts in roleParams field of execution command.
> On 2k node cluster this results in 170kB overhead for each task and 170kB more after deserialization, which seem to be enough to cause OutOfMemory error.
> Workaround is to increase max heap size to 4096m.
> For tasks generated by server similar trouble was resolved by moving clusterHostInfo to Stage level.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java b9a67b7
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/Stage.java c4bbb46
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/StageFactory.java fab2e96
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java fdf96df
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java 64f132e
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 339194f
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 3b80515
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java 245c9e9
> ambari-server/src/main/java/org/apache/ambari/server/stageplanner/RoleGraph.java 913b878
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog170.java eeba932
> ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java 7160bcf
> ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 29fa041
> ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 457404d
> ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql b58fd21
> ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 664bbe9
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/ExecutionCommandWrapperTest.java 034ac5b
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionDBAccessorImpl.java 2850897
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionManager.java 39bc762
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java a536bef
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestStage.java ba6286f
> ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java 5c4a4f1
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java 3aafb9a
> ambari-server/src/test/java/org/apache/ambari/server/stageplanner/TestStagePlanner.java cf1e412
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog170Test.java 13cb231
> ambari-server/src/test/java/org/apache/ambari/server/utils/TestStageUtils.java 6e587f1
> ambari-web/app/controllers/wizard/step3_controller.js 88a328e
>
> Diff: https://reviews.apache.org/r/25062/diff/
>
>
> Testing
> -------
>
> [INFO]
> [INFO] Ambari Views ...................................... SUCCESS [8.805s]
> [INFO] Ambari Server ..................................... SUCCESS [24:48.010s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 24:59.801s
> [INFO] Finished at: Tue Aug 26 15:28:41 FET 2014
> [INFO] Final Memory: 23M/256M
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 25062: OutOfMemoryError during host checks on 2k
nodes cluster
Posted by Myroslav Papirkovskyy <mp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25062/#review51525
-----------------------------------------------------------
Ship it!
Ship It!
- Myroslav Papirkovskyy
On Сер. 26, 2014, 3:48 після полудня, Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25062/
> -----------------------------------------------------------
>
> (Updated Сер. 26, 2014, 3:48 після полудня)
>
>
> Review request for Ambari, Andrew Onischuk and Myroslav Papirkovskyy.
>
>
> Bugs: AMBARI-6061
> https://issues.apache.org/jira/browse/AMBARI-6061
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Host checks are executed as custom actions. But each task contains list of all hosts in roleParams field of execution command.
> On 2k node cluster this results in 170kB overhead for each task and 170kB more after deserialization, which seem to be enough to cause OutOfMemory error.
> Workaround is to increase max heap size to 4096m.
> For tasks generated by server similar trouble was resolved by moving clusterHostInfo to Stage level.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java b9a67b7
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/Stage.java c4bbb46
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/StageFactory.java fab2e96
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java fdf96df
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java 64f132e
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 339194f
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 3b80515
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java 245c9e9
> ambari-server/src/main/java/org/apache/ambari/server/stageplanner/RoleGraph.java 913b878
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog170.java eeba932
> ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java 7160bcf
> ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 29fa041
> ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 457404d
> ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql b58fd21
> ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 664bbe9
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/ExecutionCommandWrapperTest.java 034ac5b
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionDBAccessorImpl.java 2850897
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionManager.java 39bc762
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java a536bef
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestStage.java ba6286f
> ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java 5c4a4f1
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java 3aafb9a
> ambari-server/src/test/java/org/apache/ambari/server/stageplanner/TestStagePlanner.java cf1e412
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog170Test.java 13cb231
> ambari-server/src/test/java/org/apache/ambari/server/utils/TestStageUtils.java 6e587f1
> ambari-web/app/controllers/wizard/step3_controller.js 88a328e
>
> Diff: https://reviews.apache.org/r/25062/diff/
>
>
> Testing
> -------
>
> [INFO]
> [INFO] Ambari Views ...................................... SUCCESS [8.805s]
> [INFO] Ambari Server ..................................... SUCCESS [24:48.010s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 24:59.801s
> [INFO] Finished at: Tue Aug 26 15:28:41 FET 2014
> [INFO] Final Memory: 23M/256M
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>