You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Dmitro Lisnichenko <dl...@hortonworks.com> on 2014/01/16 16:40:26 UTC
Review Request 16970: Task timeout should be a configurable knob at the
ambari-server
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16970/
-----------------------------------------------------------
Review request for Ambari, Dmytro Sen and Sumit Mohanty.
Bugs: AMBARI-4319
https://issues.apache.org/jira/browse/AMBARI-4319
Repository: ambari
Description
-------
full implementation
Also tested overriding timeout at metainfo.xml and autokilling tasks that exceed timeout
Diffs
-----
ambari-agent/conf/unix/ambari-agent.ini 95944d2
ambari-agent/src/main/python/ambari_agent/PuppetExecutor.py 00c5c98
ambari-agent/src/main/python/ambari_agent/test.json 6da53da
ambari-agent/src/test/python/ambari_agent/TestPuppetExecutor.py 9e6da16
ambari-server/conf/unix/ambari.properties 563c04f
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 47f56eb
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/Stage.java c0272ec
ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java 7bd2b41
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatMonitor.java ee6826a
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java 6af54c0
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 6548be8
ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java b4d96ee
ambari-server/src/main/java/org/apache/ambari/server/state/CommandScriptDefinition.java fd06385
ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java 6f7f465
ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestStage.java 6f74d0a
ambari-server/src/test/java/org/apache/ambari/server/api/util/StackExtensionHelperTest.java 5a3ab18
ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HBASE/metainfo.xml 0605952
Diff: https://reviews.apache.org/r/16970/diff/
Testing
-------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ....................................... SUCCESS [0.335s]
[INFO] Apache Ambari Project POM ......................... SUCCESS [0.040s]
[INFO] Ambari Web ........................................ SUCCESS [5.969s]
[INFO] Ambari Server ..................................... SUCCESS [9:55.335s]
[INFO] Ambari Agent ...................................... SUCCESS [15.767s]
[INFO] Ambari Client ..................................... SUCCESS [0.409s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10:18.405s
[INFO] Finished at: Thu Jan 16 18:25:08 FET 2014
[INFO] Final Memory: 24M/284M
[INFO] ------------------------------------------------------------------------
Thanks,
Dmitro Lisnichenko
Re: Review Request 16970: Task timeout should be a configurable knob at the
ambari-server
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
> On Jan. 16, 2014, 4:23 p.m., Sumit Mohanty wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/actionmanager/Stage.java
> > public synchronized int getStageTimeout()
> >
> > Considering all tasks execute in parallel for a stage - should we be adding them up?
> >
> > ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestStage.java
> >
> > Why its 3 times 600 second?
> >
> > assertEquals(3*600000, s.getStageTimeout());
> >
> > Alternatively, can you explain how timeouts work say for a Request that has two stages and the stages have 2 and 3 tasks respectively.
regarding getStageTimeout():
all tasks execute in parallel on different hosts. But on the same hosts they execute sequentially. That's why we iterate over all hosts and select maximum per-host sum of timeouts. This sum becomes a timeout for stage (maximum theoretical time that tasks in this stage may take until they are killed).
Regarding test:
we define stage consisting of 3 tasks at 1 host (1 is added at getATestStage() and 2 tasks are added explicitly). For every task, we define timeout of 600 seconds, so the summary timeout for stage is 3*600 seconds.
- Dmitro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16970/#review32035
-----------------------------------------------------------
On Jan. 16, 2014, 3:40 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16970/
> -----------------------------------------------------------
>
> (Updated Jan. 16, 2014, 3:40 p.m.)
>
>
> Review request for Ambari, Dmytro Sen and Sumit Mohanty.
>
>
> Bugs: AMBARI-4319
> https://issues.apache.org/jira/browse/AMBARI-4319
>
>
> Repository: ambari
>
>
> Description
> -------
>
> full implementation
> Also tested overriding timeout at metainfo.xml and autokilling tasks that exceed timeout
>
>
> Diffs
> -----
>
> ambari-agent/conf/unix/ambari-agent.ini 95944d2
> ambari-agent/src/main/python/ambari_agent/PuppetExecutor.py 00c5c98
> ambari-agent/src/main/python/ambari_agent/test.json 6da53da
> ambari-agent/src/test/python/ambari_agent/TestPuppetExecutor.py 9e6da16
> ambari-server/conf/unix/ambari.properties 563c04f
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 47f56eb
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/Stage.java c0272ec
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java 7bd2b41
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatMonitor.java ee6826a
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java 6af54c0
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 6548be8
> ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java b4d96ee
> ambari-server/src/main/java/org/apache/ambari/server/state/CommandScriptDefinition.java fd06385
> ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java 6f7f465
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestStage.java 6f74d0a
> ambari-server/src/test/java/org/apache/ambari/server/api/util/StackExtensionHelperTest.java 5a3ab18
> ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HBASE/metainfo.xml 0605952
>
> Diff: https://reviews.apache.org/r/16970/diff/
>
>
> Testing
> -------
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ....................................... SUCCESS [0.335s]
> [INFO] Apache Ambari Project POM ......................... SUCCESS [0.040s]
> [INFO] Ambari Web ........................................ SUCCESS [5.969s]
> [INFO] Ambari Server ..................................... SUCCESS [9:55.335s]
> [INFO] Ambari Agent ...................................... SUCCESS [15.767s]
> [INFO] Ambari Client ..................................... SUCCESS [0.409s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 10:18.405s
> [INFO] Finished at: Thu Jan 16 18:25:08 FET 2014
> [INFO] Final Memory: 24M/284M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 16970: Task timeout should be a configurable knob at the
ambari-server
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16970/#review32035
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/Stage.java
public synchronized int getStageTimeout()
Considering all tasks execute in parallel for a stage - should we be adding them up?
ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestStage.java
Why its 3 times 600 second?
assertEquals(3*600000, s.getStageTimeout());
Alternatively, can you explain how timeouts work say for a Request that has two stages and the stages have 2 and 3 tasks respectively.
- Sumit Mohanty
On Jan. 16, 2014, 3:40 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16970/
> -----------------------------------------------------------
>
> (Updated Jan. 16, 2014, 3:40 p.m.)
>
>
> Review request for Ambari, Dmytro Sen and Sumit Mohanty.
>
>
> Bugs: AMBARI-4319
> https://issues.apache.org/jira/browse/AMBARI-4319
>
>
> Repository: ambari
>
>
> Description
> -------
>
> full implementation
> Also tested overriding timeout at metainfo.xml and autokilling tasks that exceed timeout
>
>
> Diffs
> -----
>
> ambari-agent/conf/unix/ambari-agent.ini 95944d2
> ambari-agent/src/main/python/ambari_agent/PuppetExecutor.py 00c5c98
> ambari-agent/src/main/python/ambari_agent/test.json 6da53da
> ambari-agent/src/test/python/ambari_agent/TestPuppetExecutor.py 9e6da16
> ambari-server/conf/unix/ambari.properties 563c04f
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 47f56eb
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/Stage.java c0272ec
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java 7bd2b41
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatMonitor.java ee6826a
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java 6af54c0
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 6548be8
> ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java b4d96ee
> ambari-server/src/main/java/org/apache/ambari/server/state/CommandScriptDefinition.java fd06385
> ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java 6f7f465
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestStage.java 6f74d0a
> ambari-server/src/test/java/org/apache/ambari/server/api/util/StackExtensionHelperTest.java 5a3ab18
> ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HBASE/metainfo.xml 0605952
>
> Diff: https://reviews.apache.org/r/16970/diff/
>
>
> Testing
> -------
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ....................................... SUCCESS [0.335s]
> [INFO] Apache Ambari Project POM ......................... SUCCESS [0.040s]
> [INFO] Ambari Web ........................................ SUCCESS [5.969s]
> [INFO] Ambari Server ..................................... SUCCESS [9:55.335s]
> [INFO] Ambari Agent ...................................... SUCCESS [15.767s]
> [INFO] Ambari Client ..................................... SUCCESS [0.409s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 10:18.405s
> [INFO] Finished at: Thu Jan 16 18:25:08 FET 2014
> [INFO] Final Memory: 24M/284M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 16970: Task timeout should be a configurable knob at the
ambari-server
Posted by Dmytro Sen <ds...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16970/#review32034
-----------------------------------------------------------
Ship it!
Ship It!
- Dmytro Sen
On Jan. 16, 2014, 3:40 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16970/
> -----------------------------------------------------------
>
> (Updated Jan. 16, 2014, 3:40 p.m.)
>
>
> Review request for Ambari, Dmytro Sen and Sumit Mohanty.
>
>
> Bugs: AMBARI-4319
> https://issues.apache.org/jira/browse/AMBARI-4319
>
>
> Repository: ambari
>
>
> Description
> -------
>
> full implementation
> Also tested overriding timeout at metainfo.xml and autokilling tasks that exceed timeout
>
>
> Diffs
> -----
>
> ambari-agent/conf/unix/ambari-agent.ini 95944d2
> ambari-agent/src/main/python/ambari_agent/PuppetExecutor.py 00c5c98
> ambari-agent/src/main/python/ambari_agent/test.json 6da53da
> ambari-agent/src/test/python/ambari_agent/TestPuppetExecutor.py 9e6da16
> ambari-server/conf/unix/ambari.properties 563c04f
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 47f56eb
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/Stage.java c0272ec
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java 7bd2b41
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatMonitor.java ee6826a
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java 6af54c0
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 6548be8
> ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java b4d96ee
> ambari-server/src/main/java/org/apache/ambari/server/state/CommandScriptDefinition.java fd06385
> ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java 6f7f465
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestStage.java 6f74d0a
> ambari-server/src/test/java/org/apache/ambari/server/api/util/StackExtensionHelperTest.java 5a3ab18
> ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HBASE/metainfo.xml 0605952
>
> Diff: https://reviews.apache.org/r/16970/diff/
>
>
> Testing
> -------
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ....................................... SUCCESS [0.335s]
> [INFO] Apache Ambari Project POM ......................... SUCCESS [0.040s]
> [INFO] Ambari Web ........................................ SUCCESS [5.969s]
> [INFO] Ambari Server ..................................... SUCCESS [9:55.335s]
> [INFO] Ambari Agent ...................................... SUCCESS [15.767s]
> [INFO] Ambari Client ..................................... SUCCESS [0.409s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 10:18.405s
> [INFO] Finished at: Thu Jan 16 18:25:08 FET 2014
> [INFO] Final Memory: 24M/284M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 16970: Task timeout should be a configurable knob at the
ambari-server
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16970/#review32038
-----------------------------------------------------------
Ship it!
Ship It!
- Sumit Mohanty
On Jan. 16, 2014, 3:40 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16970/
> -----------------------------------------------------------
>
> (Updated Jan. 16, 2014, 3:40 p.m.)
>
>
> Review request for Ambari, Dmytro Sen and Sumit Mohanty.
>
>
> Bugs: AMBARI-4319
> https://issues.apache.org/jira/browse/AMBARI-4319
>
>
> Repository: ambari
>
>
> Description
> -------
>
> full implementation
> Also tested overriding timeout at metainfo.xml and autokilling tasks that exceed timeout
>
>
> Diffs
> -----
>
> ambari-agent/conf/unix/ambari-agent.ini 95944d2
> ambari-agent/src/main/python/ambari_agent/PuppetExecutor.py 00c5c98
> ambari-agent/src/main/python/ambari_agent/test.json 6da53da
> ambari-agent/src/test/python/ambari_agent/TestPuppetExecutor.py 9e6da16
> ambari-server/conf/unix/ambari.properties 563c04f
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 47f56eb
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/Stage.java c0272ec
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java 7bd2b41
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatMonitor.java ee6826a
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java 6af54c0
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 6548be8
> ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java b4d96ee
> ambari-server/src/main/java/org/apache/ambari/server/state/CommandScriptDefinition.java fd06385
> ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java 6f7f465
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestStage.java 6f74d0a
> ambari-server/src/test/java/org/apache/ambari/server/api/util/StackExtensionHelperTest.java 5a3ab18
> ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HBASE/metainfo.xml 0605952
>
> Diff: https://reviews.apache.org/r/16970/diff/
>
>
> Testing
> -------
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ....................................... SUCCESS [0.335s]
> [INFO] Apache Ambari Project POM ......................... SUCCESS [0.040s]
> [INFO] Ambari Web ........................................ SUCCESS [5.969s]
> [INFO] Ambari Server ..................................... SUCCESS [9:55.335s]
> [INFO] Ambari Agent ...................................... SUCCESS [15.767s]
> [INFO] Ambari Client ..................................... SUCCESS [0.409s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 10:18.405s
> [INFO] Finished at: Thu Jan 16 18:25:08 FET 2014
> [INFO] Final Memory: 24M/284M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>