You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Robert Levas <rl...@hortonworks.com> on 2015/10/25 23:51:24 UTC
Review Request 39637: ActionScheduler#filterParallelPerHostStages
should not filter out stages with server-side actions
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39637/
-----------------------------------------------------------
Review request for Ambari, Jonathan Hurley, Nate Cole, Robert Nettleton, and Sandor Magyari.
Bugs: AMBARI-13550
https://issues.apache.org/jira/browse/AMBARI-13550
Repository: ambari
Description
-------
org.apache.ambari.server.actionmanager.ActionScheduler#filterParallelPerHostStages should not filter out stages that contain server-side actions (to be executed on the Ambari server host). This is because tasks in these stages are typically required to complete before other stages on other hosts are executed.
For example while enabling Kerberos for an added service, principals and keytab files need to be created before the stage to distribute the key tab files. The principal and keytab file creation happens on the Ambari server (as server-side actions) and the distribution tasks happen on the relevant hosts. If the server-side stages are filtered out (in the event multiple stages are pending for the Ambari server host), then one or more might be skipped and the distribution task is queued. In this scenario, the distribution stage will fail since the required keytab files will not have been created.
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 3f289b2
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/Stage.java 8b2703c
ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java f8f9ce9
Diff: https://reviews.apache.org/r/39637/diff/
Testing
-------
Manually tested in cluster... testing kerberizing and rolling upgrades.
#Jenkins test results: PENDING
Thanks,
Robert Levas
Re: Review Request 39637: ActionScheduler#filterParallelPerHostStages
should not filter out stages with server-side actions
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39637/#review104026
-----------------------------------------------------------
Ship it!
Ship It!
- Jonathan Hurley
On Oct. 25, 2015, 6:51 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39637/
> -----------------------------------------------------------
>
> (Updated Oct. 25, 2015, 6:51 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Robert Nettleton, and Sandor Magyari.
>
>
> Bugs: AMBARI-13550
> https://issues.apache.org/jira/browse/AMBARI-13550
>
>
> Repository: ambari
>
>
> Description
> -------
>
> org.apache.ambari.server.actionmanager.ActionScheduler#filterParallelPerHostStages should not filter out stages that contain server-side actions (to be executed on the Ambari server host). This is because tasks in these stages are typically required to complete before other stages on other hosts are executed.
>
> For example while enabling Kerberos for an added service, principals and keytab files need to be created before the stage to distribute the key tab files. The principal and keytab file creation happens on the Ambari server (as server-side actions) and the distribution tasks happen on the relevant hosts. If the server-side stages are filtered out (in the event multiple stages are pending for the Ambari server host), then one or more might be skipped and the distribution task is queued. In this scenario, the distribution stage will fail since the required keytab files will not have been created.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 3f289b2
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/Stage.java 8b2703c
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java f8f9ce9
>
> Diff: https://reviews.apache.org/r/39637/diff/
>
>
> Testing
> -------
>
> Manually tested in cluster... testing kerberizing and rolling upgrades.
>
> #Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 39637: ActionScheduler#filterParallelPerHostStages
should not filter out stages with server-side actions
Posted by Robert Nettleton <rn...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39637/#review104296
-----------------------------------------------------------
Ship it!
Ship It!
- Robert Nettleton
On Oct. 28, 2015, 2:33 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39637/
> -----------------------------------------------------------
>
> (Updated Oct. 28, 2015, 2:33 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Robert Nettleton, and Sandor Magyari.
>
>
> Bugs: AMBARI-13550
> https://issues.apache.org/jira/browse/AMBARI-13550
>
>
> Repository: ambari
>
>
> Description
> -------
>
> org.apache.ambari.server.actionmanager.ActionScheduler#filterParallelPerHostStages should not filter out stages that contain server-side actions (to be executed on the Ambari server host). This is because tasks in these stages are typically required to complete before other stages on other hosts are executed.
>
> For example while enabling Kerberos for an added service, principals and keytab files need to be created before the stage to distribute the key tab files. The principal and keytab file creation happens on the Ambari server (as server-side actions) and the distribution tasks happen on the relevant hosts. If the server-side stages are filtered out (in the event multiple stages are pending for the Ambari server host), then one or more might be skipped and the distribution task is queued. In this scenario, the distribution stage will fail since the required keytab files will not have been created.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 3f289b2
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/Stage.java 8b2703c
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java f8f9ce9
>
> Diff: https://reviews.apache.org/r/39637/diff/
>
>
> Testing
> -------
>
> Manually tested in cluster... testing kerberizing and rolling upgrades.
>
> #Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 39637: ActionScheduler#filterParallelPerHostStages
should not filter out stages with server-side actions
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39637/#review104306
-----------------------------------------------------------
Ship it!
Ship It!
- Nate Cole
On Oct. 28, 2015, 10:33 a.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39637/
> -----------------------------------------------------------
>
> (Updated Oct. 28, 2015, 10:33 a.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Robert Nettleton, and Sandor Magyari.
>
>
> Bugs: AMBARI-13550
> https://issues.apache.org/jira/browse/AMBARI-13550
>
>
> Repository: ambari
>
>
> Description
> -------
>
> org.apache.ambari.server.actionmanager.ActionScheduler#filterParallelPerHostStages should not filter out stages that contain server-side actions (to be executed on the Ambari server host). This is because tasks in these stages are typically required to complete before other stages on other hosts are executed.
>
> For example while enabling Kerberos for an added service, principals and keytab files need to be created before the stage to distribute the key tab files. The principal and keytab file creation happens on the Ambari server (as server-side actions) and the distribution tasks happen on the relevant hosts. If the server-side stages are filtered out (in the event multiple stages are pending for the Ambari server host), then one or more might be skipped and the distribution task is queued. In this scenario, the distribution stage will fail since the required keytab files will not have been created.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 3f289b2
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/Stage.java 8b2703c
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java f8f9ce9
>
> Diff: https://reviews.apache.org/r/39637/diff/
>
>
> Testing
> -------
>
> Manually tested in cluster... testing kerberizing and rolling upgrades.
>
> #Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 39637: ActionScheduler#filterParallelPerHostStages
should not filter out stages with server-side actions
Posted by Robert Levas <rl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39637/
-----------------------------------------------------------
(Updated Oct. 28, 2015, 10:33 a.m.)
Review request for Ambari, Jonathan Hurley, Nate Cole, Robert Nettleton, and Sandor Magyari.
Changes
-------
Updated `filterParallelPerHostStages` to fix race contion where if multiple reqeusts are in progress, stages in later requests may execute before stages in earlier requests even if there is an intersection on hosts. This occurred because multiple stage from each request were added to the filtered list, skipping stages if intersections occured on hosts within the same request. The skipped stages may have additional hosts that were not factored in when appling rules to stages in later requests. Therefore it was possible for a START command to be executed from a laster request when the INSTALL command from the earilier request was not yet executed. This could happen if server-side actions needed to occur between INIT and INSTALL stages from the earlier request.
Bugs: AMBARI-13550
https://issues.apache.org/jira/browse/AMBARI-13550
Repository: ambari
Description
-------
org.apache.ambari.server.actionmanager.ActionScheduler#filterParallelPerHostStages should not filter out stages that contain server-side actions (to be executed on the Ambari server host). This is because tasks in these stages are typically required to complete before other stages on other hosts are executed.
For example while enabling Kerberos for an added service, principals and keytab files need to be created before the stage to distribute the key tab files. The principal and keytab file creation happens on the Ambari server (as server-side actions) and the distribution tasks happen on the relevant hosts. If the server-side stages are filtered out (in the event multiple stages are pending for the Ambari server host), then one or more might be skipped and the distribution task is queued. In this scenario, the distribution stage will fail since the required keytab files will not have been created.
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 3f289b2
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/Stage.java 8b2703c
ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java f8f9ce9
Diff: https://reviews.apache.org/r/39637/diff/
Testing
-------
Manually tested in cluster... testing kerberizing and rolling upgrades.
#Jenkins test results: PENDING
Thanks,
Robert Levas
Re: Review Request 39637: ActionScheduler#filterParallelPerHostStages
should not filter out stages with server-side actions
Posted by Robert Nettleton <rn...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39637/#review104009
-----------------------------------------------------------
Ship it!
Ship It!
- Robert Nettleton
On Oct. 25, 2015, 10:51 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39637/
> -----------------------------------------------------------
>
> (Updated Oct. 25, 2015, 10:51 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Robert Nettleton, and Sandor Magyari.
>
>
> Bugs: AMBARI-13550
> https://issues.apache.org/jira/browse/AMBARI-13550
>
>
> Repository: ambari
>
>
> Description
> -------
>
> org.apache.ambari.server.actionmanager.ActionScheduler#filterParallelPerHostStages should not filter out stages that contain server-side actions (to be executed on the Ambari server host). This is because tasks in these stages are typically required to complete before other stages on other hosts are executed.
>
> For example while enabling Kerberos for an added service, principals and keytab files need to be created before the stage to distribute the key tab files. The principal and keytab file creation happens on the Ambari server (as server-side actions) and the distribution tasks happen on the relevant hosts. If the server-side stages are filtered out (in the event multiple stages are pending for the Ambari server host), then one or more might be skipped and the distribution task is queued. In this scenario, the distribution stage will fail since the required keytab files will not have been created.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 3f289b2
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/Stage.java 8b2703c
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java f8f9ce9
>
> Diff: https://reviews.apache.org/r/39637/diff/
>
>
> Testing
> -------
>
> Manually tested in cluster... testing kerberizing and rolling upgrades.
>
> #Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 39637: ActionScheduler#filterParallelPerHostStages
should not filter out stages with server-side actions
Posted by Nate Cole <nc...@hortonworks.com>.
> On Oct. 26, 2015, 8:58 a.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java, lines 468-470
> > <https://reviews.apache.org/r/39637/diff/1/?file=1107413#file1107413line468>
> >
> > Could use affectedHosts.isEmpty() && LOG.isTraceEnabled() check instead of addStage boolean? Just a readability nit.
>
> Robert Levas wrote:
> affectedHosts wont necessarily be empty if addState is true. There may be multiple tasks in the stage, each with potentially a different host (one of which may be Stage.INTERNAL_HOSTNAME). In this case, addStage will be true but affectedHosts may contain one of more hosts from the tasks.
Very well :)
- Nate
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39637/#review104003
-----------------------------------------------------------
On Oct. 25, 2015, 6:51 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39637/
> -----------------------------------------------------------
>
> (Updated Oct. 25, 2015, 6:51 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Robert Nettleton, and Sandor Magyari.
>
>
> Bugs: AMBARI-13550
> https://issues.apache.org/jira/browse/AMBARI-13550
>
>
> Repository: ambari
>
>
> Description
> -------
>
> org.apache.ambari.server.actionmanager.ActionScheduler#filterParallelPerHostStages should not filter out stages that contain server-side actions (to be executed on the Ambari server host). This is because tasks in these stages are typically required to complete before other stages on other hosts are executed.
>
> For example while enabling Kerberos for an added service, principals and keytab files need to be created before the stage to distribute the key tab files. The principal and keytab file creation happens on the Ambari server (as server-side actions) and the distribution tasks happen on the relevant hosts. If the server-side stages are filtered out (in the event multiple stages are pending for the Ambari server host), then one or more might be skipped and the distribution task is queued. In this scenario, the distribution stage will fail since the required keytab files will not have been created.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 3f289b2
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/Stage.java 8b2703c
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java f8f9ce9
>
> Diff: https://reviews.apache.org/r/39637/diff/
>
>
> Testing
> -------
>
> Manually tested in cluster... testing kerberizing and rolling upgrades.
>
> #Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 39637: ActionScheduler#filterParallelPerHostStages
should not filter out stages with server-side actions
Posted by Robert Levas <rl...@hortonworks.com>.
> On Oct. 26, 2015, 8:58 a.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java, lines 468-470
> > <https://reviews.apache.org/r/39637/diff/1/?file=1107413#file1107413line468>
> >
> > Could use affectedHosts.isEmpty() && LOG.isTraceEnabled() check instead of addStage boolean? Just a readability nit.
affectedHosts wont necessarily be empty if addState is true. There may be multiple tasks in the stage, each with potentially a different host (one of which may be Stage.INTERNAL_HOSTNAME). In this case, addStage will be true but affectedHosts may contain one of more hosts from the tasks.
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39637/#review104003
-----------------------------------------------------------
On Oct. 25, 2015, 6:51 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39637/
> -----------------------------------------------------------
>
> (Updated Oct. 25, 2015, 6:51 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Robert Nettleton, and Sandor Magyari.
>
>
> Bugs: AMBARI-13550
> https://issues.apache.org/jira/browse/AMBARI-13550
>
>
> Repository: ambari
>
>
> Description
> -------
>
> org.apache.ambari.server.actionmanager.ActionScheduler#filterParallelPerHostStages should not filter out stages that contain server-side actions (to be executed on the Ambari server host). This is because tasks in these stages are typically required to complete before other stages on other hosts are executed.
>
> For example while enabling Kerberos for an added service, principals and keytab files need to be created before the stage to distribute the key tab files. The principal and keytab file creation happens on the Ambari server (as server-side actions) and the distribution tasks happen on the relevant hosts. If the server-side stages are filtered out (in the event multiple stages are pending for the Ambari server host), then one or more might be skipped and the distribution task is queued. In this scenario, the distribution stage will fail since the required keytab files will not have been created.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 3f289b2
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/Stage.java 8b2703c
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java f8f9ce9
>
> Diff: https://reviews.apache.org/r/39637/diff/
>
>
> Testing
> -------
>
> Manually tested in cluster... testing kerberizing and rolling upgrades.
>
> #Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 39637: ActionScheduler#filterParallelPerHostStages
should not filter out stages with server-side actions
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39637/#review104003
-----------------------------------------------------------
Ship it!
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java (lines 468 - 470)
<https://reviews.apache.org/r/39637/#comment162189>
Could use affectedHosts.isEmpty() && LOG.isTraceEnabled() check instead of addStage boolean? Just a readability nit.
- Nate Cole
On Oct. 25, 2015, 6:51 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39637/
> -----------------------------------------------------------
>
> (Updated Oct. 25, 2015, 6:51 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Robert Nettleton, and Sandor Magyari.
>
>
> Bugs: AMBARI-13550
> https://issues.apache.org/jira/browse/AMBARI-13550
>
>
> Repository: ambari
>
>
> Description
> -------
>
> org.apache.ambari.server.actionmanager.ActionScheduler#filterParallelPerHostStages should not filter out stages that contain server-side actions (to be executed on the Ambari server host). This is because tasks in these stages are typically required to complete before other stages on other hosts are executed.
>
> For example while enabling Kerberos for an added service, principals and keytab files need to be created before the stage to distribute the key tab files. The principal and keytab file creation happens on the Ambari server (as server-side actions) and the distribution tasks happen on the relevant hosts. If the server-side stages are filtered out (in the event multiple stages are pending for the Ambari server host), then one or more might be skipped and the distribution task is queued. In this scenario, the distribution stage will fail since the required keytab files will not have been created.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 3f289b2
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/Stage.java 8b2703c
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java f8f9ce9
>
> Diff: https://reviews.apache.org/r/39637/diff/
>
>
> Testing
> -------
>
> Manually tested in cluster... testing kerberizing and rolling upgrades.
>
> #Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>