You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Jonathan Hurley <jh...@hortonworks.com> on 2017/04/08 19:19:52 UTC
Review Request 58282: Parallel Requests With Intersecting Hosts Don't
Block Correctly
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58282/
-----------------------------------------------------------
Review request for Ambari and Sumit Mohanty.
Bugs: AMBARI-20712
https://issues.apache.org/jira/browse/AMBARI-20712
Repository: ambari
Description
-------
AMBARI-20646 introduced a regression which can be seen when deploying a cluster via blueprint with Kerberization. The problem is caused by the change which was made to only pull in the most recent stage in progress per request.
There was logic which prevented concurrently executing request stages to be scheduled if a prior request had an un-executed stage with a conflicting host. For example:
- Request 1
-- Stage 1 (IN_PROGRESS)
--- Host 1
--- Host 2
--- Host 3
-- Stage 2 (PENDING)
--- Host 4
- Request 2
-- Stage 1
--- Host 4
In the above scenario, the scheduler was tricked into thinking that Request 2 can run even though Host-4 has a conflict. This is because it was only looking at the stage in progress.
We can't simply look at all stages in progress since this would cause the performance bug to manifest again. So, the solution is to have the database tell us which hosts are blocking from prior requests. This SQL should be very fast and will only run in the specific cases where there are multiple concurrent requests (which is typically at blueprint time).
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 758db35
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/HostRoleCommand.java 651eb24
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 77d5bf8
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java 79b8bc9
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostRoleCommandEntity.java a809295
ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql 6744a74
ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 6279f83
ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 470eb60
ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 87ffb7f
ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql a460194
ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 237c892
Diff: https://reviews.apache.org/r/58282/diff/1/
Testing
-------
PENDING
Thanks,
Jonathan Hurley
Re: Review Request 58282: Parallel Requests With Intersecting Hosts
Don't Block Correctly
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58282/#review171405
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java
Lines 615-618 (patched)
<https://reviews.apache.org/r/58282/#comment244317>
We can't use the list of stages for this anymore since it's not complete. Instead, we can have the database tell us which hosts have commands for a given request if needed.
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java
Line 916 (original), 917-947 (patched)
<https://reviews.apache.org/r/58282/#comment244318>
Two new queries which can be used if there are concurrent requests in order to determine which hosts have commands which would block future requests for that host.
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostRoleCommandEntity.java
Lines 74-76 (patched)
<https://reviews.apache.org/r/58282/#comment244320>
Find all hosts with requests in progress
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostRoleCommandEntity.java
Lines 77-79 (patched)
<https://reviews.apache.org/r/58282/#comment244319>
Find every host which has a command in progress (which is not a background command) for any request before the specified one.
- Jonathan Hurley
On April 8, 2017, 3:19 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58282/
> -----------------------------------------------------------
>
> (Updated April 8, 2017, 3:19 p.m.)
>
>
> Review request for Ambari and Sumit Mohanty.
>
>
> Bugs: AMBARI-20712
> https://issues.apache.org/jira/browse/AMBARI-20712
>
>
> Repository: ambari
>
>
> Description
> -------
>
> AMBARI-20646 introduced a regression which can be seen when deploying a cluster via blueprint with Kerberization. The problem is caused by the change which was made to only pull in the most recent stage in progress per request.
>
> There was logic which prevented concurrently executing request stages to be scheduled if a prior request had an un-executed stage with a conflicting host. For example:
>
> - Request 1
> -- Stage 1 (IN_PROGRESS)
> --- Host 1
> --- Host 2
> --- Host 3
> -- Stage 2 (PENDING)
> --- Host 4
>
> - Request 2
> -- Stage 1
> --- Host 4
>
> In the above scenario, the scheduler was tricked into thinking that Request 2 can run even though Host-4 has a conflict. This is because it was only looking at the stage in progress.
>
> We can't simply look at all stages in progress since this would cause the performance bug to manifest again. So, the solution is to have the database tell us which hosts are blocking from prior requests. This SQL should be very fast and will only run in the specific cases where there are multiple concurrent requests (which is typically at blueprint time).
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 758db35
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/HostRoleCommand.java 651eb24
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 77d5bf8
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java 79b8bc9
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostRoleCommandEntity.java a809295
> ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql 6744a74
> ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 6279f83
> ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 470eb60
> ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 87ffb7f
> ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql a460194
> ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 237c892
>
>
> Diff: https://reviews.apache.org/r/58282/diff/1/
>
>
> Testing
> -------
>
> PENDING
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 58282: Parallel Requests With Intersecting Hosts
Don't Block Correctly
Posted by Sumit Mohanty <sm...@hortonworks.com>.
> On April 8, 2017, 9:06 p.m., Sumit Mohanty wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java
> > Lines 945 (patched)
> > <https://reviews.apache.org/r/58282/diff/1/?file=1686812#file1686812line945>
> >
> > Is in_progress sufficient? Based on the summary, we are guarding a situation when a future stage (task is not queued yet) of a currently active request has pending tasks on a host.
>
> Jonathan Hurley wrote:
> Yes; this is HostRoleStatus.IN_PROGRESS_STATUSES, which includes QUEUED, HOLDING*, IN_PROGRESS, PENDING, etc.
Got it - missed the "ES" at the end.
- Sumit
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58282/#review171407
-----------------------------------------------------------
On April 8, 2017, 8:50 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58282/
> -----------------------------------------------------------
>
> (Updated April 8, 2017, 8:50 p.m.)
>
>
> Review request for Ambari and Sumit Mohanty.
>
>
> Bugs: AMBARI-20712
> https://issues.apache.org/jira/browse/AMBARI-20712
>
>
> Repository: ambari
>
>
> Description
> -------
>
> AMBARI-20646 introduced a regression which can be seen when deploying a cluster via blueprint with Kerberization. The problem is caused by the change which was made to only pull in the most recent stage in progress per request.
>
> There was logic which prevented concurrently executing request stages to be scheduled if a prior request had an un-executed stage with a conflicting host. For example:
>
> - Request 1
> -- Stage 1 (IN_PROGRESS)
> --- Host 1
> --- Host 2
> --- Host 3
> -- Stage 2 (PENDING)
> --- Host 4
>
> - Request 2
> -- Stage 1
> --- Host 4
>
> In the above scenario, the scheduler was tricked into thinking that Request 2 can run even though Host-4 has a conflict. This is because it was only looking at the stage in progress.
>
> We can't simply look at all stages in progress since this would cause the performance bug to manifest again. So, the solution is to have the database tell us which hosts are blocking from prior requests. This SQL should be very fast and will only run in the specific cases where there are multiple concurrent requests (which is typically at blueprint time).
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 758db35
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/HostRoleCommand.java 651eb24
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 77d5bf8
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java 79b8bc9
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostRoleCommandEntity.java a809295
> ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql 6744a74
> ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 6279f83
> ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 470eb60
> ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 87ffb7f
> ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql a460194
> ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 237c892
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java d7d3d40
>
>
> Diff: https://reviews.apache.org/r/58282/diff/2/
>
>
> Testing
> -------
>
> Tests run: 4981, Failures: 0, Errors: 0, Skipped: 39
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 17:26 min
> [INFO] Finished at: 2017-04-08T15:20:57-04:00
> [INFO] Final Memory: 58M/586M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 58282: Parallel Requests With Intersecting Hosts
Don't Block Correctly
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On April 8, 2017, 5:06 p.m., Sumit Mohanty wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java
> > Lines 945 (patched)
> > <https://reviews.apache.org/r/58282/diff/1/?file=1686812#file1686812line945>
> >
> > Is in_progress sufficient? Based on the summary, we are guarding a situation when a future stage (task is not queued yet) of a currently active request has pending tasks on a host.
Yes; this is HostRoleStatus.IN_PROGRESS_STATUSES, which includes QUEUED, HOLDING*, IN_PROGRESS, PENDING, etc.
> On April 8, 2017, 5:06 p.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql
> > Lines 387 (patched)
> > <https://reviews.apache.org/r/58282/diff/1/?file=1686816#file1686816line387>
> >
> > Needs upgrade catalog changes fro 2.5.1
Once this review looked good, I was going to create a separte Jira for the upgrade work. Since we don't have an upgrade catalog yet, I felt it was too much for this one review.
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58282/#review171407
-----------------------------------------------------------
On April 8, 2017, 4:50 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58282/
> -----------------------------------------------------------
>
> (Updated April 8, 2017, 4:50 p.m.)
>
>
> Review request for Ambari and Sumit Mohanty.
>
>
> Bugs: AMBARI-20712
> https://issues.apache.org/jira/browse/AMBARI-20712
>
>
> Repository: ambari
>
>
> Description
> -------
>
> AMBARI-20646 introduced a regression which can be seen when deploying a cluster via blueprint with Kerberization. The problem is caused by the change which was made to only pull in the most recent stage in progress per request.
>
> There was logic which prevented concurrently executing request stages to be scheduled if a prior request had an un-executed stage with a conflicting host. For example:
>
> - Request 1
> -- Stage 1 (IN_PROGRESS)
> --- Host 1
> --- Host 2
> --- Host 3
> -- Stage 2 (PENDING)
> --- Host 4
>
> - Request 2
> -- Stage 1
> --- Host 4
>
> In the above scenario, the scheduler was tricked into thinking that Request 2 can run even though Host-4 has a conflict. This is because it was only looking at the stage in progress.
>
> We can't simply look at all stages in progress since this would cause the performance bug to manifest again. So, the solution is to have the database tell us which hosts are blocking from prior requests. This SQL should be very fast and will only run in the specific cases where there are multiple concurrent requests (which is typically at blueprint time).
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 758db35
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/HostRoleCommand.java 651eb24
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 77d5bf8
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java 79b8bc9
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostRoleCommandEntity.java a809295
> ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql 6744a74
> ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 6279f83
> ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 470eb60
> ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 87ffb7f
> ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql a460194
> ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 237c892
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java d7d3d40
>
>
> Diff: https://reviews.apache.org/r/58282/diff/2/
>
>
> Testing
> -------
>
> Tests run: 4981, Failures: 0, Errors: 0, Skipped: 39
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 17:26 min
> [INFO] Finished at: 2017-04-08T15:20:57-04:00
> [INFO] Final Memory: 58M/586M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 58282: Parallel Requests With Intersecting Hosts
Don't Block Correctly
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58282/#review171407
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java
Lines 945 (patched)
<https://reviews.apache.org/r/58282/#comment244328>
Is in_progress sufficient? Based on the summary, we are guarding a situation when a future stage (task is not queued yet) of a currently active request has pending tasks on a host.
ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql
Lines 387 (patched)
<https://reviews.apache.org/r/58282/#comment244327>
Needs upgrade catalog changes fro 2.5.1
- Sumit Mohanty
On April 8, 2017, 8:50 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58282/
> -----------------------------------------------------------
>
> (Updated April 8, 2017, 8:50 p.m.)
>
>
> Review request for Ambari and Sumit Mohanty.
>
>
> Bugs: AMBARI-20712
> https://issues.apache.org/jira/browse/AMBARI-20712
>
>
> Repository: ambari
>
>
> Description
> -------
>
> AMBARI-20646 introduced a regression which can be seen when deploying a cluster via blueprint with Kerberization. The problem is caused by the change which was made to only pull in the most recent stage in progress per request.
>
> There was logic which prevented concurrently executing request stages to be scheduled if a prior request had an un-executed stage with a conflicting host. For example:
>
> - Request 1
> -- Stage 1 (IN_PROGRESS)
> --- Host 1
> --- Host 2
> --- Host 3
> -- Stage 2 (PENDING)
> --- Host 4
>
> - Request 2
> -- Stage 1
> --- Host 4
>
> In the above scenario, the scheduler was tricked into thinking that Request 2 can run even though Host-4 has a conflict. This is because it was only looking at the stage in progress.
>
> We can't simply look at all stages in progress since this would cause the performance bug to manifest again. So, the solution is to have the database tell us which hosts are blocking from prior requests. This SQL should be very fast and will only run in the specific cases where there are multiple concurrent requests (which is typically at blueprint time).
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 758db35
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/HostRoleCommand.java 651eb24
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 77d5bf8
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java 79b8bc9
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostRoleCommandEntity.java a809295
> ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql 6744a74
> ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 6279f83
> ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 470eb60
> ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 87ffb7f
> ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql a460194
> ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 237c892
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java d7d3d40
>
>
> Diff: https://reviews.apache.org/r/58282/diff/2/
>
>
> Testing
> -------
>
> Tests run: 4981, Failures: 0, Errors: 0, Skipped: 39
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 17:26 min
> [INFO] Finished at: 2017-04-08T15:20:57-04:00
> [INFO] Final Memory: 58M/586M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 58282: Parallel Requests With Intersecting Hosts
Don't Block Correctly
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On April 8, 2017, 7:16 p.m., Sumit Mohanty wrote:
> > Ship It!
Thanks! I created AMBARI-20714 to track the changes for the upgrade catalog.
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58282/#review171409
-----------------------------------------------------------
On April 8, 2017, 8:22 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58282/
> -----------------------------------------------------------
>
> (Updated April 8, 2017, 8:22 p.m.)
>
>
> Review request for Ambari and Sumit Mohanty.
>
>
> Bugs: AMBARI-20712
> https://issues.apache.org/jira/browse/AMBARI-20712
>
>
> Repository: ambari
>
>
> Description
> -------
>
> AMBARI-20646 introduced a regression which can be seen when deploying a cluster via blueprint with Kerberization. The problem is caused by the change which was made to only pull in the most recent stage in progress per request.
>
> There was logic which prevented concurrently executing request stages to be scheduled if a prior request had an un-executed stage with a conflicting host. For example:
>
> - Request 1
> -- Stage 1 (IN_PROGRESS)
> --- Host 1
> --- Host 2
> --- Host 3
> -- Stage 2 (PENDING)
> --- Host 4
>
> - Request 2
> -- Stage 1
> --- Host 4
>
> In the above scenario, the scheduler was tricked into thinking that Request 2 can run even though Host-4 has a conflict. This is because it was only looking at the stage in progress.
>
> We can't simply look at all stages in progress since this would cause the performance bug to manifest again. So, the solution is to have the database tell us which hosts are blocking from prior requests. This SQL should be very fast and will only run in the specific cases where there are multiple concurrent requests (which is typically at blueprint time).
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 758db35
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/HostRoleCommand.java 651eb24
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 77d5bf8
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java 79b8bc9
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostRoleCommandEntity.java a809295
> ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql 6744a74
> ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 6279f83
> ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 470eb60
> ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 87ffb7f
> ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql a460194
> ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 237c892
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java d7d3d40
>
>
> Diff: https://reviews.apache.org/r/58282/diff/2/
>
>
> Testing
> -------
>
> Performed a blueprint install of a kerberized cluster; the START commands waiting for the INSTALL commands to complete.
>
> Tests run: 4981, Failures: 0, Errors: 0, Skipped: 39
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 17:26 min
> [INFO] Finished at: 2017-04-08T15:20:57-04:00
> [INFO] Final Memory: 58M/586M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 58282: Parallel Requests With Intersecting Hosts
Don't Block Correctly
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58282/#review171409
-----------------------------------------------------------
Ship it!
Ship It!
- Sumit Mohanty
On April 8, 2017, 8:50 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58282/
> -----------------------------------------------------------
>
> (Updated April 8, 2017, 8:50 p.m.)
>
>
> Review request for Ambari and Sumit Mohanty.
>
>
> Bugs: AMBARI-20712
> https://issues.apache.org/jira/browse/AMBARI-20712
>
>
> Repository: ambari
>
>
> Description
> -------
>
> AMBARI-20646 introduced a regression which can be seen when deploying a cluster via blueprint with Kerberization. The problem is caused by the change which was made to only pull in the most recent stage in progress per request.
>
> There was logic which prevented concurrently executing request stages to be scheduled if a prior request had an un-executed stage with a conflicting host. For example:
>
> - Request 1
> -- Stage 1 (IN_PROGRESS)
> --- Host 1
> --- Host 2
> --- Host 3
> -- Stage 2 (PENDING)
> --- Host 4
>
> - Request 2
> -- Stage 1
> --- Host 4
>
> In the above scenario, the scheduler was tricked into thinking that Request 2 can run even though Host-4 has a conflict. This is because it was only looking at the stage in progress.
>
> We can't simply look at all stages in progress since this would cause the performance bug to manifest again. So, the solution is to have the database tell us which hosts are blocking from prior requests. This SQL should be very fast and will only run in the specific cases where there are multiple concurrent requests (which is typically at blueprint time).
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 758db35
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/HostRoleCommand.java 651eb24
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 77d5bf8
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java 79b8bc9
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostRoleCommandEntity.java a809295
> ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql 6744a74
> ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 6279f83
> ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 470eb60
> ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 87ffb7f
> ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql a460194
> ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 237c892
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java d7d3d40
>
>
> Diff: https://reviews.apache.org/r/58282/diff/2/
>
>
> Testing
> -------
>
> Tests run: 4981, Failures: 0, Errors: 0, Skipped: 39
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 17:26 min
> [INFO] Finished at: 2017-04-08T15:20:57-04:00
> [INFO] Final Memory: 58M/586M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 58282: Parallel Requests With Intersecting Hosts
Don't Block Correctly
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58282/
-----------------------------------------------------------
(Updated April 8, 2017, 8:22 p.m.)
Review request for Ambari and Sumit Mohanty.
Bugs: AMBARI-20712
https://issues.apache.org/jira/browse/AMBARI-20712
Repository: ambari
Description
-------
AMBARI-20646 introduced a regression which can be seen when deploying a cluster via blueprint with Kerberization. The problem is caused by the change which was made to only pull in the most recent stage in progress per request.
There was logic which prevented concurrently executing request stages to be scheduled if a prior request had an un-executed stage with a conflicting host. For example:
- Request 1
-- Stage 1 (IN_PROGRESS)
--- Host 1
--- Host 2
--- Host 3
-- Stage 2 (PENDING)
--- Host 4
- Request 2
-- Stage 1
--- Host 4
In the above scenario, the scheduler was tricked into thinking that Request 2 can run even though Host-4 has a conflict. This is because it was only looking at the stage in progress.
We can't simply look at all stages in progress since this would cause the performance bug to manifest again. So, the solution is to have the database tell us which hosts are blocking from prior requests. This SQL should be very fast and will only run in the specific cases where there are multiple concurrent requests (which is typically at blueprint time).
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 758db35
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/HostRoleCommand.java 651eb24
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 77d5bf8
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java 79b8bc9
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostRoleCommandEntity.java a809295
ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql 6744a74
ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 6279f83
ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 470eb60
ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 87ffb7f
ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql a460194
ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 237c892
ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java d7d3d40
Diff: https://reviews.apache.org/r/58282/diff/2/
Testing (updated)
-------
Performed a blueprint install of a kerberized cluster; the START commands waiting for the INSTALL commands to complete.
Tests run: 4981, Failures: 0, Errors: 0, Skipped: 39
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 17:26 min
[INFO] Finished at: 2017-04-08T15:20:57-04:00
[INFO] Final Memory: 58M/586M
[INFO] ------------------------------------------------------------------------
Thanks,
Jonathan Hurley
Re: Review Request 58282: Parallel Requests With Intersecting Hosts
Don't Block Correctly
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58282/
-----------------------------------------------------------
(Updated April 8, 2017, 4:50 p.m.)
Review request for Ambari and Sumit Mohanty.
Bugs: AMBARI-20712
https://issues.apache.org/jira/browse/AMBARI-20712
Repository: ambari
Description
-------
AMBARI-20646 introduced a regression which can be seen when deploying a cluster via blueprint with Kerberization. The problem is caused by the change which was made to only pull in the most recent stage in progress per request.
There was logic which prevented concurrently executing request stages to be scheduled if a prior request had an un-executed stage with a conflicting host. For example:
- Request 1
-- Stage 1 (IN_PROGRESS)
--- Host 1
--- Host 2
--- Host 3
-- Stage 2 (PENDING)
--- Host 4
- Request 2
-- Stage 1
--- Host 4
In the above scenario, the scheduler was tricked into thinking that Request 2 can run even though Host-4 has a conflict. This is because it was only looking at the stage in progress.
We can't simply look at all stages in progress since this would cause the performance bug to manifest again. So, the solution is to have the database tell us which hosts are blocking from prior requests. This SQL should be very fast and will only run in the specific cases where there are multiple concurrent requests (which is typically at blueprint time).
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 758db35
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/HostRoleCommand.java 651eb24
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 77d5bf8
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java 79b8bc9
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostRoleCommandEntity.java a809295
ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql 6744a74
ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 6279f83
ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 470eb60
ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 87ffb7f
ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql a460194
ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 237c892
ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java d7d3d40
Diff: https://reviews.apache.org/r/58282/diff/2/
Changes: https://reviews.apache.org/r/58282/diff/1-2/
Testing (updated)
-------
Tests run: 4981, Failures: 0, Errors: 0, Skipped: 39
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 17:26 min
[INFO] Finished at: 2017-04-08T15:20:57-04:00
[INFO] Final Memory: 58M/586M
[INFO] ------------------------------------------------------------------------
Thanks,
Jonathan Hurley