You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Jonathan Hurley <jh...@hortonworks.com> on 2017/04/05 20:40:52 UTC
Review Request 58218: Upgrade Progress Dialog Executes Query Which
Causes StackOverflow in JPA
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58218/
-----------------------------------------------------------
Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, and Sid Wagle.
Bugs: AMBARI-20685
https://issues.apache.org/jira/browse/AMBARI-20685
Repository: ambari
Description
-------
Large rolling upgrades (1000 hosts with 10,000 host components) creates a massive request object object with 10's of 1000's of stages and tasks. The web client executes a {{GET}} command against the API to retrieve upgrade groups/items. On a large upgrade, this causes the EclipseLink SQL generation code to run out of stack frame space while recursively building a query. To work around this, the stack frame size per thread can be increased using {{-Xss}} along with a value like 32M.
{code}
http://localhost:8080/api/v1/clusters/c1/upgrades/12?
upgrade_groups/UpgradeGroup/status!=PENDING&
fields=
Upgrade/progress_percent,
Upgrade/request_context,
Upgrade/request_status,
Upgrade/direction,
Upgrade/downgrade_allowed,
upgrade_groups/UpgradeGroup,
Upgrade/*,
upgrade_groups/upgrade_items/UpgradeItem/status,
minimal_response=true&_=1489680258782
{code}
This call gets turn into a LOT of queries, but some seem to be based on the number of stages. For example, this one on a very small upgrade produces very poor SQL...
{code}
SELECT
stage_id,
cluster_host_info,
cluster_id,
command_execution_type,
command_params,
display_status,
host_params,
log_info,
request_context,
request_id,
skippable,
status,
supports_auto_skip_failure
FROM stage
WHERE ((request_id = ?)
AND ((((((((stage_id = ?)
OR (stage_id = ?))
OR (stage_id = ?))
OR (stage_id = ?))
OR (stage_id = ?))
OR (stage_id = ?))
OR (stage_id = ?))
OR (stage_id = ?)))
{code}
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java 8759844
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeItemResourceProvider.java bf0fa33
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java 49c1594
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntityPK.java 34d175c
ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java eee913a
Diff: https://reviews.apache.org/r/58218/diff/1/
Testing
-------
PENDING
Thanks,
Jonathan Hurley
Re: Review Request 58218: Upgrade Progress Dialog Executes Query Which
Causes StackOverflow in JPA
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58218/#review171164
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeItemResourceProvider.java
Lines 239-241 (original), 238-245 (patched)
<https://reviews.apache.org/r/58218/#comment244044>
This may look horrible, but it's actually incredible efficient b/c of the PK lookup:
Name: org.apache.ambari.server.orm.entities.StageEntity-findByPrimaryKey
Cache Hits: 156500224
DB Hits: 7360
ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java
Lines 670-714 (original)
<https://reviews.apache.org/r/58218/#comment244043>
This was the culprit; this recursion breaks JPA in a lot of ways. More importantly, it doesn't let us get the StageEntity from the cache.
- Jonathan Hurley
On April 5, 2017, 4:40 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58218/
> -----------------------------------------------------------
>
> (Updated April 5, 2017, 4:40 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, and Sid Wagle.
>
>
> Bugs: AMBARI-20685
> https://issues.apache.org/jira/browse/AMBARI-20685
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Large rolling upgrades (1000 hosts with 10,000 host components) creates a massive request object object with 10's of 1000's of stages and tasks. The web client executes a {{GET}} command against the API to retrieve upgrade groups/items. On a large upgrade, this causes the EclipseLink SQL generation code to run out of stack frame space while recursively building a query. To work around this, the stack frame size per thread can be increased using {{-Xss}} along with a value like 32M.
>
> {code}
> http://localhost:8080/api/v1/clusters/c1/upgrades/12?
> upgrade_groups/UpgradeGroup/status!=PENDING&
> fields=
> Upgrade/progress_percent,
> Upgrade/request_context,
> Upgrade/request_status,
> Upgrade/direction,
> Upgrade/downgrade_allowed,
> upgrade_groups/UpgradeGroup,
> Upgrade/*,
> upgrade_groups/upgrade_items/UpgradeItem/status,
> minimal_response=true&_=1489680258782
> {code}
>
> This call gets turn into a LOT of queries, but some seem to be based on the number of stages. For example, this one on a very small upgrade produces very poor SQL...
>
> {code}
> SELECT
> stage_id,
> cluster_host_info,
> cluster_id,
> command_execution_type,
> command_params,
> display_status,
> host_params,
> log_info,
> request_context,
> request_id,
> skippable,
> status,
> supports_auto_skip_failure
> FROM stage
> WHERE ((request_id = ?)
> AND ((((((((stage_id = ?)
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?)))
> {code}
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java 8759844
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeItemResourceProvider.java bf0fa33
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java 49c1594
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntityPK.java 34d175c
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java eee913a
>
>
> Diff: https://reviews.apache.org/r/58218/diff/1/
>
>
> Testing
> -------
>
> PENDING
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 58218: Upgrade Progress Dialog Executes Query Which
Causes StackOverflow in JPA
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58218/#review171173
-----------------------------------------------------------
Ship it!
Ship It!
- Alejandro Fernandez
On April 5, 2017, 11:20 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58218/
> -----------------------------------------------------------
>
> (Updated April 5, 2017, 11:20 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, and Sid Wagle.
>
>
> Bugs: AMBARI-20685
> https://issues.apache.org/jira/browse/AMBARI-20685
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Large rolling upgrades (1000 hosts with 10,000 host components) creates a massive request object object with 10's of 1000's of stages and tasks. The web client executes a {{GET}} command against the API to retrieve upgrade groups/items. On a large upgrade, this causes the EclipseLink SQL generation code to run out of stack frame space while recursively building a query. To work around this, the stack frame size per thread can be increased using {{-Xss}} along with a value like 32M.
>
> {code}
> http://localhost:8080/api/v1/clusters/c1/upgrades/12?
> upgrade_groups/UpgradeGroup/status!=PENDING&
> fields=
> Upgrade/progress_percent,
> Upgrade/request_context,
> Upgrade/request_status,
> Upgrade/direction,
> Upgrade/downgrade_allowed,
> upgrade_groups/UpgradeGroup,
> Upgrade/*,
> upgrade_groups/upgrade_items/UpgradeItem/status,
> minimal_response=true&_=1489680258782
> {code}
>
> This call gets turn into a LOT of queries, but some seem to be based on the number of stages. For example, this one on a very small upgrade produces very poor SQL...
>
> {code}
> SELECT
> stage_id,
> cluster_host_info,
> cluster_id,
> command_execution_type,
> command_params,
> display_status,
> host_params,
> log_info,
> request_context,
> request_id,
> skippable,
> status,
> supports_auto_skip_failure
> FROM stage
> WHERE ((request_id = ?)
> AND ((((((((stage_id = ?)
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?)))
> {code}
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java 8759844
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeItemResourceProvider.java bf0fa33
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java 49c1594
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntityPK.java 34d175c
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java eee913a
>
>
> Diff: https://reviews.apache.org/r/58218/diff/2/
>
>
> Testing
> -------
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 17:46 min
> [INFO] Finished at: 2017-04-05T17:30:09-04:00
> [INFO] Final Memory: 67M/861M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 58218: Upgrade Progress Dialog Executes Query Which
Causes StackOverflow in JPA
Posted by Robert Levas <rl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58218/#review171192
-----------------------------------------------------------
Ship it!
Ship It!
- Robert Levas
On April 5, 2017, 7:20 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58218/
> -----------------------------------------------------------
>
> (Updated April 5, 2017, 7:20 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, and Sid Wagle.
>
>
> Bugs: AMBARI-20685
> https://issues.apache.org/jira/browse/AMBARI-20685
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Large rolling upgrades (1000 hosts with 10,000 host components) creates a massive request object object with 10's of 1000's of stages and tasks. The web client executes a {{GET}} command against the API to retrieve upgrade groups/items. On a large upgrade, this causes the EclipseLink SQL generation code to run out of stack frame space while recursively building a query. To work around this, the stack frame size per thread can be increased using {{-Xss}} along with a value like 32M.
>
> {code}
> http://localhost:8080/api/v1/clusters/c1/upgrades/12?
> upgrade_groups/UpgradeGroup/status!=PENDING&
> fields=
> Upgrade/progress_percent,
> Upgrade/request_context,
> Upgrade/request_status,
> Upgrade/direction,
> Upgrade/downgrade_allowed,
> upgrade_groups/UpgradeGroup,
> Upgrade/*,
> upgrade_groups/upgrade_items/UpgradeItem/status,
> minimal_response=true&_=1489680258782
> {code}
>
> This call gets turn into a LOT of queries, but some seem to be based on the number of stages. For example, this one on a very small upgrade produces very poor SQL...
>
> {code}
> SELECT
> stage_id,
> cluster_host_info,
> cluster_id,
> command_execution_type,
> command_params,
> display_status,
> host_params,
> log_info,
> request_context,
> request_id,
> skippable,
> status,
> supports_auto_skip_failure
> FROM stage
> WHERE ((request_id = ?)
> AND ((((((((stage_id = ?)
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?)))
> {code}
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java 8759844
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeItemResourceProvider.java bf0fa33
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java 49c1594
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntityPK.java 34d175c
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java eee913a
>
>
> Diff: https://reviews.apache.org/r/58218/diff/2/
>
>
> Testing
> -------
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 17:46 min
> [INFO] Finished at: 2017-04-05T17:30:09-04:00
> [INFO] Final Memory: 67M/861M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 58218: Upgrade Progress Dialog Executes Query Which
Causes StackOverflow in JPA
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58218/
-----------------------------------------------------------
(Updated April 5, 2017, 7:20 p.m.)
Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, and Sid Wagle.
Bugs: AMBARI-20685
https://issues.apache.org/jira/browse/AMBARI-20685
Repository: ambari
Description
-------
Large rolling upgrades (1000 hosts with 10,000 host components) creates a massive request object object with 10's of 1000's of stages and tasks. The web client executes a {{GET}} command against the API to retrieve upgrade groups/items. On a large upgrade, this causes the EclipseLink SQL generation code to run out of stack frame space while recursively building a query. To work around this, the stack frame size per thread can be increased using {{-Xss}} along with a value like 32M.
{code}
http://localhost:8080/api/v1/clusters/c1/upgrades/12?
upgrade_groups/UpgradeGroup/status!=PENDING&
fields=
Upgrade/progress_percent,
Upgrade/request_context,
Upgrade/request_status,
Upgrade/direction,
Upgrade/downgrade_allowed,
upgrade_groups/UpgradeGroup,
Upgrade/*,
upgrade_groups/upgrade_items/UpgradeItem/status,
minimal_response=true&_=1489680258782
{code}
This call gets turn into a LOT of queries, but some seem to be based on the number of stages. For example, this one on a very small upgrade produces very poor SQL...
{code}
SELECT
stage_id,
cluster_host_info,
cluster_id,
command_execution_type,
command_params,
display_status,
host_params,
log_info,
request_context,
request_id,
skippable,
status,
supports_auto_skip_failure
FROM stage
WHERE ((request_id = ?)
AND ((((((((stage_id = ?)
OR (stage_id = ?))
OR (stage_id = ?))
OR (stage_id = ?))
OR (stage_id = ?))
OR (stage_id = ?))
OR (stage_id = ?))
OR (stage_id = ?)))
{code}
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java 8759844
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeItemResourceProvider.java bf0fa33
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java 49c1594
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntityPK.java 34d175c
ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java eee913a
Diff: https://reviews.apache.org/r/58218/diff/2/
Testing (updated)
-------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 17:46 min
[INFO] Finished at: 2017-04-05T17:30:09-04:00
[INFO] Final Memory: 67M/861M
[INFO] ------------------------------------------------------------------------
Thanks,
Jonathan Hurley
Re: Review Request 58218: Upgrade Progress Dialog Executes Query Which
Causes StackOverflow in JPA
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58218/
-----------------------------------------------------------
(Updated April 5, 2017, 7:19 p.m.)
Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, and Sid Wagle.
Bugs: AMBARI-20685
https://issues.apache.org/jira/browse/AMBARI-20685
Repository: ambari
Description
-------
Large rolling upgrades (1000 hosts with 10,000 host components) creates a massive request object object with 10's of 1000's of stages and tasks. The web client executes a {{GET}} command against the API to retrieve upgrade groups/items. On a large upgrade, this causes the EclipseLink SQL generation code to run out of stack frame space while recursively building a query. To work around this, the stack frame size per thread can be increased using {{-Xss}} along with a value like 32M.
{code}
http://localhost:8080/api/v1/clusters/c1/upgrades/12?
upgrade_groups/UpgradeGroup/status!=PENDING&
fields=
Upgrade/progress_percent,
Upgrade/request_context,
Upgrade/request_status,
Upgrade/direction,
Upgrade/downgrade_allowed,
upgrade_groups/UpgradeGroup,
Upgrade/*,
upgrade_groups/upgrade_items/UpgradeItem/status,
minimal_response=true&_=1489680258782
{code}
This call gets turn into a LOT of queries, but some seem to be based on the number of stages. For example, this one on a very small upgrade produces very poor SQL...
{code}
SELECT
stage_id,
cluster_host_info,
cluster_id,
command_execution_type,
command_params,
display_status,
host_params,
log_info,
request_context,
request_id,
skippable,
status,
supports_auto_skip_failure
FROM stage
WHERE ((request_id = ?)
AND ((((((((stage_id = ?)
OR (stage_id = ?))
OR (stage_id = ?))
OR (stage_id = ?))
OR (stage_id = ?))
OR (stage_id = ?))
OR (stage_id = ?))
OR (stage_id = ?)))
{code}
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java 8759844
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeItemResourceProvider.java bf0fa33
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java 49c1594
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntityPK.java 34d175c
ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java eee913a
Diff: https://reviews.apache.org/r/58218/diff/2/
Changes: https://reviews.apache.org/r/58218/diff/1-2/
Testing
-------
PENDING
Thanks,
Jonathan Hurley
Re: Review Request 58218: Upgrade Progress Dialog Executes Query Which
Causes StackOverflow in JPA
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58218/#review171165
-----------------------------------------------------------
Ship it!
Ship It!
- Nate Cole
On April 5, 2017, 4:40 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58218/
> -----------------------------------------------------------
>
> (Updated April 5, 2017, 4:40 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, and Sid Wagle.
>
>
> Bugs: AMBARI-20685
> https://issues.apache.org/jira/browse/AMBARI-20685
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Large rolling upgrades (1000 hosts with 10,000 host components) creates a massive request object object with 10's of 1000's of stages and tasks. The web client executes a {{GET}} command against the API to retrieve upgrade groups/items. On a large upgrade, this causes the EclipseLink SQL generation code to run out of stack frame space while recursively building a query. To work around this, the stack frame size per thread can be increased using {{-Xss}} along with a value like 32M.
>
> {code}
> http://localhost:8080/api/v1/clusters/c1/upgrades/12?
> upgrade_groups/UpgradeGroup/status!=PENDING&
> fields=
> Upgrade/progress_percent,
> Upgrade/request_context,
> Upgrade/request_status,
> Upgrade/direction,
> Upgrade/downgrade_allowed,
> upgrade_groups/UpgradeGroup,
> Upgrade/*,
> upgrade_groups/upgrade_items/UpgradeItem/status,
> minimal_response=true&_=1489680258782
> {code}
>
> This call gets turn into a LOT of queries, but some seem to be based on the number of stages. For example, this one on a very small upgrade produces very poor SQL...
>
> {code}
> SELECT
> stage_id,
> cluster_host_info,
> cluster_id,
> command_execution_type,
> command_params,
> display_status,
> host_params,
> log_info,
> request_context,
> request_id,
> skippable,
> status,
> supports_auto_skip_failure
> FROM stage
> WHERE ((request_id = ?)
> AND ((((((((stage_id = ?)
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?)))
> {code}
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java 8759844
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeItemResourceProvider.java bf0fa33
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java 49c1594
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntityPK.java 34d175c
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java eee913a
>
>
> Diff: https://reviews.apache.org/r/58218/diff/1/
>
>
> Testing
> -------
>
> PENDING
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 58218: Upgrade Progress Dialog Executes Query Which
Causes StackOverflow in JPA
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On April 5, 2017, 5:16 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeItemResourceProvider.java
> > Lines 264 (patched)
> > <https://reviews.apache.org/r/58218/diff/1/?file=1685525#file1685525line271>
> >
> > Let's remove dead code
Ugh - I did remove it, but I didn't stage the change before creating the patch. Thanks!
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58218/#review171167
-----------------------------------------------------------
On April 5, 2017, 4:40 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58218/
> -----------------------------------------------------------
>
> (Updated April 5, 2017, 4:40 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, and Sid Wagle.
>
>
> Bugs: AMBARI-20685
> https://issues.apache.org/jira/browse/AMBARI-20685
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Large rolling upgrades (1000 hosts with 10,000 host components) creates a massive request object object with 10's of 1000's of stages and tasks. The web client executes a {{GET}} command against the API to retrieve upgrade groups/items. On a large upgrade, this causes the EclipseLink SQL generation code to run out of stack frame space while recursively building a query. To work around this, the stack frame size per thread can be increased using {{-Xss}} along with a value like 32M.
>
> {code}
> http://localhost:8080/api/v1/clusters/c1/upgrades/12?
> upgrade_groups/UpgradeGroup/status!=PENDING&
> fields=
> Upgrade/progress_percent,
> Upgrade/request_context,
> Upgrade/request_status,
> Upgrade/direction,
> Upgrade/downgrade_allowed,
> upgrade_groups/UpgradeGroup,
> Upgrade/*,
> upgrade_groups/upgrade_items/UpgradeItem/status,
> minimal_response=true&_=1489680258782
> {code}
>
> This call gets turn into a LOT of queries, but some seem to be based on the number of stages. For example, this one on a very small upgrade produces very poor SQL...
>
> {code}
> SELECT
> stage_id,
> cluster_host_info,
> cluster_id,
> command_execution_type,
> command_params,
> display_status,
> host_params,
> log_info,
> request_context,
> request_id,
> skippable,
> status,
> supports_auto_skip_failure
> FROM stage
> WHERE ((request_id = ?)
> AND ((((((((stage_id = ?)
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?)))
> {code}
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java 8759844
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeItemResourceProvider.java bf0fa33
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java 49c1594
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntityPK.java 34d175c
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java eee913a
>
>
> Diff: https://reviews.apache.org/r/58218/diff/1/
>
>
> Testing
> -------
>
> PENDING
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 58218: Upgrade Progress Dialog Executes Query Which
Causes StackOverflow in JPA
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58218/#review171167
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeItemResourceProvider.java
Lines 264 (patched)
<https://reviews.apache.org/r/58218/#comment244046>
Let's remove dead code
- Alejandro Fernandez
On April 5, 2017, 8:40 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58218/
> -----------------------------------------------------------
>
> (Updated April 5, 2017, 8:40 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, and Sid Wagle.
>
>
> Bugs: AMBARI-20685
> https://issues.apache.org/jira/browse/AMBARI-20685
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Large rolling upgrades (1000 hosts with 10,000 host components) creates a massive request object object with 10's of 1000's of stages and tasks. The web client executes a {{GET}} command against the API to retrieve upgrade groups/items. On a large upgrade, this causes the EclipseLink SQL generation code to run out of stack frame space while recursively building a query. To work around this, the stack frame size per thread can be increased using {{-Xss}} along with a value like 32M.
>
> {code}
> http://localhost:8080/api/v1/clusters/c1/upgrades/12?
> upgrade_groups/UpgradeGroup/status!=PENDING&
> fields=
> Upgrade/progress_percent,
> Upgrade/request_context,
> Upgrade/request_status,
> Upgrade/direction,
> Upgrade/downgrade_allowed,
> upgrade_groups/UpgradeGroup,
> Upgrade/*,
> upgrade_groups/upgrade_items/UpgradeItem/status,
> minimal_response=true&_=1489680258782
> {code}
>
> This call gets turn into a LOT of queries, but some seem to be based on the number of stages. For example, this one on a very small upgrade produces very poor SQL...
>
> {code}
> SELECT
> stage_id,
> cluster_host_info,
> cluster_id,
> command_execution_type,
> command_params,
> display_status,
> host_params,
> log_info,
> request_context,
> request_id,
> skippable,
> status,
> supports_auto_skip_failure
> FROM stage
> WHERE ((request_id = ?)
> AND ((((((((stage_id = ?)
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?))
> OR (stage_id = ?)))
> {code}
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java 8759844
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeItemResourceProvider.java bf0fa33
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java 49c1594
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntityPK.java 34d175c
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java eee913a
>
>
> Diff: https://reviews.apache.org/r/58218/diff/1/
>
>
> Testing
> -------
>
> PENDING
>
>
> Thanks,
>
> Jonathan Hurley
>
>