You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@ambari.apache.org by Jonathan Hurley <jh...@hortonworks.com> on 2015/06/26 23:57:04 UTC

Review Request 35945: Memory Exhausted During Upgrade Of Large Cluster

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35945/
-----------------------------------------------------------

Review request for Ambari, Nate Cole and Tom Beerbower.

Bugs: AMBARI-12178
https://issues.apache.org/jira/browse/AMBARI-12178

Repository: ambari

Description
-------

During an upgrade of a large cluster, the memory used by Ambari grows until it is fully consumed. This, however, only happens when the Upgrade Dialog page is open. If that popup is closed, the memory usage stays relatively constant. Based on heap dumps, the larges offenders are StageEnity and, as a result, byte[]

Long story short here is that we have a cache in ActionDBAccessorImpl that holds onto entities. Because of this, the underlying UnitOfWork map is never released and holds onto all StageEntity instances. Eventually, items are purged from this cache, but it's not fast enough to free up memory resources.

Without ripping apart Ambari or making dangerous cache reference changes, the easiest solution was to ensure that the fields causing the StageEntity to be large were lazy loaded since most of the time these entities are just sitting around in the EntityManager.

Diffs
-----

ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java 664fae3
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeGroupResourceProvider.java eb34d63
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/StageDAO.java b354841
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostEntity.java 9f3f70c
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java c2b97d6
ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java 31363b4

Diff: https://reviews.apache.org/r/35945/diff/

Testing
-------

Performed a cluster upgrade and monitored the memory usage. 200,000 StageEntity used to occupy 3.5GB of heap; now they take up about 150MB.

Thanks,

Jonathan Hurley

Re: Review Request 35945: Memory Exhausted During Upgrade Of Large Cluster

Posted by Jonathan Hurley <jh...@hortonworks.com>.


> On June 26, 2015, 6:09 p.m., Tom Beerbower wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java, lines 358-359
> > <https://reviews.apache.org/r/35945/diff/1/?file=993620#file993620line358>
> >
> >     Doesn't setResourceProperty make the same check?

Yes it does, but with `setResourceProperty`, we have to pass in `entity.getCommandParamsStage()` which actually causes it to be populated. So, we'd be populating it even if it wasn't requested.


- Jonathan


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35945/#review89579
-----------------------------------------------------------


On June 26, 2015, 5:57 p.m., Jonathan Hurley wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35945/
> -----------------------------------------------------------
> 
> (Updated June 26, 2015, 5:57 p.m.)
> 
> 
> Review request for Ambari, Nate Cole and Tom Beerbower.
> 
> 
> Bugs: AMBARI-12178
>     https://issues.apache.org/jira/browse/AMBARI-12178
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> During an upgrade of a large cluster, the memory used by Ambari grows until it is fully consumed. This, however, only happens when the Upgrade Dialog page is open. If that popup is closed, the memory usage stays relatively constant. Based on heap dumps, the larges offenders are StageEnity and, as a result, byte[]
> 
> Long story short here is that we have a cache in ActionDBAccessorImpl that holds onto entities. Because of this, the underlying UnitOfWork map is never released and holds onto all StageEntity instances. Eventually, items are purged from this cache, but it's not fast enough to free up memory resources.
> 
> Without ripping apart Ambari or making dangerous cache reference changes, the easiest solution was to ensure that the fields causing the StageEntity to be large were lazy loaded since most of the time these entities are just sitting around in the EntityManager.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java 664fae3 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeGroupResourceProvider.java eb34d63 
>   ambari-server/src/main/java/org/apache/ambari/server/orm/dao/StageDAO.java b354841 
>   ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostEntity.java 9f3f70c 
>   ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java c2b97d6 
>   ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java 31363b4 
> 
> Diff: https://reviews.apache.org/r/35945/diff/
> 
> 
> Testing
> -------
> 
> Performed a cluster upgrade and monitored the memory usage. 200,000 StageEntity used to occupy 3.5GB of heap; now they take up about 150MB.
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 35945: Memory Exhausted During Upgrade Of Large Cluster

Posted by Tom Beerbower <tb...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35945/#review89579
-----------------------------------------------------------

Ship it!



ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java (lines 349 - 350)
<https://reviews.apache.org/r/35945/#comment142207>

    Doesn't setResourceProperty make the same check?


- Tom Beerbower


On June 26, 2015, 9:57 p.m., Jonathan Hurley wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35945/
> -----------------------------------------------------------
> 
> (Updated June 26, 2015, 9:57 p.m.)
> 
> 
> Review request for Ambari, Nate Cole and Tom Beerbower.
> 
> 
> Bugs: AMBARI-12178
>     https://issues.apache.org/jira/browse/AMBARI-12178
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> During an upgrade of a large cluster, the memory used by Ambari grows until it is fully consumed. This, however, only happens when the Upgrade Dialog page is open. If that popup is closed, the memory usage stays relatively constant. Based on heap dumps, the larges offenders are StageEnity and, as a result, byte[]
> 
> Long story short here is that we have a cache in ActionDBAccessorImpl that holds onto entities. Because of this, the underlying UnitOfWork map is never released and holds onto all StageEntity instances. Eventually, items are purged from this cache, but it's not fast enough to free up memory resources.
> 
> Without ripping apart Ambari or making dangerous cache reference changes, the easiest solution was to ensure that the fields causing the StageEntity to be large were lazy loaded since most of the time these entities are just sitting around in the EntityManager.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java 664fae3 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeGroupResourceProvider.java eb34d63 
>   ambari-server/src/main/java/org/apache/ambari/server/orm/dao/StageDAO.java b354841 
>   ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostEntity.java 9f3f70c 
>   ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java c2b97d6 
>   ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java 31363b4 
> 
> Diff: https://reviews.apache.org/r/35945/diff/
> 
> 
> Testing
> -------
> 
> Performed a cluster upgrade and monitored the memory usage. 200,000 StageEntity used to occupy 3.5GB of heap; now they take up about 150MB.
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 35945: Memory Exhausted During Upgrade Of Large Cluster

Posted by Jonathan Hurley <jh...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35945/
-----------------------------------------------------------

(Updated June 26, 2015, 10:34 p.m.)


Review request for Ambari, Mahadev Konar, Nate Cole, Sumit Mohanty, and Tom Beerbower.


Bugs: AMBARI-12178
    https://issues.apache.org/jira/browse/AMBARI-12178


Repository: ambari


Description (updated)
-------

During an upgrade of a large cluster, the memory used by Ambari grows until it is fully consumed. This, however, only happens when the Upgrade Dialog page is open. If that popup is closed, the memory usage stays relatively constant. Based on heap dumps, the largest offenders are StageEnity and, as a result, byte[]

Long story short here is that we have a cache in ActionDBAccessorImpl that holds onto entities. Because of this, the underlying UnitOfWork map is never released and holds onto all StageEntity instances. Eventually, items are purged from this cache, but it's not fast enough to free up memory resources.

Without ripping apart Ambari or making dangerous cache reference changes, the easiest solution was to ensure that the fields causing the StageEntity to be large were lazy loaded since most of the time these entities are just sitting around in the EntityManager.

Before change
```
Class Name                                       | Objects | Shallow Heap | Retained Heap
------------------------------------------------------------------------------------------
org.apache.ambari.server.orm.entities.StageEntity| 292,356 |   18,466,176 | 3,575,693,136
------------------------------------------------------------------------------------------
```

First patch
```
Class Name                                       | Objects | Shallow Heap | Retained Heap
------------------------------------------------------------------------------------------
org.apache.ambari.server.orm.entities.StageEntity| 193,715 |   15,716,640 |   255,318,392
------------------------------------------------------------------------------------------
```

Second patch
```
Class Name                                       | Objects | Shallow Heap | Retained Heap
------------------------------------------------------------------------------------------
org.apache.ambari.server.orm.entities.StageEntity|   4,410 |      423,360 |    61,598,736
------------------------------------------------------------------------------------------
```


Diffs
-----

  ambari-server/src/main/java/org/apache/ambari/server/actionmanager/HostRoleCommand.java 20ec9ea 
  ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java 664fae3 
  ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeGroupResourceProvider.java eb34d63 
  ambari-server/src/main/java/org/apache/ambari/server/orm/dao/StageDAO.java b354841 
  ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostEntity.java 9f3f70c 
  ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java c2b97d6 
  ambari-server/src/main/java/org/apache/ambari/server/topology/HostRequest.java f63ba3f 
  ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java 31363b4 

Diff: https://reviews.apache.org/r/35945/diff/


Testing
-------

Performed a cluster upgrade and monitored the memory usage. 200,000 StageEntity used to occupy 3.5GB of heap; now they take up about 150MB.

Tests run: 3099, Failures: 0, Errors: 0, Skipped: 28

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 24:13 min
[INFO] Finished at: 2015-06-26T21:32:27-04:00
[INFO] Final Memory: 46M/1414M
[INFO] ------------------------------------------------------------------------


Thanks,

Jonathan Hurley

Re: Review Request 35945: Memory Exhausted During Upgrade Of Large Cluster

Posted by Jonathan Hurley <jh...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35945/
-----------------------------------------------------------

(Updated June 26, 2015, 10:29 p.m.)


Review request for Ambari, Mahadev Konar, Nate Cole, Sumit Mohanty, and Tom Beerbower.


Bugs: AMBARI-12178
    https://issues.apache.org/jira/browse/AMBARI-12178


Repository: ambari


Description
-------

During an upgrade of a large cluster, the memory used by Ambari grows until it is fully consumed. This, however, only happens when the Upgrade Dialog page is open. If that popup is closed, the memory usage stays relatively constant. Based on heap dumps, the larges offenders are StageEnity and, as a result, byte[]

Long story short here is that we have a cache in ActionDBAccessorImpl that holds onto entities. Because of this, the underlying UnitOfWork map is never released and holds onto all StageEntity instances. Eventually, items are purged from this cache, but it's not fast enough to free up memory resources.

Without ripping apart Ambari or making dangerous cache reference changes, the easiest solution was to ensure that the fields causing the StageEntity to be large were lazy loaded since most of the time these entities are just sitting around in the EntityManager.


Diffs
-----

  ambari-server/src/main/java/org/apache/ambari/server/actionmanager/HostRoleCommand.java 20ec9ea 
  ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java 664fae3 
  ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeGroupResourceProvider.java eb34d63 
  ambari-server/src/main/java/org/apache/ambari/server/orm/dao/StageDAO.java b354841 
  ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostEntity.java 9f3f70c 
  ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java c2b97d6 
  ambari-server/src/main/java/org/apache/ambari/server/topology/HostRequest.java f63ba3f 
  ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java 31363b4 

Diff: https://reviews.apache.org/r/35945/diff/


Testing (updated)
-------

Performed a cluster upgrade and monitored the memory usage. 200,000 StageEntity used to occupy 3.5GB of heap; now they take up about 150MB.

Tests run: 3099, Failures: 0, Errors: 0, Skipped: 28

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 24:13 min
[INFO] Finished at: 2015-06-26T21:32:27-04:00
[INFO] Final Memory: 46M/1414M
[INFO] ------------------------------------------------------------------------


Thanks,

Jonathan Hurley

Re: Review Request 35945: Memory Exhausted During Upgrade Of Large Cluster

Posted by Jonathan Hurley <jh...@hortonworks.com>.

(Updated June 26, 2015, 9:35 p.m.)

Review request for Ambari, Nate Cole and Tom Beerbower.

Changes
-------

In addition to lazy loading the byte[] fields, it makes sense to elimintate the inadvertent caching of the JPA entities as well. This cuts down dramatically on the objects being held onto. The Stage instances still hold onto entities, so we need the lazy byte[] fields anyway.

Bugs: AMBARI-12178
https://issues.apache.org/jira/browse/AMBARI-12178

Repository: ambari

Description
-------

Diffs (updated)
-----

ambari-server/src/main/java/org/apache/ambari/server/actionmanager/HostRoleCommand.java 20ec9ea
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java 664fae3
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeGroupResourceProvider.java eb34d63
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/StageDAO.java b354841
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostEntity.java 9f3f70c
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java c2b97d6
ambari-server/src/main/java/org/apache/ambari/server/topology/HostRequest.java f63ba3f
ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java 31363b4

Diff: https://reviews.apache.org/r/35945/diff/

Testing
-------

Performed a cluster upgrade and monitored the memory usage. 200,000 StageEntity used to occupy 3.5GB of heap; now they take up about 150MB.

Thanks,

Jonathan Hurley

Re: Review Request 35945: Memory Exhausted During Upgrade Of Large Cluster

Posted by Jonathan Hurley <jh...@hortonworks.com>.

(Updated June 26, 2015, 5:57 p.m.)

Review request for Ambari, Nate Cole and Tom Beerbower.

Bugs: AMBARI-12178
https://issues.apache.org/jira/browse/AMBARI-12178

Repository: ambari

Description
-------

Diffs
-----

Diff: https://reviews.apache.org/r/35945/diff/

Testing
-------

Performed a cluster upgrade and monitored the memory usage. 200,000 StageEntity used to occupy 3.5GB of heap; now they take up about 150MB.

Thanks,

Jonathan Hurley