You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Jonathan Hurley <jh...@hortonworks.com> on 2014/09/10 05:15:50 UTC
Review Request 25490: Ambari Server REST API Memory Leak
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25490/
-----------------------------------------------------------
Review request for Ambari, Sid Wagle and Yusaku Sako.
Bugs: AMBARI-7221
https://issues.apache.org/jira/browse/AMBARI-7221
Repository: ambari
Description
-------
There is a memory leak for Jetty HashedSession instances created by Spring Security for requests to the REST endpoints that do not use AMBARISESSIONID to manage credentials. The web client does not see this memory leak since they use AMBARISESSIONID and Spring recognizes this token and re-uses an existing session instance.
However, pure stateless REST requests will create a brand new session for each request. These sessions are never cleaned up by the HashSessionManager.
The best solution here is to have Spring only create a session when AMBARISESSIONID is used, but there's no way to know that since an initial request is not required to contain a random ID. Setting the inactive timeout value for sessions alleviates the problem. Although sessions are created for pure stateless REST calls, they are reaped correctly.
Diffs
-----
ambari-server/conf/unix/ambari.properties b77ae320507aa5fd0a8b86bcf59841841bdf5690
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java a21f98c59d68346f4f8878ee813299049b399e61
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java fc74e000b9b189ae7623712db05872efa10a540c
Diff: https://reviews.apache.org/r/25490/diff/
Testing
-------
Measured JVM heap usage with jconsole with REST endpoints under stress testing in addition to normal web client stress. The heap climbed slowly for new sessions but as sessions became invalidated, they were moved to WeakReferences and then eventually garbage collected.
[INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 approved: 35 licence.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ........................................ SUCCESS [ 3.665 s]
[INFO] Apache Ambari Project POM .......................... SUCCESS [ 0.325 s]
[INFO] Ambari Web ......................................... SUCCESS [ 24.192 s]
[INFO] Ambari Views ....................................... SUCCESS [ 1.700 s]
[INFO] Ambari Admin View .................................. SUCCESS [ 9.078 s]
[INFO] Ambari Server ...................................... SUCCESS [16:58 min]
[INFO] Ambari Agent ....................................... SUCCESS [ 7.617 s]
[INFO] Ambari Client ...................................... SUCCESS [ 0.031 s]
[INFO] Ambari Python Client ............................... SUCCESS [ 0.368 s]
[INFO] Ambari Groovy Client ............................... SUCCESS [ 9.322 s]
[INFO] Ambari Shell ....................................... SUCCESS [ 0.032 s]
[INFO] Ambari Python Shell ................................ SUCCESS [ 0.040 s]
[INFO] Ambari Groovy Shell ................................ SUCCESS [ 5.153 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18:00 min
[INFO] Finished at: 2014-09-09T22:43:56-04:00
[INFO] Final Memory: 44M/236M
[INFO] ------------------------------------------------------------------------
Thanks,
Jonathan Hurley
Re: Review Request 25490: Ambari Server REST API Memory Leak
Posted by Yusaku Sako <yu...@hortonworks.com>.
> On Sept. 11, 2014, 5:09 p.m., Yusaku Sako wrote:
> > ambari-server/conf/unix/ambari.properties, line 58
> > <https://reviews.apache.org/r/25490/diff/2/?file=685560#file685560line58>
> >
> > This seems a bit aggressive.
> > I would suggest at least one hour.
>
> Jonathan Hurley wrote:
> This is an inactive timeout; not a total timeout. As long as you make a REST request with your AMBARISESSIONID within the timeout period, the session is refreshed. This timeout period only affects Ambari during a cluster install. If you wait 300 seconds in between steps, then you'll need to log back in.
>
> I can see 5 minutes being aggresive; but I think that 60 minutes would prevent the sessions from being reclaimed fast enough. I'm willing to move this to 10 minutes.
Yep, I understood it as inactivity timeout.
Consider the following scenario...
The user is in Cluster Install Wizard.
He gets to the page where you need to tweak configs, and spends a lot of time carefully configuring numerous parameters.
This typically takes longer than 5-10 minutes.
Then the user proceeds to deploy. The user is logged out. This is frustrating UX. Furthermore, not sure how this plays out today with the current code, but potentially the user can lose all the data configured.
Now, we can send "keep alive" packets from the web client to make sure the session does not get invalidated and also to manage session inactivity explicitly. However, it does not do that today so such change would have to accompy this patch if we were to keep the inactivity timeout short.
FWIW, even after implementing keep alive on the web client, we will have a smaller issue; if the user logs in, and then explicitly changes the browser location to some API end point to see JSON output for debugging purposes. Then goes to SSH, does some stuff. 5 minutes (or 10 minutes) later, he tries to make another call, and the user gets "403".
So these are issues that we need to consider when setting the inactivity timeout.
- Yusaku
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25490/#review53053
-----------------------------------------------------------
On Sept. 11, 2014, 3:09 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25490/
> -----------------------------------------------------------
>
> (Updated Sept. 11, 2014, 3:09 p.m.)
>
>
> Review request for Ambari, Nate Cole, Sid Wagle, and Yusaku Sako.
>
>
> Bugs: AMBARI-7221
> https://issues.apache.org/jira/browse/AMBARI-7221
>
>
> Repository: ambari
>
>
> Description
> -------
>
> There is a memory leak for Jetty HashedSession instances created by Spring Security for requests to the REST endpoints that do not use AMBARISESSIONID to manage credentials. The web client does not see this memory leak since they use AMBARISESSIONID and Spring recognizes this token and re-uses an existing session instance.
>
> However, pure stateless REST requests will create a brand new session for each request. These sessions are never cleaned up by the HashSessionManager.
>
> The best solution here is to have Spring only create a session when AMBARISESSIONID is used, but there's no way to know that since an initial request is not required to contain a random ID. Setting the inactive timeout value for sessions alleviates the problem. Although sessions are created for pure stateless REST calls, they are reaped correctly.
>
>
> Diffs
> -----
>
> ambari-server/conf/unix/ambari.properties 41cada7f32fad0ac9a6a12aee5233f13d7882390
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java a3a07b005ec3415f401d8ec0119844910135fd71
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java fc74e000b9b189ae7623712db05872efa10a540c
>
> Diff: https://reviews.apache.org/r/25490/diff/
>
>
> Testing
> -------
>
> Measured JVM heap usage with jconsole with REST endpoints under stress testing in addition to normal web client stress. The heap climbed slowly for new sessions but as sessions became invalidated, they were moved to WeakReferences and then eventually garbage collected.
>
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 approved: 35 licence.
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ........................................ SUCCESS [ 3.665 s]
> [INFO] Apache Ambari Project POM .......................... SUCCESS [ 0.325 s]
> [INFO] Ambari Web ......................................... SUCCESS [ 24.192 s]
> [INFO] Ambari Views ....................................... SUCCESS [ 1.700 s]
> [INFO] Ambari Admin View .................................. SUCCESS [ 9.078 s]
> [INFO] Ambari Server ...................................... SUCCESS [16:58 min]
> [INFO] Ambari Agent ....................................... SUCCESS [ 7.617 s]
> [INFO] Ambari Client ...................................... SUCCESS [ 0.031 s]
> [INFO] Ambari Python Client ............................... SUCCESS [ 0.368 s]
> [INFO] Ambari Groovy Client ............................... SUCCESS [ 9.322 s]
> [INFO] Ambari Shell ....................................... SUCCESS [ 0.032 s]
> [INFO] Ambari Python Shell ................................ SUCCESS [ 0.040 s]
> [INFO] Ambari Groovy Shell ................................ SUCCESS [ 5.153 s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 18:00 min
> [INFO] Finished at: 2014-09-09T22:43:56-04:00
> [INFO] Final Memory: 44M/236M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 25490: Ambari Server REST API Memory Leak
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On Sept. 11, 2014, 1:09 p.m., Yusaku Sako wrote:
> > ambari-server/conf/unix/ambari.properties, line 58
> > <https://reviews.apache.org/r/25490/diff/2/?file=685560#file685560line58>
> >
> > This seems a bit aggressive.
> > I would suggest at least one hour.
This is an inactive timeout; not a total timeout. As long as you make a REST request with your AMBARISESSIONID within the timeout period, the session is refreshed. This timeout period only affects Ambari during a cluster install. If you wait 300 seconds in between steps, then you'll need to log back in.
I can see 5 minutes being aggresive; but I think that 60 minutes would prevent the sessions from being reclaimed fast enough. I'm willing to move this to 10 minutes.
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25490/#review53053
-----------------------------------------------------------
On Sept. 11, 2014, 11:09 a.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25490/
> -----------------------------------------------------------
>
> (Updated Sept. 11, 2014, 11:09 a.m.)
>
>
> Review request for Ambari, Nate Cole, Sid Wagle, and Yusaku Sako.
>
>
> Bugs: AMBARI-7221
> https://issues.apache.org/jira/browse/AMBARI-7221
>
>
> Repository: ambari
>
>
> Description
> -------
>
> There is a memory leak for Jetty HashedSession instances created by Spring Security for requests to the REST endpoints that do not use AMBARISESSIONID to manage credentials. The web client does not see this memory leak since they use AMBARISESSIONID and Spring recognizes this token and re-uses an existing session instance.
>
> However, pure stateless REST requests will create a brand new session for each request. These sessions are never cleaned up by the HashSessionManager.
>
> The best solution here is to have Spring only create a session when AMBARISESSIONID is used, but there's no way to know that since an initial request is not required to contain a random ID. Setting the inactive timeout value for sessions alleviates the problem. Although sessions are created for pure stateless REST calls, they are reaped correctly.
>
>
> Diffs
> -----
>
> ambari-server/conf/unix/ambari.properties 41cada7f32fad0ac9a6a12aee5233f13d7882390
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java a3a07b005ec3415f401d8ec0119844910135fd71
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java fc74e000b9b189ae7623712db05872efa10a540c
>
> Diff: https://reviews.apache.org/r/25490/diff/
>
>
> Testing
> -------
>
> Measured JVM heap usage with jconsole with REST endpoints under stress testing in addition to normal web client stress. The heap climbed slowly for new sessions but as sessions became invalidated, they were moved to WeakReferences and then eventually garbage collected.
>
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 approved: 35 licence.
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ........................................ SUCCESS [ 3.665 s]
> [INFO] Apache Ambari Project POM .......................... SUCCESS [ 0.325 s]
> [INFO] Ambari Web ......................................... SUCCESS [ 24.192 s]
> [INFO] Ambari Views ....................................... SUCCESS [ 1.700 s]
> [INFO] Ambari Admin View .................................. SUCCESS [ 9.078 s]
> [INFO] Ambari Server ...................................... SUCCESS [16:58 min]
> [INFO] Ambari Agent ....................................... SUCCESS [ 7.617 s]
> [INFO] Ambari Client ...................................... SUCCESS [ 0.031 s]
> [INFO] Ambari Python Client ............................... SUCCESS [ 0.368 s]
> [INFO] Ambari Groovy Client ............................... SUCCESS [ 9.322 s]
> [INFO] Ambari Shell ....................................... SUCCESS [ 0.032 s]
> [INFO] Ambari Python Shell ................................ SUCCESS [ 0.040 s]
> [INFO] Ambari Groovy Shell ................................ SUCCESS [ 5.153 s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 18:00 min
> [INFO] Finished at: 2014-09-09T22:43:56-04:00
> [INFO] Final Memory: 44M/236M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 25490: Ambari Server REST API Memory Leak
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On Sept. 11, 2014, 1:09 p.m., Yusaku Sako wrote:
> > ambari-server/conf/unix/ambari.properties, line 58
> > <https://reviews.apache.org/r/25490/diff/2/?file=685560#file685560line58>
> >
> > This seems a bit aggressive.
> > I would suggest at least one hour.
>
> Jonathan Hurley wrote:
> This is an inactive timeout; not a total timeout. As long as you make a REST request with your AMBARISESSIONID within the timeout period, the session is refreshed. This timeout period only affects Ambari during a cluster install. If you wait 300 seconds in between steps, then you'll need to log back in.
>
> I can see 5 minutes being aggresive; but I think that 60 minutes would prevent the sessions from being reclaimed fast enough. I'm willing to move this to 10 minutes.
>
> Yusaku Sako wrote:
> Yep, I understood it as inactivity timeout.
> Consider the following scenario...
> The user is in Cluster Install Wizard.
> He gets to the page where you need to tweak configs, and spends a lot of time carefully configuring numerous parameters.
> This typically takes longer than 5-10 minutes.
> Then the user proceeds to deploy. The user is logged out. This is frustrating UX. Furthermore, not sure how this plays out today with the current code, but potentially the user can lose all the data configured.
> Now, we can send "keep alive" packets from the web client to make sure the session does not get invalidated and also to manage session inactivity explicitly. However, it does not do that today so such change would have to accompy this patch if we were to keep the inactivity timeout short.
> FWIW, even after implementing keep alive on the web client, we will have a smaller issue; if the user logs in, and then explicitly changes the browser location to some API end point to see JSON output for debugging purposes. Then goes to SSH, does some stuff. 5 minutes (or 10 minutes) later, he tries to make another call, and the user gets "403".
> So these are issues that we need to consider when setting the inactivity timeout.
I'm fine increasing the timeout a bit, but this discussion is also a reason we're exposing it; so that a customer can increase it in their environment if necessary. Let's address your points:
1) I think that if a client is choosing to use AMBARISESSIONID then they need to send keep-alive requests so that the session is refreshed. This should hold true for configuring a cluster where we can see a timeout problem.
2) For the scenario where a user is making manual requests and switching back and forth, they can always specify the credentials or the AMBARISESSIONID in their requests (or just keep the web client open). When using curl, they can also use the -cookie-store option so that curl requests will use AMBARISESSIONID automatically.
I turned up the timeout to 30 minutes and then 60 mintues and ran tests against the REST endpoints every second. After 30 minutes, the heap increased by about 35MB. For the 60 minute timout, the heap amount held hostage by the sessions was close to 75MB. Would a happy medium here be 30 minutes?
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25490/#review53053
-----------------------------------------------------------
On Sept. 11, 2014, 11:09 a.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25490/
> -----------------------------------------------------------
>
> (Updated Sept. 11, 2014, 11:09 a.m.)
>
>
> Review request for Ambari, Nate Cole, Sid Wagle, and Yusaku Sako.
>
>
> Bugs: AMBARI-7221
> https://issues.apache.org/jira/browse/AMBARI-7221
>
>
> Repository: ambari
>
>
> Description
> -------
>
> There is a memory leak for Jetty HashedSession instances created by Spring Security for requests to the REST endpoints that do not use AMBARISESSIONID to manage credentials. The web client does not see this memory leak since they use AMBARISESSIONID and Spring recognizes this token and re-uses an existing session instance.
>
> However, pure stateless REST requests will create a brand new session for each request. These sessions are never cleaned up by the HashSessionManager.
>
> The best solution here is to have Spring only create a session when AMBARISESSIONID is used, but there's no way to know that since an initial request is not required to contain a random ID. Setting the inactive timeout value for sessions alleviates the problem. Although sessions are created for pure stateless REST calls, they are reaped correctly.
>
>
> Diffs
> -----
>
> ambari-server/conf/unix/ambari.properties 41cada7f32fad0ac9a6a12aee5233f13d7882390
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java a3a07b005ec3415f401d8ec0119844910135fd71
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java fc74e000b9b189ae7623712db05872efa10a540c
>
> Diff: https://reviews.apache.org/r/25490/diff/
>
>
> Testing
> -------
>
> Measured JVM heap usage with jconsole with REST endpoints under stress testing in addition to normal web client stress. The heap climbed slowly for new sessions but as sessions became invalidated, they were moved to WeakReferences and then eventually garbage collected.
>
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 approved: 35 licence.
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ........................................ SUCCESS [ 3.665 s]
> [INFO] Apache Ambari Project POM .......................... SUCCESS [ 0.325 s]
> [INFO] Ambari Web ......................................... SUCCESS [ 24.192 s]
> [INFO] Ambari Views ....................................... SUCCESS [ 1.700 s]
> [INFO] Ambari Admin View .................................. SUCCESS [ 9.078 s]
> [INFO] Ambari Server ...................................... SUCCESS [16:58 min]
> [INFO] Ambari Agent ....................................... SUCCESS [ 7.617 s]
> [INFO] Ambari Client ...................................... SUCCESS [ 0.031 s]
> [INFO] Ambari Python Client ............................... SUCCESS [ 0.368 s]
> [INFO] Ambari Groovy Client ............................... SUCCESS [ 9.322 s]
> [INFO] Ambari Shell ....................................... SUCCESS [ 0.032 s]
> [INFO] Ambari Python Shell ................................ SUCCESS [ 0.040 s]
> [INFO] Ambari Groovy Shell ................................ SUCCESS [ 5.153 s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 18:00 min
> [INFO] Finished at: 2014-09-09T22:43:56-04:00
> [INFO] Final Memory: 44M/236M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 25490: Ambari Server REST API Memory Leak
Posted by Yusaku Sako <yu...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25490/#review53053
-----------------------------------------------------------
ambari-server/conf/unix/ambari.properties
<https://reviews.apache.org/r/25490/#comment92428>
This seems a bit aggressive.
I would suggest at least one hour.
- Yusaku Sako
On Sept. 11, 2014, 3:09 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25490/
> -----------------------------------------------------------
>
> (Updated Sept. 11, 2014, 3:09 p.m.)
>
>
> Review request for Ambari, Nate Cole, Sid Wagle, and Yusaku Sako.
>
>
> Bugs: AMBARI-7221
> https://issues.apache.org/jira/browse/AMBARI-7221
>
>
> Repository: ambari
>
>
> Description
> -------
>
> There is a memory leak for Jetty HashedSession instances created by Spring Security for requests to the REST endpoints that do not use AMBARISESSIONID to manage credentials. The web client does not see this memory leak since they use AMBARISESSIONID and Spring recognizes this token and re-uses an existing session instance.
>
> However, pure stateless REST requests will create a brand new session for each request. These sessions are never cleaned up by the HashSessionManager.
>
> The best solution here is to have Spring only create a session when AMBARISESSIONID is used, but there's no way to know that since an initial request is not required to contain a random ID. Setting the inactive timeout value for sessions alleviates the problem. Although sessions are created for pure stateless REST calls, they are reaped correctly.
>
>
> Diffs
> -----
>
> ambari-server/conf/unix/ambari.properties 41cada7f32fad0ac9a6a12aee5233f13d7882390
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java a3a07b005ec3415f401d8ec0119844910135fd71
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java fc74e000b9b189ae7623712db05872efa10a540c
>
> Diff: https://reviews.apache.org/r/25490/diff/
>
>
> Testing
> -------
>
> Measured JVM heap usage with jconsole with REST endpoints under stress testing in addition to normal web client stress. The heap climbed slowly for new sessions but as sessions became invalidated, they were moved to WeakReferences and then eventually garbage collected.
>
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 approved: 35 licence.
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ........................................ SUCCESS [ 3.665 s]
> [INFO] Apache Ambari Project POM .......................... SUCCESS [ 0.325 s]
> [INFO] Ambari Web ......................................... SUCCESS [ 24.192 s]
> [INFO] Ambari Views ....................................... SUCCESS [ 1.700 s]
> [INFO] Ambari Admin View .................................. SUCCESS [ 9.078 s]
> [INFO] Ambari Server ...................................... SUCCESS [16:58 min]
> [INFO] Ambari Agent ....................................... SUCCESS [ 7.617 s]
> [INFO] Ambari Client ...................................... SUCCESS [ 0.031 s]
> [INFO] Ambari Python Client ............................... SUCCESS [ 0.368 s]
> [INFO] Ambari Groovy Client ............................... SUCCESS [ 9.322 s]
> [INFO] Ambari Shell ....................................... SUCCESS [ 0.032 s]
> [INFO] Ambari Python Shell ................................ SUCCESS [ 0.040 s]
> [INFO] Ambari Groovy Shell ................................ SUCCESS [ 5.153 s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 18:00 min
> [INFO] Finished at: 2014-09-09T22:43:56-04:00
> [INFO] Final Memory: 44M/236M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 25490: Ambari Server REST API Memory Leak
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25490/#review53037
-----------------------------------------------------------
Ship it!
Ship It!
- Nate Cole
On Sept. 11, 2014, 11:09 a.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25490/
> -----------------------------------------------------------
>
> (Updated Sept. 11, 2014, 11:09 a.m.)
>
>
> Review request for Ambari, Nate Cole, Sid Wagle, and Yusaku Sako.
>
>
> Bugs: AMBARI-7221
> https://issues.apache.org/jira/browse/AMBARI-7221
>
>
> Repository: ambari
>
>
> Description
> -------
>
> There is a memory leak for Jetty HashedSession instances created by Spring Security for requests to the REST endpoints that do not use AMBARISESSIONID to manage credentials. The web client does not see this memory leak since they use AMBARISESSIONID and Spring recognizes this token and re-uses an existing session instance.
>
> However, pure stateless REST requests will create a brand new session for each request. These sessions are never cleaned up by the HashSessionManager.
>
> The best solution here is to have Spring only create a session when AMBARISESSIONID is used, but there's no way to know that since an initial request is not required to contain a random ID. Setting the inactive timeout value for sessions alleviates the problem. Although sessions are created for pure stateless REST calls, they are reaped correctly.
>
>
> Diffs
> -----
>
> ambari-server/conf/unix/ambari.properties 41cada7f32fad0ac9a6a12aee5233f13d7882390
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java a3a07b005ec3415f401d8ec0119844910135fd71
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java fc74e000b9b189ae7623712db05872efa10a540c
>
> Diff: https://reviews.apache.org/r/25490/diff/
>
>
> Testing
> -------
>
> Measured JVM heap usage with jconsole with REST endpoints under stress testing in addition to normal web client stress. The heap climbed slowly for new sessions but as sessions became invalidated, they were moved to WeakReferences and then eventually garbage collected.
>
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 approved: 35 licence.
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ........................................ SUCCESS [ 3.665 s]
> [INFO] Apache Ambari Project POM .......................... SUCCESS [ 0.325 s]
> [INFO] Ambari Web ......................................... SUCCESS [ 24.192 s]
> [INFO] Ambari Views ....................................... SUCCESS [ 1.700 s]
> [INFO] Ambari Admin View .................................. SUCCESS [ 9.078 s]
> [INFO] Ambari Server ...................................... SUCCESS [16:58 min]
> [INFO] Ambari Agent ....................................... SUCCESS [ 7.617 s]
> [INFO] Ambari Client ...................................... SUCCESS [ 0.031 s]
> [INFO] Ambari Python Client ............................... SUCCESS [ 0.368 s]
> [INFO] Ambari Groovy Client ............................... SUCCESS [ 9.322 s]
> [INFO] Ambari Shell ....................................... SUCCESS [ 0.032 s]
> [INFO] Ambari Python Shell ................................ SUCCESS [ 0.040 s]
> [INFO] Ambari Groovy Shell ................................ SUCCESS [ 5.153 s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 18:00 min
> [INFO] Finished at: 2014-09-09T22:43:56-04:00
> [INFO] Final Memory: 44M/236M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 25490: Ambari Server REST API Memory Leak
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25490/
-----------------------------------------------------------
(Updated Sept. 11, 2014, 7:57 p.m.)
Review request for Ambari, Nate Cole, Sid Wagle, and Yusaku Sako.
Changes
-------
Increased timeout to 30 minutes by default; This provides a good balance of heap to UX. Although we could try to do something in the future using Spring's concurrent session management, preliminary tests showed a worse leak of AmbariGrantedAuthority. I will open another Jira to track this.
Bugs: AMBARI-7221
https://issues.apache.org/jira/browse/AMBARI-7221
Repository: ambari
Description
-------
There is a memory leak for Jetty HashedSession instances created by Spring Security for requests to the REST endpoints that do not use AMBARISESSIONID to manage credentials. The web client does not see this memory leak since they use AMBARISESSIONID and Spring recognizes this token and re-uses an existing session instance.
However, pure stateless REST requests will create a brand new session for each request. These sessions are never cleaned up by the HashSessionManager.
The best solution here is to have Spring only create a session when AMBARISESSIONID is used, but there's no way to know that since an initial request is not required to contain a random ID. Setting the inactive timeout value for sessions alleviates the problem. Although sessions are created for pure stateless REST calls, they are reaped correctly.
Diffs (updated)
-----
ambari-server/conf/unix/ambari.properties 41cada7f32fad0ac9a6a12aee5233f13d7882390
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java a3a07b005ec3415f401d8ec0119844910135fd71
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java fc74e000b9b189ae7623712db05872efa10a540c
Diff: https://reviews.apache.org/r/25490/diff/
Testing
-------
Measured JVM heap usage with jconsole with REST endpoints under stress testing in addition to normal web client stress. The heap climbed slowly for new sessions but as sessions became invalidated, they were moved to WeakReferences and then eventually garbage collected.
[INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 approved: 35 licence.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ........................................ SUCCESS [ 3.665 s]
[INFO] Apache Ambari Project POM .......................... SUCCESS [ 0.325 s]
[INFO] Ambari Web ......................................... SUCCESS [ 24.192 s]
[INFO] Ambari Views ....................................... SUCCESS [ 1.700 s]
[INFO] Ambari Admin View .................................. SUCCESS [ 9.078 s]
[INFO] Ambari Server ...................................... SUCCESS [16:58 min]
[INFO] Ambari Agent ....................................... SUCCESS [ 7.617 s]
[INFO] Ambari Client ...................................... SUCCESS [ 0.031 s]
[INFO] Ambari Python Client ............................... SUCCESS [ 0.368 s]
[INFO] Ambari Groovy Client ............................... SUCCESS [ 9.322 s]
[INFO] Ambari Shell ....................................... SUCCESS [ 0.032 s]
[INFO] Ambari Python Shell ................................ SUCCESS [ 0.040 s]
[INFO] Ambari Groovy Shell ................................ SUCCESS [ 5.153 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18:00 min
[INFO] Finished at: 2014-09-09T22:43:56-04:00
[INFO] Final Memory: 44M/236M
[INFO] ------------------------------------------------------------------------
Thanks,
Jonathan Hurley
Re: Review Request 25490: Ambari Server REST API Memory Leak
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25490/
-----------------------------------------------------------
(Updated Sept. 11, 2014, 11:09 a.m.)
Review request for Ambari, Nate Cole, Sid Wagle, and Yusaku Sako.
Bugs: AMBARI-7221
https://issues.apache.org/jira/browse/AMBARI-7221
Repository: ambari
Description
-------
There is a memory leak for Jetty HashedSession instances created by Spring Security for requests to the REST endpoints that do not use AMBARISESSIONID to manage credentials. The web client does not see this memory leak since they use AMBARISESSIONID and Spring recognizes this token and re-uses an existing session instance.
However, pure stateless REST requests will create a brand new session for each request. These sessions are never cleaned up by the HashSessionManager.
The best solution here is to have Spring only create a session when AMBARISESSIONID is used, but there's no way to know that since an initial request is not required to contain a random ID. Setting the inactive timeout value for sessions alleviates the problem. Although sessions are created for pure stateless REST calls, they are reaped correctly.
Diffs
-----
ambari-server/conf/unix/ambari.properties 41cada7f32fad0ac9a6a12aee5233f13d7882390
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java a3a07b005ec3415f401d8ec0119844910135fd71
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java fc74e000b9b189ae7623712db05872efa10a540c
Diff: https://reviews.apache.org/r/25490/diff/
Testing
-------
Measured JVM heap usage with jconsole with REST endpoints under stress testing in addition to normal web client stress. The heap climbed slowly for new sessions but as sessions became invalidated, they were moved to WeakReferences and then eventually garbage collected.
[INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 approved: 35 licence.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ........................................ SUCCESS [ 3.665 s]
[INFO] Apache Ambari Project POM .......................... SUCCESS [ 0.325 s]
[INFO] Ambari Web ......................................... SUCCESS [ 24.192 s]
[INFO] Ambari Views ....................................... SUCCESS [ 1.700 s]
[INFO] Ambari Admin View .................................. SUCCESS [ 9.078 s]
[INFO] Ambari Server ...................................... SUCCESS [16:58 min]
[INFO] Ambari Agent ....................................... SUCCESS [ 7.617 s]
[INFO] Ambari Client ...................................... SUCCESS [ 0.031 s]
[INFO] Ambari Python Client ............................... SUCCESS [ 0.368 s]
[INFO] Ambari Groovy Client ............................... SUCCESS [ 9.322 s]
[INFO] Ambari Shell ....................................... SUCCESS [ 0.032 s]
[INFO] Ambari Python Shell ................................ SUCCESS [ 0.040 s]
[INFO] Ambari Groovy Shell ................................ SUCCESS [ 5.153 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18:00 min
[INFO] Finished at: 2014-09-09T22:43:56-04:00
[INFO] Final Memory: 44M/236M
[INFO] ------------------------------------------------------------------------
Thanks,
Jonathan Hurley
Re: Review Request 25490: Ambari Server REST API Memory Leak
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25490/
-----------------------------------------------------------
(Updated Sept. 11, 2014, 11:09 a.m.)
Review request for Ambari, Sid Wagle and Yusaku Sako.
Changes
-------
Increased default timeout to 5 minutes as this affects the wizard screens as well.
Bugs: AMBARI-7221
https://issues.apache.org/jira/browse/AMBARI-7221
Repository: ambari
Description
-------
There is a memory leak for Jetty HashedSession instances created by Spring Security for requests to the REST endpoints that do not use AMBARISESSIONID to manage credentials. The web client does not see this memory leak since they use AMBARISESSIONID and Spring recognizes this token and re-uses an existing session instance.
However, pure stateless REST requests will create a brand new session for each request. These sessions are never cleaned up by the HashSessionManager.
The best solution here is to have Spring only create a session when AMBARISESSIONID is used, but there's no way to know that since an initial request is not required to contain a random ID. Setting the inactive timeout value for sessions alleviates the problem. Although sessions are created for pure stateless REST calls, they are reaped correctly.
Diffs (updated)
-----
ambari-server/conf/unix/ambari.properties 41cada7f32fad0ac9a6a12aee5233f13d7882390
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java a3a07b005ec3415f401d8ec0119844910135fd71
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java fc74e000b9b189ae7623712db05872efa10a540c
Diff: https://reviews.apache.org/r/25490/diff/
Testing
-------
Measured JVM heap usage with jconsole with REST endpoints under stress testing in addition to normal web client stress. The heap climbed slowly for new sessions but as sessions became invalidated, they were moved to WeakReferences and then eventually garbage collected.
[INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 approved: 35 licence.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ........................................ SUCCESS [ 3.665 s]
[INFO] Apache Ambari Project POM .......................... SUCCESS [ 0.325 s]
[INFO] Ambari Web ......................................... SUCCESS [ 24.192 s]
[INFO] Ambari Views ....................................... SUCCESS [ 1.700 s]
[INFO] Ambari Admin View .................................. SUCCESS [ 9.078 s]
[INFO] Ambari Server ...................................... SUCCESS [16:58 min]
[INFO] Ambari Agent ....................................... SUCCESS [ 7.617 s]
[INFO] Ambari Client ...................................... SUCCESS [ 0.031 s]
[INFO] Ambari Python Client ............................... SUCCESS [ 0.368 s]
[INFO] Ambari Groovy Client ............................... SUCCESS [ 9.322 s]
[INFO] Ambari Shell ....................................... SUCCESS [ 0.032 s]
[INFO] Ambari Python Shell ................................ SUCCESS [ 0.040 s]
[INFO] Ambari Groovy Shell ................................ SUCCESS [ 5.153 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18:00 min
[INFO] Finished at: 2014-09-09T22:43:56-04:00
[INFO] Final Memory: 44M/236M
[INFO] ------------------------------------------------------------------------
Thanks,
Jonathan Hurley
Re: Review Request 25490: Ambari Server REST API Memory Leak
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25490/#review52815
-----------------------------------------------------------
Ship it!
Ship It!
- Sid Wagle
On Sept. 10, 2014, 3:15 a.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25490/
> -----------------------------------------------------------
>
> (Updated Sept. 10, 2014, 3:15 a.m.)
>
>
> Review request for Ambari, Sid Wagle and Yusaku Sako.
>
>
> Bugs: AMBARI-7221
> https://issues.apache.org/jira/browse/AMBARI-7221
>
>
> Repository: ambari
>
>
> Description
> -------
>
> There is a memory leak for Jetty HashedSession instances created by Spring Security for requests to the REST endpoints that do not use AMBARISESSIONID to manage credentials. The web client does not see this memory leak since they use AMBARISESSIONID and Spring recognizes this token and re-uses an existing session instance.
>
> However, pure stateless REST requests will create a brand new session for each request. These sessions are never cleaned up by the HashSessionManager.
>
> The best solution here is to have Spring only create a session when AMBARISESSIONID is used, but there's no way to know that since an initial request is not required to contain a random ID. Setting the inactive timeout value for sessions alleviates the problem. Although sessions are created for pure stateless REST calls, they are reaped correctly.
>
>
> Diffs
> -----
>
> ambari-server/conf/unix/ambari.properties b77ae320507aa5fd0a8b86bcf59841841bdf5690
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java a21f98c59d68346f4f8878ee813299049b399e61
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java fc74e000b9b189ae7623712db05872efa10a540c
>
> Diff: https://reviews.apache.org/r/25490/diff/
>
>
> Testing
> -------
>
> Measured JVM heap usage with jconsole with REST endpoints under stress testing in addition to normal web client stress. The heap climbed slowly for new sessions but as sessions became invalidated, they were moved to WeakReferences and then eventually garbage collected.
>
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 approved: 35 licence.
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ........................................ SUCCESS [ 3.665 s]
> [INFO] Apache Ambari Project POM .......................... SUCCESS [ 0.325 s]
> [INFO] Ambari Web ......................................... SUCCESS [ 24.192 s]
> [INFO] Ambari Views ....................................... SUCCESS [ 1.700 s]
> [INFO] Ambari Admin View .................................. SUCCESS [ 9.078 s]
> [INFO] Ambari Server ...................................... SUCCESS [16:58 min]
> [INFO] Ambari Agent ....................................... SUCCESS [ 7.617 s]
> [INFO] Ambari Client ...................................... SUCCESS [ 0.031 s]
> [INFO] Ambari Python Client ............................... SUCCESS [ 0.368 s]
> [INFO] Ambari Groovy Client ............................... SUCCESS [ 9.322 s]
> [INFO] Ambari Shell ....................................... SUCCESS [ 0.032 s]
> [INFO] Ambari Python Shell ................................ SUCCESS [ 0.040 s]
> [INFO] Ambari Groovy Shell ................................ SUCCESS [ 5.153 s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 18:00 min
> [INFO] Finished at: 2014-09-09T22:43:56-04:00
> [INFO] Final Memory: 44M/236M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jonathan Hurley
>
>