You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@isis.apache.org by Leandro D'Agostino <L....@pocos.nl> on 2019/09/05 10:27:50 UTC
Re: Memory leak related to permissions
Hi Andi,
So from your answer I understand that tomcat's session persistence mode
is the cause of accumulating sessions without releasing them. But I then
would expect when sessions expire, after the session-timeout period,
that related objects will eventually be garbage collected. But objects
just keep accumulating over a period way longer then the session-timeout
period.
Could you tell where you did find the hints that for basic-auth no
HttpSession objects should be created?
Thanks,
Leandro
On 8/3/19 4:47 PM, Andi Huber wrote:
> Again, I'm not entirely sure, but I've found some hints that for basic-auth strategy, we don't want the servlet container to create any HttpSession objects at all. So a possible fix would be to tell Shiro not to create a HttpSession object, whenever we are using basic-auth strategy.
>
> I'm working on this ...
>
> // Basic auth should never create sessions ...
>
> request.setAttribute("org.apache.shiro.subject.support.DefaultSubjectContext.SESSION_CREATION_ENABLED", Boolean.FALSE);
>
> On 2019/08/03 12:52:49, Andi Huber <ah...@apache.org> wrote:
>> I think I tracked down the reason why these PrincipalForApplicationUser instances won't get garbage collected:
>>
>> With Basic-Auth as the authentication strategy, each request to the REST endpoint spawns a new HTTP Session, which holds a reference to a collection of PrincipalForApplicationUser instances.
>>
>> Now when tomcat runs in a mode, where it keeps all the sessions (session persistence mode) these objects cannot be garbage collected.
>>
>> I'm not a 100% sure but, it seems the described behavior is as it should be and hence a non-issue.
>>
>> However, I'm currently investigating, whether I got it all wrong, or there is a convenient solution to this, eg. don't create that many HTTP session objects.
>>
>> Let me know what you think!
>>
>> Cheers, Andi
>>
>> On 2019/07/23 07:29:26, Leandro D'Agostino <L....@pocos.nl> wrote:
>>> Thanks Andi!
>>>
>>> We look forward to your findings.
>>>
>>> Leandro
>>>
>>> On 7/23/19 9:11 AM, Andi Huber wrote:
>>>> We will certainly investigate this.
>>>>
>>>> Thanks for the effort of tracking this down!
>>>>
>>>> I've opened a Jira ticket [1].
>>>>
>>>> KR Andi
>>>>
>>>> [1] https://issues.apache.org/jira/browse/ISIS-2156
>>>>
>>>> On 2019/07/22 14:30:44, Leandro D'Agostino <L....@pocos.nl> wrote:
>>>>> Hi,
>>>>>
>>>>> We ran into the issue that our application keeps building up memory but
>>>>> never releases it.
>>>>> We could track it down to the class PrincipalForApplicationUser. Every
>>>>> time a user is authenticated, a new PrincipalForApplicationUser object
>>>>> is created and it is then never released. We experience this memory
>>>>> issue after a change of increasing the number of permissions from 20 to
>>>>> about 200. When comparing the behaviour of the application before and
>>>>> after the change we see that the retained size of the
>>>>> PrincipalForApplicationUser class has increased from about 15KB per
>>>>> instance to about 210KB per instance.
>>>>>
>>>>> For our investigation we created a test environment based on the
>>>>> simpleapp application to be able to isolate the issue and reproduce the
>>>>> issue in a minimalistic environment. So the simpleapp application was
>>>>> extended with 200 fields on which permissions are set. In our test we
>>>>> set the maximum memory size of the VM to 200MB. The test application
>>>>> uses about 40MB initially. We then start firing requests to the
>>>>> application (using jmeter), to:
>>>>> /restful/services/simple.SimpleObjectMenu
>>>>> What we observe then is that memory usage grows rather quickly and
>>>>> eventually it is exhausted.
>>>>>
>>>>> Can you help with a solution for this issue?
>>>>>
>>>>> The simpleapp test application we used is available on github:
>>>>> https://github.com/pocos-nl/isis-simpleapp-memoryissues
>>>>>
>>>>> It also includes the jmeter test script:
>>>>> memoryLeak-bareApp-SecurityPerformanceTestObject.jmx
>>>>>
>>>>> Thanks,
>>>>> Leandro D'Agostino
>>>>>
>>>>>
>>>>>
Re: Memory leak related to permissions
Posted by Martin Grigorov <mg...@apache.org>.
Hi,
Keeping your requests stateless is a best practice indeed. It helps to
scale your application easily because there is no need of session
replication, sticky sessions/nodes, etc.
Each request can go to any server node without any preparations.
One can easily debug which part of the code creates an http session by
registering a HttpSessionListener. If
https://docs.oracle.com/javaee/6/api/javax/servlet/http/HttpSessionListener.html#sessionCreated(javax.servlet.http.HttpSessionEvent)
is
called then throw an exception and see the stacktrace.
If the objects stay after the session expiration then most probably there
is another problem.
I'd recommend to use Eclipse MAT (Memory Analyzer Tool) to investigate
what's in the heap memory of the application and what keeps hard references
to the problematic instances.
Cheers,
Martin
On Tue, Sep 24, 2019 at 12:38 PM Andi Huber <ah...@apache.org> wrote:
> Sorry for the late reply!
>
> I'm certainly no expert on this. It only seems best practice to consider
> talking to
> REST endpoints a stateless conversation. Meaning the http request/response
> cycle should not carry over any state from one request to the next. There
> is of
> course state, the state of the domain-objects, but that belongs to a
> different tier.
>
> Not sure if that answers your question. I'm also open if someone can shine
> more light on this topic.
>
> On 2019/09/05 10:27:50, Leandro D'Agostino <L....@pocos.nl> wrote:
> > Hi Andi,
> >
> > So from your answer I understand that tomcat's session persistence mode
> > is the cause of accumulating sessions without releasing them. But I then
> > would expect when sessions expire, after the session-timeout period,
> > that related objects will eventually be garbage collected. But objects
> > just keep accumulating over a period way longer then the session-timeout
> > period.
> >
> > Could you tell where you did find the hints that for basic-auth no
> > HttpSession objects should be created?
> >
> > Thanks,
> > Leandro
> >
> >
> > On 8/3/19 4:47 PM, Andi Huber wrote:
> > > Again, I'm not entirely sure, but I've found some hints that for
> basic-auth strategy, we don't want the servlet container to create any
> HttpSession objects at all. So a possible fix would be to tell Shiro not to
> create a HttpSession object, whenever we are using basic-auth strategy.
> > >
> > > I'm working on this ...
> > >
> > > // Basic auth should never create sessions ...
> > >
> > >
> request.setAttribute("org.apache.shiro.subject.support.DefaultSubjectContext.SESSION_CREATION_ENABLED",
> Boolean.FALSE);
> > >
> > > On 2019/08/03 12:52:49, Andi Huber <ah...@apache.org> wrote:
> > >> I think I tracked down the reason why these
> PrincipalForApplicationUser instances won't get garbage collected:
> > >>
> > >> With Basic-Auth as the authentication strategy, each request to the
> REST endpoint spawns a new HTTP Session, which holds a reference to a
> collection of PrincipalForApplicationUser instances.
> > >>
> > >> Now when tomcat runs in a mode, where it keeps all the sessions
> (session persistence mode) these objects cannot be garbage collected.
> > >>
> > >> I'm not a 100% sure but, it seems the described behavior is as it
> should be and hence a non-issue.
> > >>
> > >> However, I'm currently investigating, whether I got it all wrong, or
> there is a convenient solution to this, eg. don't create that many HTTP
> session objects.
> > >>
> > >> Let me know what you think!
> > >>
> > >> Cheers, Andi
> > >>
> > >> On 2019/07/23 07:29:26, Leandro D'Agostino <L....@pocos.nl>
> wrote:
> > >>> Thanks Andi!
> > >>>
> > >>> We look forward to your findings.
> > >>>
> > >>> Leandro
> > >>>
> > >>> On 7/23/19 9:11 AM, Andi Huber wrote:
> > >>>> We will certainly investigate this.
> > >>>>
> > >>>> Thanks for the effort of tracking this down!
> > >>>>
> > >>>> I've opened a Jira ticket [1].
> > >>>>
> > >>>> KR Andi
> > >>>>
> > >>>> [1] https://issues.apache.org/jira/browse/ISIS-2156
> > >>>>
> > >>>> On 2019/07/22 14:30:44, Leandro D'Agostino <L....@pocos.nl>
> wrote:
> > >>>>> Hi,
> > >>>>>
> > >>>>> We ran into the issue that our application keeps building up
> memory but
> > >>>>> never releases it.
> > >>>>> We could track it down to the class PrincipalForApplicationUser.
> Every
> > >>>>> time a user is authenticated, a new PrincipalForApplicationUser
> object
> > >>>>> is created and it is then never released. We experience this memory
> > >>>>> issue after a change of increasing the number of permissions from
> 20 to
> > >>>>> about 200. When comparing the behaviour of the application before
> and
> > >>>>> after the change we see that the retained size of the
> > >>>>> PrincipalForApplicationUser class has increased from about 15KB per
> > >>>>> instance to about 210KB per instance.
> > >>>>>
> > >>>>> For our investigation we created a test environment based on the
> > >>>>> simpleapp application to be able to isolate the issue and
> reproduce the
> > >>>>> issue in a minimalistic environment. So the simpleapp application
> was
> > >>>>> extended with 200 fields on which permissions are set. In our test
> we
> > >>>>> set the maximum memory size of the VM to 200MB. The test
> application
> > >>>>> uses about 40MB initially. We then start firing requests to the
> > >>>>> application (using jmeter), to:
> > >>>>> /restful/services/simple.SimpleObjectMenu
> > >>>>> What we observe then is that memory usage grows rather quickly and
> > >>>>> eventually it is exhausted.
> > >>>>>
> > >>>>> Can you help with a solution for this issue?
> > >>>>>
> > >>>>> The simpleapp test application we used is available on github:
> > >>>>> https://github.com/pocos-nl/isis-simpleapp-memoryissues
> > >>>>>
> > >>>>> It also includes the jmeter test script:
> > >>>>> memoryLeak-bareApp-SecurityPerformanceTestObject.jmx
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Leandro D'Agostino
> > >>>>>
> > >>>>>
> > >>>>>
> >
>
Re: Memory leak related to permissions
Posted by Andi Huber <ah...@apache.org>.
Sorry for the late reply!
I'm certainly no expert on this. It only seems best practice to consider talking to
REST endpoints a stateless conversation. Meaning the http request/response
cycle should not carry over any state from one request to the next. There is of
course state, the state of the domain-objects, but that belongs to a different tier.
Not sure if that answers your question. I'm also open if someone can shine
more light on this topic.
On 2019/09/05 10:27:50, Leandro D'Agostino <L....@pocos.nl> wrote:
> Hi Andi,
>
> So from your answer I understand that tomcat's session persistence mode
> is the cause of accumulating sessions without releasing them. But I then
> would expect when sessions expire, after the session-timeout period,
> that related objects will eventually be garbage collected. But objects
> just keep accumulating over a period way longer then the session-timeout
> period.
>
> Could you tell where you did find the hints that for basic-auth no
> HttpSession objects should be created?
>
> Thanks,
> Leandro
>
>
> On 8/3/19 4:47 PM, Andi Huber wrote:
> > Again, I'm not entirely sure, but I've found some hints that for basic-auth strategy, we don't want the servlet container to create any HttpSession objects at all. So a possible fix would be to tell Shiro not to create a HttpSession object, whenever we are using basic-auth strategy.
> >
> > I'm working on this ...
> >
> > // Basic auth should never create sessions ...
> >
> > request.setAttribute("org.apache.shiro.subject.support.DefaultSubjectContext.SESSION_CREATION_ENABLED", Boolean.FALSE);
> >
> > On 2019/08/03 12:52:49, Andi Huber <ah...@apache.org> wrote:
> >> I think I tracked down the reason why these PrincipalForApplicationUser instances won't get garbage collected:
> >>
> >> With Basic-Auth as the authentication strategy, each request to the REST endpoint spawns a new HTTP Session, which holds a reference to a collection of PrincipalForApplicationUser instances.
> >>
> >> Now when tomcat runs in a mode, where it keeps all the sessions (session persistence mode) these objects cannot be garbage collected.
> >>
> >> I'm not a 100% sure but, it seems the described behavior is as it should be and hence a non-issue.
> >>
> >> However, I'm currently investigating, whether I got it all wrong, or there is a convenient solution to this, eg. don't create that many HTTP session objects.
> >>
> >> Let me know what you think!
> >>
> >> Cheers, Andi
> >>
> >> On 2019/07/23 07:29:26, Leandro D'Agostino <L....@pocos.nl> wrote:
> >>> Thanks Andi!
> >>>
> >>> We look forward to your findings.
> >>>
> >>> Leandro
> >>>
> >>> On 7/23/19 9:11 AM, Andi Huber wrote:
> >>>> We will certainly investigate this.
> >>>>
> >>>> Thanks for the effort of tracking this down!
> >>>>
> >>>> I've opened a Jira ticket [1].
> >>>>
> >>>> KR Andi
> >>>>
> >>>> [1] https://issues.apache.org/jira/browse/ISIS-2156
> >>>>
> >>>> On 2019/07/22 14:30:44, Leandro D'Agostino <L....@pocos.nl> wrote:
> >>>>> Hi,
> >>>>>
> >>>>> We ran into the issue that our application keeps building up memory but
> >>>>> never releases it.
> >>>>> We could track it down to the class PrincipalForApplicationUser. Every
> >>>>> time a user is authenticated, a new PrincipalForApplicationUser object
> >>>>> is created and it is then never released. We experience this memory
> >>>>> issue after a change of increasing the number of permissions from 20 to
> >>>>> about 200. When comparing the behaviour of the application before and
> >>>>> after the change we see that the retained size of the
> >>>>> PrincipalForApplicationUser class has increased from about 15KB per
> >>>>> instance to about 210KB per instance.
> >>>>>
> >>>>> For our investigation we created a test environment based on the
> >>>>> simpleapp application to be able to isolate the issue and reproduce the
> >>>>> issue in a minimalistic environment. So the simpleapp application was
> >>>>> extended with 200 fields on which permissions are set. In our test we
> >>>>> set the maximum memory size of the VM to 200MB. The test application
> >>>>> uses about 40MB initially. We then start firing requests to the
> >>>>> application (using jmeter), to:
> >>>>> /restful/services/simple.SimpleObjectMenu
> >>>>> What we observe then is that memory usage grows rather quickly and
> >>>>> eventually it is exhausted.
> >>>>>
> >>>>> Can you help with a solution for this issue?
> >>>>>
> >>>>> The simpleapp test application we used is available on github:
> >>>>> https://github.com/pocos-nl/isis-simpleapp-memoryissues
> >>>>>
> >>>>> It also includes the jmeter test script:
> >>>>> memoryLeak-bareApp-SecurityPerformanceTestObject.jmx
> >>>>>
> >>>>> Thanks,
> >>>>> Leandro D'Agostino
> >>>>>
> >>>>>
> >>>>>
>