You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@wicket.apache.org by Tobias Gierke <to...@voipfuture.com.INVALID> on 2023/01/11 12:16:32 UTC
Strange memory leak in CachingResourceStreamLocator
Hi guys,
I'm currently investigating a memory leak inside our application
(running Wicket 8.14.0). Sorry for the large amount of images in this
(HTML) mail but the issue is rather complicated and as the saying goes,
"A picture is worth a thousand words".
Inspecting the CachingResourceStreamLocator's internal ConcurrentHashMap
reveals that around 500.000 of the cache entries are all related to a
set of only 5 *static* resources:
Further digging into the cache map's content reveals that the reason for
this duplication is that the
org.apache.wicket.core.util.resource.locator.caching.CachingResourceStreamLocator$CacheKey
instances used as keys in the map for these 5 static resources only
deviate by their "style" attribute:
Note that this output was generated by parsing the heap dump using some
custom tooling I wrote (as Eclipse MAT is not powerful enough to do this
via OQL/Calcite) where I parsed all the CacheKey field values and then
concatenated them as a string like so:
to generate CSV output in "map key;map value" format. Also note the
somewhat mangled looking style values with the trailing '.' (I
double-checked my program output against Eclipse MAT and both agree that
the strings are indeed ending with a trailing dot).
Since we're never calling WebSession#setStyle(), this looks weird to me.
One of the URLs being parsed is an AJAX callback URL generated by an
onclick AjaxEventBehaviour attached to a button:
I stepped through the Wicket code and this URL gets parsed by
org.apache.wicket.resource.ResourceUtil
I checked how the AJAX callback URL is generated and noticed that what
is mistaken as "style" by ResourceUtil is "<render count>.<behaviour id>":
The weird thing about all of this is that the issue seems to only affect
5 static application-level resources (though we obviously use a lot more
static application-level resources everywhere) and 4 out of those 5
resources are used by a single WebPage of our application only (the
login page).
Since our application is rather old (>10 years) and - for various
reasons - quite deeply integrated with Wicket internals, I wouldn't rule
out that we're doing something wrong (I checked our custom CSRFMapper
since it shows up in the stack trace but couldn't see any obvious
relationship to the observed behaviour).
Any ideas ?
Cheers,
Tobias
Re: Strange memory leak in CachingResourceStreamLocator
Posted by Tobias Gierke <to...@voipfuture.com.INVALID>.
Ah, good news - so the behaviour was not related to our IResourceFinder
implementation :)
Thanks for your time and investigating the issue !
> https://issues.apache.org/jira/browse/WICKET-7024
> The attached quickstart there does not use your finder.
>
> On Thu, Jan 12, 2023 at 12:27 PM Tobias Gierke
> <to...@voipfuture.com.invalid> wrote:
>
>> I see!
>>
>> Can you reproduce this in a quickstart ?
>>
>>
>>
>> Yes, attached a quickstart to this e-mail (I hope it goes through). It
>> somehow seems to be related to the custom IResourceFinder we've registered
>> *and* <wicket:link/> because if either of those is *not* present, the
>> issue goes away. I'm reluctant to file a bug report for that just yet
>> because maybe our IResourceFinder implementation does something weird/wrong.
>>
>> To reproduce:
>>
>> 1.) Put a conditional "key.style != null" breakpoint on
>> CachingResourceStreamLocator#updateCache , line 107 (where the "stream
>> instanceof UrlResourceStream") case is getting handled
>> 2.) Run the quickstart and click the AJAX button that redirects to the
>> homepage
>> 3.) Observe that the breakpoint gets hit with style != null
>>
>>
--
Tobias Gierke
Software Developer
Voipfuture GmbH Wendenstr. 4 20097 Hamburg Germany
Phone +49 40 688 9001 64 Fax +49 40 688 9001 99
Managing Directors Jan Bastian Eyal Ullert
Commercial Court AG Hamburg HRB 109896 VAT ID DE263738086
Re: Strange memory leak in CachingResourceStreamLocator
Posted by Martin Grigorov <mg...@apache.org>.
https://issues.apache.org/jira/browse/WICKET-7024
The attached quickstart there does not use your finder.
On Thu, Jan 12, 2023 at 12:27 PM Tobias Gierke
<to...@voipfuture.com.invalid> wrote:
>
> I see!
>
> Can you reproduce this in a quickstart ?
>
>
>
> Yes, attached a quickstart to this e-mail (I hope it goes through). It
> somehow seems to be related to the custom IResourceFinder we've registered
> *and* <wicket:link/> because if either of those is *not* present, the
> issue goes away. I'm reluctant to file a bug report for that just yet
> because maybe our IResourceFinder implementation does something weird/wrong.
>
> To reproduce:
>
> 1.) Put a conditional "key.style != null" breakpoint on
> CachingResourceStreamLocator#updateCache , line 107 (where the "stream
> instanceof UrlResourceStream") case is getting handled
> 2.) Run the quickstart and click the AJAX button that redirects to the
> homepage
> 3.) Observe that the breakpoint gets hit with style != null
>
>
Re: Strange memory leak in CachingResourceStreamLocator
Posted by Tobias Gierke <to...@voipfuture.com.INVALID>.
> I see!
>
> Can you reproduce this in a quickstart ?
>
Yes, attached a quickstart to this e-mail (I hope it goes through). It
somehow seems to be related to the custom IResourceFinder we've
registered *and* <wicket:link/> because if either of those is *not*
present, the issue goes away. I'm reluctant to file a bug report for
that just yet because maybe our IResourceFinder implementation does
something weird/wrong.
To reproduce:
1.) Put a conditional "key.style != null" breakpoint on
CachingResourceStreamLocator#updateCache , line 107 (where the "stream
instanceof UrlResourceStream") case is getting handled
2.) Run the quickstart and click the AJAX button that redirects to the
homepage
3.) Observe that the breakpoint gets hit with style != null
Re: Strange memory leak in CachingResourceStreamLocator
Posted by Martin Grigorov <mg...@apache.org>.
On Thu, Jan 12, 2023 at 10:21 AM Tobias Gierke
<to...@voipfuture.com.invalid> wrote:
> Hi Martin,
>
> I wonder why people create images of plain text to paste it in text
> context (email) ... It just makes it hard to do anything with this text,
> e.g. to copy/paste the relevant part!
>
> Good point, will try to better myself ;)
>
You did it again! :-)
>
> In your first email the CacheKey says that the scope is o.a.w.Application,
> but here we see that it actually is
> com.vodecc.voipmng....PageWithLoginScreenLayout
>
> The heap dump I'm talking about in my first e-mail is from one of our
> largest customers and was provided to me by one of our support guys.
> Because the code is running inside a highly secured internal network (telco
> business), I cannot do any remote debugging there. So my debugger
> screenshots are taken on my local dev machine running the same GIT tag.
> There's only a single page in the whole application where those resources
> (except for the PNG image which is used elsewhere as well) are being used -
> and that is the application's login page.
>
> the "name" is "css/qrystal.css"
> and there are no locale/style/variation at all in the url (link's href),
> as expected
>
> I probably didn't explain this very well, the memory leak happens during
> processing of the "redirect to login page" AJAX request triggered when a
> user clicks a button on our "session expired" page (see below) and we
> perform an internal redirect to the login page.
>
>
> Does it help if you change the above to
setResponsePage(WicketLoginPage.class, p) ?
> also there is no page instance information in the css' href (pageId,
> renderCount, behaviorId)...
> the hashing 167344540504 is not related to the CacheKey
>
> I'd suggest you to put a conditional breakpoint at
> WicketFilter#processFilter() and see whether there are requests to url that
> contains both "o.a.w.Application" and "qrystal-login"
> If this does not break then do the same in CacheKey constructor
>
>
> 1.) Put a conditional breakpoint on "key.style != null" in
> CachingResourceStreamLocator#updateCache()
>
> As I've said before, we're never calling WebSession#setStyle() so this
> breakpoint should never trigger.
>
> 2.) After starting the application in debug mode, I enter the URL of a
> mounted,protected (=login required) page into the browser's navigation bar
> and am being redirected to our application's "Page expired/Login required"
> page with the AJAX button mentioned above.
>
> 3.) Clicking the AJAX button immediately makes the debugger stop at the
> "impossible" breakpoint
>
>
> 3.) Going up the call stack to the WicketFilter class shows that the
> request currently being processed is in fact the AJAX request triggered by
> the "Go to login page" button
>
>
> The "CacheKey.style" is "1.0" which matches what is present in the request
> URL.
>
> So the memory leak happens during rendering the response for "onclick"
> AJAX request, which is our login page.
>
I see!
Can you reproduce this in a quickstart ?
Re: Strange memory leak in CachingResourceStreamLocator
Posted by Tobias Gierke <to...@voipfuture.com.INVALID>.
Hi Martin,
> I wonder why people create images of plain text to paste it in text
> context (email) ... It just makes it hard to do anything with this
> text, e.g. to copy/paste the relevant part!
Good point, will try to better myself ;)
>
> In your first email the CacheKey says that the scope is
> o.a.w.Application, but here we see that it actually is
> com.vodecc.voipmng....PageWithLoginScreenLayout
The heap dump I'm talking about in my first e-mail is from one of our
largest customers and was provided to me by one of our support guys.
Because the code is running inside a highly secured internal network
(telco business), I cannot do any remote debugging there. So my debugger
screenshots are taken on my local dev machine running the same GIT tag.
There's only a single page in the whole application where those
resources (except for the PNG image which is used elsewhere as well) are
being used - and that is the application's login page.
> the "name" is "css/qrystal.css"
> and there are no locale/style/variation at all in the url (link's
> href), as expected
I probably didn't explain this very well, the memory leak happens during
processing of the "redirect to login page" AJAX request triggered when a
user clicks a button on our "session expired" page (see below) and we
perform an internal redirect to the login page.
> also there is no page instance information in the css' href (pageId,
> renderCount, behaviorId)...
> the hashing 167344540504 is not related to the CacheKey
>
> I'd suggest you to put a conditional breakpoint at
> WicketFilter#processFilter() and see whether there are requests to url
> that contains both "o.a.w.Application" and "qrystal-login"
> If this does not break then do the same in CacheKey constructor
1.) Put a conditional breakpoint on "key.style != null" in
CachingResourceStreamLocator#updateCache()
As I've said before, we're never calling WebSession#setStyle() so this
breakpoint should never trigger.
2.) After starting the application in debug mode, I enter the URL of a
mounted,protected (=login required) page into the browser's navigation
bar and am being redirected to our application's "Page expired/Login
required" page with the AJAX button mentioned above.
3.) Clicking the AJAX button immediately makes the debugger stop at the
"impossible" breakpoint
3.) Going up the call stack to the WicketFilter class shows that the
request currently being processed is in fact the AJAX request triggered
by the "Go to login page" button
The "CacheKey.style" is "1.0" which matches what is present in the
request URL.
So the memory leak happens during rendering the response for "onclick"
AJAX request, which is our login page.
Re: Strange memory leak in CachingResourceStreamLocator
Posted by Martin Grigorov <mg...@apache.org>.
I wonder why people create images of plain text to paste it in text context
(email) ... It just makes it hard to do anything with this text, e.g. to
copy/paste the relevant part!
In your first email the CacheKey says that the scope is o.a.w.Application,
but here we see that it actually is
com.vodecc.voipmng....PageWithLoginScreenLayout
the "name" is "css/qrystal.css"
and there are no locale/style/variation at all in the url (link's href), as
expected
also there is no page instance information in the css' href (pageId,
renderCount, behaviorId)...
the hashing 167344540504 is not related to the CacheKey
I'd suggest you to put a conditional breakpoint at
WicketFilter#processFilter() and see whether there are requests to url that
contains both "o.a.w.Application" and "qrystal-login"
If this does not break then do the same in CacheKey constructor
The requests to resources (js, css) know nothing about the page (instance)
that triggered them.
I suspect that your custom parsing of the heap dump might be broken
somehow. Do you see o.a.w.Application in MAT when expanding the fields of
the cache items ?
On Wed, Jan 11, 2023 at 5:03 PM Tobias Gierke
<to...@voipfuture.com.invalid> wrote:
> Hi Martin,
>
> What is the url for any of the problematic resources in the generated HTML
> ?
> The resources' urls are not related to page *instances*, so I don't
> expect to see anything like pageId, renderCount and behaviorId in them.
>
> Not really sure I understood your question correctly... the final markup
> (debug mode enabled, hence the wicket tags are still visible) received by
> the browser looks like this:
>
> while the XHTML template looks like this:
>
> Not sure this is relevant here but for historic reasons (many years ago
> our application got migrated gradually from Struts2 to Wicket) we're
> registering a custom IResourceFinder in WebApplication#init() that handles
> "css/*" , "script/*" and "img/*" paths.
>
>
>
>
> On Wed, Jan 11, 2023 at 2:17 PM Tobias Gierke
> <to...@voipfuture.com.invalid>
> <to...@voipfuture.com.invalid> wrote:
>
>> Hi guys,
>>
>> I'm currently investigating a memory leak inside our application (running
>> Wicket 8.14.0). Sorry for the large amount of images in this (HTML) mail
>> but the issue is rather complicated and as the saying goes, "A picture is
>> worth a thousand words".
>>
>> Inspecting the CachingResourceStreamLocator's internal ConcurrentHashMap
>> reveals that around 500.000 of the cache entries are all related to a set
>> of only 5 *static* resources:
>>
>> Further digging into the cache map's content reveals that the reason for
>> this duplication is that the
>> org.apache.wicket.core.util.resource.locator.caching.CachingResourceStreamLocator$CacheKey
>> instances used as keys in the map for these 5 static resources only deviate
>> by their "style" attribute:
>>
>> Note that this output was generated by parsing the heap dump using some
>> custom tooling I wrote (as Eclipse MAT is not powerful enough to do this
>> via OQL/Calcite) where I parsed all the CacheKey field values and then
>> concatenated them as a string like so:
>>
>> to generate CSV output in "map key;map value" format. Also note the
>> somewhat mangled looking style values with the trailing '.' (I
>> double-checked my program output against Eclipse MAT and both agree that
>> the strings are indeed ending with a trailing dot).
>>
>> Since we're never calling WebSession#setStyle(), this looks weird to me.
>> One of the URLs being parsed is an AJAX callback URL generated by an
>> onclick AjaxEventBehaviour attached to a button:
>>
>> I stepped through the Wicket code and this URL gets parsed by
>> org.apache.wicket.resource.ResourceUtil
>>
>> I checked how the AJAX callback URL is generated and noticed that what is
>> mistaken as "style" by ResourceUtil is "<render count>.<behaviour id>":
>>
>>
>> The weird thing about all of this is that the issue seems to only affect
>> 5 static application-level resources (though we obviously use a lot more
>> static application-level resources everywhere) and 4 out of those 5
>> resources are used by a single WebPage of our application only (the login
>> page).
>>
>> Since our application is rather old (>10 years) and - for various reasons
>> - quite deeply integrated with Wicket internals, I wouldn't rule out that
>> we're doing something wrong (I checked our custom CSRFMapper since it shows
>> up in the stack trace but couldn't see any obvious relationship to the
>> observed behaviour).
>>
>> Any ideas ?
>>
>> Cheers,
>> Tobias
>>
> --
> Tobias Gierke
> Software Developer
>
> Voipfuture GmbH Wendenstr. 4 20097 Hamburg Germany
> Phone +49 40 688 9001 64 Fax +49 40 688 9001 99
> Managing Directors Jan Bastian Eyal Ullert
> Commercial Court AG Hamburg HRB 109896 VAT ID DE263738086
>
>
Re: Strange memory leak in CachingResourceStreamLocator
Posted by Tobias Gierke <to...@voipfuture.com.INVALID>.
Hi Martin,
> What is the url for any of the problematic resources in the generated
> HTML ?
> The resources' urls are not related to page *instances*, so I don't
> expect to see anything like pageId, renderCount and behaviorId in them.
Not really sure I understood your question correctly... the final
markup (debug mode enabled, hence the wicket tags are still visible)
received by the browser looks like this:
while the XHTML template looks like this:
Not sure this is relevant here but for historic reasons (many years ago
our application got migrated gradually from Struts2 to Wicket) we're
registering a custom IResourceFinder in WebApplication#init() that
handles "css/*" , "script/*" and "img/*" paths.
>
> On Wed, Jan 11, 2023 at 2:17 PM Tobias Gierke
> <to...@voipfuture.com.invalid> wrote:
>
> Hi guys,
>
> I'm currently investigating a memory leak inside our application
> (running Wicket 8.14.0). Sorry for the large amount of images in
> this (HTML) mail but the issue is rather complicated and as the
> saying goes, "A picture is worth a thousand words".
>
> Inspecting the CachingResourceStreamLocator's internal
> ConcurrentHashMap reveals that around 500.000 of the cache entries
> are all related to a set of only 5 *static* resources:
>
> Further digging into the cache map's content reveals that the
> reason for this duplication is that the
> org.apache.wicket.core.util.resource.locator.caching.CachingResourceStreamLocator$CacheKey
> instances used as keys in the map for these 5 static resources
> only deviate by their "style" attribute:
>
> Note that this output was generated by parsing the heap dump using
> some custom tooling I wrote (as Eclipse MAT is not powerful enough
> to do this via OQL/Calcite) where I parsed all the CacheKey field
> values and then concatenated them as a string like so:
>
> to generate CSV output in "map key;map value" format. Also note
> the somewhat mangled looking style values with the trailing '.' (I
> double-checked my program output against Eclipse MAT and both
> agree that the strings are indeed ending with a trailing dot).
>
> Since we're never calling WebSession#setStyle(), this looks weird
> to me. One of the URLs being parsed is an AJAX callback URL
> generated by an onclick AjaxEventBehaviour attached to a button:
>
> I stepped through the Wicket code and this URL gets parsed by
> org.apache.wicket.resource.ResourceUtil
>
> I checked how the AJAX callback URL is generated and noticed that
> what is mistaken as "style" by ResourceUtil is "<render
> count>.<behaviour id>":
>
>
> The weird thing about all of this is that the issue seems to only
> affect 5 static application-level resources (though we obviously
> use a lot more static application-level resources everywhere) and
> 4 out of those 5 resources are used by a single WebPage of our
> application only (the login page).
>
> Since our application is rather old (>10 years) and - for various
> reasons - quite deeply integrated with Wicket internals, I
> wouldn't rule out that we're doing something wrong (I checked our
> custom CSRFMapper since it shows up in the stack trace but
> couldn't see any obvious relationship to the observed behaviour).
>
> Any ideas ?
>
> Cheers,
> Tobias
>
--
Tobias Gierke
Software Developer
Voipfuture GmbH Wendenstr. 4 20097 Hamburg Germany
Phone +49 40 688 9001 64 Fax +49 40 688 9001 99
Managing Directors Jan Bastian Eyal Ullert
Commercial Court AG Hamburg HRB 109896 VAT ID DE263738086
Re: Strange memory leak in CachingResourceStreamLocator
Posted by Martin Grigorov <mg...@apache.org>.
What is the url for any of the problematic resources in the generated HTML ?
The resources' urls are not related to page *instances*, so I don't expect
to see anything like pageId, renderCount and behaviorId in them.
On Wed, Jan 11, 2023 at 2:17 PM Tobias Gierke
<to...@voipfuture.com.invalid> wrote:
> Hi guys,
>
> I'm currently investigating a memory leak inside our application (running
> Wicket 8.14.0). Sorry for the large amount of images in this (HTML) mail
> but the issue is rather complicated and as the saying goes, "A picture is
> worth a thousand words".
>
> Inspecting the CachingResourceStreamLocator's internal ConcurrentHashMap
> reveals that around 500.000 of the cache entries are all related to a set
> of only 5 *static* resources:
>
> Further digging into the cache map's content reveals that the reason for
> this duplication is that the
> org.apache.wicket.core.util.resource.locator.caching.CachingResourceStreamLocator$CacheKey
> instances used as keys in the map for these 5 static resources only deviate
> by their "style" attribute:
>
> Note that this output was generated by parsing the heap dump using some
> custom tooling I wrote (as Eclipse MAT is not powerful enough to do this
> via OQL/Calcite) where I parsed all the CacheKey field values and then
> concatenated them as a string like so:
>
> to generate CSV output in "map key;map value" format. Also note the
> somewhat mangled looking style values with the trailing '.' (I
> double-checked my program output against Eclipse MAT and both agree that
> the strings are indeed ending with a trailing dot).
>
> Since we're never calling WebSession#setStyle(), this looks weird to me.
> One of the URLs being parsed is an AJAX callback URL generated by an
> onclick AjaxEventBehaviour attached to a button:
>
> I stepped through the Wicket code and this URL gets parsed by
> org.apache.wicket.resource.ResourceUtil
>
> I checked how the AJAX callback URL is generated and noticed that what is
> mistaken as "style" by ResourceUtil is "<render count>.<behaviour id>":
>
>
> The weird thing about all of this is that the issue seems to only affect 5
> static application-level resources (though we obviously use a lot more
> static application-level resources everywhere) and 4 out of those 5
> resources are used by a single WebPage of our application only (the login
> page).
>
> Since our application is rather old (>10 years) and - for various reasons
> - quite deeply integrated with Wicket internals, I wouldn't rule out that
> we're doing something wrong (I checked our custom CSRFMapper since it shows
> up in the stack trace but couldn't see any obvious relationship to the
> observed behaviour).
>
> Any ideas ?
>
> Cheers,
> Tobias
>