You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomee.apache.org by tonywestonuk <to...@totspics.com> on 2015/09/02 17:10:01 UTC
Permgen issue?
Hi,
I am coding an application using TOMEE..... I notice when I make changes in
eclipse, with tomee running, the tomee hot-deploy doesn't GC the previous
application before the change.
Using VisualVM, it appears that there are a bunch of tomee threads that are
keeping hold of the LazyStopWebappClassLoader. Is this something I should
be worried about? Does TOMEE gradually rotate these threads so the previous
applications can be GC'd?
Thanks
--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/Permgen-issue-tp4676065.html
Sent from the TomEE Users mailing list archive at Nabble.com.
Re: Permgen issue?
Posted by Romain Manni-Bucau <rm...@gmail.com>.
2015-09-03 17:02 GMT+02:00 tonywestonuk <to...@totspics.com>:
> .......
>
> Romain. I am sorry.
>
> I was using version 1.7.1 - I have upgraded to 1.7.2, and the bug has
> gone.
>
>
> Thankyou for helping me, just happy to know it is fixed. Next time I'll
> make sure im on the latest version!
>
>
hehe, sometimes we cant, think we have a trace of it on jira but between
searching and finding on jira there is a gap sometimes ;). Happy it works
and thanks for the investigation!
>
> Cheers,
> Tony.
>
>
>
> --
> View this message in context:
> http://tomee-openejb.979440.n4.nabble.com/Permgen-issue-tp4676065p4676097.html
> Sent from the TomEE Users mailing list archive at Nabble.com.
>
Re: Permgen issue?
Posted by tonywestonuk <to...@totspics.com>.
.......
Romain. I am sorry.
I was using version 1.7.1 - I have upgraded to 1.7.2, and the bug has
gone.
Thankyou for helping me, just happy to know it is fixed. Next time I'll
make sure im on the latest version!
Cheers,
Tony.
--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/Permgen-issue-tp4676065p4676097.html
Sent from the TomEE Users mailing list archive at Nabble.com.
Re: Permgen issue?
Posted by Romain Manni-Bucau <rm...@gmail.com>.
tested with the snapshot calling reload() through JMX and didnt get the
issue at all - threads and classloader are cleaned up properly
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
<http://www.tomitribe.com>
2015-09-03 16:45 GMT+02:00 tonywestonuk <to...@totspics.com>:
> Though I have not confirmed this, I think you probably need to refresh the
> home page, then reload the app in tomee, for the bug to be seen.
>
> Nothing shows on the homepage..... keep your eye on the console.
>
> Thanks,
> Tony.
>
>
>
> --
> View this message in context:
> http://tomee-openejb.979440.n4.nabble.com/Permgen-issue-tp4676065p4676095.html
> Sent from the TomEE Users mailing list archive at Nabble.com.
>
Re: Permgen issue?
Posted by tonywestonuk <to...@totspics.com>.
Though I have not confirmed this, I think you probably need to refresh the
home page, then reload the app in tomee, for the bug to be seen.
Nothing shows on the homepage..... keep your eye on the console.
Thanks,
Tony.
--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/Permgen-issue-tp4676065p4676095.html
Sent from the TomEE Users mailing list archive at Nabble.com.
Re: Permgen issue?
Posted by tonywestonuk <to...@totspics.com>.
There you go - complete with source!
https://www.tonyweston.co.uk/FriendlyStock/gifts/TomeePermgenBug.war
--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/Permgen-issue-tp4676065p4676094.html
Sent from the TomEE Users mailing list archive at Nabble.com.
Re: Permgen issue?
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Do you care preparing a maven war project on github reproducing it?
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
<http://www.tomitribe.com>
2015-09-03 16:24 GMT+02:00 tonywestonuk <to...@totspics.com>:
> Ok,
>
> I can confirm it is still leaking permgen memory, on a standalone Tomee.
>
>
> :-/
>
> Thanks,
> Tony.
>
>
>
> --
> View this message in context:
> http://tomee-openejb.979440.n4.nabble.com/Permgen-issue-tp4676065p4676092.html
> Sent from the TomEE Users mailing list archive at Nabble.com.
>
Re: Permgen issue?
Posted by tonywestonuk <to...@totspics.com>.
Ok,
I can confirm it is still leaking permgen memory, on a standalone Tomee.
:-/
Thanks,
Tony.
--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/Permgen-issue-tp4676065p4676092.html
Sent from the TomEE Users mailing list archive at Nabble.com.
Re: Permgen issue?
Posted by tonywestonuk <to...@totspics.com>.
Ok, I will try this on a standalone TOMEE installation.....
Though, it appears that even on WTP tomee, the pool is tied to the webapp -
the classloader is the LazyStopWebappClassLoader.
The problem is that the org.apache.catalina.loader.StandardClassLoader
retains a reference to LazyStopWebappClassloader, and prevents it from
GC'ing.
I will try now anyhow, to see what happens. I will return!!
Tony.
--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/Permgen-issue-tp4676065p4676091.html
Sent from the TomEE Users mailing list archive at Nabble.com.
Re: Permgen issue?
Posted by Romain Manni-Bucau <rm...@gmail.com>.
About the pool itself: on modern tomee version the pool (AsynchronousPool-*)
is bound to the app and stopped when undeploying the application waiting
max AsynchronousPool.ShutdownWaitDuration . So basically a real
undeployment/redeployment should work smoothly. Can you validate it quickly
(ie dont use WTP to test it). Our async pool thread have the container
classloader as context classloader by default so the thread shouldnt kep
track of the classloader and bindings should be resetted properly in the
async task. Said otherwise Im trying to check if WTP hot deploy doesnt
follow the full lifecycle.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
<http://www.tomitribe.com>
2015-09-03 15:54 GMT+02:00 Romain Manni-Bucau <rm...@gmail.com>:
> ok so 2 cases:
> - bug is in tomee when creating the thread -> Ill check it
> - bug is due to your task as explained before, for this case you can do
> something like:
>
> @Singleton
> public class MyAsyncService {
> @Resource SessionContext ctx;
> AtomicReference<Future<?>> future = new AtomicReference<>();
>
> public Future<Void> doIt(){
> future.set(ctx. getBusinessObject(MyAsyncService.class)
> .doItAsync());
> System.out.println("Done");
> return null;
> }
>
> @Asynchronous
> public void doItAsync(){
>
> System.out.println("Doing it async");
> }
>
> @PreDestoy public void cancel()
> {ofNullable(future).map(AtomicReference::get).ifPresent(Future::cancel);}
> }
>
>
>
>
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | Tomitriber
> <http://www.tomitribe.com>
>
> 2015-09-03 15:39 GMT+02:00 tonywestonuk <to...@totspics.com>:
>
>> I am sorry, but I don't really understand how I can kill the task....
>>
>> In my example, above, the async task is just a single System.out !! -
>> How
>> can I kill this, it executes in milliseconds!!
>>
>> Looking at the threads using VisualVM, I can see the asynchronous thread
>> in
>> there, but it is marked as completed, and no longer running. there is no
>> task to kill.... everything is done! It just needs to be removed from
>> the
>> ContextBindings to allow the old application to be GC'd.
>>
>> Or, if i am completely missing the point of what you are telling me,
>> please
>> feel free to show me how I can make my example MyAsyncService work so the
>> app correctly undeploys.
>>
>> Thank you for helping..... :-D
>>
>>
>>
>> --
>> View this message in context:
>> http://tomee-openejb.979440.n4.nabble.com/Permgen-issue-tp4676065p4676088.html
>> Sent from the TomEE Users mailing list archive at Nabble.com.
>>
>
>
Re: Permgen issue?
Posted by Romain Manni-Bucau <rm...@gmail.com>.
ok so 2 cases:
- bug is in tomee when creating the thread -> Ill check it
- bug is due to your task as explained before, for this case you can do
something like:
@Singleton
public class MyAsyncService {
@Resource SessionContext ctx;
AtomicReference<Future<?>> future = new AtomicReference<>();
public Future<Void> doIt(){
future.set(ctx. getBusinessObject(MyAsyncService.class)
.doItAsync());
System.out.println("Done");
return null;
}
@Asynchronous
public void doItAsync(){
System.out.println("Doing it async");
}
@PreDestoy public void cancel()
{ofNullable(future).map(AtomicReference::get).ifPresent(Future::cancel);}
}
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
<http://www.tomitribe.com>
2015-09-03 15:39 GMT+02:00 tonywestonuk <to...@totspics.com>:
> I am sorry, but I don't really understand how I can kill the task....
>
> In my example, above, the async task is just a single System.out !! -
> How
> can I kill this, it executes in milliseconds!!
>
> Looking at the threads using VisualVM, I can see the asynchronous thread in
> there, but it is marked as completed, and no longer running. there is no
> task to kill.... everything is done! It just needs to be removed from the
> ContextBindings to allow the old application to be GC'd.
>
> Or, if i am completely missing the point of what you are telling me, please
> feel free to show me how I can make my example MyAsyncService work so the
> app correctly undeploys.
>
> Thank you for helping..... :-D
>
>
>
> --
> View this message in context:
> http://tomee-openejb.979440.n4.nabble.com/Permgen-issue-tp4676065p4676088.html
> Sent from the TomEE Users mailing list archive at Nabble.com.
>
Re: Permgen issue?
Posted by tonywestonuk <to...@totspics.com>.
I am sorry, but I don't really understand how I can kill the task....
In my example, above, the async task is just a single System.out !! - How
can I kill this, it executes in milliseconds!!
Looking at the threads using VisualVM, I can see the asynchronous thread in
there, but it is marked as completed, and no longer running. there is no
task to kill.... everything is done! It just needs to be removed from the
ContextBindings to allow the old application to be GC'd.
Or, if i am completely missing the point of what you are telling me, please
feel free to show me how I can make my example MyAsyncService work so the
app correctly undeploys.
Thank you for helping..... :-D
--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/Permgen-issue-tp4676065p4676088.html
Sent from the TomEE Users mailing list archive at Nabble.com.
Re: Permgen issue?
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Hi
I got the same analysis on the "where is the leak" but if you want us to
prevent this leak we would kill your task. This behavior is outside EE
specification and I'm tempted to not do it since you can add a @PreDestroy
keeping track of your task using a Future<Void> or Future<Boolean> and
cancel the task through the future in predestroy if necessary. If we kill
it and you are actually doing something important then you will blame the
container :p.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
<http://www.tomitribe.com>
2015-09-03 15:23 GMT+02:00 tonywestonuk <to...@totspics.com>:
> Ok..... the Async Bug, I replicated it in a example app. If you have
> something like the following, this will cause the Webapplication to not be
> unloaded correctly, and permgen leak will result.
>
> The problem appears to be with ContextBindings which is part of the vector
> of classes in the StandardClassLoader.
>
> This ContextBindings class has a static map of Threads to threadnames....
> one of these threads in the map is the Asynchrounous thread that is used to
> process the async method. This is not being unloaded, and so hangs around
> preventing the unloading of the webapp.
>
> I think this is a bug in TOMEE, unless you can see a problem with my code
> below....
>
>
>
>
>
> @Singleton
> public class MyAsyncService {
> @Inject MyAsyncService asyncService;
>
> public void doIt(){
> asyncService.doItAsync();
> System.out.println("Done");
> }
>
> @Asynchronous
> public void doItAsync(){
>
> System.out.println("Doing it async");
> }
> }
>
>
>
> --
> View this message in context:
> http://tomee-openejb.979440.n4.nabble.com/Permgen-issue-tp4676065p4676086.html
> Sent from the TomEE Users mailing list archive at Nabble.com.
>
Re: Permgen issue?
Posted by tonywestonuk <to...@totspics.com>.
Ok..... the Async Bug, I replicated it in a example app. If you have
something like the following, this will cause the Webapplication to not be
unloaded correctly, and permgen leak will result.
The problem appears to be with ContextBindings which is part of the vector
of classes in the StandardClassLoader.
This ContextBindings class has a static map of Threads to threadnames....
one of these threads in the map is the Asynchrounous thread that is used to
process the async method. This is not being unloaded, and so hangs around
preventing the unloading of the webapp.
I think this is a bug in TOMEE, unless you can see a problem with my code
below....
@Singleton
public class MyAsyncService {
@Inject MyAsyncService asyncService;
public void doIt(){
asyncService.doItAsync();
System.out.println("Done");
}
@Asynchronous
public void doItAsync(){
System.out.println("Doing it async");
}
}
--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/Permgen-issue-tp4676065p4676086.html
Sent from the TomEE Users mailing list archive at Nabble.com.
Re: Permgen issue?
Posted by Romain Manni-Bucau <rm...@gmail.com>.
maybe try with johnzon or jackson as provider to check the behavior is the
same. Clearly johnzon doesnt keep any reference.
About your async: the task is under you control (means the container will
not kill it if it is not done) so you need to cancel it if it is running
while the app is undeployed. This is relatively unlikely in prod but in dev
it happens.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
<http://www.tomitribe.com>
2015-09-03 11:49 GMT+02:00 tonywestonuk <to...@totspics.com>:
> Ok.... I think I have a couple of places where this memory could be
> leaking.
>
> When I remove the async, this no longer appears in the heap dump. I dont
> have a @PreDestroy, but I wouldn't know what to put in that method anyhow.
> All I want is for my async method to run outside the main HTTPRequest
> thread
> so it doesn't slow up the response, and once that method is done (it only
> takes a second or so to run), then that is the end of that!... There is no
> cancelling anything.... Or is there something I do have to do to destroy
> the background thread that processes these async methods?
>
> Anyhow, as I say, this is only one of my permgen issues. After removing
> the async method, the application still didn't GC on redeploy. I tracked
> this down to using JAXRS. I made a very simple webapp that simply returns
> a
> java object serialised as JSON, in a get request. This GC's ok on
> redeploy.
> However, if I add the configuration according to adam-bien,
>
>
> http://www.adam-bien.com/roller/abien/entry/configuring_the_json_default_provider
>
> This stops it hot deploying, it appears that the JSONProvider doesn't GC,
> and this retains a reference to the JAXB class that is serialised to
> JSON......
>
>
>
>
> --
> View this message in context:
> http://tomee-openejb.979440.n4.nabble.com/Permgen-issue-tp4676065p4676080.html
> Sent from the TomEE Users mailing list archive at Nabble.com.
>
Re: Permgen issue?
Posted by tonywestonuk <to...@totspics.com>.
Ok.... I think I have a couple of places where this memory could be leaking.
When I remove the async, this no longer appears in the heap dump. I dont
have a @PreDestroy, but I wouldn't know what to put in that method anyhow.
All I want is for my async method to run outside the main HTTPRequest thread
so it doesn't slow up the response, and once that method is done (it only
takes a second or so to run), then that is the end of that!... There is no
cancelling anything.... Or is there something I do have to do to destroy
the background thread that processes these async methods?
Anyhow, as I say, this is only one of my permgen issues. After removing
the async method, the application still didn't GC on redeploy. I tracked
this down to using JAXRS. I made a very simple webapp that simply returns a
java object serialised as JSON, in a get request. This GC's ok on redeploy.
However, if I add the configuration according to adam-bien,
http://www.adam-bien.com/roller/abien/entry/configuring_the_json_default_provider
This stops it hot deploying, it appears that the JSONProvider doesn't GC,
and this retains a reference to the JAXB class that is serialised to
JSON......
--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/Permgen-issue-tp4676065p4676080.html
Sent from the TomEE Users mailing list archive at Nabble.com.
Re: Permgen issue?
Posted by Romain Manni-Bucau <rm...@gmail.com>.
2015-09-03 10:27 GMT+02:00 tonywestonuk <to...@totspics.com>:
> Yeh, I noticed that. However I don't do anything 'illegal' in the async
> task..... and even when I get rid of the @async annotation, it still
> doesn't GC on redeploys.
>
>
sure but if the thread olds the task during the underploy then the
classloader will not be resetted in tomcat. Do you have a @PreDestroy or
something to cancel this task on redeploys?
> I also tried a very simple tomee project with a simple async task that just
> printed 'helloworld' to the console.... this did GC correctly.
>
> I am wondering if it could be MyBatis (which I use in the aync task), that
> is not freeing itself when the application shuts down. I couldn't see
> anything related to this in the heap dumps however :-/
>
>
>
>
>
>
>
> --
> View this message in context:
> http://tomee-openejb.979440.n4.nabble.com/Permgen-issue-tp4676065p4676077.html
> Sent from the TomEE Users mailing list archive at Nabble.com.
>
Re: Permgen issue?
Posted by tonywestonuk <to...@totspics.com>.
Yeh, I noticed that. However I don't do anything 'illegal' in the async
task..... and even when I get rid of the @async annotation, it still
doesn't GC on redeploys.
I also tried a very simple tomee project with a simple async task that just
printed 'helloworld' to the console.... this did GC correctly.
I am wondering if it could be MyBatis (which I use in the aync task), that
is not freeing itself when the application shuts down. I couldn't see
anything related to this in the heap dumps however :-/
--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/Permgen-issue-tp4676065p4676077.html
Sent from the TomEE Users mailing list archive at Nabble.com.
Re: Permgen issue?
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Hmm
wonder if it cant be linked to your @Async task running while eclipse
redeploys which would be understandable. Do you think you can check it
quickly?
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
<http://www.tomitribe.com>
2015-09-03 8:39 GMT+02:00 tonywestonuk <to...@totspics.com>:
> I dropped a visualvm heapdump here:
>
> https://www.tonyweston.co.uk/FriendlyStock/gifts/heapdump.hprof.zip
> <https://www.tonyweston.co.uk/FriendlyStock/gifts/heapdump.hprof.zip>
>
> I see 9 instances of the LazyStopWebAppClassLoader - they are attached to
> the http-8080 acceptor thread.....(I think....I don't have much experience
> with VisualVM)..... I guess if the threads rotate, the old webapp
> instances
> should be GC'd?
>
> Thanks
> Tony.
>
>
>
> --
> View this message in context:
> http://tomee-openejb.979440.n4.nabble.com/Permgen-issue-tp4676065p4676072.html
> Sent from the TomEE Users mailing list archive at Nabble.com.
>
Re: Permgen issue?
Posted by tonywestonuk <to...@totspics.com>.
I dropped a visualvm heapdump here:
https://www.tonyweston.co.uk/FriendlyStock/gifts/heapdump.hprof.zip
<https://www.tonyweston.co.uk/FriendlyStock/gifts/heapdump.hprof.zip>
I see 9 instances of the LazyStopWebAppClassLoader - they are attached to
the http-8080 acceptor thread.....(I think....I don't have much experience
with VisualVM)..... I guess if the threads rotate, the old webapp instances
should be GC'd?
Thanks
Tony.
--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/Permgen-issue-tp4676065p4676072.html
Sent from the TomEE Users mailing list archive at Nabble.com.
Re: Permgen issue?
Posted by Romain Manni-Bucau <rm...@gmail.com>.
most of tomee threads have keep alive so they should rotate, that said if
you can spot where exactly we can validate it on a precise case
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
<http://www.tomitribe.com>
2015-09-02 17:10 GMT+02:00 tonywestonuk <to...@totspics.com>:
> Hi,
>
> I am coding an application using TOMEE..... I notice when I make changes
> in
> eclipse, with tomee running, the tomee hot-deploy doesn't GC the previous
> application before the change.
>
> Using VisualVM, it appears that there are a bunch of tomee threads that are
> keeping hold of the LazyStopWebappClassLoader. Is this something I
> should
> be worried about? Does TOMEE gradually rotate these threads so the previous
> applications can be GC'd?
>
>
> Thanks
>
>
>
> --
> View this message in context:
> http://tomee-openejb.979440.n4.nabble.com/Permgen-issue-tp4676065.html
> Sent from the TomEE Users mailing list archive at Nabble.com.
>