You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@shindig.apache.org by Pablo Graña <pa...@globant.com> on 2010/09/22 04:39:52 UTC
Multiple hot deploys and shindig
Hi guys.
I get an oom after some hot deploys. I traced one problem to Guice (
http://www.mail-archive.com/google-guice@googlegroups.com/msg02987.html -
there is a workaround) and another to shindig: the DefaultGuiceModule
installs a shutdown hook, apparently with the idea to shut down a thread
pool. Is this hook correct? It should probably go in a
ContextLoaderListener, right?
thanks a lot.
--
Pablo Gra\~na
Chief Architect
Globant
Arg Office: +54 (11) 4109 1743
UK Office: +44 (20) 7043 8269 int 8043
US Office: +1 (212) 400 7686 int 8043
Re: Multiple hot deploys and shindig
Posted by Pablo Graña <pa...@globant.com>.
I created ticket https://issues.apache.org/jira/browse/SHINDIG-1439
for the shutdown hook.
I read a little bit more about thread locals, and I think that caching
the DocumentBuilder in a ThreadLocal should not create a memory leak.
See this blog post:
http://blog.crazybob.org/2006/07/hard-core-java-threadlocal.html (I
think I understand why
https://issues.apache.org/jira/browse/SHINDIG-1132 is fixed).
Should I open a jira issue for the guava thing?
On Mon, Oct 4, 2010 at 4:23 AM, Paul Lindner <li...@inuus.com> wrote:
> Can you open a jira issue with this information so that it doesn't get
> forgotten. I'd like to see a solution, just don't have the time to
> tackle it myself at the moment.
>
>
> 2010/9/23 Pablo Graña <pa...@globant.com>:
>> Here is more info on the problem and the current status:
>>
>> http://code.google.com/p/guava-libraries/issues/detail?id=92
>>
>> Guys from guice copied the classes from guava (or google-collections),
>> bringing the same problem to guice.
>>
>> The memory leaks manifests with hot redeploys in tomcat, after a couple of
>> which you get a perm gen memory error. The only safe thing to do is to
>> restart tomcat for every deploy.
>>
>> The problem is that for each web application, tomcat creates a class loader.
>> When you redeploy the same application, tomcat creates a new class loader
>> and removes the references it holds to the old class loader. But in some
>> cases, there could be some reference from, for example, the system class
>> loader to the old web app class loader, stopping the garbage collector from
>> reclaiming the memory. A classic example is when you package the database
>> driver in WEB-INF/lib in your webapp: the driver manager (in the system
>> class loader) will hold a reference to a class in the webapp classloader.
>>
>> The same happens with shutdown hooks: you have a system class loader
>> (Runtime) that holds a reference to the shutdown thread, keeping the webapp
>> classloader always alive.
>>
>> Anyway, a shutdown hook will only cleanup resources when the jvm ends. And
>> you probably want the cleanup to happen when the web application is
>> undeployed.
>>
>> That is why I think the best option is to use a context loader listener: add
>> a shutdown operation to DefaultGuiceModule and call it in from
>> GuiceServletContextListener.contextDestroyed.
>>
>> BTW: I found another problem. XmlUtil is caching a DocumentBuilder in a
>> ThreadLocal, and never released. This also keeps the web app classloader
>> alive.
>>
>> Thanks
>>
>> On Thu, Sep 23, 2010 at 11:18 PM, Gagandeep singh <ga...@gmail.com>wrote:
>>
>>> Hi Pablo
>>>
>>> I couldn't understand the memory leak you found in Guice. Hopefully crazy
>>> bob will fix it soon (it has been 3 months already).
>>> As for the shindig shutdown hook, it looks like an innocent hook which
>>> tries
>>> to stop the executor service which is where all the threads are scheduled.
>>> Did you find any particular instance in which it creates a memory leak ?
>>> Also, please explain a bit more on your ContextLoaderListener idea.
>>>
>>> Thanks
>>> Gagan
>>>
>>> 2010/9/22 Pablo Graña <pa...@globant.com>
>>>
>>> > Hi guys.
>>> >
>>> > I get an oom after some hot deploys. I traced one problem to Guice (
>>> > http://www.mail-archive.com/google-guice@googlegroups.com/msg02987.html-
>>> > there is a workaround) and another to shindig: the DefaultGuiceModule
>>> > installs a shutdown hook, apparently with the idea to shut down a thread
>>> > pool. Is this hook correct? It should probably go in a
>>> > ContextLoaderListener, right?
>>> >
>>> > thanks a lot.
>>> >
>>> > --
>>> > Pablo Gra\~na
>>> > Chief Architect
>>> > Globant
>>> > Arg Office: +54 (11) 4109 1743
>>> > UK Office: +44 (20) 7043 8269 int 8043
>>> > US Office: +1 (212) 400 7686 int 8043
>>> >
>>>
>>
>>
>>
>> --
>> Pablo Gra\~na
>> Chief Architect
>> Globant
>> Arg Office: +54 (11) 4109 1743
>> UK Office: +44 (20) 7043 8269 int 8043
>> US Office: +1 (212) 400 7686 int 8043
>>
>
>
>
> --
> Paul Lindner -- lindner@inuus.com -- linkedin.com/in/plindner
>
--
Pablo Gra\~na
Chief Architect
Globant
Arg Office: +54 (11) 4109 1743
UK Office: +44 (20) 7043 8269 int 8043
US Office: +1 (212) 400 7686 int 8043
Re: Multiple hot deploys and shindig
Posted by Paul Lindner <li...@inuus.com>.
Can you open a jira issue with this information so that it doesn't get
forgotten. I'd like to see a solution, just don't have the time to
tackle it myself at the moment.
2010/9/23 Pablo Graña <pa...@globant.com>:
> Here is more info on the problem and the current status:
>
> http://code.google.com/p/guava-libraries/issues/detail?id=92
>
> Guys from guice copied the classes from guava (or google-collections),
> bringing the same problem to guice.
>
> The memory leaks manifests with hot redeploys in tomcat, after a couple of
> which you get a perm gen memory error. The only safe thing to do is to
> restart tomcat for every deploy.
>
> The problem is that for each web application, tomcat creates a class loader.
> When you redeploy the same application, tomcat creates a new class loader
> and removes the references it holds to the old class loader. But in some
> cases, there could be some reference from, for example, the system class
> loader to the old web app class loader, stopping the garbage collector from
> reclaiming the memory. A classic example is when you package the database
> driver in WEB-INF/lib in your webapp: the driver manager (in the system
> class loader) will hold a reference to a class in the webapp classloader.
>
> The same happens with shutdown hooks: you have a system class loader
> (Runtime) that holds a reference to the shutdown thread, keeping the webapp
> classloader always alive.
>
> Anyway, a shutdown hook will only cleanup resources when the jvm ends. And
> you probably want the cleanup to happen when the web application is
> undeployed.
>
> That is why I think the best option is to use a context loader listener: add
> a shutdown operation to DefaultGuiceModule and call it in from
> GuiceServletContextListener.contextDestroyed.
>
> BTW: I found another problem. XmlUtil is caching a DocumentBuilder in a
> ThreadLocal, and never released. This also keeps the web app classloader
> alive.
>
> Thanks
>
> On Thu, Sep 23, 2010 at 11:18 PM, Gagandeep singh <ga...@gmail.com>wrote:
>
>> Hi Pablo
>>
>> I couldn't understand the memory leak you found in Guice. Hopefully crazy
>> bob will fix it soon (it has been 3 months already).
>> As for the shindig shutdown hook, it looks like an innocent hook which
>> tries
>> to stop the executor service which is where all the threads are scheduled.
>> Did you find any particular instance in which it creates a memory leak ?
>> Also, please explain a bit more on your ContextLoaderListener idea.
>>
>> Thanks
>> Gagan
>>
>> 2010/9/22 Pablo Graña <pa...@globant.com>
>>
>> > Hi guys.
>> >
>> > I get an oom after some hot deploys. I traced one problem to Guice (
>> > http://www.mail-archive.com/google-guice@googlegroups.com/msg02987.html-
>> > there is a workaround) and another to shindig: the DefaultGuiceModule
>> > installs a shutdown hook, apparently with the idea to shut down a thread
>> > pool. Is this hook correct? It should probably go in a
>> > ContextLoaderListener, right?
>> >
>> > thanks a lot.
>> >
>> > --
>> > Pablo Gra\~na
>> > Chief Architect
>> > Globant
>> > Arg Office: +54 (11) 4109 1743
>> > UK Office: +44 (20) 7043 8269 int 8043
>> > US Office: +1 (212) 400 7686 int 8043
>> >
>>
>
>
>
> --
> Pablo Gra\~na
> Chief Architect
> Globant
> Arg Office: +54 (11) 4109 1743
> UK Office: +44 (20) 7043 8269 int 8043
> US Office: +1 (212) 400 7686 int 8043
>
--
Paul Lindner -- lindner@inuus.com -- linkedin.com/in/plindner
Re: Multiple hot deploys and shindig
Posted by Pablo Graña <pa...@globant.com>.
Here is more info on the problem and the current status:
http://code.google.com/p/guava-libraries/issues/detail?id=92
Guys from guice copied the classes from guava (or google-collections),
bringing the same problem to guice.
The memory leaks manifests with hot redeploys in tomcat, after a couple of
which you get a perm gen memory error. The only safe thing to do is to
restart tomcat for every deploy.
The problem is that for each web application, tomcat creates a class loader.
When you redeploy the same application, tomcat creates a new class loader
and removes the references it holds to the old class loader. But in some
cases, there could be some reference from, for example, the system class
loader to the old web app class loader, stopping the garbage collector from
reclaiming the memory. A classic example is when you package the database
driver in WEB-INF/lib in your webapp: the driver manager (in the system
class loader) will hold a reference to a class in the webapp classloader.
The same happens with shutdown hooks: you have a system class loader
(Runtime) that holds a reference to the shutdown thread, keeping the webapp
classloader always alive.
Anyway, a shutdown hook will only cleanup resources when the jvm ends. And
you probably want the cleanup to happen when the web application is
undeployed.
That is why I think the best option is to use a context loader listener: add
a shutdown operation to DefaultGuiceModule and call it in from
GuiceServletContextListener.contextDestroyed.
BTW: I found another problem. XmlUtil is caching a DocumentBuilder in a
ThreadLocal, and never released. This also keeps the web app classloader
alive.
Thanks
On Thu, Sep 23, 2010 at 11:18 PM, Gagandeep singh <ga...@gmail.com>wrote:
> Hi Pablo
>
> I couldn't understand the memory leak you found in Guice. Hopefully crazy
> bob will fix it soon (it has been 3 months already).
> As for the shindig shutdown hook, it looks like an innocent hook which
> tries
> to stop the executor service which is where all the threads are scheduled.
> Did you find any particular instance in which it creates a memory leak ?
> Also, please explain a bit more on your ContextLoaderListener idea.
>
> Thanks
> Gagan
>
> 2010/9/22 Pablo Graña <pa...@globant.com>
>
> > Hi guys.
> >
> > I get an oom after some hot deploys. I traced one problem to Guice (
> > http://www.mail-archive.com/google-guice@googlegroups.com/msg02987.html-
> > there is a workaround) and another to shindig: the DefaultGuiceModule
> > installs a shutdown hook, apparently with the idea to shut down a thread
> > pool. Is this hook correct? It should probably go in a
> > ContextLoaderListener, right?
> >
> > thanks a lot.
> >
> > --
> > Pablo Gra\~na
> > Chief Architect
> > Globant
> > Arg Office: +54 (11) 4109 1743
> > UK Office: +44 (20) 7043 8269 int 8043
> > US Office: +1 (212) 400 7686 int 8043
> >
>
--
Pablo Gra\~na
Chief Architect
Globant
Arg Office: +54 (11) 4109 1743
UK Office: +44 (20) 7043 8269 int 8043
US Office: +1 (212) 400 7686 int 8043
Re: Multiple hot deploys and shindig
Posted by Gagandeep singh <ga...@gmail.com>.
Hi Pablo
I couldn't understand the memory leak you found in Guice. Hopefully crazy
bob will fix it soon (it has been 3 months already).
As for the shindig shutdown hook, it looks like an innocent hook which tries
to stop the executor service which is where all the threads are scheduled.
Did you find any particular instance in which it creates a memory leak ?
Also, please explain a bit more on your ContextLoaderListener idea.
Thanks
Gagan
2010/9/22 Pablo Graña <pa...@globant.com>
> Hi guys.
>
> I get an oom after some hot deploys. I traced one problem to Guice (
> http://www.mail-archive.com/google-guice@googlegroups.com/msg02987.html -
> there is a workaround) and another to shindig: the DefaultGuiceModule
> installs a shutdown hook, apparently with the idea to shut down a thread
> pool. Is this hook correct? It should probably go in a
> ContextLoaderListener, right?
>
> thanks a lot.
>
> --
> Pablo Gra\~na
> Chief Architect
> Globant
> Arg Office: +54 (11) 4109 1743
> UK Office: +44 (20) 7043 8269 int 8043
> US Office: +1 (212) 400 7686 int 8043
>