You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geronimo.apache.org by Kevan Miller <ke...@gmail.com> on 2005/11/07 19:56:52 UTC
ClassLoader memory leaks
I've been battling a variety of ClassLoader-based memory leaks that occur
during the deploy/undeploy of DayTrader. With the patches for the following
problems, I no longer see MultiParentClassLoaders being leaked:
http://issues.apache.org/jira/browse/GERONIMO-1125 (patch just posted)
http://issues.apache.org/jira/browse/GERONIMO-1118 (fix committed)
http://issues.apache.org/jira/browse/AXIS-2232 (Old patch didn't fix the
problem. I've attached a new patch that fixes the problem. Yet to be
committed)
http://issues.apache.org/jira/browse/AXIS-2278 (Fix committed, but not
picked up by Geronimo build)
Deployment of DayTrader is still leaking memory (but only 200 kbytes per
deploy/undeploy cycle. "only" is a relative term... ;-). So, there's still
work to be done.
Finally, there is an intermittent problem which will cause a Thread to
temporarily hold on to a ClassLoader. Since the problem is intermittent and
temporary, I'm not actively pursuing. I'll document the problem in a jira,
but here's the chain of references keeping the ClassLoader alive... The
Thread is in the RMI Runtime ThreadGroup.
java.lang.Thread.inheritedAccessControlContext -->
java.security.AccessControlContext.context -->
java.security.ProtectionDomain[4] -->
java.security.ProtectionDomain.classLoader -->
org.apache.geronimo.kernel.config.MultiParentClassLoader
--kevan
Re: ClassLoader memory leaks
Posted by Kevan Miller <ke...@gmail.com>.
Apologies for talking to myself... ;-)
Regarding Sun bug id 4167874, there is the possibility of turning of caching
to avoid the problem. I'll investigate...
--kevan
On 11/8/05, Kevan Miller <ke...@gmail.com> wrote:
>
> Just a quick update. There is an initial bump in memory usage after the
> first deploy (15 megs -> 19 megs). After 5 deploys, memory growth levels off
> at around 20 megs. There is memory growth after that. A number of Strings,
> some TimerTasks. However, the most serious problem is that we're leaking
> sun.net.www.protocol.jar.URLJarFiles (kept alive by sun's JarFileFactory).
> See http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4167874 for more
> info. Near term, I'm not sure that there's much we can do about this one...
> Bug description indicates that this is also hanging onto file descriptors.
> If true, we'll run out of file descriptors before running out of memory.
>
> After ~ 50 deploys, my server process froze. Good chance that this was
> caused by profiling instrumentation (I was tracking allocations). I'll set
> up an overnight test and see how long it can run for...
>
> --kevan
>
>
> On 11/7/05, Kevan Miller <ke...@gmail.com> wrote:
> >
> > Hi Jeff,
> > Yes, I had, but only for a few deploy/undeploy cycles. I just ran a more
> > extended test, with much better results. I'm not seeing unbounded growth in
> > memory. At least not at the rate I reported earlier. I'm seeing growth of
> > about 500 bytes per deploy/undeploy cycle. I think there may be an initial
> > ramp-up for the first several deploys. Afterwards, growth is very small --
> > practically in the noise range.
> >
> > I know there's a 30 minute timer task that will hold on to a few
> > objects. I also see a number of WeakHashMap Entries which are pending a Map
> > operation before they will be cleared out. It's possible that we aren't
> > leaking memory at all.
> >
> > I'll run a more extended test. However, unless I get dramatically
> > different results, I think there are bigger fish to fry...
> >
> > --kevan
> >
> > On 11/7/05, Jeff Genender < jgenender@savoirtech.com> wrote:
> > >
> > > Have you run a profiler to see where the leaks are occurring?
> > >
> > > Matt Hogstrom wrote:
> > > > Kevan,
> > > >
> > > > Big thanks for tracking down these issues. I know they aren't easy.
> > > >
> > > > Matt
> > > >
> > > > Kevan Miller wrote:
> > > >> I've been battling a variety of ClassLoader-based memory leaks that
> > > occur
> > > >> during the deploy/undeploy of DayTrader. With the patches for the
> > > >> following
> > > >> problems, I no longer see MultiParentClassLoaders being leaked:
> > > >>
> > > >> http://issues.apache.org/jira/browse/GERONIMO-1125 (patch just
> > > posted)
> > > >> http://issues.apache.org/jira/browse/GERONIMO-1118 (fix committed)
> > > >> http://issues.apache.org/jira/browse/AXIS-2232 (Old patch didn't
> > > fix the
> > > >> problem. I've attached a new patch that fixes the problem. Yet to
> > > be
> > > >> committed)
> > > >> http://issues.apache.org/jira/browse/AXIS-2278 (Fix committed, but
> > > not
> > > >> picked up by Geronimo build)
> > > >>
> > > >> Deployment of DayTrader is still leaking memory (but only 200
> > > kbytes per
> > > >> deploy/undeploy cycle. "only" is a relative term... ;-). So,
> > > there's
> > > >> still
> > > >> work to be done.
> > > >>
> > > >> Finally, there is an intermittent problem which will cause a Thread
> > > to
> > > >> temporarily hold on to a ClassLoader. Since the problem is
> > > >> intermittent and
> > > >> temporary, I'm not actively pursuing. I'll document the problem in
> > > a
> > > >> jira,
> > > >> but here's the chain of references keeping the ClassLoader alive...
> > > The
> > > >> Thread is in the RMI Runtime ThreadGroup.
> > > >>
> > > >> java.lang.Thread.inheritedAccessControlContext -->
> > > >> java.security.AccessControlContext.context -->
> > > >> java.security.ProtectionDomain[4] -->
> > > >> java.security.ProtectionDomain.classLoader -->
> > > >> org.apache.geronimo.kernel.config.MultiParentClassLoader
> > > >>
> > > >> --kevan
> > > >>
> > >
> >
> >
>
Re: ClassLoader memory leaks
Posted by Kevan Miller <ke...@gmail.com>.
Just a quick update. There is an initial bump in memory usage after the
first deploy (15 megs -> 19 megs). After 5 deploys, memory growth levels off
at around 20 megs. There is memory growth after that. A number of Strings,
some TimerTasks. However, the most serious problem is that we're leaking
sun.net.www.protocol.jar.URLJarFiles (kept alive by sun's JarFileFactory).
See http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4167874 for more
info. Near term, I'm not sure that there's much we can do about this one...
Bug description indicates that this is also hanging onto file descriptors.
If true, we'll run out of file descriptors before running out of memory.
After ~ 50 deploys, my server process froze. Good chance that this was
caused by profiling instrumentation (I was tracking allocations). I'll set
up an overnight test and see how long it can run for...
--kevan
On 11/7/05, Kevan Miller <ke...@gmail.com> wrote:
>
> Hi Jeff,
> Yes, I had, but only for a few deploy/undeploy cycles. I just ran a more
> extended test, with much better results. I'm not seeing unbounded growth in
> memory. At least not at the rate I reported earlier. I'm seeing growth of
> about 500 bytes per deploy/undeploy cycle. I think there may be an initial
> ramp-up for the first several deploys. Afterwards, growth is very small --
> practically in the noise range.
>
> I know there's a 30 minute timer task that will hold on to a few objects.
> I also see a number of WeakHashMap Entries which are pending a Map operation
> before they will be cleared out. It's possible that we aren't leaking memory
> at all.
>
> I'll run a more extended test. However, unless I get dramatically
> different results, I think there are bigger fish to fry...
>
> --kevan
>
> On 11/7/05, Jeff Genender <jg...@savoirtech.com> wrote:
> >
> > Have you run a profiler to see where the leaks are occurring?
> >
> > Matt Hogstrom wrote:
> > > Kevan,
> > >
> > > Big thanks for tracking down these issues. I know they aren't easy.
> > >
> > > Matt
> > >
> > > Kevan Miller wrote:
> > >> I've been battling a variety of ClassLoader-based memory leaks that
> > occur
> > >> during the deploy/undeploy of DayTrader. With the patches for the
> > >> following
> > >> problems, I no longer see MultiParentClassLoaders being leaked:
> > >>
> > >> http://issues.apache.org/jira/browse/GERONIMO-1125 (patch just
> > posted)
> > >> http://issues.apache.org/jira/browse/GERONIMO-1118 (fix committed)
> > >> http://issues.apache.org/jira/browse/AXIS-2232 (Old patch didn't fix
> > the
> > >> problem. I've attached a new patch that fixes the problem. Yet to be
> > >> committed)
> > >> http://issues.apache.org/jira/browse/AXIS-2278 (Fix committed, but
> > not
> > >> picked up by Geronimo build)
> > >>
> > >> Deployment of DayTrader is still leaking memory (but only 200 kbytes
> > per
> > >> deploy/undeploy cycle. "only" is a relative term... ;-). So, there's
> > >> still
> > >> work to be done.
> > >>
> > >> Finally, there is an intermittent problem which will cause a Thread
> > to
> > >> temporarily hold on to a ClassLoader. Since the problem is
> > >> intermittent and
> > >> temporary, I'm not actively pursuing. I'll document the problem in a
> > >> jira,
> > >> but here's the chain of references keeping the ClassLoader alive...
> > The
> > >> Thread is in the RMI Runtime ThreadGroup.
> > >>
> > >> java.lang.Thread.inheritedAccessControlContext -->
> > >> java.security.AccessControlContext.context -->
> > >> java.security.ProtectionDomain[4] -->
> > >> java.security.ProtectionDomain.classLoader -->
> > >> org.apache.geronimo.kernel.config.MultiParentClassLoader
> > >>
> > >> --kevan
> > >>
> >
>
>
Re: ClassLoader memory leaks
Posted by Kevan Miller <ke...@gmail.com>.
Hi Jeff,
Yes, I had, but only for a few deploy/undeploy cycles. I just ran a more
extended test, with much better results. I'm not seeing unbounded growth in
memory. At least not at the rate I reported earlier. I'm seeing growth of
about 500 bytes per deploy/undeploy cycle. I think there may be an initial
ramp-up for the first several deploys. Afterwards, growth is very small --
practically in the noise range.
I know there's a 30 minute timer task that will hold on to a few objects. I
also see a number of WeakHashMap Entries which are pending a Map operation
before they will be cleared out. It's possible that we aren't leaking memory
at all.
I'll run a more extended test. However, unless I get dramatically different
results, I think there are bigger fish to fry...
--kevan
On 11/7/05, Jeff Genender <jg...@savoirtech.com> wrote:
>
> Have you run a profiler to see where the leaks are occurring?
>
> Matt Hogstrom wrote:
> > Kevan,
> >
> > Big thanks for tracking down these issues. I know they aren't easy.
> >
> > Matt
> >
> > Kevan Miller wrote:
> >> I've been battling a variety of ClassLoader-based memory leaks that
> occur
> >> during the deploy/undeploy of DayTrader. With the patches for the
> >> following
> >> problems, I no longer see MultiParentClassLoaders being leaked:
> >>
> >> http://issues.apache.org/jira/browse/GERONIMO-1125 (patch just posted)
> >> http://issues.apache.org/jira/browse/GERONIMO-1118 (fix committed)
> >> http://issues.apache.org/jira/browse/AXIS-2232 (Old patch didn't fix
> the
> >> problem. I've attached a new patch that fixes the problem. Yet to be
> >> committed)
> >> http://issues.apache.org/jira/browse/AXIS-2278 (Fix committed, but not
> >> picked up by Geronimo build)
> >>
> >> Deployment of DayTrader is still leaking memory (but only 200 kbytes
> per
> >> deploy/undeploy cycle. "only" is a relative term... ;-). So, there's
> >> still
> >> work to be done.
> >>
> >> Finally, there is an intermittent problem which will cause a Thread to
> >> temporarily hold on to a ClassLoader. Since the problem is
> >> intermittent and
> >> temporary, I'm not actively pursuing. I'll document the problem in a
> >> jira,
> >> but here's the chain of references keeping the ClassLoader alive... The
> >> Thread is in the RMI Runtime ThreadGroup.
> >>
> >> java.lang.Thread.inheritedAccessControlContext -->
> >> java.security.AccessControlContext.context -->
> >> java.security.ProtectionDomain[4] -->
> >> java.security.ProtectionDomain.classLoader -->
> >> org.apache.geronimo.kernel.config.MultiParentClassLoader
> >>
> >> --kevan
> >>
>
Re: ClassLoader memory leaks
Posted by Jeff Genender <jg...@savoirtech.com>.
Have you run a profiler to see where the leaks are occurring?
Matt Hogstrom wrote:
> Kevan,
>
> Big thanks for tracking down these issues. I know they aren't easy.
>
> Matt
>
> Kevan Miller wrote:
>> I've been battling a variety of ClassLoader-based memory leaks that occur
>> during the deploy/undeploy of DayTrader. With the patches for the
>> following
>> problems, I no longer see MultiParentClassLoaders being leaked:
>>
>> http://issues.apache.org/jira/browse/GERONIMO-1125 (patch just posted)
>> http://issues.apache.org/jira/browse/GERONIMO-1118 (fix committed)
>> http://issues.apache.org/jira/browse/AXIS-2232 (Old patch didn't fix the
>> problem. I've attached a new patch that fixes the problem. Yet to be
>> committed)
>> http://issues.apache.org/jira/browse/AXIS-2278 (Fix committed, but not
>> picked up by Geronimo build)
>>
>> Deployment of DayTrader is still leaking memory (but only 200 kbytes per
>> deploy/undeploy cycle. "only" is a relative term... ;-). So, there's
>> still
>> work to be done.
>>
>> Finally, there is an intermittent problem which will cause a Thread to
>> temporarily hold on to a ClassLoader. Since the problem is
>> intermittent and
>> temporary, I'm not actively pursuing. I'll document the problem in a
>> jira,
>> but here's the chain of references keeping the ClassLoader alive... The
>> Thread is in the RMI Runtime ThreadGroup.
>>
>> java.lang.Thread.inheritedAccessControlContext -->
>> java.security.AccessControlContext.context -->
>> java.security.ProtectionDomain[4] -->
>> java.security.ProtectionDomain.classLoader -->
>> org.apache.geronimo.kernel.config.MultiParentClassLoader
>>
>> --kevan
>>
Re: ClassLoader memory leaks
Posted by Matt Hogstrom <ma...@hogstrom.org>.
Kevan,
Big thanks for tracking down these issues. I know they aren't easy.
Matt
Kevan Miller wrote:
> I've been battling a variety of ClassLoader-based memory leaks that occur
> during the deploy/undeploy of DayTrader. With the patches for the following
> problems, I no longer see MultiParentClassLoaders being leaked:
>
> http://issues.apache.org/jira/browse/GERONIMO-1125 (patch just posted)
> http://issues.apache.org/jira/browse/GERONIMO-1118 (fix committed)
> http://issues.apache.org/jira/browse/AXIS-2232 (Old patch didn't fix the
> problem. I've attached a new patch that fixes the problem. Yet to be
> committed)
> http://issues.apache.org/jira/browse/AXIS-2278 (Fix committed, but not
> picked up by Geronimo build)
>
> Deployment of DayTrader is still leaking memory (but only 200 kbytes per
> deploy/undeploy cycle. "only" is a relative term... ;-). So, there's still
> work to be done.
>
> Finally, there is an intermittent problem which will cause a Thread to
> temporarily hold on to a ClassLoader. Since the problem is intermittent and
> temporary, I'm not actively pursuing. I'll document the problem in a jira,
> but here's the chain of references keeping the ClassLoader alive... The
> Thread is in the RMI Runtime ThreadGroup.
>
> java.lang.Thread.inheritedAccessControlContext -->
> java.security.AccessControlContext.context -->
> java.security.ProtectionDomain[4] -->
> java.security.ProtectionDomain.classLoader -->
> org.apache.geronimo.kernel.config.MultiParentClassLoader
>
> --kevan
>