You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@groovy.apache.org by Kyle Boon <ky...@gmail.com> on 2015/06/01 22:13:06 UTC

Memory leaks, GroovyClassLoader, and the MetaClassRegistry

We have a memory leak that I'm fairly confident that can be traced back to
this issue:

http://stackoverflow.com/questions/14343393/using-groovyclassloader-from-java-classes-not-gcd

We have a lot of REST services that result in code being compiled and
executed during the HTTP request.  I'm exploring a few different options to
try to fix the memory leak by removing the class from the MetaClassRegistry
which (should) allow the class to be garbage collected during the next full
GC.  At the end of the request, while there is still an instance of the
compiled Script class, I could remove the class from the MetaClassRegistry
explicitly. Or, I could create a background job that would find all
compiled Script classes and remove them from the MetaClassRegistry in bulk.

Complicating this slightly is that in order to reduce the number of times
we actually need to compile code, we put the compiled class into a Guava
cache. Then we can avoid the compilation step the majority of the time. My
question is, what happens if the class is removed from the
MetaClassRegistry and then another instance of that class is created later?
Would it be worth it to try to recreate the meta class for the script class
if it was missing? Or is this just a fool's errand?

Thanks,

Kyle

Re: Memory leaks, GroovyClassLoader, and the MetaClassRegistry

Posted by Jochen Theodorou <bl...@gmx.org>.
Am 03.06.2015 17:01, schrieb Kyle Boon:
> We're using 1.7.0_80 and Grails 2.3.11 which brings in Groovy 2.1.9. Now
> that we finished a spring security upgrade, we're planning moving to
> grails 2.5 and java 8 in the near future.
>
> I'd have to double check but I'm pretty sure we don't mutate the
> metaclass and the Scripts being executed are preventing from accessing
> the metaclass at all.

I was mainly asking because in Groovy 2.4 we got a user contribution to 
use ClassValue in JDK7+ which makes the metaclass a part of the class, 
instead of having global structure managed by us. That would be the case 
with Grails 2.5 then as well

bye blackdrag


-- 
Jochen "blackdrag" Theodorou
blog: http://blackdragsview.blogspot.com/


Re: Memory leaks, GroovyClassLoader, and the MetaClassRegistry

Posted by Kyle Boon <ky...@gmail.com>.
We're using 1.7.0_80 and Grails 2.3.11 which brings in Groovy 2.1.9. Now
that we finished a spring security upgrade, we're planning moving to grails
2.5 and java 8 in the near future.

I'd have to double check but I'm pretty sure we don't mutate the metaclass
and the Scripts being executed are preventing from accessing the metaclass
at all.

Kyle

On Tue, Jun 2, 2015 at 7:00 AM, Jochen Theodorou <bl...@gmx.org> wrote:

> Am 01.06.2015 22:13, schrieb Kyle Boon:
>
>> We have a memory leak that I'm fairly confident that can be traced back
>> to this issue:
>>
>>
>> http://stackoverflow.com/questions/14343393/using-groovyclassloader-from-java-classes-not-gcd
>>
>
> can you tell me Groovy and Java versions as well as if you tried the vm
> switch -XX:+CMSClassUnloadingEnabled ?
>
>  We have a lot of REST services that result in code being compiled and
>> executed during the HTTP request.  I'm exploring a few different options
>> to try to fix the memory leak by removing the class from the
>> MetaClassRegistry which (should) allow the class to be garbage collected
>> during the next full GC.  At the end of the request, while there is
>> still an instance of the compiled Script class, I could remove the class
>> from the MetaClassRegistry explicitly. Or, I could create a background
>> job that would find all compiled Script classes and remove them from the
>> MetaClassRegistry in bulk.
>>
>> Complicating this slightly is that in order to reduce the number of
>> times we actually need to compile code, we put the compiled class into a
>> Guava cache. Then we can avoid the compilation step the majority of the
>> time. My question is, what happens if the class is removed from the
>> MetaClassRegistry and then another instance of that class is created
>> later? Would it be worth it to try to recreate the meta class for the
>> script class if it was missing? Or is this just a fool's errand?
>>
>
> It really depends on what meta class you are using here... namely
> MetaClassImpl (default), ExpandoMetaClass or a custom one (which I assume
> you don't do).
>
> The default meta class handling is designed to be able to "loose" the meta
> class and have it automatically recreated at a later point. For this all
> the default meta classes are only reachable through soft references. So it
> is perfectly fine to remove it from the registry. It will be recreated on
> demand... of course that has a performance penalty. But you don't have to
> recreate it manually... actually setting the meta class manually can result
> in the meta class not being soft referenced anymore. Also groovy objects
> store the meta class in a instance field (groovy has per instance meta
> classes). So as long as the script instance exists, the meta class usually
> does as well, unless you null the field of course. But the instance is not
> the problem here I guess.
>
> If ExpandoMetaClass is used (default in Grails), things can be slightly
> different. Here usually strong references are used, if not from the
> beginning, then from the point of mutation of the meta class. Would be good
> to know if we can exclude the ExpandoMetaClass case.
>
> bye blackdrag
>
> --
> Jochen "blackdrag" Theodorou
> blog: http://blackdragsview.blogspot.com/
>
>

Re: Memory leaks, GroovyClassLoader, and the MetaClassRegistry

Posted by Jochen Theodorou <bl...@gmx.org>.
Am 01.06.2015 22:13, schrieb Kyle Boon:
> We have a memory leak that I'm fairly confident that can be traced back
> to this issue:
>
> http://stackoverflow.com/questions/14343393/using-groovyclassloader-from-java-classes-not-gcd

can you tell me Groovy and Java versions as well as if you tried the vm 
switch -XX:+CMSClassUnloadingEnabled ?

> We have a lot of REST services that result in code being compiled and
> executed during the HTTP request.  I'm exploring a few different options
> to try to fix the memory leak by removing the class from the
> MetaClassRegistry which (should) allow the class to be garbage collected
> during the next full GC.  At the end of the request, while there is
> still an instance of the compiled Script class, I could remove the class
> from the MetaClassRegistry explicitly. Or, I could create a background
> job that would find all compiled Script classes and remove them from the
> MetaClassRegistry in bulk.
>
> Complicating this slightly is that in order to reduce the number of
> times we actually need to compile code, we put the compiled class into a
> Guava cache. Then we can avoid the compilation step the majority of the
> time. My question is, what happens if the class is removed from the
> MetaClassRegistry and then another instance of that class is created
> later? Would it be worth it to try to recreate the meta class for the
> script class if it was missing? Or is this just a fool's errand?

It really depends on what meta class you are using here... namely 
MetaClassImpl (default), ExpandoMetaClass or a custom one (which I 
assume you don't do).

The default meta class handling is designed to be able to "loose" the 
meta class and have it automatically recreated at a later point. For 
this all the default meta classes are only reachable through soft 
references. So it is perfectly fine to remove it from the registry. It 
will be recreated on demand... of course that has a performance penalty. 
But you don't have to recreate it manually... actually setting the meta 
class manually can result in the meta class not being soft referenced 
anymore. Also groovy objects store the meta class in a instance field 
(groovy has per instance meta classes). So as long as the script 
instance exists, the meta class usually does as well, unless you null 
the field of course. But the instance is not the problem here I guess.

If ExpandoMetaClass is used (default in Grails), things can be slightly 
different. Here usually strong references are used, if not from the 
beginning, then from the point of mutation of the meta class. Would be 
good to know if we can exclude the ExpandoMetaClass case.

bye blackdrag

-- 
Jochen "blackdrag" Theodorou
blog: http://blackdragsview.blogspot.com/