You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cayenne.apache.org by Andrew Lindesay <ap...@lindesay.co.nz> on 2012/02/14 11:30:33 UTC
Question on ObjectContext and Deserialization
Hello;
I have a little question on the ObjectContext in 3.0.2.
I'm keeping an ObjectContext in an HttpSession instance's attributes for
re-use between request-response cycles. When I serialize this
ObjectContext instance, it (+ other objects) stores out to < 8k in my
test case, but then deserializing the ObjectContext causes an
OutOfMemory -- even with substantial quantities of memory available.
Has anybody else seen this or is there anything to be weary of before I
look into it further?
Regards
--
Andrew Lindesay
Re: Question on ObjectContext and Deserialization
Posted by Andrew Lindesay <ap...@lindesay.co.nz>.
Hi Andrus;
For what it is worth; I have tested commons 3.1 and 3.2 with a small
test rig and the problem appears to be gone in 3.2 -- although not
needing to serialize at all would be ideal if you can see a way forward
that way!
Regards;
On 23/02/12 12:22 AM, Andrus Adamchik wrote:
> I think we can improve deserialization performance in general and solve this specific issue if we make ObjectStore.objectMap (this is the AbstractReferenceMap causing trouble) transient. There's no value is saving and restoring this map. There's another map called "changes", which is a regular HashMap, and that needs to be preserved.
> Now the implementation can be a little tricky, as "objectMap" is provided by the DI factory and we need to create an empty map of the same type on deserialization. This may require refactoring of ObjectStore creation flow. Should be doable though...
--
Andrew Lindesay
www.silvereye.co.nz
Re: Question on ObjectContext and Deserialization
Posted by Andrus Adamchik <an...@objectstyle.org>.
I think we can improve deserialization performance in general and solve this specific issue if we make ObjectStore.objectMap (this is the AbstractReferenceMap causing trouble) transient. There's no value is saving and restoring this map. There's another map called "changes", which is a regular HashMap, and that needs to be preserved.
Now the implementation can be a little tricky, as "objectMap" is provided by the DI factory and we need to create an empty map of the same type on deserialization. This may require refactoring of ObjectStore creation flow. Should be doable though...
Andrus
On Feb 22, 2012, at 1:34 PM, Andrew Lindesay wrote:
> Hi there;
>
> It does appear that the problem is in an Apache Commons abstract superclass. What is happening is that whilst running;
>
> AbstractReferenceMap.doReadObject(..)
>
> ...the method...
>
> AbstractReferenceMap.checkCapacity(..)
>
> ...is invoked and it seems to be doubling the size of the Map's entry array on de-serialization. Through sequential serialization + de-serialization this yields the exponential growth in the memory consumption.
>
> It seems that the problem is caused by the 'threshold' value being calculated after to populating the Map during de-serialization.
>
> I have re-built 3.0-STABLE with commons 3.2.1 and it still seems to exhibit the same problem.
>
> The release cycles for commons seems to be quite long these days. Hmmm.....
>
> Regards;
>
> On 18/02/12 11:50 AM, Andrew Lindesay wrote:
>> Hi Andrus;
>>
>> It does seem rather strange.
>>
>> I'm starting the application with maven (from Eclipse) and
>> jetty-maven-plugin. No hot redeploy. No actual container per se. The
>> class-loader should be the same so shouldn't be re-creating "isolated
>> singletons or statics". I'm using a Filter to serialize-->deserialize
>> the attributes on the HttpSession in order to test how some
>> infrastructure handles serialization-->deserialization of the session
>> attributes.
>>
>> If I explicitly don't serialize-->deserialize the ObjectContext then
>> there is no problem; so serializing ObjectContext as the root object
>> does seem to cause the problem.
>>
>> The incremental increase in the memory consumption seems to be roughly
>> exponential in relation to the number of deserializations of the
>> ObjectContext. That may be a clue; I may download Apache commons and see
>> if I can see anything obvious there in the LRU map. Like you say though;
>> it probably shouldn't be serializing this stuff anyway though.
>>
>> It looks like "SQLTemplateResourceManager" is created via reflection so
>> it may be hard to profile for memory usage.
>>
>> I'll log a Jira when I get a moment.
>>
>> Regards;
>>
>> On 17/02/12 11:29 PM, Andrus Adamchik wrote:
>>> Interesting... I didn't realize we have a static Velocity runtime in
>>> the app. This must to be changed to instance variable with correct
>>> scope, (and if needed shut down properly when ServerRuntime is
>>> shutdown). Could you please log a Jira - we need to investigate.
>>>
>>> BTW, are you doing hot redeploy of your app? This in combination with
>>> static Velocity runtime might produce the observed behavior.
>>>
>>> Andrus
>>>
>>>
>>> On Feb 17, 2012, at 1:19 PM, Andrew Lindesay wrote:
>>>
>>>> Hi;
>>>>
>>>> It looks like it is holding onto lots (hundreds of megs) of;
>>>>
>>>> org.apache.commons.collections.map.AbstractHashedMap$HashEntry[]
>>>> > org.apache.commons.collecitons.map.LRUMap
>>>> >> org.apache.cayenne.access.jdbc.SQLTemplateResourceManager
>>>> >>> ...velocity
>>>>
>>>> Not quite understanding yet... as it seems like this wouldn't be
>>>> something stored on an object context.
>>>>
>>>> Regards;
>>>>
>>>> On 16/02/12 7:50 PM, Andrus Adamchik wrote:
>>>>>>> an OutOfMemory -- even with substantial quantities of memory
>>>>>>> available.
>>>>>
>>>>> This is a hint actually.. Maybe the problem is not heap size, but
>>>>> perm gen size instead? Try increasing that:
>>>>>
>>>>> -XX:MaxPermSize=256m
>>>>>
>>>>> On Feb 16, 2012, at 9:44 AM, Andrus Adamchik wrote:
>>>>>
>>>>>> This is strange indeed. Haven't seen such behavior. May need some
>>>>>> profiling on your end I guess.
>>>>>>
>>>>>> Andrus
>>>>>>
>>>>>>
>>>>>> On Feb 14, 2012, at 1:30 PM, Andrew Lindesay wrote:
>>>>>>> Hello;
>>>>>>>
>>>>>>> I have a little question on the ObjectContext in 3.0.2.
>>>>>>>
>>>>>>> I'm keeping an ObjectContext in an HttpSession instance's
>>>>>>> attributes for re-use between request-response cycles. When I
>>>>>>> serialize this ObjectContext instance, it (+ other objects) stores
>>>>>>> out to< 8k in my test case, but then deserializing the
>>>>>>> ObjectContext causes an OutOfMemory -- even with substantial
>>>>>>> quantities of memory available.
>>>>>>>
>>>>>>> Has anybody else seen this or is there anything to be weary of
>>>>>>> before I look into it further?
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>> --
>>>>>>> Andrew Lindesay
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Andrew Lindesay
>>>> www.silvereye.co.nz
>>>>
>>>
>>>
>>
>>
>
>
> --
> Andrew Lindesay
> www.silvereye.co.nz
>
Re: Question on ObjectContext and Deserialization
Posted by Andrew Lindesay <ap...@lindesay.co.nz>.
Hi there;
It does appear that the problem is in an Apache Commons abstract
superclass. What is happening is that whilst running;
AbstractReferenceMap.doReadObject(..)
...the method...
AbstractReferenceMap.checkCapacity(..)
...is invoked and it seems to be doubling the size of the Map's entry
array on de-serialization. Through sequential serialization +
de-serialization this yields the exponential growth in the memory
consumption.
It seems that the problem is caused by the 'threshold' value being
calculated after to populating the Map during de-serialization.
I have re-built 3.0-STABLE with commons 3.2.1 and it still seems to
exhibit the same problem.
The release cycles for commons seems to be quite long these days. Hmmm.....
Regards;
On 18/02/12 11:50 AM, Andrew Lindesay wrote:
> Hi Andrus;
>
> It does seem rather strange.
>
> I'm starting the application with maven (from Eclipse) and
> jetty-maven-plugin. No hot redeploy. No actual container per se. The
> class-loader should be the same so shouldn't be re-creating "isolated
> singletons or statics". I'm using a Filter to serialize-->deserialize
> the attributes on the HttpSession in order to test how some
> infrastructure handles serialization-->deserialization of the session
> attributes.
>
> If I explicitly don't serialize-->deserialize the ObjectContext then
> there is no problem; so serializing ObjectContext as the root object
> does seem to cause the problem.
>
> The incremental increase in the memory consumption seems to be roughly
> exponential in relation to the number of deserializations of the
> ObjectContext. That may be a clue; I may download Apache commons and see
> if I can see anything obvious there in the LRU map. Like you say though;
> it probably shouldn't be serializing this stuff anyway though.
>
> It looks like "SQLTemplateResourceManager" is created via reflection so
> it may be hard to profile for memory usage.
>
> I'll log a Jira when I get a moment.
>
> Regards;
>
> On 17/02/12 11:29 PM, Andrus Adamchik wrote:
>> Interesting... I didn't realize we have a static Velocity runtime in
>> the app. This must to be changed to instance variable with correct
>> scope, (and if needed shut down properly when ServerRuntime is
>> shutdown). Could you please log a Jira - we need to investigate.
>>
>> BTW, are you doing hot redeploy of your app? This in combination with
>> static Velocity runtime might produce the observed behavior.
>>
>> Andrus
>>
>>
>> On Feb 17, 2012, at 1:19 PM, Andrew Lindesay wrote:
>>
>>> Hi;
>>>
>>> It looks like it is holding onto lots (hundreds of megs) of;
>>>
>>> org.apache.commons.collections.map.AbstractHashedMap$HashEntry[]
>>> > org.apache.commons.collecitons.map.LRUMap
>>> >> org.apache.cayenne.access.jdbc.SQLTemplateResourceManager
>>> >>> ...velocity
>>>
>>> Not quite understanding yet... as it seems like this wouldn't be
>>> something stored on an object context.
>>>
>>> Regards;
>>>
>>> On 16/02/12 7:50 PM, Andrus Adamchik wrote:
>>>>>> an OutOfMemory -- even with substantial quantities of memory
>>>>>> available.
>>>>
>>>> This is a hint actually.. Maybe the problem is not heap size, but
>>>> perm gen size instead? Try increasing that:
>>>>
>>>> -XX:MaxPermSize=256m
>>>>
>>>> On Feb 16, 2012, at 9:44 AM, Andrus Adamchik wrote:
>>>>
>>>>> This is strange indeed. Haven't seen such behavior. May need some
>>>>> profiling on your end I guess.
>>>>>
>>>>> Andrus
>>>>>
>>>>>
>>>>> On Feb 14, 2012, at 1:30 PM, Andrew Lindesay wrote:
>>>>>> Hello;
>>>>>>
>>>>>> I have a little question on the ObjectContext in 3.0.2.
>>>>>>
>>>>>> I'm keeping an ObjectContext in an HttpSession instance's
>>>>>> attributes for re-use between request-response cycles. When I
>>>>>> serialize this ObjectContext instance, it (+ other objects) stores
>>>>>> out to< 8k in my test case, but then deserializing the
>>>>>> ObjectContext causes an OutOfMemory -- even with substantial
>>>>>> quantities of memory available.
>>>>>>
>>>>>> Has anybody else seen this or is there anything to be weary of
>>>>>> before I look into it further?
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> --
>>>>>> Andrew Lindesay
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Andrew Lindesay
>>> www.silvereye.co.nz
>>>
>>
>>
>
>
--
Andrew Lindesay
www.silvereye.co.nz
Re: Question on ObjectContext and Deserialization
Posted by Andrew Lindesay <ap...@lindesay.co.nz>.
Hi Andrus;
It does seem rather strange.
I'm starting the application with maven (from Eclipse) and
jetty-maven-plugin. No hot redeploy. No actual container per se. The
class-loader should be the same so shouldn't be re-creating "isolated
singletons or statics". I'm using a Filter to serialize-->deserialize
the attributes on the HttpSession in order to test how some
infrastructure handles serialization-->deserialization of the session
attributes.
If I explicitly don't serialize-->deserialize the ObjectContext then
there is no problem; so serializing ObjectContext as the root object
does seem to cause the problem.
The incremental increase in the memory consumption seems to be roughly
exponential in relation to the number of deserializations of the
ObjectContext. That may be a clue; I may download Apache commons and
see if I can see anything obvious there in the LRU map. Like you say
though; it probably shouldn't be serializing this stuff anyway though.
It looks like "SQLTemplateResourceManager" is created via reflection so
it may be hard to profile for memory usage.
I'll log a Jira when I get a moment.
Regards;
On 17/02/12 11:29 PM, Andrus Adamchik wrote:
> Interesting... I didn't realize we have a static Velocity runtime in the app. This must to be changed to instance variable with correct scope, (and if needed shut down properly when ServerRuntime is shutdown). Could you please log a Jira - we need to investigate.
>
> BTW, are you doing hot redeploy of your app? This in combination with static Velocity runtime might produce the observed behavior.
>
> Andrus
>
>
> On Feb 17, 2012, at 1:19 PM, Andrew Lindesay wrote:
>
>> Hi;
>>
>> It looks like it is holding onto lots (hundreds of megs) of;
>>
>> org.apache.commons.collections.map.AbstractHashedMap$HashEntry[]
>> > org.apache.commons.collecitons.map.LRUMap
>> >> org.apache.cayenne.access.jdbc.SQLTemplateResourceManager
>> >>> ...velocity
>>
>> Not quite understanding yet... as it seems like this wouldn't be something stored on an object context.
>>
>> Regards;
>>
>> On 16/02/12 7:50 PM, Andrus Adamchik wrote:
>>>>> an OutOfMemory -- even with substantial quantities of memory available.
>>>
>>> This is a hint actually.. Maybe the problem is not heap size, but perm gen size instead? Try increasing that:
>>>
>>> -XX:MaxPermSize=256m
>>>
>>> On Feb 16, 2012, at 9:44 AM, Andrus Adamchik wrote:
>>>
>>>> This is strange indeed. Haven't seen such behavior. May need some profiling on your end I guess.
>>>>
>>>> Andrus
>>>>
>>>>
>>>> On Feb 14, 2012, at 1:30 PM, Andrew Lindesay wrote:
>>>>> Hello;
>>>>>
>>>>> I have a little question on the ObjectContext in 3.0.2.
>>>>>
>>>>> I'm keeping an ObjectContext in an HttpSession instance's attributes for re-use between request-response cycles. When I serialize this ObjectContext instance, it (+ other objects) stores out to< 8k in my test case, but then deserializing the ObjectContext causes an OutOfMemory -- even with substantial quantities of memory available.
>>>>>
>>>>> Has anybody else seen this or is there anything to be weary of before I look into it further?
>>>>>
>>>>> Regards
>>>>>
>>>>> --
>>>>> Andrew Lindesay
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>> --
>> Andrew Lindesay
>> www.silvereye.co.nz
>>
>
>
--
Andrew Lindesay
www.silvereye.co.nz
Re: Question on ObjectContext and Deserialization
Posted by Andrus Adamchik <an...@objectstyle.org>.
Interesting... I didn't realize we have a static Velocity runtime in the app. This must to be changed to instance variable with correct scope, (and if needed shut down properly when ServerRuntime is shutdown). Could you please log a Jira - we need to investigate.
BTW, are you doing hot redeploy of your app? This in combination with static Velocity runtime might produce the observed behavior.
Andrus
On Feb 17, 2012, at 1:19 PM, Andrew Lindesay wrote:
> Hi;
>
> It looks like it is holding onto lots (hundreds of megs) of;
>
> org.apache.commons.collections.map.AbstractHashedMap$HashEntry[]
> > org.apache.commons.collecitons.map.LRUMap
> >> org.apache.cayenne.access.jdbc.SQLTemplateResourceManager
> >>> ...velocity
>
> Not quite understanding yet... as it seems like this wouldn't be something stored on an object context.
>
> Regards;
>
> On 16/02/12 7:50 PM, Andrus Adamchik wrote:
>>>> an OutOfMemory -- even with substantial quantities of memory available.
>>
>> This is a hint actually.. Maybe the problem is not heap size, but perm gen size instead? Try increasing that:
>>
>> -XX:MaxPermSize=256m
>>
>> On Feb 16, 2012, at 9:44 AM, Andrus Adamchik wrote:
>>
>>> This is strange indeed. Haven't seen such behavior. May need some profiling on your end I guess.
>>>
>>> Andrus
>>>
>>>
>>> On Feb 14, 2012, at 1:30 PM, Andrew Lindesay wrote:
>>>> Hello;
>>>>
>>>> I have a little question on the ObjectContext in 3.0.2.
>>>>
>>>> I'm keeping an ObjectContext in an HttpSession instance's attributes for re-use between request-response cycles. When I serialize this ObjectContext instance, it (+ other objects) stores out to< 8k in my test case, but then deserializing the ObjectContext causes an OutOfMemory -- even with substantial quantities of memory available.
>>>>
>>>> Has anybody else seen this or is there anything to be weary of before I look into it further?
>>>>
>>>> Regards
>>>>
>>>> --
>>>> Andrew Lindesay
>>>>
>>>
>>>
>>
>>
>
>
> --
> Andrew Lindesay
> www.silvereye.co.nz
>
Re: Question on ObjectContext and Deserialization
Posted by Andrew Lindesay <ap...@lindesay.co.nz>.
Hi;
It looks like it is holding onto lots (hundreds of megs) of;
org.apache.commons.collections.map.AbstractHashedMap$HashEntry[]
> org.apache.commons.collecitons.map.LRUMap
>> org.apache.cayenne.access.jdbc.SQLTemplateResourceManager
>>> ...velocity
Not quite understanding yet... as it seems like this wouldn't be
something stored on an object context.
Regards;
On 16/02/12 7:50 PM, Andrus Adamchik wrote:
>>> an OutOfMemory -- even with substantial quantities of memory available.
>
> This is a hint actually.. Maybe the problem is not heap size, but perm gen size instead? Try increasing that:
>
> -XX:MaxPermSize=256m
>
> On Feb 16, 2012, at 9:44 AM, Andrus Adamchik wrote:
>
>> This is strange indeed. Haven't seen such behavior. May need some profiling on your end I guess.
>>
>> Andrus
>>
>>
>> On Feb 14, 2012, at 1:30 PM, Andrew Lindesay wrote:
>>> Hello;
>>>
>>> I have a little question on the ObjectContext in 3.0.2.
>>>
>>> I'm keeping an ObjectContext in an HttpSession instance's attributes for re-use between request-response cycles. When I serialize this ObjectContext instance, it (+ other objects) stores out to< 8k in my test case, but then deserializing the ObjectContext causes an OutOfMemory -- even with substantial quantities of memory available.
>>>
>>> Has anybody else seen this or is there anything to be weary of before I look into it further?
>>>
>>> Regards
>>>
>>> --
>>> Andrew Lindesay
>>>
>>
>>
>
>
--
Andrew Lindesay
www.silvereye.co.nz
Re: Question on ObjectContext and Deserialization
Posted by Andrew Lindesay <ap...@lindesay.co.nz>.
Hi Andrus;
Thanks for getting back.
I tried this, but it did not change anything. It is most unusual. When
I get some more time, I will see if I am able to attach jProfiler and
see what is actually going on!
Regards;
On 16/02/12 7:50 PM, Andrus Adamchik wrote:
>>> an OutOfMemory -- even with substantial quantities of memory available.
>
> This is a hint actually.. Maybe the problem is not heap size, but perm gen size instead? Try increasing that:
>
> -XX:MaxPermSize=256m
>
> On Feb 16, 2012, at 9:44 AM, Andrus Adamchik wrote:
>
>> This is strange indeed. Haven't seen such behavior. May need some profiling on your end I guess.
>>
>> Andrus
>>
>>
>> On Feb 14, 2012, at 1:30 PM, Andrew Lindesay wrote:
>>> Hello;
>>>
>>> I have a little question on the ObjectContext in 3.0.2.
>>>
>>> I'm keeping an ObjectContext in an HttpSession instance's attributes for re-use between request-response cycles. When I serialize this ObjectContext instance, it (+ other objects) stores out to< 8k in my test case, but then deserializing the ObjectContext causes an OutOfMemory -- even with substantial quantities of memory available.
>>>
>>> Has anybody else seen this or is there anything to be weary of before I look into it further?
>>>
>>> Regards
>>>
>>> --
>>> Andrew Lindesay
>>>
>>
>>
>
>
--
Andrew Lindesay
www.silvereye.co.nz
Re: Question on ObjectContext and Deserialization
Posted by Andrus Adamchik <an...@objectstyle.org>.
>> an OutOfMemory -- even with substantial quantities of memory available.
This is a hint actually.. Maybe the problem is not heap size, but perm gen size instead? Try increasing that:
-XX:MaxPermSize=256m
On Feb 16, 2012, at 9:44 AM, Andrus Adamchik wrote:
> This is strange indeed. Haven't seen such behavior. May need some profiling on your end I guess.
>
> Andrus
>
>
> On Feb 14, 2012, at 1:30 PM, Andrew Lindesay wrote:
>> Hello;
>>
>> I have a little question on the ObjectContext in 3.0.2.
>>
>> I'm keeping an ObjectContext in an HttpSession instance's attributes for re-use between request-response cycles. When I serialize this ObjectContext instance, it (+ other objects) stores out to < 8k in my test case, but then deserializing the ObjectContext causes an OutOfMemory -- even with substantial quantities of memory available.
>>
>> Has anybody else seen this or is there anything to be weary of before I look into it further?
>>
>> Regards
>>
>> --
>> Andrew Lindesay
>>
>
>
Re: Question on ObjectContext and Deserialization
Posted by Andrus Adamchik <an...@objectstyle.org>.
This is strange indeed. Haven't seen such behavior. May need some profiling on your end I guess.
Andrus
On Feb 14, 2012, at 1:30 PM, Andrew Lindesay wrote:
> Hello;
>
> I have a little question on the ObjectContext in 3.0.2.
>
> I'm keeping an ObjectContext in an HttpSession instance's attributes for re-use between request-response cycles. When I serialize this ObjectContext instance, it (+ other objects) stores out to < 8k in my test case, but then deserializing the ObjectContext causes an OutOfMemory -- even with substantial quantities of memory available.
>
> Has anybody else seen this or is there anything to be weary of before I look into it further?
>
> Regards
>
> --
> Andrew Lindesay
>