You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cayenne.apache.org by "Andrew Lindesay (Created) (JIRA)" <ji...@apache.org> on 2012/02/19 12:44:34 UTC
[jira] [Created] (CAY-1668) Memory Exhaustion Problem with
Deserialization of ObjectContext
Memory Exhaustion Problem with Deserialization of ObjectContext
---------------------------------------------------------------
Key: CAY-1668
URL: https://issues.apache.org/jira/browse/CAY-1668
Project: Cayenne
Issue Type: Bug
Components: Core Library
Affects Versions: 3.0.2
Environment: MacOS-X 10.7, Java 1.6.0_29, Simple Web Application run in the Maven plugin "jetty-maven-plugin"
Reporter: Andrew Lindesay
I am running a simple web application launched from maven using "jetty-maven-plugin". The simple web application has a Filter which does this;
1) Copies the HttpSession's attributes to a Map
2) Serializes the Map
3) Deserializes the Map
4) Copies the entries of the Map back into HttpSession
In each case, the ClassLoader should be the same (no stranded singletons etc...) and there is no "hot deploy" happening. The purpose of this undertaking is to ensure that the application is able to handle serialized sessions. In doing this experiment, I have observed a memory exhaustion issue around serializing and de-serializing ObjectContext. If I explicitly stop serializing "ObjectContext" then the problem stops.
Using jprofiler, I have observed that the memory consumption increases roughly exponentially in relation to the number of serialization/de-serialization events. Also from jquery analysis I see the memory is being referenced from;
org.apache.commons.collections.map.AbstractHashedMap$HashEntry[]
> org.apache.commons.collecitons.map.LRUMap
>> org.apache.cayenne.access.jdbc.SQLTemplateResourceManager
>>> ...velocity
The problem manifests itself quite quickly; a few dozen serialize + de-serialize phases.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAY-1668) Memory Exhaustion Problem with
Deserialization of ObjectContext
Posted by "Andrus Adamchik (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAY-1668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222515#comment-13222515 ]
Andrus Adamchik commented on CAY-1668:
--------------------------------------
This is good news. Then we can simply flip the dependency version in Cayenne POM
> Memory Exhaustion Problem with Deserialization of ObjectContext
> ---------------------------------------------------------------
>
> Key: CAY-1668
> URL: https://issues.apache.org/jira/browse/CAY-1668
> Project: Cayenne
> Issue Type: Bug
> Components: Core Library
> Affects Versions: 3.0.2
> Environment: MacOS-X 10.7, Java 1.6.0_29, Simple Web Application run in the Maven plugin "jetty-maven-plugin"
> Reporter: Andrew Lindesay
>
> I am running a simple web application launched from maven using "jetty-maven-plugin". The simple web application has a Filter which does this;
> 1) Copies the HttpSession's attributes to a Map
> 2) Serializes the Map
> 3) Deserializes the Map
> 4) Copies the entries of the Map back into HttpSession
> In each case, the ClassLoader should be the same (no stranded singletons etc...) and there is no "hot deploy" happening. The purpose of this undertaking is to ensure that the application is able to handle serialized sessions. In doing this experiment, I have observed a memory exhaustion issue around serializing and de-serializing ObjectContext. If I explicitly stop serializing "ObjectContext" then the problem stops.
> Using jprofiler, I have observed that the memory consumption increases roughly exponentially in relation to the number of serialization/de-serialization events. Also from jquery analysis I see the memory is being referenced from;
> org.apache.commons.collections.map.AbstractHashedMap$HashEntry[]
> > org.apache.commons.collecitons.map.LRUMap
> >> org.apache.cayenne.access.jdbc.SQLTemplateResourceManager
> >>> ...velocity
> The problem manifests itself quite quickly; a few dozen serialize + de-serialize phases.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (CAY-1668) Memory Exhaustion Problem with
Deserialization of ObjectContext
Posted by "Andrus Adamchik (Closed) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAY-1668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrus Adamchik closed CAY-1668.
--------------------------------
Resolution: Fixed
Fix Version/s: 3.1M4
Should be fixed per CAY-1684
Andrew, fill free to reopen if you keep encountering this leak.
> Memory Exhaustion Problem with Deserialization of ObjectContext
> ---------------------------------------------------------------
>
> Key: CAY-1668
> URL: https://issues.apache.org/jira/browse/CAY-1668
> Project: Cayenne
> Issue Type: Bug
> Components: Core Library
> Affects Versions: 3.0.2
> Environment: MacOS-X 10.7, Java 1.6.0_29, Simple Web Application run in the Maven plugin "jetty-maven-plugin"
> Reporter: Andrew Lindesay
> Fix For: 3.1M4
>
>
> I am running a simple web application launched from maven using "jetty-maven-plugin". The simple web application has a Filter which does this;
> 1) Copies the HttpSession's attributes to a Map
> 2) Serializes the Map
> 3) Deserializes the Map
> 4) Copies the entries of the Map back into HttpSession
> In each case, the ClassLoader should be the same (no stranded singletons etc...) and there is no "hot deploy" happening. The purpose of this undertaking is to ensure that the application is able to handle serialized sessions. In doing this experiment, I have observed a memory exhaustion issue around serializing and de-serializing ObjectContext. If I explicitly stop serializing "ObjectContext" then the problem stops.
> Using jprofiler, I have observed that the memory consumption increases roughly exponentially in relation to the number of serialization/de-serialization events. Also from jquery analysis I see the memory is being referenced from;
> org.apache.commons.collections.map.AbstractHashedMap$HashEntry[]
> > org.apache.commons.collecitons.map.LRUMap
> >> org.apache.cayenne.access.jdbc.SQLTemplateResourceManager
> >>> ...velocity
> The problem manifests itself quite quickly; a few dozen serialize + de-serialize phases.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CAY-1668) Memory Exhaustion Problem
with Deserialization of ObjectContext
Posted by "Andrew Lindesay (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAY-1668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222503#comment-13222503 ]
Andrew Lindesay edited comment on CAY-1668 at 3/5/12 6:59 PM:
--------------------------------------------------------------
For what it is worth; I have tested commons 3.1 and 3.2 with a small test rig and the problem appears to be gone in 3.2. By code introspection, I can't easily see how they resolved it though.
was (Author: apl):
For what it is worth; I have tested commons 3.1 and 3.2 with a small test rig and the problem appears to be gone in 3.2.
> Memory Exhaustion Problem with Deserialization of ObjectContext
> ---------------------------------------------------------------
>
> Key: CAY-1668
> URL: https://issues.apache.org/jira/browse/CAY-1668
> Project: Cayenne
> Issue Type: Bug
> Components: Core Library
> Affects Versions: 3.0.2
> Environment: MacOS-X 10.7, Java 1.6.0_29, Simple Web Application run in the Maven plugin "jetty-maven-plugin"
> Reporter: Andrew Lindesay
>
> I am running a simple web application launched from maven using "jetty-maven-plugin". The simple web application has a Filter which does this;
> 1) Copies the HttpSession's attributes to a Map
> 2) Serializes the Map
> 3) Deserializes the Map
> 4) Copies the entries of the Map back into HttpSession
> In each case, the ClassLoader should be the same (no stranded singletons etc...) and there is no "hot deploy" happening. The purpose of this undertaking is to ensure that the application is able to handle serialized sessions. In doing this experiment, I have observed a memory exhaustion issue around serializing and de-serializing ObjectContext. If I explicitly stop serializing "ObjectContext" then the problem stops.
> Using jprofiler, I have observed that the memory consumption increases roughly exponentially in relation to the number of serialization/de-serialization events. Also from jquery analysis I see the memory is being referenced from;
> org.apache.commons.collections.map.AbstractHashedMap$HashEntry[]
> > org.apache.commons.collecitons.map.LRUMap
> >> org.apache.cayenne.access.jdbc.SQLTemplateResourceManager
> >>> ...velocity
> The problem manifests itself quite quickly; a few dozen serialize + de-serialize phases.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAY-1668) Memory Exhaustion Problem with
Deserialization of ObjectContext
Posted by "Andrew Lindesay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAY-1668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222503#comment-13222503 ]
Andrew Lindesay commented on CAY-1668:
--------------------------------------
For what it is worth; I have tested commons 3.1 and 3.2 with a small test rig and the problem appears to be gone in 3.2.
> Memory Exhaustion Problem with Deserialization of ObjectContext
> ---------------------------------------------------------------
>
> Key: CAY-1668
> URL: https://issues.apache.org/jira/browse/CAY-1668
> Project: Cayenne
> Issue Type: Bug
> Components: Core Library
> Affects Versions: 3.0.2
> Environment: MacOS-X 10.7, Java 1.6.0_29, Simple Web Application run in the Maven plugin "jetty-maven-plugin"
> Reporter: Andrew Lindesay
>
> I am running a simple web application launched from maven using "jetty-maven-plugin". The simple web application has a Filter which does this;
> 1) Copies the HttpSession's attributes to a Map
> 2) Serializes the Map
> 3) Deserializes the Map
> 4) Copies the entries of the Map back into HttpSession
> In each case, the ClassLoader should be the same (no stranded singletons etc...) and there is no "hot deploy" happening. The purpose of this undertaking is to ensure that the application is able to handle serialized sessions. In doing this experiment, I have observed a memory exhaustion issue around serializing and de-serializing ObjectContext. If I explicitly stop serializing "ObjectContext" then the problem stops.
> Using jprofiler, I have observed that the memory consumption increases roughly exponentially in relation to the number of serialization/de-serialization events. Also from jquery analysis I see the memory is being referenced from;
> org.apache.commons.collections.map.AbstractHashedMap$HashEntry[]
> > org.apache.commons.collecitons.map.LRUMap
> >> org.apache.cayenne.access.jdbc.SQLTemplateResourceManager
> >>> ...velocity
> The problem manifests itself quite quickly; a few dozen serialize + de-serialize phases.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAY-1668) Memory Exhaustion Problem with
Deserialization of ObjectContext
Posted by "Andrus Adamchik (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAY-1668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222398#comment-13222398 ]
Andrus Adamchik commented on CAY-1668:
--------------------------------------
BTW, after researching that I'll probably follow up with commons project and open a Jira there... Ideally they should fix it themselves at some point...
> Memory Exhaustion Problem with Deserialization of ObjectContext
> ---------------------------------------------------------------
>
> Key: CAY-1668
> URL: https://issues.apache.org/jira/browse/CAY-1668
> Project: Cayenne
> Issue Type: Bug
> Components: Core Library
> Affects Versions: 3.0.2
> Environment: MacOS-X 10.7, Java 1.6.0_29, Simple Web Application run in the Maven plugin "jetty-maven-plugin"
> Reporter: Andrew Lindesay
>
> I am running a simple web application launched from maven using "jetty-maven-plugin". The simple web application has a Filter which does this;
> 1) Copies the HttpSession's attributes to a Map
> 2) Serializes the Map
> 3) Deserializes the Map
> 4) Copies the entries of the Map back into HttpSession
> In each case, the ClassLoader should be the same (no stranded singletons etc...) and there is no "hot deploy" happening. The purpose of this undertaking is to ensure that the application is able to handle serialized sessions. In doing this experiment, I have observed a memory exhaustion issue around serializing and de-serializing ObjectContext. If I explicitly stop serializing "ObjectContext" then the problem stops.
> Using jprofiler, I have observed that the memory consumption increases roughly exponentially in relation to the number of serialization/de-serialization events. Also from jquery analysis I see the memory is being referenced from;
> org.apache.commons.collections.map.AbstractHashedMap$HashEntry[]
> > org.apache.commons.collecitons.map.LRUMap
> >> org.apache.cayenne.access.jdbc.SQLTemplateResourceManager
> >>> ...velocity
> The problem manifests itself quite quickly; a few dozen serialize + de-serialize phases.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAY-1668) Memory Exhaustion Problem with
Deserialization of ObjectContext
Posted by "Andrew Lindesay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAY-1668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13213512#comment-13213512 ]
Andrew Lindesay commented on CAY-1668:
--------------------------------------
It does appear that the problem is in an Apache Commons abstract superclass. What is happening is that whilst running;
AbstractReferenceMap.doReadObject(..)
...the method...
AbstractReferenceMap.checkCapacity(..)
...is invoked and it seems to be doubling the size of the Map's entry array on de-serialization. Through sequential serialization + de-serialization this yields the exponential growth in the memory consumption.
It seems that the problem is caused by the 'threshold' value being calculated after to populating the Map during de-serialization.
I have re-built 3.0-STABLE with commons 3.2.1 and it still seems to exhibit the same problem.
The release cycles for commons seems to be quite long these days. Hmmm.....
> Memory Exhaustion Problem with Deserialization of ObjectContext
> ---------------------------------------------------------------
>
> Key: CAY-1668
> URL: https://issues.apache.org/jira/browse/CAY-1668
> Project: Cayenne
> Issue Type: Bug
> Components: Core Library
> Affects Versions: 3.0.2
> Environment: MacOS-X 10.7, Java 1.6.0_29, Simple Web Application run in the Maven plugin "jetty-maven-plugin"
> Reporter: Andrew Lindesay
>
> I am running a simple web application launched from maven using "jetty-maven-plugin". The simple web application has a Filter which does this;
> 1) Copies the HttpSession's attributes to a Map
> 2) Serializes the Map
> 3) Deserializes the Map
> 4) Copies the entries of the Map back into HttpSession
> In each case, the ClassLoader should be the same (no stranded singletons etc...) and there is no "hot deploy" happening. The purpose of this undertaking is to ensure that the application is able to handle serialized sessions. In doing this experiment, I have observed a memory exhaustion issue around serializing and de-serializing ObjectContext. If I explicitly stop serializing "ObjectContext" then the problem stops.
> Using jprofiler, I have observed that the memory consumption increases roughly exponentially in relation to the number of serialization/de-serialization events. Also from jquery analysis I see the memory is being referenced from;
> org.apache.commons.collections.map.AbstractHashedMap$HashEntry[]
> > org.apache.commons.collecitons.map.LRUMap
> >> org.apache.cayenne.access.jdbc.SQLTemplateResourceManager
> >>> ...velocity
> The problem manifests itself quite quickly; a few dozen serialize + de-serialize phases.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAY-1668) Memory Exhaustion Problem with
Deserialization of ObjectContext
Posted by "Andrus Adamchik (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAY-1668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13213536#comment-13213536 ]
Andrus Adamchik commented on CAY-1668:
--------------------------------------
I think we can improve deserialization performance in general and solve this specific issue if we make ObjectStore.objectMap (this is the AbstractReferenceMap causing trouble) transient. There's no value in saving and restoring this map. There's another map called "changes", which is a regular HashMap, and that needs to be preserved.
Now the implementation can be a little tricky, as "objectMap" is provided by the DI factory and we need to create an empty map of the same type on deserialization. This may require refactoring of ObjectStore creation flow. Should be doable though...
> Memory Exhaustion Problem with Deserialization of ObjectContext
> ---------------------------------------------------------------
>
> Key: CAY-1668
> URL: https://issues.apache.org/jira/browse/CAY-1668
> Project: Cayenne
> Issue Type: Bug
> Components: Core Library
> Affects Versions: 3.0.2
> Environment: MacOS-X 10.7, Java 1.6.0_29, Simple Web Application run in the Maven plugin "jetty-maven-plugin"
> Reporter: Andrew Lindesay
>
> I am running a simple web application launched from maven using "jetty-maven-plugin". The simple web application has a Filter which does this;
> 1) Copies the HttpSession's attributes to a Map
> 2) Serializes the Map
> 3) Deserializes the Map
> 4) Copies the entries of the Map back into HttpSession
> In each case, the ClassLoader should be the same (no stranded singletons etc...) and there is no "hot deploy" happening. The purpose of this undertaking is to ensure that the application is able to handle serialized sessions. In doing this experiment, I have observed a memory exhaustion issue around serializing and de-serializing ObjectContext. If I explicitly stop serializing "ObjectContext" then the problem stops.
> Using jprofiler, I have observed that the memory consumption increases roughly exponentially in relation to the number of serialization/de-serialization events. Also from jquery analysis I see the memory is being referenced from;
> org.apache.commons.collections.map.AbstractHashedMap$HashEntry[]
> > org.apache.commons.collecitons.map.LRUMap
> >> org.apache.cayenne.access.jdbc.SQLTemplateResourceManager
> >>> ...velocity
> The problem manifests itself quite quickly; a few dozen serialize + de-serialize phases.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira