You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@openjpa.apache.org by Alejandro Abdelnur <tu...@gmail.com> on 2013/05/09 01:34:28 UTC
OOM with JDBCConfigurationImpl holding 1+ GB of heap
We are seeing this using OpenJPA 2.1.0 in Oozie.
A JDBCConfigurationImpl seems to be holding a large amount of memory via
its dataCacheTimeout (then listeners) instance variable keeping
Class # Objects Used Heap Retained Heap
-------------------------------------------------------------------
o.a.o.j.meta.MappingRepository 1,902 304,320 904,410,184
o.a.o.j.meta.ClassMapping 3,824 1,223,680 105,073,624
-------------------------------------------------------------------
We are using a single EntityManagerFactory and we are closing all
EntityManager instances correctly (via a command pattern).
All caching settings are default ones.
Any hint where to look would be appreciated it.
Thanks
Alejandro
Re: OOM with JDBCConfigurationImpl holding 1+ GB of heap
Posted by Alejandro Abdelnur <tu...@gmail.com>.
Hi Kevin,
We are enhancing classes as built time (via maven plugin).
So far we've seen this memory leak only in a single setup using embedded
Derby.
A hprof dump is available at
http://people.apache.org/~tucu/tmp/openjpaleak.hprof.tar.gz
I'll remove the entity specifications via properties in our next release.
Thanks for helping with this.
Alejandro
On Mon, May 20, 2013 at 9:38 AM, Kevin Sutter <kw...@gmail.com> wrote:
> Hi Alejandro,
> It looks like something is holding onto references... How are you
> enhancing your classes? At build time? Or, some other mechanism?
>
> Does this memory leak happen with any of your configured databases? Or,
> just select ones?
>
> This shouldn't make a difference, but specifying your Entity classes via
> both <class> and the openjpa.MetaDataFactory property is overkill. The
> <class> elements are sufficient and is the standard, so I'd suggest
> sticking with that mechanism and removing the openjpa.MetaDataFactory
> property.
>
> We do considerable memory leak testing both as a standalone JPA provider as
> well as the JPA provider for WebSphere Application Server. We could have
> missed something due to your configuration requirements, but this looks to
> be a pretty major leak...
>
> Can you narrow down or simplify your scenario to the point of sharing a
> project that demonstrates the leak? Or, maybe there is memory data dumps
> that can be shared to help diagnose the issue?
>
> Kevin
>
>
> On Fri, May 17, 2013 at 4:50 PM, Alejandro Abdelnur <tucu00@gmail.com
> >wrote:
>
> > And .... I'm back.....
> >
> > It seems that removing runtime enhancement has some effect, but the heap
> > keeps building up in the same spot.
> >
> > The following link is show where is the memory sink
> >
> > http://people.apache.org/~tucu/tmp/OpenJPALeak.png
> >
> > I'd appreciate any ideas/suggestions.
> >
> > Thanks
> >
> > Alejandro
> >
> >
> >
> > On Fri, May 10, 2013 at 10:11 AM, Kevin Sutter <kw...@gmail.com>
> wrote:
> >
> > > It shoudn't... But, with that flag set as "supported", we can't be
> sure
> > > that the entities were actually enhanced at build time. By setting
> this
> > > property to "unsupported" or removing it altogether, then if an
> > unenhanced
> > > entity class is loaded, we'll log a message and quit processing. So,
> > let's
> > > go this route first and then we'll deal with any potential memory leak.
> > >
> > > Thanks, Kevin
> > >
> > >
> > > On Fri, May 10, 2013 at 12:05 PM, Alejandro Abdelnur <tucu00@gmail.com
> > > >wrote:
> > >
> > > > I'm in the process of verifying if Kevin's suggestion fixes the
> > problem.
> > > >
> > > > While making the necessary changes, I've notice we were already
> > enhancing
> > > > the classes at build time.
> > > >
> > > > If the classes were enhanced at build time, still would be we see the
> > > leak
> > > > if the runtime enhancement flag is enabled?
> > > >
> > > > Thanks
> > > >
> > > >
> > > > On Thu, May 9, 2013 at 9:15 AM, Alejandro Abdelnur <tucu00@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Thanks Kevin, will try that.
> > > > >
> > > > >
> > > > > On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter <kw...@gmail.com>
> > > wrote:
> > > > >
> > > > >> Don't use this:
> > > > >>
> > > > >> <property name="openjpa.RuntimeUnenhancedClasses"
> > value="supported"/>
> > > > >>
> > > > >> That is probably your issue. Reference this page for more
> > > information:
> > > > >> http://openjpa.apache.org/entity-enhancement.html
> > > > >>
> > > > >> Good luck, Kevin
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Thu, May 9, 2013 at 10:17 AM, Alejandro Abdelnur <
> > tucu00@gmail.com
> > > > >> >wrote:
> > > > >>
> > > > >> > Hi Rick,
> > > > >> >
> > > > >> > Thanks for looking into this, the persistence.xml file we are
> > using
> > > > is:
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml
> > > > >> >
> > > > >> > The EMF is instatiated here:
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156
> > > > >> >
> > > > >> >
> > > > >> > Thanks again.
> > > > >> >
> > > > >> > Alejandro
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Thu, May 9, 2013 at 6:06 AM, Rick Curtis <curtisr7@gmail.com
> >
> > > > wrote:
> > > > >> >
> > > > >> > > Can we see the contents of your persistence.xml file?
> > > > >> > >
> > > > >> > >
> > > > >> > > On Wed, May 8, 2013 at 6:34 PM, Alejandro Abdelnur <
> > > > tucu00@gmail.com>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > We are seeing this using OpenJPA 2.1.0 in Oozie.
> > > > >> > > >
> > > > >> > > > A JDBCConfigurationImpl seems to be holding a large amount
> of
> > > > memory
> > > > >> > via
> > > > >> > > > its dataCacheTimeout (then listeners) instance variable
> > keeping
> > > > >> > > >
> > > > >> > > > Class # Objects Used Heap
> Retained
> > > > Heap
> > > > >> > > >
> > > > -------------------------------------------------------------------
> > > > >> > > >
> > > > >> > > > o.a.o.j.meta.MappingRepository 1,902 304,320
> > > 904,410,184
> > > > >> > > > o.a.o.j.meta.ClassMapping 3,824 1,223,680
> > > 105,073,624
> > > > >> > > >
> > > > -------------------------------------------------------------------
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > We are using a single EntityManagerFactory and we are
> closing
> > > all
> > > > >> > > > EntityManager instances correctly (via a command pattern).
> > > > >> > > >
> > > > >> > > > All caching settings are default ones.
> > > > >> > > >
> > > > >> > > > Any hint where to look would be appreciated it.
> > > > >> > > >
> > > > >> > > > Thanks
> > > > >> > > >
> > > > >> > > > Alejandro
> > > > >> > > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > --
> > > > >> > > *Rick Curtis*
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: OOM with JDBCConfigurationImpl holding 1+ GB of heap
Posted by Alejandro Abdelnur <tu...@gmail.com>.
rick, thanks for following up, i've already scanned oozie source and we create a single EMF and use it for the lifetime of the process, so i'm discarding this as the root cause at the moment. the intereesting thing is that this has been running without any issues for 2+ years in heavy loaded prod systems. and still is running without any issues in all but a couple of system, i'm trying to identify any app usage pattern that could gove me a clue on something diff we ate doing with openjpa conns in those setups.
thx
Alejandro
(phone typing)
On Jun 10, 2013, at 7:26, Rick Curtis <cu...@gmail.com> wrote:
> No reason why I suggested 2.2.0 as opposed to 2.2.2. Either should work
> just fine.
>
> I'd be more concerned that something is happening in your application that
> you don't fully understand. Even if you move to 2.2.x, you should really
> spend some time to figure out why these EMFs aren't being closed. EMF
> creation is fairly expensive and if your application is performance
> sensitive, you might want to figure out what is going on.
>
> I am glad we are able to point in the direction of the leak though. Good
> luck!
>
> Thanks,
> Rick
>
>
> On Fri, Jun 7, 2013 at 6:37 PM, Alejandro Abdelnur <tu...@gmail.com> wrote:
>
>> Rick,
>>
>> Following up on this, it seems that using 2.2.0 is fixing the OOM issues.
>>
>> BTW, any reason you sugested 2.2.0 instead 2.2.2?
>>
>> Thanks.
>>
>>
>> On Tue, May 28, 2013 at 9:33 AM, Rick Curtis <cu...@gmail.com> wrote:
>>
>>> You are reading too far into my analysis. The leak is NOT
>>> EntityManagerFactoryImpl instances, it is a data structure
>>> (MappingRepository) that gets created as a result of creating an EMF. The
>>> MappingRepository is statically referenced elsewhere in our code and if
>> you
>>> fail to close all EntityManagerFactoryImpl instances, the
>> MappingRepository
>>> instances don't get cleaned up.
>>>
>>> Thanks,
>>> Rick
>>>
>>>
>>> On Tue, May 28, 2013 at 11:24 AM, Alejandro Abdelnur <tucu00@gmail.com
>>>> wrote:
>>>
>>>> Rick,
>>>>
>>>> Thanks for following up.
>>>>
>>>> After you email I've looked again at the hprof dump and the number
>>>> of org.apache.openjpa.persistence.EntityManagerFactoryImpl isntances is
>>> 1.
>>>> I've also noticed the heap has
>>>> 500+ org.apache.openjpa.persistence.EntityManagerImpl instances, but
>> they
>>>> are referenced from a finalizer; thus in queue to be GC. So, it would
>>>> argue, the leak does not come from there.
>>>>
>>>> I'll give it a ry using 2.2.0 to see if that makes a difference.
>>>>
>>>> Thanks again.
>>>>
>>>> Alejandro
>>>>
>>>>
>>>>
>>>> On Fri, May 24, 2013 at 12:05 PM, Rick Curtis <cu...@gmail.com>
>>> wrote:
>>>>
>>>>> After digging into the heapdump, I'm fairly certain that somewhere in
>>>> your
>>>>> application EntityManagerFactories are being created, yet they are
>>> never
>>>>> closed. I know you pointed me at your JPA service source code, but
>>> there
>>>>> must be something else going on. For giggles you could move to
>> version
>>>>> 2.2.0, where we made some changes[1] to better tolerate applications
>>> that
>>>>> fail to close EMFs. The JIRA talks about additional diagnostics, but
>> we
>>>>> just changed a bit of our code to use WeakReferences in hopes of
>>> avoiding
>>>>> these sorts of OOMs. If moving to 2.2.0 makes your problem go away,
>>>> you'll
>>>>> need to look elsewhere in your code for EMF creation. Hope this
>> helps.
>>>>>
>>>>> Thanks,
>>>>> Rick
>>>>>
>>>>>
>>>>> [1] https://issues.apache.org/jira/browse/OPENJPA-2042
>>>>>
>>>>>
>>>>> On Fri, May 24, 2013 at 10:49 AM, Alejandro Abdelnur <
>> tucu00@gmail.com
>>>>>> wrote:
>>>>>
>>>>>> Hi Rick, a hprof dump is available at
>>>>>> http://people.apache.org/~tucu/tmp/openjpaleak.hprof.tar.gz
>>>>>>
>>>>>> Thank you very much for looking into this.
>>>>>>
>>>>>> Alejandro
>>>>>>
>>>>>>
>>>>>> On Fri, May 24, 2013 at 7:46 AM, Rick Curtis <cu...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>>> Can you post a heapdump somewhere for review?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Rick
>>>>>>>
>>>>>>>
>>>>>>> On Thu, May 23, 2013 at 6:53 PM, Alejandro Abdelnur <
>>>> tucu00@gmail.com
>>>>>>>> wrote:
>>>>>>>
>>>>>>>> Rick,
>>>>>>>>
>>>>>>>> We create a single EMF:
>> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L160
>>>>>>>>
>>>>>>>> And we use create/use/dispose all EMs using a command patter:
>> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L209
>>>>>>>>
>>>>>>>> Any other idea/hint?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, May 22, 2013 at 11:32 AM, Rick Curtis <
>>> curtisr7@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Alejandro -
>>>>>>>>>
>>>>>>>>> It appears that your application is creating
>>>>> EntityManagerFactories,
>>>>>>> yet
>>>>>>>>> they are never getting closed.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Rick
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, May 22, 2013 at 12:40 PM, Alejandro Abdelnur <
>>>>>> tucu00@gmail.com
>>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Rick, by looking at the other thread memory analyzer image,
>>> the
>>>>>>> issues
>>>>>>>>> seem
>>>>>>>>>> to be different.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, May 20, 2013 at 9:47 AM, Rick Curtis <
>>>> curtisr7@gmail.com
>>>>>>
>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Alejandro --
>>>>>>>>>>>
>>>>>>>>>>> Is the problem that you're asking about the same as
>> Hovan?
>>> If
>>>>> so,
>>>>>>>> lets
>>>>>>>>>>> consolidate this conversation down to a single thread.
>>> http://openjpa.208410.n2.nabble.com/OOM-caused-by-OpenJPA-td7583869.html
>> http://openjpa.208410.n2.nabble.com/OOM-with-JDBCConfigurationImpl-holding-1-GB-of-heap-td7583862.html
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Rick
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, May 20, 2013 at 11:38 AM, Kevin Sutter <
>>>>>> kwsutter@gmail.com
>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Alejandro,
>>>>>>>>>>>> It looks like something is holding onto references...
>>> How
>>>>> are
>>>>>>> you
>>>>>>>>>>>> enhancing your classes? At build time? Or, some other
>>>>>>> mechanism?
>>>>>>>>>>>>
>>>>>>>>>>>> Does this memory leak happen with any of your
>> configured
>>>>>>> databases?
>>>>>>>>>> Or,
>>>>>>>>>>>> just select ones?
>>>>>>>>>>>>
>>>>>>>>>>>> This shouldn't make a difference, but specifying your
>>>> Entity
>>>>>>>> classes
>>>>>>>>>> via
>>>>>>>>>>>> both <class> and the openjpa.MetaDataFactory property
>> is
>>>>>>> overkill.
>>>>>>>>> The
>>>>>>>>>>>> <class> elements are sufficient and is the standard, so
>>> I'd
>>>>>>> suggest
>>>>>>>>>>>> sticking with that mechanism and removing the
>>>>>>>> openjpa.MetaDataFactory
>>>>>>>>>>>> property.
>>>>>>>>>>>>
>>>>>>>>>>>> We do considerable memory leak testing both as a
>>> standalone
>>>>> JPA
>>>>>>>>>> provider
>>>>>>>>>>> as
>>>>>>>>>>>> well as the JPA provider for WebSphere Application
>>> Server.
>>>>> We
>>>>>>>> could
>>>>>>>>>> have
>>>>>>>>>>>> missed something due to your configuration
>> requirements,
>>>> but
>>>>>> this
>>>>>>>>> looks
>>>>>>>>>>> to
>>>>>>>>>>>> be a pretty major leak...
>>>>>>>>>>>>
>>>>>>>>>>>> Can you narrow down or simplify your scenario to the
>>> point
>>>> of
>>>>>>>>> sharing a
>>>>>>>>>>>> project that demonstrates the leak? Or, maybe there is
>>>>> memory
>>>>>>> data
>>>>>>>>>> dumps
>>>>>>>>>>>> that can be shared to help diagnose the issue?
>>>>>>>>>>>>
>>>>>>>>>>>> Kevin
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, May 17, 2013 at 4:50 PM, Alejandro Abdelnur <
>>>>>>>>> tucu00@gmail.com
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> And .... I'm back.....
>>>>>>>>>>>>>
>>>>>>>>>>>>> It seems that removing runtime enhancement has some
>>>> effect,
>>>>>> but
>>>>>>>> the
>>>>>>>>>>> heap
>>>>>>>>>>>>> keeps building up in the same spot.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The following link is show where is the memory sink
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://people.apache.org/~tucu/tmp/OpenJPALeak.png
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'd appreciate any ideas/suggestions.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>> Alejandro
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, May 10, 2013 at 10:11 AM, Kevin Sutter <
>>>>>>>> kwsutter@gmail.com
>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> It shoudn't... But, with that flag set as
>>> "supported",
>>>>> we
>>>>>>>> can't
>>>>>>>>> be
>>>>>>>>>>>> sure
>>>>>>>>>>>>>> that the entities were actually enhanced at build
>>> time.
>>>>> By
>>>>>>>>> setting
>>>>>>>>>>>> this
>>>>>>>>>>>>>> property to "unsupported" or removing it
>> altogether,
>>>> then
>>>>>> if
>>>>>>> an
>>>>>>>>>>>>> unenhanced
>>>>>>>>>>>>>> entity class is loaded, we'll log a message and
>> quit
>>>>>>>> processing.
>>>>>>>>>> So,
>>>>>>>>>>>>> let's
>>>>>>>>>>>>>> go this route first and then we'll deal with any
>>>>> potential
>>>>>>>> memory
>>>>>>>>>>> leak.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks, Kevin
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, May 10, 2013 at 12:05 PM, Alejandro
>> Abdelnur
>>> <
>>>>>>>>>>> tucu00@gmail.com
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm in the process of verifying if Kevin's
>>> suggestion
>>>>>> fixes
>>>>>>>> the
>>>>>>>>>>>>> problem.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> While making the necessary changes, I've notice
>> we
>>>> were
>>>>>>>> already
>>>>>>>>>>>>> enhancing
>>>>>>>>>>>>>>> the classes at build time.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If the classes were enhanced at build time, still
>>>> would
>>>>>> be
>>>>>>> we
>>>>>>>>> see
>>>>>>>>>>> the
>>>>>>>>>>>>>> leak
>>>>>>>>>>>>>>> if the runtime enhancement flag is enabled?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, May 9, 2013 at 9:15 AM, Alejandro
>> Abdelnur
>>> <
>>>>>>>>>>> tucu00@gmail.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks Kevin, will try that.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter <
>>>>>>>>>> kwsutter@gmail.com
>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Don't use this:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> <property
>>> name="openjpa.RuntimeUnenhancedClasses"
>>>>>>>>>>>>> value="supported"/>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> That is probably your issue. Reference this
>>> page
>>>>> for
>>>>>>> more
>>>>>>>>>>>>>> information:
>>> http://openjpa.apache.org/entity-enhancement.html
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Good luck, Kevin
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, May 9, 2013 at 10:17 AM, Alejandro
>>>> Abdelnur
>>>>> <
>>>>>>>>>>>>> tucu00@gmail.com
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Rick,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for looking into this, the
>>>> persistence.xml
>>>>>> file
>>>>>>>> we
>>>>>>>>>> are
>>>>>>>>>>>>> using
>>>>>>>>>>>>>>> is:
>> https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The EMF is instatiated here:
>> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks again.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Alejandro
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, May 9, 2013 at 6:06 AM, Rick Curtis
>> <
>>>>>>>>>>> curtisr7@gmail.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Can we see the contents of your
>>>> persistence.xml
>>>>>>> file?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Wed, May 8, 2013 at 6:34 PM, Alejandro
>>>>>> Abdelnur <
>>>>>>>>>>>>>>> tucu00@gmail.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> We are seeing this using OpenJPA 2.1.0
>> in
>>>>> Oozie.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> A JDBCConfigurationImpl seems to be
>>> holding
>>>> a
>>>>>>> large
>>>>>>>>>> amount
>>>>>>>>>>>> of
>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>>> via
>>>>>>>>>>>>>>>>>>>> its dataCacheTimeout (then listeners)
>>>> instance
>>>>>>>>> variable
>>>>>>>>>>>>> keeping
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Class # Objects
>>>> Used
>>>>>>> Heap
>>>>>>>>>>>> Retained
>>>>>>>>>>>>>>> Heap
>>>>> -------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> o.a.o.j.meta.MappingRepository 1,902
>>>>>> 304,320
>>>>>>>>>>>>>> 904,410,184
>>>>>>>>>>>>>>>>>>>> o.a.o.j.meta.ClassMapping 3,824
>>>>>>> 1,223,680
>>>>>>>>>>>>>> 105,073,624
>>>>> -------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> We are using a single
>> EntityManagerFactory
>>>> and
>>>>>> we
>>>>>>>> are
>>>>>>>>>>>> closing
>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>> EntityManager instances correctly (via a
>>>>> command
>>>>>>>>>> pattern).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> All caching settings are default ones.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Any hint where to look would be
>>> appreciated
>>>>> it.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Alejandro
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> *Rick Curtis*
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> *Rick Curtis*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Rick Curtis*
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Rick Curtis*
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Rick Curtis*
>>>
>>>
>>>
>>> --
>>> *Rick Curtis*
>
>
>
> --
> *Rick Curtis*
Re: OOM with JDBCConfigurationImpl holding 1+ GB of heap
Posted by Rick Curtis <cu...@gmail.com>.
No reason why I suggested 2.2.0 as opposed to 2.2.2. Either should work
just fine.
I'd be more concerned that something is happening in your application that
you don't fully understand. Even if you move to 2.2.x, you should really
spend some time to figure out why these EMFs aren't being closed. EMF
creation is fairly expensive and if your application is performance
sensitive, you might want to figure out what is going on.
I am glad we are able to point in the direction of the leak though. Good
luck!
Thanks,
Rick
On Fri, Jun 7, 2013 at 6:37 PM, Alejandro Abdelnur <tu...@gmail.com> wrote:
> Rick,
>
> Following up on this, it seems that using 2.2.0 is fixing the OOM issues.
>
> BTW, any reason you sugested 2.2.0 instead 2.2.2?
>
> Thanks.
>
>
> On Tue, May 28, 2013 at 9:33 AM, Rick Curtis <cu...@gmail.com> wrote:
>
> > You are reading too far into my analysis. The leak is NOT
> > EntityManagerFactoryImpl instances, it is a data structure
> > (MappingRepository) that gets created as a result of creating an EMF. The
> > MappingRepository is statically referenced elsewhere in our code and if
> you
> > fail to close all EntityManagerFactoryImpl instances, the
> MappingRepository
> > instances don't get cleaned up.
> >
> > Thanks,
> > Rick
> >
> >
> > On Tue, May 28, 2013 at 11:24 AM, Alejandro Abdelnur <tucu00@gmail.com
> > >wrote:
> >
> > > Rick,
> > >
> > > Thanks for following up.
> > >
> > > After you email I've looked again at the hprof dump and the number
> > > of org.apache.openjpa.persistence.EntityManagerFactoryImpl isntances is
> > 1.
> > > I've also noticed the heap has
> > > 500+ org.apache.openjpa.persistence.EntityManagerImpl instances, but
> they
> > > are referenced from a finalizer; thus in queue to be GC. So, it would
> > > argue, the leak does not come from there.
> > >
> > > I'll give it a ry using 2.2.0 to see if that makes a difference.
> > >
> > > Thanks again.
> > >
> > > Alejandro
> > >
> > >
> > >
> > > On Fri, May 24, 2013 at 12:05 PM, Rick Curtis <cu...@gmail.com>
> > wrote:
> > >
> > > > After digging into the heapdump, I'm fairly certain that somewhere in
> > > your
> > > > application EntityManagerFactories are being created, yet they are
> > never
> > > > closed. I know you pointed me at your JPA service source code, but
> > there
> > > > must be something else going on. For giggles you could move to
> version
> > > > 2.2.0, where we made some changes[1] to better tolerate applications
> > that
> > > > fail to close EMFs. The JIRA talks about additional diagnostics, but
> we
> > > > just changed a bit of our code to use WeakReferences in hopes of
> > avoiding
> > > > these sorts of OOMs. If moving to 2.2.0 makes your problem go away,
> > > you'll
> > > > need to look elsewhere in your code for EMF creation. Hope this
> helps.
> > > >
> > > > Thanks,
> > > > Rick
> > > >
> > > >
> > > > [1] https://issues.apache.org/jira/browse/OPENJPA-2042
> > > >
> > > >
> > > > On Fri, May 24, 2013 at 10:49 AM, Alejandro Abdelnur <
> tucu00@gmail.com
> > > > >wrote:
> > > >
> > > > > Hi Rick, a hprof dump is available at
> > > > > http://people.apache.org/~tucu/tmp/openjpaleak.hprof.tar.gz
> > > > >
> > > > > Thank you very much for looking into this.
> > > > >
> > > > > Alejandro
> > > > >
> > > > >
> > > > > On Fri, May 24, 2013 at 7:46 AM, Rick Curtis <cu...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Can you post a heapdump somewhere for review?
> > > > > >
> > > > > > Thanks,
> > > > > > Rick
> > > > > >
> > > > > >
> > > > > > On Thu, May 23, 2013 at 6:53 PM, Alejandro Abdelnur <
> > > tucu00@gmail.com
> > > > > > >wrote:
> > > > > >
> > > > > > > Rick,
> > > > > > >
> > > > > > > We create a single EMF:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L160
> > > > > > >
> > > > > > > And we use create/use/dispose all EMs using a command patter:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L209
> > > > > > >
> > > > > > > Any other idea/hint?
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > >
> > > > > > > On Wed, May 22, 2013 at 11:32 AM, Rick Curtis <
> > curtisr7@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Alejandro -
> > > > > > > >
> > > > > > > > It appears that your application is creating
> > > > EntityManagerFactories,
> > > > > > yet
> > > > > > > > they are never getting closed.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Rick
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, May 22, 2013 at 12:40 PM, Alejandro Abdelnur <
> > > > > tucu00@gmail.com
> > > > > > > > >wrote:
> > > > > > > >
> > > > > > > > > Rick, by looking at the other thread memory analyzer image,
> > the
> > > > > > issues
> > > > > > > > seem
> > > > > > > > > to be different.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, May 20, 2013 at 9:47 AM, Rick Curtis <
> > > curtisr7@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Alejandro --
> > > > > > > > > >
> > > > > > > > > > Is the problem that you're asking about the same as
> Hovan?
> > If
> > > > so,
> > > > > > > lets
> > > > > > > > > > consolidate this conversation down to a single thread.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > http://openjpa.208410.n2.nabble.com/OOM-caused-by-OpenJPA-td7583869.html
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://openjpa.208410.n2.nabble.com/OOM-with-JDBCConfigurationImpl-holding-1-GB-of-heap-td7583862.html
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Rick
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, May 20, 2013 at 11:38 AM, Kevin Sutter <
> > > > > kwsutter@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Alejandro,
> > > > > > > > > > > It looks like something is holding onto references...
> > How
> > > > are
> > > > > > you
> > > > > > > > > > > enhancing your classes? At build time? Or, some other
> > > > > > mechanism?
> > > > > > > > > > >
> > > > > > > > > > > Does this memory leak happen with any of your
> configured
> > > > > > databases?
> > > > > > > > > Or,
> > > > > > > > > > > just select ones?
> > > > > > > > > > >
> > > > > > > > > > > This shouldn't make a difference, but specifying your
> > > Entity
> > > > > > > classes
> > > > > > > > > via
> > > > > > > > > > > both <class> and the openjpa.MetaDataFactory property
> is
> > > > > > overkill.
> > > > > > > > The
> > > > > > > > > > > <class> elements are sufficient and is the standard, so
> > I'd
> > > > > > suggest
> > > > > > > > > > > sticking with that mechanism and removing the
> > > > > > > openjpa.MetaDataFactory
> > > > > > > > > > > property.
> > > > > > > > > > >
> > > > > > > > > > > We do considerable memory leak testing both as a
> > standalone
> > > > JPA
> > > > > > > > > provider
> > > > > > > > > > as
> > > > > > > > > > > well as the JPA provider for WebSphere Application
> > Server.
> > > > We
> > > > > > > could
> > > > > > > > > have
> > > > > > > > > > > missed something due to your configuration
> requirements,
> > > but
> > > > > this
> > > > > > > > looks
> > > > > > > > > > to
> > > > > > > > > > > be a pretty major leak...
> > > > > > > > > > >
> > > > > > > > > > > Can you narrow down or simplify your scenario to the
> > point
> > > of
> > > > > > > > sharing a
> > > > > > > > > > > project that demonstrates the leak? Or, maybe there is
> > > > memory
> > > > > > data
> > > > > > > > > dumps
> > > > > > > > > > > that can be shared to help diagnose the issue?
> > > > > > > > > > >
> > > > > > > > > > > Kevin
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, May 17, 2013 at 4:50 PM, Alejandro Abdelnur <
> > > > > > > > tucu00@gmail.com
> > > > > > > > > > > >wrote:
> > > > > > > > > > >
> > > > > > > > > > > > And .... I'm back.....
> > > > > > > > > > > >
> > > > > > > > > > > > It seems that removing runtime enhancement has some
> > > effect,
> > > > > but
> > > > > > > the
> > > > > > > > > > heap
> > > > > > > > > > > > keeps building up in the same spot.
> > > > > > > > > > > >
> > > > > > > > > > > > The following link is show where is the memory sink
> > > > > > > > > > > >
> > > > > > > > > > > > http://people.apache.org/~tucu/tmp/OpenJPALeak.png
> > > > > > > > > > > >
> > > > > > > > > > > > I'd appreciate any ideas/suggestions.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > > > > Alejandro
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, May 10, 2013 at 10:11 AM, Kevin Sutter <
> > > > > > > kwsutter@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > It shoudn't... But, with that flag set as
> > "supported",
> > > > we
> > > > > > > can't
> > > > > > > > be
> > > > > > > > > > > sure
> > > > > > > > > > > > > that the entities were actually enhanced at build
> > time.
> > > > By
> > > > > > > > setting
> > > > > > > > > > > this
> > > > > > > > > > > > > property to "unsupported" or removing it
> altogether,
> > > then
> > > > > if
> > > > > > an
> > > > > > > > > > > > unenhanced
> > > > > > > > > > > > > entity class is loaded, we'll log a message and
> quit
> > > > > > > processing.
> > > > > > > > > So,
> > > > > > > > > > > > let's
> > > > > > > > > > > > > go this route first and then we'll deal with any
> > > > potential
> > > > > > > memory
> > > > > > > > > > leak.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks, Kevin
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, May 10, 2013 at 12:05 PM, Alejandro
> Abdelnur
> > <
> > > > > > > > > > tucu00@gmail.com
> > > > > > > > > > > > > >wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I'm in the process of verifying if Kevin's
> > suggestion
> > > > > fixes
> > > > > > > the
> > > > > > > > > > > > problem.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > While making the necessary changes, I've notice
> we
> > > were
> > > > > > > already
> > > > > > > > > > > > enhancing
> > > > > > > > > > > > > > the classes at build time.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > If the classes were enhanced at build time, still
> > > would
> > > > > be
> > > > > > we
> > > > > > > > see
> > > > > > > > > > the
> > > > > > > > > > > > > leak
> > > > > > > > > > > > > > if the runtime enhancement flag is enabled?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, May 9, 2013 at 9:15 AM, Alejandro
> Abdelnur
> > <
> > > > > > > > > > tucu00@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks Kevin, will try that.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter <
> > > > > > > > > kwsutter@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> Don't use this:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> <property
> > name="openjpa.RuntimeUnenhancedClasses"
> > > > > > > > > > > > value="supported"/>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> That is probably your issue. Reference this
> > page
> > > > for
> > > > > > more
> > > > > > > > > > > > > information:
> > > > > > > > > > > > > > >>
> > http://openjpa.apache.org/entity-enhancement.html
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Good luck, Kevin
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> On Thu, May 9, 2013 at 10:17 AM, Alejandro
> > > Abdelnur
> > > > <
> > > > > > > > > > > > tucu00@gmail.com
> > > > > > > > > > > > > > >> >wrote:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> > Hi Rick,
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > Thanks for looking into this, the
> > > persistence.xml
> > > > > file
> > > > > > > we
> > > > > > > > > are
> > > > > > > > > > > > using
> > > > > > > > > > > > > > is:
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > The EMF is instatiated here:
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > Thanks again.
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > Alejandro
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > On Thu, May 9, 2013 at 6:06 AM, Rick Curtis
> <
> > > > > > > > > > curtisr7@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > > Can we see the contents of your
> > > persistence.xml
> > > > > > file?
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > > On Wed, May 8, 2013 at 6:34 PM, Alejandro
> > > > > Abdelnur <
> > > > > > > > > > > > > > tucu00@gmail.com>
> > > > > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > > > We are seeing this using OpenJPA 2.1.0
> in
> > > > Oozie.
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > > > A JDBCConfigurationImpl seems to be
> > holding
> > > a
> > > > > > large
> > > > > > > > > amount
> > > > > > > > > > > of
> > > > > > > > > > > > > > memory
> > > > > > > > > > > > > > >> > via
> > > > > > > > > > > > > > >> > > > its dataCacheTimeout (then listeners)
> > > instance
> > > > > > > > variable
> > > > > > > > > > > > keeping
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > > > Class # Objects
> > > Used
> > > > > > Heap
> > > > > > > > > > > Retained
> > > > > > > > > > > > > > Heap
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >
> > > > > > > > >
> > > > -------------------------------------------------------------------
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > > > o.a.o.j.meta.MappingRepository 1,902
> > > > > 304,320
> > > > > > > > > > > > > 904,410,184
> > > > > > > > > > > > > > >> > > > o.a.o.j.meta.ClassMapping 3,824
> > > > > > 1,223,680
> > > > > > > > > > > > > 105,073,624
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >
> > > > > > > > >
> > > > -------------------------------------------------------------------
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > > > We are using a single
> EntityManagerFactory
> > > and
> > > > > we
> > > > > > > are
> > > > > > > > > > > closing
> > > > > > > > > > > > > all
> > > > > > > > > > > > > > >> > > > EntityManager instances correctly (via a
> > > > command
> > > > > > > > > pattern).
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > > > All caching settings are default ones.
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > > > Any hint where to look would be
> > appreciated
> > > > it.
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > > > Thanks
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > > > Alejandro
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > > --
> > > > > > > > > > > > > > >> > > *Rick Curtis*
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > *Rick Curtis*
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > *Rick Curtis*
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Rick Curtis*
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Rick Curtis*
> > > >
> > >
> >
> >
> >
> > --
> > *Rick Curtis*
> >
>
--
*Rick Curtis*
Re: OOM with JDBCConfigurationImpl holding 1+ GB of heap
Posted by Alejandro Abdelnur <tu...@gmail.com>.
Rick,
Following up on this, it seems that using 2.2.0 is fixing the OOM issues.
BTW, any reason you sugested 2.2.0 instead 2.2.2?
Thanks.
On Tue, May 28, 2013 at 9:33 AM, Rick Curtis <cu...@gmail.com> wrote:
> You are reading too far into my analysis. The leak is NOT
> EntityManagerFactoryImpl instances, it is a data structure
> (MappingRepository) that gets created as a result of creating an EMF. The
> MappingRepository is statically referenced elsewhere in our code and if you
> fail to close all EntityManagerFactoryImpl instances, the MappingRepository
> instances don't get cleaned up.
>
> Thanks,
> Rick
>
>
> On Tue, May 28, 2013 at 11:24 AM, Alejandro Abdelnur <tucu00@gmail.com
> >wrote:
>
> > Rick,
> >
> > Thanks for following up.
> >
> > After you email I've looked again at the hprof dump and the number
> > of org.apache.openjpa.persistence.EntityManagerFactoryImpl isntances is
> 1.
> > I've also noticed the heap has
> > 500+ org.apache.openjpa.persistence.EntityManagerImpl instances, but they
> > are referenced from a finalizer; thus in queue to be GC. So, it would
> > argue, the leak does not come from there.
> >
> > I'll give it a ry using 2.2.0 to see if that makes a difference.
> >
> > Thanks again.
> >
> > Alejandro
> >
> >
> >
> > On Fri, May 24, 2013 at 12:05 PM, Rick Curtis <cu...@gmail.com>
> wrote:
> >
> > > After digging into the heapdump, I'm fairly certain that somewhere in
> > your
> > > application EntityManagerFactories are being created, yet they are
> never
> > > closed. I know you pointed me at your JPA service source code, but
> there
> > > must be something else going on. For giggles you could move to version
> > > 2.2.0, where we made some changes[1] to better tolerate applications
> that
> > > fail to close EMFs. The JIRA talks about additional diagnostics, but we
> > > just changed a bit of our code to use WeakReferences in hopes of
> avoiding
> > > these sorts of OOMs. If moving to 2.2.0 makes your problem go away,
> > you'll
> > > need to look elsewhere in your code for EMF creation. Hope this helps.
> > >
> > > Thanks,
> > > Rick
> > >
> > >
> > > [1] https://issues.apache.org/jira/browse/OPENJPA-2042
> > >
> > >
> > > On Fri, May 24, 2013 at 10:49 AM, Alejandro Abdelnur <tucu00@gmail.com
> > > >wrote:
> > >
> > > > Hi Rick, a hprof dump is available at
> > > > http://people.apache.org/~tucu/tmp/openjpaleak.hprof.tar.gz
> > > >
> > > > Thank you very much for looking into this.
> > > >
> > > > Alejandro
> > > >
> > > >
> > > > On Fri, May 24, 2013 at 7:46 AM, Rick Curtis <cu...@gmail.com>
> > wrote:
> > > >
> > > > > Can you post a heapdump somewhere for review?
> > > > >
> > > > > Thanks,
> > > > > Rick
> > > > >
> > > > >
> > > > > On Thu, May 23, 2013 at 6:53 PM, Alejandro Abdelnur <
> > tucu00@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Rick,
> > > > > >
> > > > > > We create a single EMF:
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L160
> > > > > >
> > > > > > And we use create/use/dispose all EMs using a command patter:
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L209
> > > > > >
> > > > > > Any other idea/hint?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > >
> > > > > > On Wed, May 22, 2013 at 11:32 AM, Rick Curtis <
> curtisr7@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Alejandro -
> > > > > > >
> > > > > > > It appears that your application is creating
> > > EntityManagerFactories,
> > > > > yet
> > > > > > > they are never getting closed.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Rick
> > > > > > >
> > > > > > >
> > > > > > > On Wed, May 22, 2013 at 12:40 PM, Alejandro Abdelnur <
> > > > tucu00@gmail.com
> > > > > > > >wrote:
> > > > > > >
> > > > > > > > Rick, by looking at the other thread memory analyzer image,
> the
> > > > > issues
> > > > > > > seem
> > > > > > > > to be different.
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, May 20, 2013 at 9:47 AM, Rick Curtis <
> > curtisr7@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Alejandro --
> > > > > > > > >
> > > > > > > > > Is the problem that you're asking about the same as Hovan?
> If
> > > so,
> > > > > > lets
> > > > > > > > > consolidate this conversation down to a single thread.
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> http://openjpa.208410.n2.nabble.com/OOM-caused-by-OpenJPA-td7583869.html
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://openjpa.208410.n2.nabble.com/OOM-with-JDBCConfigurationImpl-holding-1-GB-of-heap-td7583862.html
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Rick
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, May 20, 2013 at 11:38 AM, Kevin Sutter <
> > > > kwsutter@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Alejandro,
> > > > > > > > > > It looks like something is holding onto references...
> How
> > > are
> > > > > you
> > > > > > > > > > enhancing your classes? At build time? Or, some other
> > > > > mechanism?
> > > > > > > > > >
> > > > > > > > > > Does this memory leak happen with any of your configured
> > > > > databases?
> > > > > > > > Or,
> > > > > > > > > > just select ones?
> > > > > > > > > >
> > > > > > > > > > This shouldn't make a difference, but specifying your
> > Entity
> > > > > > classes
> > > > > > > > via
> > > > > > > > > > both <class> and the openjpa.MetaDataFactory property is
> > > > > overkill.
> > > > > > > The
> > > > > > > > > > <class> elements are sufficient and is the standard, so
> I'd
> > > > > suggest
> > > > > > > > > > sticking with that mechanism and removing the
> > > > > > openjpa.MetaDataFactory
> > > > > > > > > > property.
> > > > > > > > > >
> > > > > > > > > > We do considerable memory leak testing both as a
> standalone
> > > JPA
> > > > > > > > provider
> > > > > > > > > as
> > > > > > > > > > well as the JPA provider for WebSphere Application
> Server.
> > > We
> > > > > > could
> > > > > > > > have
> > > > > > > > > > missed something due to your configuration requirements,
> > but
> > > > this
> > > > > > > looks
> > > > > > > > > to
> > > > > > > > > > be a pretty major leak...
> > > > > > > > > >
> > > > > > > > > > Can you narrow down or simplify your scenario to the
> point
> > of
> > > > > > > sharing a
> > > > > > > > > > project that demonstrates the leak? Or, maybe there is
> > > memory
> > > > > data
> > > > > > > > dumps
> > > > > > > > > > that can be shared to help diagnose the issue?
> > > > > > > > > >
> > > > > > > > > > Kevin
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, May 17, 2013 at 4:50 PM, Alejandro Abdelnur <
> > > > > > > tucu00@gmail.com
> > > > > > > > > > >wrote:
> > > > > > > > > >
> > > > > > > > > > > And .... I'm back.....
> > > > > > > > > > >
> > > > > > > > > > > It seems that removing runtime enhancement has some
> > effect,
> > > > but
> > > > > > the
> > > > > > > > > heap
> > > > > > > > > > > keeps building up in the same spot.
> > > > > > > > > > >
> > > > > > > > > > > The following link is show where is the memory sink
> > > > > > > > > > >
> > > > > > > > > > > http://people.apache.org/~tucu/tmp/OpenJPALeak.png
> > > > > > > > > > >
> > > > > > > > > > > I'd appreciate any ideas/suggestions.
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > > > > Alejandro
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, May 10, 2013 at 10:11 AM, Kevin Sutter <
> > > > > > kwsutter@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > It shoudn't... But, with that flag set as
> "supported",
> > > we
> > > > > > can't
> > > > > > > be
> > > > > > > > > > sure
> > > > > > > > > > > > that the entities were actually enhanced at build
> time.
> > > By
> > > > > > > setting
> > > > > > > > > > this
> > > > > > > > > > > > property to "unsupported" or removing it altogether,
> > then
> > > > if
> > > > > an
> > > > > > > > > > > unenhanced
> > > > > > > > > > > > entity class is loaded, we'll log a message and quit
> > > > > > processing.
> > > > > > > > So,
> > > > > > > > > > > let's
> > > > > > > > > > > > go this route first and then we'll deal with any
> > > potential
> > > > > > memory
> > > > > > > > > leak.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks, Kevin
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, May 10, 2013 at 12:05 PM, Alejandro Abdelnur
> <
> > > > > > > > > tucu00@gmail.com
> > > > > > > > > > > > >wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > I'm in the process of verifying if Kevin's
> suggestion
> > > > fixes
> > > > > > the
> > > > > > > > > > > problem.
> > > > > > > > > > > > >
> > > > > > > > > > > > > While making the necessary changes, I've notice we
> > were
> > > > > > already
> > > > > > > > > > > enhancing
> > > > > > > > > > > > > the classes at build time.
> > > > > > > > > > > > >
> > > > > > > > > > > > > If the classes were enhanced at build time, still
> > would
> > > > be
> > > > > we
> > > > > > > see
> > > > > > > > > the
> > > > > > > > > > > > leak
> > > > > > > > > > > > > if the runtime enhancement flag is enabled?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, May 9, 2013 at 9:15 AM, Alejandro Abdelnur
> <
> > > > > > > > > tucu00@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks Kevin, will try that.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter <
> > > > > > > > kwsutter@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> Don't use this:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> <property
> name="openjpa.RuntimeUnenhancedClasses"
> > > > > > > > > > > value="supported"/>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> That is probably your issue. Reference this
> page
> > > for
> > > > > more
> > > > > > > > > > > > information:
> > > > > > > > > > > > > >>
> http://openjpa.apache.org/entity-enhancement.html
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Good luck, Kevin
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> On Thu, May 9, 2013 at 10:17 AM, Alejandro
> > Abdelnur
> > > <
> > > > > > > > > > > tucu00@gmail.com
> > > > > > > > > > > > > >> >wrote:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> > Hi Rick,
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > Thanks for looking into this, the
> > persistence.xml
> > > > file
> > > > > > we
> > > > > > > > are
> > > > > > > > > > > using
> > > > > > > > > > > > > is:
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > The EMF is instatiated here:
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > Thanks again.
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > Alejandro
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > On Thu, May 9, 2013 at 6:06 AM, Rick Curtis <
> > > > > > > > > curtisr7@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > > Can we see the contents of your
> > persistence.xml
> > > > > file?
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > On Wed, May 8, 2013 at 6:34 PM, Alejandro
> > > > Abdelnur <
> > > > > > > > > > > > > tucu00@gmail.com>
> > > > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > > We are seeing this using OpenJPA 2.1.0 in
> > > Oozie.
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > > A JDBCConfigurationImpl seems to be
> holding
> > a
> > > > > large
> > > > > > > > amount
> > > > > > > > > > of
> > > > > > > > > > > > > memory
> > > > > > > > > > > > > >> > via
> > > > > > > > > > > > > >> > > > its dataCacheTimeout (then listeners)
> > instance
> > > > > > > variable
> > > > > > > > > > > keeping
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > > Class # Objects
> > Used
> > > > > Heap
> > > > > > > > > > Retained
> > > > > > > > > > > > > Heap
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >
> > > > > > > >
> > > -------------------------------------------------------------------
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > > o.a.o.j.meta.MappingRepository 1,902
> > > > 304,320
> > > > > > > > > > > > 904,410,184
> > > > > > > > > > > > > >> > > > o.a.o.j.meta.ClassMapping 3,824
> > > > > 1,223,680
> > > > > > > > > > > > 105,073,624
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >
> > > > > > > >
> > > -------------------------------------------------------------------
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > > We are using a single EntityManagerFactory
> > and
> > > > we
> > > > > > are
> > > > > > > > > > closing
> > > > > > > > > > > > all
> > > > > > > > > > > > > >> > > > EntityManager instances correctly (via a
> > > command
> > > > > > > > pattern).
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > > All caching settings are default ones.
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > > Any hint where to look would be
> appreciated
> > > it.
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > > Thanks
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > > Alejandro
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > --
> > > > > > > > > > > > > >> > > *Rick Curtis*
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > *Rick Curtis*
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *Rick Curtis*
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Rick Curtis*
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Rick Curtis*
> > >
> >
>
>
>
> --
> *Rick Curtis*
>
Re: OOM with JDBCConfigurationImpl holding 1+ GB of heap
Posted by Rick Curtis <cu...@gmail.com>.
You are reading too far into my analysis. The leak is NOT
EntityManagerFactoryImpl instances, it is a data structure
(MappingRepository) that gets created as a result of creating an EMF. The
MappingRepository is statically referenced elsewhere in our code and if you
fail to close all EntityManagerFactoryImpl instances, the MappingRepository
instances don't get cleaned up.
Thanks,
Rick
On Tue, May 28, 2013 at 11:24 AM, Alejandro Abdelnur <tu...@gmail.com>wrote:
> Rick,
>
> Thanks for following up.
>
> After you email I've looked again at the hprof dump and the number
> of org.apache.openjpa.persistence.EntityManagerFactoryImpl isntances is 1.
> I've also noticed the heap has
> 500+ org.apache.openjpa.persistence.EntityManagerImpl instances, but they
> are referenced from a finalizer; thus in queue to be GC. So, it would
> argue, the leak does not come from there.
>
> I'll give it a ry using 2.2.0 to see if that makes a difference.
>
> Thanks again.
>
> Alejandro
>
>
>
> On Fri, May 24, 2013 at 12:05 PM, Rick Curtis <cu...@gmail.com> wrote:
>
> > After digging into the heapdump, I'm fairly certain that somewhere in
> your
> > application EntityManagerFactories are being created, yet they are never
> > closed. I know you pointed me at your JPA service source code, but there
> > must be something else going on. For giggles you could move to version
> > 2.2.0, where we made some changes[1] to better tolerate applications that
> > fail to close EMFs. The JIRA talks about additional diagnostics, but we
> > just changed a bit of our code to use WeakReferences in hopes of avoiding
> > these sorts of OOMs. If moving to 2.2.0 makes your problem go away,
> you'll
> > need to look elsewhere in your code for EMF creation. Hope this helps.
> >
> > Thanks,
> > Rick
> >
> >
> > [1] https://issues.apache.org/jira/browse/OPENJPA-2042
> >
> >
> > On Fri, May 24, 2013 at 10:49 AM, Alejandro Abdelnur <tucu00@gmail.com
> > >wrote:
> >
> > > Hi Rick, a hprof dump is available at
> > > http://people.apache.org/~tucu/tmp/openjpaleak.hprof.tar.gz
> > >
> > > Thank you very much for looking into this.
> > >
> > > Alejandro
> > >
> > >
> > > On Fri, May 24, 2013 at 7:46 AM, Rick Curtis <cu...@gmail.com>
> wrote:
> > >
> > > > Can you post a heapdump somewhere for review?
> > > >
> > > > Thanks,
> > > > Rick
> > > >
> > > >
> > > > On Thu, May 23, 2013 at 6:53 PM, Alejandro Abdelnur <
> tucu00@gmail.com
> > > > >wrote:
> > > >
> > > > > Rick,
> > > > >
> > > > > We create a single EMF:
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L160
> > > > >
> > > > > And we use create/use/dispose all EMs using a command patter:
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L209
> > > > >
> > > > > Any other idea/hint?
> > > > >
> > > > > Thanks
> > > > >
> > > > >
> > > > > On Wed, May 22, 2013 at 11:32 AM, Rick Curtis <cu...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Alejandro -
> > > > > >
> > > > > > It appears that your application is creating
> > EntityManagerFactories,
> > > > yet
> > > > > > they are never getting closed.
> > > > > >
> > > > > > Thanks,
> > > > > > Rick
> > > > > >
> > > > > >
> > > > > > On Wed, May 22, 2013 at 12:40 PM, Alejandro Abdelnur <
> > > tucu00@gmail.com
> > > > > > >wrote:
> > > > > >
> > > > > > > Rick, by looking at the other thread memory analyzer image, the
> > > > issues
> > > > > > seem
> > > > > > > to be different.
> > > > > > >
> > > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > >
> > > > > > > On Mon, May 20, 2013 at 9:47 AM, Rick Curtis <
> curtisr7@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Alejandro --
> > > > > > > >
> > > > > > > > Is the problem that you're asking about the same as Hovan? If
> > so,
> > > > > lets
> > > > > > > > consolidate this conversation down to a single thread.
> > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > http://openjpa.208410.n2.nabble.com/OOM-caused-by-OpenJPA-td7583869.html
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://openjpa.208410.n2.nabble.com/OOM-with-JDBCConfigurationImpl-holding-1-GB-of-heap-td7583862.html
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Rick
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, May 20, 2013 at 11:38 AM, Kevin Sutter <
> > > kwsutter@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Alejandro,
> > > > > > > > > It looks like something is holding onto references... How
> > are
> > > > you
> > > > > > > > > enhancing your classes? At build time? Or, some other
> > > > mechanism?
> > > > > > > > >
> > > > > > > > > Does this memory leak happen with any of your configured
> > > > databases?
> > > > > > > Or,
> > > > > > > > > just select ones?
> > > > > > > > >
> > > > > > > > > This shouldn't make a difference, but specifying your
> Entity
> > > > > classes
> > > > > > > via
> > > > > > > > > both <class> and the openjpa.MetaDataFactory property is
> > > > overkill.
> > > > > > The
> > > > > > > > > <class> elements are sufficient and is the standard, so I'd
> > > > suggest
> > > > > > > > > sticking with that mechanism and removing the
> > > > > openjpa.MetaDataFactory
> > > > > > > > > property.
> > > > > > > > >
> > > > > > > > > We do considerable memory leak testing both as a standalone
> > JPA
> > > > > > > provider
> > > > > > > > as
> > > > > > > > > well as the JPA provider for WebSphere Application Server.
> > We
> > > > > could
> > > > > > > have
> > > > > > > > > missed something due to your configuration requirements,
> but
> > > this
> > > > > > looks
> > > > > > > > to
> > > > > > > > > be a pretty major leak...
> > > > > > > > >
> > > > > > > > > Can you narrow down or simplify your scenario to the point
> of
> > > > > > sharing a
> > > > > > > > > project that demonstrates the leak? Or, maybe there is
> > memory
> > > > data
> > > > > > > dumps
> > > > > > > > > that can be shared to help diagnose the issue?
> > > > > > > > >
> > > > > > > > > Kevin
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, May 17, 2013 at 4:50 PM, Alejandro Abdelnur <
> > > > > > tucu00@gmail.com
> > > > > > > > > >wrote:
> > > > > > > > >
> > > > > > > > > > And .... I'm back.....
> > > > > > > > > >
> > > > > > > > > > It seems that removing runtime enhancement has some
> effect,
> > > but
> > > > > the
> > > > > > > > heap
> > > > > > > > > > keeps building up in the same spot.
> > > > > > > > > >
> > > > > > > > > > The following link is show where is the memory sink
> > > > > > > > > >
> > > > > > > > > > http://people.apache.org/~tucu/tmp/OpenJPALeak.png
> > > > > > > > > >
> > > > > > > > > > I'd appreciate any ideas/suggestions.
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > > Alejandro
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, May 10, 2013 at 10:11 AM, Kevin Sutter <
> > > > > kwsutter@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > It shoudn't... But, with that flag set as "supported",
> > we
> > > > > can't
> > > > > > be
> > > > > > > > > sure
> > > > > > > > > > > that the entities were actually enhanced at build time.
> > By
> > > > > > setting
> > > > > > > > > this
> > > > > > > > > > > property to "unsupported" or removing it altogether,
> then
> > > if
> > > > an
> > > > > > > > > > unenhanced
> > > > > > > > > > > entity class is loaded, we'll log a message and quit
> > > > > processing.
> > > > > > > So,
> > > > > > > > > > let's
> > > > > > > > > > > go this route first and then we'll deal with any
> > potential
> > > > > memory
> > > > > > > > leak.
> > > > > > > > > > >
> > > > > > > > > > > Thanks, Kevin
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, May 10, 2013 at 12:05 PM, Alejandro Abdelnur <
> > > > > > > > tucu00@gmail.com
> > > > > > > > > > > >wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I'm in the process of verifying if Kevin's suggestion
> > > fixes
> > > > > the
> > > > > > > > > > problem.
> > > > > > > > > > > >
> > > > > > > > > > > > While making the necessary changes, I've notice we
> were
> > > > > already
> > > > > > > > > > enhancing
> > > > > > > > > > > > the classes at build time.
> > > > > > > > > > > >
> > > > > > > > > > > > If the classes were enhanced at build time, still
> would
> > > be
> > > > we
> > > > > > see
> > > > > > > > the
> > > > > > > > > > > leak
> > > > > > > > > > > > if the runtime enhancement flag is enabled?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, May 9, 2013 at 9:15 AM, Alejandro Abdelnur <
> > > > > > > > tucu00@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thanks Kevin, will try that.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter <
> > > > > > > kwsutter@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Don't use this:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> <property name="openjpa.RuntimeUnenhancedClasses"
> > > > > > > > > > value="supported"/>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> That is probably your issue. Reference this page
> > for
> > > > more
> > > > > > > > > > > information:
> > > > > > > > > > > > >> http://openjpa.apache.org/entity-enhancement.html
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Good luck, Kevin
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On Thu, May 9, 2013 at 10:17 AM, Alejandro
> Abdelnur
> > <
> > > > > > > > > > tucu00@gmail.com
> > > > > > > > > > > > >> >wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> > Hi Rick,
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Thanks for looking into this, the
> persistence.xml
> > > file
> > > > > we
> > > > > > > are
> > > > > > > > > > using
> > > > > > > > > > > > is:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > The EMF is instatiated here:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Thanks again.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Alejandro
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > On Thu, May 9, 2013 at 6:06 AM, Rick Curtis <
> > > > > > > > curtisr7@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > > Can we see the contents of your
> persistence.xml
> > > > file?
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > On Wed, May 8, 2013 at 6:34 PM, Alejandro
> > > Abdelnur <
> > > > > > > > > > > > tucu00@gmail.com>
> > > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > > We are seeing this using OpenJPA 2.1.0 in
> > Oozie.
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > A JDBCConfigurationImpl seems to be holding
> a
> > > > large
> > > > > > > amount
> > > > > > > > > of
> > > > > > > > > > > > memory
> > > > > > > > > > > > >> > via
> > > > > > > > > > > > >> > > > its dataCacheTimeout (then listeners)
> instance
> > > > > > variable
> > > > > > > > > > keeping
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > Class # Objects
> Used
> > > > Heap
> > > > > > > > > Retained
> > > > > > > > > > > > Heap
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > >
> > > > > > >
> > -------------------------------------------------------------------
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > o.a.o.j.meta.MappingRepository 1,902
> > > 304,320
> > > > > > > > > > > 904,410,184
> > > > > > > > > > > > >> > > > o.a.o.j.meta.ClassMapping 3,824
> > > > 1,223,680
> > > > > > > > > > > 105,073,624
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > >
> > > > > > >
> > -------------------------------------------------------------------
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > We are using a single EntityManagerFactory
> and
> > > we
> > > > > are
> > > > > > > > > closing
> > > > > > > > > > > all
> > > > > > > > > > > > >> > > > EntityManager instances correctly (via a
> > command
> > > > > > > pattern).
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > All caching settings are default ones.
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > Any hint where to look would be appreciated
> > it.
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > Thanks
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > Alejandro
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > --
> > > > > > > > > > > > >> > > *Rick Curtis*
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > *Rick Curtis*
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Rick Curtis*
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Rick Curtis*
> > > >
> > >
> >
> >
> >
> > --
> > *Rick Curtis*
> >
>
--
*Rick Curtis*
Re: OOM with JDBCConfigurationImpl holding 1+ GB of heap
Posted by Alejandro Abdelnur <tu...@gmail.com>.
Rick,
Thanks for following up.
After you email I've looked again at the hprof dump and the number
of org.apache.openjpa.persistence.EntityManagerFactoryImpl isntances is 1.
I've also noticed the heap has
500+ org.apache.openjpa.persistence.EntityManagerImpl instances, but they
are referenced from a finalizer; thus in queue to be GC. So, it would
argue, the leak does not come from there.
I'll give it a ry using 2.2.0 to see if that makes a difference.
Thanks again.
Alejandro
On Fri, May 24, 2013 at 12:05 PM, Rick Curtis <cu...@gmail.com> wrote:
> After digging into the heapdump, I'm fairly certain that somewhere in your
> application EntityManagerFactories are being created, yet they are never
> closed. I know you pointed me at your JPA service source code, but there
> must be something else going on. For giggles you could move to version
> 2.2.0, where we made some changes[1] to better tolerate applications that
> fail to close EMFs. The JIRA talks about additional diagnostics, but we
> just changed a bit of our code to use WeakReferences in hopes of avoiding
> these sorts of OOMs. If moving to 2.2.0 makes your problem go away, you'll
> need to look elsewhere in your code for EMF creation. Hope this helps.
>
> Thanks,
> Rick
>
>
> [1] https://issues.apache.org/jira/browse/OPENJPA-2042
>
>
> On Fri, May 24, 2013 at 10:49 AM, Alejandro Abdelnur <tucu00@gmail.com
> >wrote:
>
> > Hi Rick, a hprof dump is available at
> > http://people.apache.org/~tucu/tmp/openjpaleak.hprof.tar.gz
> >
> > Thank you very much for looking into this.
> >
> > Alejandro
> >
> >
> > On Fri, May 24, 2013 at 7:46 AM, Rick Curtis <cu...@gmail.com> wrote:
> >
> > > Can you post a heapdump somewhere for review?
> > >
> > > Thanks,
> > > Rick
> > >
> > >
> > > On Thu, May 23, 2013 at 6:53 PM, Alejandro Abdelnur <tucu00@gmail.com
> > > >wrote:
> > >
> > > > Rick,
> > > >
> > > > We create a single EMF:
> > > >
> > > >
> > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L160
> > > >
> > > > And we use create/use/dispose all EMs using a command patter:
> > > >
> > > >
> > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L209
> > > >
> > > > Any other idea/hint?
> > > >
> > > > Thanks
> > > >
> > > >
> > > > On Wed, May 22, 2013 at 11:32 AM, Rick Curtis <cu...@gmail.com>
> > > wrote:
> > > >
> > > > > Alejandro -
> > > > >
> > > > > It appears that your application is creating
> EntityManagerFactories,
> > > yet
> > > > > they are never getting closed.
> > > > >
> > > > > Thanks,
> > > > > Rick
> > > > >
> > > > >
> > > > > On Wed, May 22, 2013 at 12:40 PM, Alejandro Abdelnur <
> > tucu00@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Rick, by looking at the other thread memory analyzer image, the
> > > issues
> > > > > seem
> > > > > > to be different.
> > > > > >
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > >
> > > > > > On Mon, May 20, 2013 at 9:47 AM, Rick Curtis <curtisr7@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Alejandro --
> > > > > > >
> > > > > > > Is the problem that you're asking about the same as Hovan? If
> so,
> > > > lets
> > > > > > > consolidate this conversation down to a single thread.
> > > > > > >
> > > > > > >
> > > > >
> > >
> http://openjpa.208410.n2.nabble.com/OOM-caused-by-OpenJPA-td7583869.html
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://openjpa.208410.n2.nabble.com/OOM-with-JDBCConfigurationImpl-holding-1-GB-of-heap-td7583862.html
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Rick
> > > > > > >
> > > > > > >
> > > > > > > On Mon, May 20, 2013 at 11:38 AM, Kevin Sutter <
> > kwsutter@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Alejandro,
> > > > > > > > It looks like something is holding onto references... How
> are
> > > you
> > > > > > > > enhancing your classes? At build time? Or, some other
> > > mechanism?
> > > > > > > >
> > > > > > > > Does this memory leak happen with any of your configured
> > > databases?
> > > > > > Or,
> > > > > > > > just select ones?
> > > > > > > >
> > > > > > > > This shouldn't make a difference, but specifying your Entity
> > > > classes
> > > > > > via
> > > > > > > > both <class> and the openjpa.MetaDataFactory property is
> > > overkill.
> > > > > The
> > > > > > > > <class> elements are sufficient and is the standard, so I'd
> > > suggest
> > > > > > > > sticking with that mechanism and removing the
> > > > openjpa.MetaDataFactory
> > > > > > > > property.
> > > > > > > >
> > > > > > > > We do considerable memory leak testing both as a standalone
> JPA
> > > > > > provider
> > > > > > > as
> > > > > > > > well as the JPA provider for WebSphere Application Server.
> We
> > > > could
> > > > > > have
> > > > > > > > missed something due to your configuration requirements, but
> > this
> > > > > looks
> > > > > > > to
> > > > > > > > be a pretty major leak...
> > > > > > > >
> > > > > > > > Can you narrow down or simplify your scenario to the point of
> > > > > sharing a
> > > > > > > > project that demonstrates the leak? Or, maybe there is
> memory
> > > data
> > > > > > dumps
> > > > > > > > that can be shared to help diagnose the issue?
> > > > > > > >
> > > > > > > > Kevin
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, May 17, 2013 at 4:50 PM, Alejandro Abdelnur <
> > > > > tucu00@gmail.com
> > > > > > > > >wrote:
> > > > > > > >
> > > > > > > > > And .... I'm back.....
> > > > > > > > >
> > > > > > > > > It seems that removing runtime enhancement has some effect,
> > but
> > > > the
> > > > > > > heap
> > > > > > > > > keeps building up in the same spot.
> > > > > > > > >
> > > > > > > > > The following link is show where is the memory sink
> > > > > > > > >
> > > > > > > > > http://people.apache.org/~tucu/tmp/OpenJPALeak.png
> > > > > > > > >
> > > > > > > > > I'd appreciate any ideas/suggestions.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > > Alejandro
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, May 10, 2013 at 10:11 AM, Kevin Sutter <
> > > > kwsutter@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > It shoudn't... But, with that flag set as "supported",
> we
> > > > can't
> > > > > be
> > > > > > > > sure
> > > > > > > > > > that the entities were actually enhanced at build time.
> By
> > > > > setting
> > > > > > > > this
> > > > > > > > > > property to "unsupported" or removing it altogether, then
> > if
> > > an
> > > > > > > > > unenhanced
> > > > > > > > > > entity class is loaded, we'll log a message and quit
> > > > processing.
> > > > > > So,
> > > > > > > > > let's
> > > > > > > > > > go this route first and then we'll deal with any
> potential
> > > > memory
> > > > > > > leak.
> > > > > > > > > >
> > > > > > > > > > Thanks, Kevin
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, May 10, 2013 at 12:05 PM, Alejandro Abdelnur <
> > > > > > > tucu00@gmail.com
> > > > > > > > > > >wrote:
> > > > > > > > > >
> > > > > > > > > > > I'm in the process of verifying if Kevin's suggestion
> > fixes
> > > > the
> > > > > > > > > problem.
> > > > > > > > > > >
> > > > > > > > > > > While making the necessary changes, I've notice we were
> > > > already
> > > > > > > > > enhancing
> > > > > > > > > > > the classes at build time.
> > > > > > > > > > >
> > > > > > > > > > > If the classes were enhanced at build time, still would
> > be
> > > we
> > > > > see
> > > > > > > the
> > > > > > > > > > leak
> > > > > > > > > > > if the runtime enhancement flag is enabled?
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, May 9, 2013 at 9:15 AM, Alejandro Abdelnur <
> > > > > > > tucu00@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks Kevin, will try that.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter <
> > > > > > kwsutter@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> Don't use this:
> > > > > > > > > > > >>
> > > > > > > > > > > >> <property name="openjpa.RuntimeUnenhancedClasses"
> > > > > > > > > value="supported"/>
> > > > > > > > > > > >>
> > > > > > > > > > > >> That is probably your issue. Reference this page
> for
> > > more
> > > > > > > > > > information:
> > > > > > > > > > > >> http://openjpa.apache.org/entity-enhancement.html
> > > > > > > > > > > >>
> > > > > > > > > > > >> Good luck, Kevin
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Thu, May 9, 2013 at 10:17 AM, Alejandro Abdelnur
> <
> > > > > > > > > tucu00@gmail.com
> > > > > > > > > > > >> >wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >> > Hi Rick,
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Thanks for looking into this, the persistence.xml
> > file
> > > > we
> > > > > > are
> > > > > > > > > using
> > > > > > > > > > > is:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > The EMF is instatiated here:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Thanks again.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Alejandro
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > On Thu, May 9, 2013 at 6:06 AM, Rick Curtis <
> > > > > > > curtisr7@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > > Can we see the contents of your persistence.xml
> > > file?
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > On Wed, May 8, 2013 at 6:34 PM, Alejandro
> > Abdelnur <
> > > > > > > > > > > tucu00@gmail.com>
> > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > > We are seeing this using OpenJPA 2.1.0 in
> Oozie.
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > A JDBCConfigurationImpl seems to be holding a
> > > large
> > > > > > amount
> > > > > > > > of
> > > > > > > > > > > memory
> > > > > > > > > > > >> > via
> > > > > > > > > > > >> > > > its dataCacheTimeout (then listeners) instance
> > > > > variable
> > > > > > > > > keeping
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > Class # Objects Used
> > > Heap
> > > > > > > > Retained
> > > > > > > > > > > Heap
> > > > > > > > > > > >> > > >
> > > > > > > > > > >
> > > > > >
> -------------------------------------------------------------------
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > o.a.o.j.meta.MappingRepository 1,902
> > 304,320
> > > > > > > > > > 904,410,184
> > > > > > > > > > > >> > > > o.a.o.j.meta.ClassMapping 3,824
> > > 1,223,680
> > > > > > > > > > 105,073,624
> > > > > > > > > > > >> > > >
> > > > > > > > > > >
> > > > > >
> -------------------------------------------------------------------
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > We are using a single EntityManagerFactory and
> > we
> > > > are
> > > > > > > > closing
> > > > > > > > > > all
> > > > > > > > > > > >> > > > EntityManager instances correctly (via a
> command
> > > > > > pattern).
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > All caching settings are default ones.
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > Any hint where to look would be appreciated
> it.
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > Thanks
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > Alejandro
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > --
> > > > > > > > > > > >> > > *Rick Curtis*
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *Rick Curtis*
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Rick Curtis*
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Rick Curtis*
> > >
> >
>
>
>
> --
> *Rick Curtis*
>
Re: OOM with JDBCConfigurationImpl holding 1+ GB of heap
Posted by Rick Curtis <cu...@gmail.com>.
After digging into the heapdump, I'm fairly certain that somewhere in your
application EntityManagerFactories are being created, yet they are never
closed. I know you pointed me at your JPA service source code, but there
must be something else going on. For giggles you could move to version
2.2.0, where we made some changes[1] to better tolerate applications that
fail to close EMFs. The JIRA talks about additional diagnostics, but we
just changed a bit of our code to use WeakReferences in hopes of avoiding
these sorts of OOMs. If moving to 2.2.0 makes your problem go away, you'll
need to look elsewhere in your code for EMF creation. Hope this helps.
Thanks,
Rick
[1] https://issues.apache.org/jira/browse/OPENJPA-2042
On Fri, May 24, 2013 at 10:49 AM, Alejandro Abdelnur <tu...@gmail.com>wrote:
> Hi Rick, a hprof dump is available at
> http://people.apache.org/~tucu/tmp/openjpaleak.hprof.tar.gz
>
> Thank you very much for looking into this.
>
> Alejandro
>
>
> On Fri, May 24, 2013 at 7:46 AM, Rick Curtis <cu...@gmail.com> wrote:
>
> > Can you post a heapdump somewhere for review?
> >
> > Thanks,
> > Rick
> >
> >
> > On Thu, May 23, 2013 at 6:53 PM, Alejandro Abdelnur <tucu00@gmail.com
> > >wrote:
> >
> > > Rick,
> > >
> > > We create a single EMF:
> > >
> > >
> > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L160
> > >
> > > And we use create/use/dispose all EMs using a command patter:
> > >
> > >
> > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L209
> > >
> > > Any other idea/hint?
> > >
> > > Thanks
> > >
> > >
> > > On Wed, May 22, 2013 at 11:32 AM, Rick Curtis <cu...@gmail.com>
> > wrote:
> > >
> > > > Alejandro -
> > > >
> > > > It appears that your application is creating EntityManagerFactories,
> > yet
> > > > they are never getting closed.
> > > >
> > > > Thanks,
> > > > Rick
> > > >
> > > >
> > > > On Wed, May 22, 2013 at 12:40 PM, Alejandro Abdelnur <
> tucu00@gmail.com
> > > > >wrote:
> > > >
> > > > > Rick, by looking at the other thread memory analyzer image, the
> > issues
> > > > seem
> > > > > to be different.
> > > > >
> > > > >
> > > > > Thanks.
> > > > >
> > > > >
> > > > > On Mon, May 20, 2013 at 9:47 AM, Rick Curtis <cu...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Alejandro --
> > > > > >
> > > > > > Is the problem that you're asking about the same as Hovan? If so,
> > > lets
> > > > > > consolidate this conversation down to a single thread.
> > > > > >
> > > > > >
> > > >
> > http://openjpa.208410.n2.nabble.com/OOM-caused-by-OpenJPA-td7583869.html
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://openjpa.208410.n2.nabble.com/OOM-with-JDBCConfigurationImpl-holding-1-GB-of-heap-td7583862.html
> > > > > >
> > > > > > Thanks,
> > > > > > Rick
> > > > > >
> > > > > >
> > > > > > On Mon, May 20, 2013 at 11:38 AM, Kevin Sutter <
> kwsutter@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi Alejandro,
> > > > > > > It looks like something is holding onto references... How are
> > you
> > > > > > > enhancing your classes? At build time? Or, some other
> > mechanism?
> > > > > > >
> > > > > > > Does this memory leak happen with any of your configured
> > databases?
> > > > > Or,
> > > > > > > just select ones?
> > > > > > >
> > > > > > > This shouldn't make a difference, but specifying your Entity
> > > classes
> > > > > via
> > > > > > > both <class> and the openjpa.MetaDataFactory property is
> > overkill.
> > > > The
> > > > > > > <class> elements are sufficient and is the standard, so I'd
> > suggest
> > > > > > > sticking with that mechanism and removing the
> > > openjpa.MetaDataFactory
> > > > > > > property.
> > > > > > >
> > > > > > > We do considerable memory leak testing both as a standalone JPA
> > > > > provider
> > > > > > as
> > > > > > > well as the JPA provider for WebSphere Application Server. We
> > > could
> > > > > have
> > > > > > > missed something due to your configuration requirements, but
> this
> > > > looks
> > > > > > to
> > > > > > > be a pretty major leak...
> > > > > > >
> > > > > > > Can you narrow down or simplify your scenario to the point of
> > > > sharing a
> > > > > > > project that demonstrates the leak? Or, maybe there is memory
> > data
> > > > > dumps
> > > > > > > that can be shared to help diagnose the issue?
> > > > > > >
> > > > > > > Kevin
> > > > > > >
> > > > > > >
> > > > > > > On Fri, May 17, 2013 at 4:50 PM, Alejandro Abdelnur <
> > > > tucu00@gmail.com
> > > > > > > >wrote:
> > > > > > >
> > > > > > > > And .... I'm back.....
> > > > > > > >
> > > > > > > > It seems that removing runtime enhancement has some effect,
> but
> > > the
> > > > > > heap
> > > > > > > > keeps building up in the same spot.
> > > > > > > >
> > > > > > > > The following link is show where is the memory sink
> > > > > > > >
> > > > > > > > http://people.apache.org/~tucu/tmp/OpenJPALeak.png
> > > > > > > >
> > > > > > > > I'd appreciate any ideas/suggestions.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > Alejandro
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, May 10, 2013 at 10:11 AM, Kevin Sutter <
> > > kwsutter@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > It shoudn't... But, with that flag set as "supported", we
> > > can't
> > > > be
> > > > > > > sure
> > > > > > > > > that the entities were actually enhanced at build time. By
> > > > setting
> > > > > > > this
> > > > > > > > > property to "unsupported" or removing it altogether, then
> if
> > an
> > > > > > > > unenhanced
> > > > > > > > > entity class is loaded, we'll log a message and quit
> > > processing.
> > > > > So,
> > > > > > > > let's
> > > > > > > > > go this route first and then we'll deal with any potential
> > > memory
> > > > > > leak.
> > > > > > > > >
> > > > > > > > > Thanks, Kevin
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, May 10, 2013 at 12:05 PM, Alejandro Abdelnur <
> > > > > > tucu00@gmail.com
> > > > > > > > > >wrote:
> > > > > > > > >
> > > > > > > > > > I'm in the process of verifying if Kevin's suggestion
> fixes
> > > the
> > > > > > > > problem.
> > > > > > > > > >
> > > > > > > > > > While making the necessary changes, I've notice we were
> > > already
> > > > > > > > enhancing
> > > > > > > > > > the classes at build time.
> > > > > > > > > >
> > > > > > > > > > If the classes were enhanced at build time, still would
> be
> > we
> > > > see
> > > > > > the
> > > > > > > > > leak
> > > > > > > > > > if the runtime enhancement flag is enabled?
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, May 9, 2013 at 9:15 AM, Alejandro Abdelnur <
> > > > > > tucu00@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks Kevin, will try that.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter <
> > > > > kwsutter@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Don't use this:
> > > > > > > > > > >>
> > > > > > > > > > >> <property name="openjpa.RuntimeUnenhancedClasses"
> > > > > > > > value="supported"/>
> > > > > > > > > > >>
> > > > > > > > > > >> That is probably your issue. Reference this page for
> > more
> > > > > > > > > information:
> > > > > > > > > > >> http://openjpa.apache.org/entity-enhancement.html
> > > > > > > > > > >>
> > > > > > > > > > >> Good luck, Kevin
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> On Thu, May 9, 2013 at 10:17 AM, Alejandro Abdelnur <
> > > > > > > > tucu00@gmail.com
> > > > > > > > > > >> >wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> > Hi Rick,
> > > > > > > > > > >> >
> > > > > > > > > > >> > Thanks for looking into this, the persistence.xml
> file
> > > we
> > > > > are
> > > > > > > > using
> > > > > > > > > > is:
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml
> > > > > > > > > > >> >
> > > > > > > > > > >> > The EMF is instatiated here:
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > Thanks again.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Alejandro
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > On Thu, May 9, 2013 at 6:06 AM, Rick Curtis <
> > > > > > curtisr7@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> > > Can we see the contents of your persistence.xml
> > file?
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > On Wed, May 8, 2013 at 6:34 PM, Alejandro
> Abdelnur <
> > > > > > > > > > tucu00@gmail.com>
> > > > > > > > > > >> > > wrote:
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > We are seeing this using OpenJPA 2.1.0 in Oozie.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > A JDBCConfigurationImpl seems to be holding a
> > large
> > > > > amount
> > > > > > > of
> > > > > > > > > > memory
> > > > > > > > > > >> > via
> > > > > > > > > > >> > > > its dataCacheTimeout (then listeners) instance
> > > > variable
> > > > > > > > keeping
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Class # Objects Used
> > Heap
> > > > > > > Retained
> > > > > > > > > > Heap
> > > > > > > > > > >> > > >
> > > > > > > > > >
> > > > > -------------------------------------------------------------------
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > o.a.o.j.meta.MappingRepository 1,902
> 304,320
> > > > > > > > > 904,410,184
> > > > > > > > > > >> > > > o.a.o.j.meta.ClassMapping 3,824
> > 1,223,680
> > > > > > > > > 105,073,624
> > > > > > > > > > >> > > >
> > > > > > > > > >
> > > > > -------------------------------------------------------------------
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > We are using a single EntityManagerFactory and
> we
> > > are
> > > > > > > closing
> > > > > > > > > all
> > > > > > > > > > >> > > > EntityManager instances correctly (via a command
> > > > > pattern).
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > All caching settings are default ones.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Any hint where to look would be appreciated it.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Thanks
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Alejandro
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > --
> > > > > > > > > > >> > > *Rick Curtis*
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Rick Curtis*
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Rick Curtis*
> > > >
> > >
> >
> >
> >
> > --
> > *Rick Curtis*
> >
>
--
*Rick Curtis*
Re: OOM with JDBCConfigurationImpl holding 1+ GB of heap
Posted by Alejandro Abdelnur <tu...@gmail.com>.
Hi Rick, a hprof dump is available at
http://people.apache.org/~tucu/tmp/openjpaleak.hprof.tar.gz
Thank you very much for looking into this.
Alejandro
On Fri, May 24, 2013 at 7:46 AM, Rick Curtis <cu...@gmail.com> wrote:
> Can you post a heapdump somewhere for review?
>
> Thanks,
> Rick
>
>
> On Thu, May 23, 2013 at 6:53 PM, Alejandro Abdelnur <tucu00@gmail.com
> >wrote:
>
> > Rick,
> >
> > We create a single EMF:
> >
> >
> >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L160
> >
> > And we use create/use/dispose all EMs using a command patter:
> >
> >
> >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L209
> >
> > Any other idea/hint?
> >
> > Thanks
> >
> >
> > On Wed, May 22, 2013 at 11:32 AM, Rick Curtis <cu...@gmail.com>
> wrote:
> >
> > > Alejandro -
> > >
> > > It appears that your application is creating EntityManagerFactories,
> yet
> > > they are never getting closed.
> > >
> > > Thanks,
> > > Rick
> > >
> > >
> > > On Wed, May 22, 2013 at 12:40 PM, Alejandro Abdelnur <tucu00@gmail.com
> > > >wrote:
> > >
> > > > Rick, by looking at the other thread memory analyzer image, the
> issues
> > > seem
> > > > to be different.
> > > >
> > > >
> > > > Thanks.
> > > >
> > > >
> > > > On Mon, May 20, 2013 at 9:47 AM, Rick Curtis <cu...@gmail.com>
> > wrote:
> > > >
> > > > > Alejandro --
> > > > >
> > > > > Is the problem that you're asking about the same as Hovan? If so,
> > lets
> > > > > consolidate this conversation down to a single thread.
> > > > >
> > > > >
> > >
> http://openjpa.208410.n2.nabble.com/OOM-caused-by-OpenJPA-td7583869.html
> > > > >
> > > > >
> > > >
> > >
> >
> http://openjpa.208410.n2.nabble.com/OOM-with-JDBCConfigurationImpl-holding-1-GB-of-heap-td7583862.html
> > > > >
> > > > > Thanks,
> > > > > Rick
> > > > >
> > > > >
> > > > > On Mon, May 20, 2013 at 11:38 AM, Kevin Sutter <kwsutter@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Hi Alejandro,
> > > > > > It looks like something is holding onto references... How are
> you
> > > > > > enhancing your classes? At build time? Or, some other
> mechanism?
> > > > > >
> > > > > > Does this memory leak happen with any of your configured
> databases?
> > > > Or,
> > > > > > just select ones?
> > > > > >
> > > > > > This shouldn't make a difference, but specifying your Entity
> > classes
> > > > via
> > > > > > both <class> and the openjpa.MetaDataFactory property is
> overkill.
> > > The
> > > > > > <class> elements are sufficient and is the standard, so I'd
> suggest
> > > > > > sticking with that mechanism and removing the
> > openjpa.MetaDataFactory
> > > > > > property.
> > > > > >
> > > > > > We do considerable memory leak testing both as a standalone JPA
> > > > provider
> > > > > as
> > > > > > well as the JPA provider for WebSphere Application Server. We
> > could
> > > > have
> > > > > > missed something due to your configuration requirements, but this
> > > looks
> > > > > to
> > > > > > be a pretty major leak...
> > > > > >
> > > > > > Can you narrow down or simplify your scenario to the point of
> > > sharing a
> > > > > > project that demonstrates the leak? Or, maybe there is memory
> data
> > > > dumps
> > > > > > that can be shared to help diagnose the issue?
> > > > > >
> > > > > > Kevin
> > > > > >
> > > > > >
> > > > > > On Fri, May 17, 2013 at 4:50 PM, Alejandro Abdelnur <
> > > tucu00@gmail.com
> > > > > > >wrote:
> > > > > >
> > > > > > > And .... I'm back.....
> > > > > > >
> > > > > > > It seems that removing runtime enhancement has some effect, but
> > the
> > > > > heap
> > > > > > > keeps building up in the same spot.
> > > > > > >
> > > > > > > The following link is show where is the memory sink
> > > > > > >
> > > > > > > http://people.apache.org/~tucu/tmp/OpenJPALeak.png
> > > > > > >
> > > > > > > I'd appreciate any ideas/suggestions.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > Alejandro
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, May 10, 2013 at 10:11 AM, Kevin Sutter <
> > kwsutter@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > It shoudn't... But, with that flag set as "supported", we
> > can't
> > > be
> > > > > > sure
> > > > > > > > that the entities were actually enhanced at build time. By
> > > setting
> > > > > > this
> > > > > > > > property to "unsupported" or removing it altogether, then if
> an
> > > > > > > unenhanced
> > > > > > > > entity class is loaded, we'll log a message and quit
> > processing.
> > > > So,
> > > > > > > let's
> > > > > > > > go this route first and then we'll deal with any potential
> > memory
> > > > > leak.
> > > > > > > >
> > > > > > > > Thanks, Kevin
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, May 10, 2013 at 12:05 PM, Alejandro Abdelnur <
> > > > > tucu00@gmail.com
> > > > > > > > >wrote:
> > > > > > > >
> > > > > > > > > I'm in the process of verifying if Kevin's suggestion fixes
> > the
> > > > > > > problem.
> > > > > > > > >
> > > > > > > > > While making the necessary changes, I've notice we were
> > already
> > > > > > > enhancing
> > > > > > > > > the classes at build time.
> > > > > > > > >
> > > > > > > > > If the classes were enhanced at build time, still would be
> we
> > > see
> > > > > the
> > > > > > > > leak
> > > > > > > > > if the runtime enhancement flag is enabled?
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, May 9, 2013 at 9:15 AM, Alejandro Abdelnur <
> > > > > tucu00@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks Kevin, will try that.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter <
> > > > kwsutter@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Don't use this:
> > > > > > > > > >>
> > > > > > > > > >> <property name="openjpa.RuntimeUnenhancedClasses"
> > > > > > > value="supported"/>
> > > > > > > > > >>
> > > > > > > > > >> That is probably your issue. Reference this page for
> more
> > > > > > > > information:
> > > > > > > > > >> http://openjpa.apache.org/entity-enhancement.html
> > > > > > > > > >>
> > > > > > > > > >> Good luck, Kevin
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> On Thu, May 9, 2013 at 10:17 AM, Alejandro Abdelnur <
> > > > > > > tucu00@gmail.com
> > > > > > > > > >> >wrote:
> > > > > > > > > >>
> > > > > > > > > >> > Hi Rick,
> > > > > > > > > >> >
> > > > > > > > > >> > Thanks for looking into this, the persistence.xml file
> > we
> > > > are
> > > > > > > using
> > > > > > > > > is:
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml
> > > > > > > > > >> >
> > > > > > > > > >> > The EMF is instatiated here:
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > Thanks again.
> > > > > > > > > >> >
> > > > > > > > > >> > Alejandro
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > On Thu, May 9, 2013 at 6:06 AM, Rick Curtis <
> > > > > curtisr7@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >> >
> > > > > > > > > >> > > Can we see the contents of your persistence.xml
> file?
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > > On Wed, May 8, 2013 at 6:34 PM, Alejandro Abdelnur <
> > > > > > > > > tucu00@gmail.com>
> > > > > > > > > >> > > wrote:
> > > > > > > > > >> > >
> > > > > > > > > >> > > > We are seeing this using OpenJPA 2.1.0 in Oozie.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > A JDBCConfigurationImpl seems to be holding a
> large
> > > > amount
> > > > > > of
> > > > > > > > > memory
> > > > > > > > > >> > via
> > > > > > > > > >> > > > its dataCacheTimeout (then listeners) instance
> > > variable
> > > > > > > keeping
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Class # Objects Used
> Heap
> > > > > > Retained
> > > > > > > > > Heap
> > > > > > > > > >> > > >
> > > > > > > > >
> > > > -------------------------------------------------------------------
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > o.a.o.j.meta.MappingRepository 1,902 304,320
> > > > > > > > 904,410,184
> > > > > > > > > >> > > > o.a.o.j.meta.ClassMapping 3,824
> 1,223,680
> > > > > > > > 105,073,624
> > > > > > > > > >> > > >
> > > > > > > > >
> > > > -------------------------------------------------------------------
> > > > > > > > > >> > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > We are using a single EntityManagerFactory and we
> > are
> > > > > > closing
> > > > > > > > all
> > > > > > > > > >> > > > EntityManager instances correctly (via a command
> > > > pattern).
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > All caching settings are default ones.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Any hint where to look would be appreciated it.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Thanks
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Alejandro
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > > --
> > > > > > > > > >> > > *Rick Curtis*
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Rick Curtis*
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Rick Curtis*
> > >
> >
>
>
>
> --
> *Rick Curtis*
>
Re: OOM with JDBCConfigurationImpl holding 1+ GB of heap
Posted by Rick Curtis <cu...@gmail.com>.
Can you post a heapdump somewhere for review?
Thanks,
Rick
On Thu, May 23, 2013 at 6:53 PM, Alejandro Abdelnur <tu...@gmail.com>wrote:
> Rick,
>
> We create a single EMF:
>
>
>
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L160
>
> And we use create/use/dispose all EMs using a command patter:
>
>
>
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L209
>
> Any other idea/hint?
>
> Thanks
>
>
> On Wed, May 22, 2013 at 11:32 AM, Rick Curtis <cu...@gmail.com> wrote:
>
> > Alejandro -
> >
> > It appears that your application is creating EntityManagerFactories, yet
> > they are never getting closed.
> >
> > Thanks,
> > Rick
> >
> >
> > On Wed, May 22, 2013 at 12:40 PM, Alejandro Abdelnur <tucu00@gmail.com
> > >wrote:
> >
> > > Rick, by looking at the other thread memory analyzer image, the issues
> > seem
> > > to be different.
> > >
> > >
> > > Thanks.
> > >
> > >
> > > On Mon, May 20, 2013 at 9:47 AM, Rick Curtis <cu...@gmail.com>
> wrote:
> > >
> > > > Alejandro --
> > > >
> > > > Is the problem that you're asking about the same as Hovan? If so,
> lets
> > > > consolidate this conversation down to a single thread.
> > > >
> > > >
> > http://openjpa.208410.n2.nabble.com/OOM-caused-by-OpenJPA-td7583869.html
> > > >
> > > >
> > >
> >
> http://openjpa.208410.n2.nabble.com/OOM-with-JDBCConfigurationImpl-holding-1-GB-of-heap-td7583862.html
> > > >
> > > > Thanks,
> > > > Rick
> > > >
> > > >
> > > > On Mon, May 20, 2013 at 11:38 AM, Kevin Sutter <kw...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi Alejandro,
> > > > > It looks like something is holding onto references... How are you
> > > > > enhancing your classes? At build time? Or, some other mechanism?
> > > > >
> > > > > Does this memory leak happen with any of your configured databases?
> > > Or,
> > > > > just select ones?
> > > > >
> > > > > This shouldn't make a difference, but specifying your Entity
> classes
> > > via
> > > > > both <class> and the openjpa.MetaDataFactory property is overkill.
> > The
> > > > > <class> elements are sufficient and is the standard, so I'd suggest
> > > > > sticking with that mechanism and removing the
> openjpa.MetaDataFactory
> > > > > property.
> > > > >
> > > > > We do considerable memory leak testing both as a standalone JPA
> > > provider
> > > > as
> > > > > well as the JPA provider for WebSphere Application Server. We
> could
> > > have
> > > > > missed something due to your configuration requirements, but this
> > looks
> > > > to
> > > > > be a pretty major leak...
> > > > >
> > > > > Can you narrow down or simplify your scenario to the point of
> > sharing a
> > > > > project that demonstrates the leak? Or, maybe there is memory data
> > > dumps
> > > > > that can be shared to help diagnose the issue?
> > > > >
> > > > > Kevin
> > > > >
> > > > >
> > > > > On Fri, May 17, 2013 at 4:50 PM, Alejandro Abdelnur <
> > tucu00@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > And .... I'm back.....
> > > > > >
> > > > > > It seems that removing runtime enhancement has some effect, but
> the
> > > > heap
> > > > > > keeps building up in the same spot.
> > > > > >
> > > > > > The following link is show where is the memory sink
> > > > > >
> > > > > > http://people.apache.org/~tucu/tmp/OpenJPALeak.png
> > > > > >
> > > > > > I'd appreciate any ideas/suggestions.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > Alejandro
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, May 10, 2013 at 10:11 AM, Kevin Sutter <
> kwsutter@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > It shoudn't... But, with that flag set as "supported", we
> can't
> > be
> > > > > sure
> > > > > > > that the entities were actually enhanced at build time. By
> > setting
> > > > > this
> > > > > > > property to "unsupported" or removing it altogether, then if an
> > > > > > unenhanced
> > > > > > > entity class is loaded, we'll log a message and quit
> processing.
> > > So,
> > > > > > let's
> > > > > > > go this route first and then we'll deal with any potential
> memory
> > > > leak.
> > > > > > >
> > > > > > > Thanks, Kevin
> > > > > > >
> > > > > > >
> > > > > > > On Fri, May 10, 2013 at 12:05 PM, Alejandro Abdelnur <
> > > > tucu00@gmail.com
> > > > > > > >wrote:
> > > > > > >
> > > > > > > > I'm in the process of verifying if Kevin's suggestion fixes
> the
> > > > > > problem.
> > > > > > > >
> > > > > > > > While making the necessary changes, I've notice we were
> already
> > > > > > enhancing
> > > > > > > > the classes at build time.
> > > > > > > >
> > > > > > > > If the classes were enhanced at build time, still would be we
> > see
> > > > the
> > > > > > > leak
> > > > > > > > if the runtime enhancement flag is enabled?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, May 9, 2013 at 9:15 AM, Alejandro Abdelnur <
> > > > tucu00@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks Kevin, will try that.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter <
> > > kwsutter@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Don't use this:
> > > > > > > > >>
> > > > > > > > >> <property name="openjpa.RuntimeUnenhancedClasses"
> > > > > > value="supported"/>
> > > > > > > > >>
> > > > > > > > >> That is probably your issue. Reference this page for more
> > > > > > > information:
> > > > > > > > >> http://openjpa.apache.org/entity-enhancement.html
> > > > > > > > >>
> > > > > > > > >> Good luck, Kevin
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Thu, May 9, 2013 at 10:17 AM, Alejandro Abdelnur <
> > > > > > tucu00@gmail.com
> > > > > > > > >> >wrote:
> > > > > > > > >>
> > > > > > > > >> > Hi Rick,
> > > > > > > > >> >
> > > > > > > > >> > Thanks for looking into this, the persistence.xml file
> we
> > > are
> > > > > > using
> > > > > > > > is:
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml
> > > > > > > > >> >
> > > > > > > > >> > The EMF is instatiated here:
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > Thanks again.
> > > > > > > > >> >
> > > > > > > > >> > Alejandro
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > On Thu, May 9, 2013 at 6:06 AM, Rick Curtis <
> > > > curtisr7@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >> >
> > > > > > > > >> > > Can we see the contents of your persistence.xml file?
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > On Wed, May 8, 2013 at 6:34 PM, Alejandro Abdelnur <
> > > > > > > > tucu00@gmail.com>
> > > > > > > > >> > > wrote:
> > > > > > > > >> > >
> > > > > > > > >> > > > We are seeing this using OpenJPA 2.1.0 in Oozie.
> > > > > > > > >> > > >
> > > > > > > > >> > > > A JDBCConfigurationImpl seems to be holding a large
> > > amount
> > > > > of
> > > > > > > > memory
> > > > > > > > >> > via
> > > > > > > > >> > > > its dataCacheTimeout (then listeners) instance
> > variable
> > > > > > keeping
> > > > > > > > >> > > >
> > > > > > > > >> > > > Class # Objects Used Heap
> > > > > Retained
> > > > > > > > Heap
> > > > > > > > >> > > >
> > > > > > > >
> > > -------------------------------------------------------------------
> > > > > > > > >> > > >
> > > > > > > > >> > > > o.a.o.j.meta.MappingRepository 1,902 304,320
> > > > > > > 904,410,184
> > > > > > > > >> > > > o.a.o.j.meta.ClassMapping 3,824 1,223,680
> > > > > > > 105,073,624
> > > > > > > > >> > > >
> > > > > > > >
> > > -------------------------------------------------------------------
> > > > > > > > >> > > >
> > > > > > > > >> > > >
> > > > > > > > >> > > > We are using a single EntityManagerFactory and we
> are
> > > > > closing
> > > > > > > all
> > > > > > > > >> > > > EntityManager instances correctly (via a command
> > > pattern).
> > > > > > > > >> > > >
> > > > > > > > >> > > > All caching settings are default ones.
> > > > > > > > >> > > >
> > > > > > > > >> > > > Any hint where to look would be appreciated it.
> > > > > > > > >> > > >
> > > > > > > > >> > > > Thanks
> > > > > > > > >> > > >
> > > > > > > > >> > > > Alejandro
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > --
> > > > > > > > >> > > *Rick Curtis*
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Rick Curtis*
> > > >
> > >
> >
> >
> >
> > --
> > *Rick Curtis*
> >
>
--
*Rick Curtis*
Re: OOM with JDBCConfigurationImpl holding 1+ GB of heap
Posted by Alejandro Abdelnur <tu...@gmail.com>.
Rick,
We create a single EMF:
https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L160
And we use create/use/dispose all EMs using a command patter:
https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L209
Any other idea/hint?
Thanks
On Wed, May 22, 2013 at 11:32 AM, Rick Curtis <cu...@gmail.com> wrote:
> Alejandro -
>
> It appears that your application is creating EntityManagerFactories, yet
> they are never getting closed.
>
> Thanks,
> Rick
>
>
> On Wed, May 22, 2013 at 12:40 PM, Alejandro Abdelnur <tucu00@gmail.com
> >wrote:
>
> > Rick, by looking at the other thread memory analyzer image, the issues
> seem
> > to be different.
> >
> >
> > Thanks.
> >
> >
> > On Mon, May 20, 2013 at 9:47 AM, Rick Curtis <cu...@gmail.com> wrote:
> >
> > > Alejandro --
> > >
> > > Is the problem that you're asking about the same as Hovan? If so, lets
> > > consolidate this conversation down to a single thread.
> > >
> > >
> http://openjpa.208410.n2.nabble.com/OOM-caused-by-OpenJPA-td7583869.html
> > >
> > >
> >
> http://openjpa.208410.n2.nabble.com/OOM-with-JDBCConfigurationImpl-holding-1-GB-of-heap-td7583862.html
> > >
> > > Thanks,
> > > Rick
> > >
> > >
> > > On Mon, May 20, 2013 at 11:38 AM, Kevin Sutter <kw...@gmail.com>
> > wrote:
> > >
> > > > Hi Alejandro,
> > > > It looks like something is holding onto references... How are you
> > > > enhancing your classes? At build time? Or, some other mechanism?
> > > >
> > > > Does this memory leak happen with any of your configured databases?
> > Or,
> > > > just select ones?
> > > >
> > > > This shouldn't make a difference, but specifying your Entity classes
> > via
> > > > both <class> and the openjpa.MetaDataFactory property is overkill.
> The
> > > > <class> elements are sufficient and is the standard, so I'd suggest
> > > > sticking with that mechanism and removing the openjpa.MetaDataFactory
> > > > property.
> > > >
> > > > We do considerable memory leak testing both as a standalone JPA
> > provider
> > > as
> > > > well as the JPA provider for WebSphere Application Server. We could
> > have
> > > > missed something due to your configuration requirements, but this
> looks
> > > to
> > > > be a pretty major leak...
> > > >
> > > > Can you narrow down or simplify your scenario to the point of
> sharing a
> > > > project that demonstrates the leak? Or, maybe there is memory data
> > dumps
> > > > that can be shared to help diagnose the issue?
> > > >
> > > > Kevin
> > > >
> > > >
> > > > On Fri, May 17, 2013 at 4:50 PM, Alejandro Abdelnur <
> tucu00@gmail.com
> > > > >wrote:
> > > >
> > > > > And .... I'm back.....
> > > > >
> > > > > It seems that removing runtime enhancement has some effect, but the
> > > heap
> > > > > keeps building up in the same spot.
> > > > >
> > > > > The following link is show where is the memory sink
> > > > >
> > > > > http://people.apache.org/~tucu/tmp/OpenJPALeak.png
> > > > >
> > > > > I'd appreciate any ideas/suggestions.
> > > > >
> > > > > Thanks
> > > > >
> > > > > Alejandro
> > > > >
> > > > >
> > > > >
> > > > > On Fri, May 10, 2013 at 10:11 AM, Kevin Sutter <kwsutter@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > It shoudn't... But, with that flag set as "supported", we can't
> be
> > > > sure
> > > > > > that the entities were actually enhanced at build time. By
> setting
> > > > this
> > > > > > property to "unsupported" or removing it altogether, then if an
> > > > > unenhanced
> > > > > > entity class is loaded, we'll log a message and quit processing.
> > So,
> > > > > let's
> > > > > > go this route first and then we'll deal with any potential memory
> > > leak.
> > > > > >
> > > > > > Thanks, Kevin
> > > > > >
> > > > > >
> > > > > > On Fri, May 10, 2013 at 12:05 PM, Alejandro Abdelnur <
> > > tucu00@gmail.com
> > > > > > >wrote:
> > > > > >
> > > > > > > I'm in the process of verifying if Kevin's suggestion fixes the
> > > > > problem.
> > > > > > >
> > > > > > > While making the necessary changes, I've notice we were already
> > > > > enhancing
> > > > > > > the classes at build time.
> > > > > > >
> > > > > > > If the classes were enhanced at build time, still would be we
> see
> > > the
> > > > > > leak
> > > > > > > if the runtime enhancement flag is enabled?
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > >
> > > > > > > On Thu, May 9, 2013 at 9:15 AM, Alejandro Abdelnur <
> > > tucu00@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks Kevin, will try that.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter <
> > kwsutter@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > >> Don't use this:
> > > > > > > >>
> > > > > > > >> <property name="openjpa.RuntimeUnenhancedClasses"
> > > > > value="supported"/>
> > > > > > > >>
> > > > > > > >> That is probably your issue. Reference this page for more
> > > > > > information:
> > > > > > > >> http://openjpa.apache.org/entity-enhancement.html
> > > > > > > >>
> > > > > > > >> Good luck, Kevin
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Thu, May 9, 2013 at 10:17 AM, Alejandro Abdelnur <
> > > > > tucu00@gmail.com
> > > > > > > >> >wrote:
> > > > > > > >>
> > > > > > > >> > Hi Rick,
> > > > > > > >> >
> > > > > > > >> > Thanks for looking into this, the persistence.xml file we
> > are
> > > > > using
> > > > > > > is:
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml
> > > > > > > >> >
> > > > > > > >> > The EMF is instatiated here:
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > Thanks again.
> > > > > > > >> >
> > > > > > > >> > Alejandro
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Thu, May 9, 2013 at 6:06 AM, Rick Curtis <
> > > curtisr7@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > > Can we see the contents of your persistence.xml file?
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > On Wed, May 8, 2013 at 6:34 PM, Alejandro Abdelnur <
> > > > > > > tucu00@gmail.com>
> > > > > > > >> > > wrote:
> > > > > > > >> > >
> > > > > > > >> > > > We are seeing this using OpenJPA 2.1.0 in Oozie.
> > > > > > > >> > > >
> > > > > > > >> > > > A JDBCConfigurationImpl seems to be holding a large
> > amount
> > > > of
> > > > > > > memory
> > > > > > > >> > via
> > > > > > > >> > > > its dataCacheTimeout (then listeners) instance
> variable
> > > > > keeping
> > > > > > > >> > > >
> > > > > > > >> > > > Class # Objects Used Heap
> > > > Retained
> > > > > > > Heap
> > > > > > > >> > > >
> > > > > > >
> > -------------------------------------------------------------------
> > > > > > > >> > > >
> > > > > > > >> > > > o.a.o.j.meta.MappingRepository 1,902 304,320
> > > > > > 904,410,184
> > > > > > > >> > > > o.a.o.j.meta.ClassMapping 3,824 1,223,680
> > > > > > 105,073,624
> > > > > > > >> > > >
> > > > > > >
> > -------------------------------------------------------------------
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > > We are using a single EntityManagerFactory and we are
> > > > closing
> > > > > > all
> > > > > > > >> > > > EntityManager instances correctly (via a command
> > pattern).
> > > > > > > >> > > >
> > > > > > > >> > > > All caching settings are default ones.
> > > > > > > >> > > >
> > > > > > > >> > > > Any hint where to look would be appreciated it.
> > > > > > > >> > > >
> > > > > > > >> > > > Thanks
> > > > > > > >> > > >
> > > > > > > >> > > > Alejandro
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > --
> > > > > > > >> > > *Rick Curtis*
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Rick Curtis*
> > >
> >
>
>
>
> --
> *Rick Curtis*
>
Re: OOM with JDBCConfigurationImpl holding 1+ GB of heap
Posted by Rick Curtis <cu...@gmail.com>.
Alejandro -
It appears that your application is creating EntityManagerFactories, yet
they are never getting closed.
Thanks,
Rick
On Wed, May 22, 2013 at 12:40 PM, Alejandro Abdelnur <tu...@gmail.com>wrote:
> Rick, by looking at the other thread memory analyzer image, the issues seem
> to be different.
>
>
> Thanks.
>
>
> On Mon, May 20, 2013 at 9:47 AM, Rick Curtis <cu...@gmail.com> wrote:
>
> > Alejandro --
> >
> > Is the problem that you're asking about the same as Hovan? If so, lets
> > consolidate this conversation down to a single thread.
> >
> > http://openjpa.208410.n2.nabble.com/OOM-caused-by-OpenJPA-td7583869.html
> >
> >
> http://openjpa.208410.n2.nabble.com/OOM-with-JDBCConfigurationImpl-holding-1-GB-of-heap-td7583862.html
> >
> > Thanks,
> > Rick
> >
> >
> > On Mon, May 20, 2013 at 11:38 AM, Kevin Sutter <kw...@gmail.com>
> wrote:
> >
> > > Hi Alejandro,
> > > It looks like something is holding onto references... How are you
> > > enhancing your classes? At build time? Or, some other mechanism?
> > >
> > > Does this memory leak happen with any of your configured databases?
> Or,
> > > just select ones?
> > >
> > > This shouldn't make a difference, but specifying your Entity classes
> via
> > > both <class> and the openjpa.MetaDataFactory property is overkill. The
> > > <class> elements are sufficient and is the standard, so I'd suggest
> > > sticking with that mechanism and removing the openjpa.MetaDataFactory
> > > property.
> > >
> > > We do considerable memory leak testing both as a standalone JPA
> provider
> > as
> > > well as the JPA provider for WebSphere Application Server. We could
> have
> > > missed something due to your configuration requirements, but this looks
> > to
> > > be a pretty major leak...
> > >
> > > Can you narrow down or simplify your scenario to the point of sharing a
> > > project that demonstrates the leak? Or, maybe there is memory data
> dumps
> > > that can be shared to help diagnose the issue?
> > >
> > > Kevin
> > >
> > >
> > > On Fri, May 17, 2013 at 4:50 PM, Alejandro Abdelnur <tucu00@gmail.com
> > > >wrote:
> > >
> > > > And .... I'm back.....
> > > >
> > > > It seems that removing runtime enhancement has some effect, but the
> > heap
> > > > keeps building up in the same spot.
> > > >
> > > > The following link is show where is the memory sink
> > > >
> > > > http://people.apache.org/~tucu/tmp/OpenJPALeak.png
> > > >
> > > > I'd appreciate any ideas/suggestions.
> > > >
> > > > Thanks
> > > >
> > > > Alejandro
> > > >
> > > >
> > > >
> > > > On Fri, May 10, 2013 at 10:11 AM, Kevin Sutter <kw...@gmail.com>
> > > wrote:
> > > >
> > > > > It shoudn't... But, with that flag set as "supported", we can't be
> > > sure
> > > > > that the entities were actually enhanced at build time. By setting
> > > this
> > > > > property to "unsupported" or removing it altogether, then if an
> > > > unenhanced
> > > > > entity class is loaded, we'll log a message and quit processing.
> So,
> > > > let's
> > > > > go this route first and then we'll deal with any potential memory
> > leak.
> > > > >
> > > > > Thanks, Kevin
> > > > >
> > > > >
> > > > > On Fri, May 10, 2013 at 12:05 PM, Alejandro Abdelnur <
> > tucu00@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > I'm in the process of verifying if Kevin's suggestion fixes the
> > > > problem.
> > > > > >
> > > > > > While making the necessary changes, I've notice we were already
> > > > enhancing
> > > > > > the classes at build time.
> > > > > >
> > > > > > If the classes were enhanced at build time, still would be we see
> > the
> > > > > leak
> > > > > > if the runtime enhancement flag is enabled?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > >
> > > > > > On Thu, May 9, 2013 at 9:15 AM, Alejandro Abdelnur <
> > tucu00@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks Kevin, will try that.
> > > > > > >
> > > > > > >
> > > > > > > On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter <
> kwsutter@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > >> Don't use this:
> > > > > > >>
> > > > > > >> <property name="openjpa.RuntimeUnenhancedClasses"
> > > > value="supported"/>
> > > > > > >>
> > > > > > >> That is probably your issue. Reference this page for more
> > > > > information:
> > > > > > >> http://openjpa.apache.org/entity-enhancement.html
> > > > > > >>
> > > > > > >> Good luck, Kevin
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> On Thu, May 9, 2013 at 10:17 AM, Alejandro Abdelnur <
> > > > tucu00@gmail.com
> > > > > > >> >wrote:
> > > > > > >>
> > > > > > >> > Hi Rick,
> > > > > > >> >
> > > > > > >> > Thanks for looking into this, the persistence.xml file we
> are
> > > > using
> > > > > > is:
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml
> > > > > > >> >
> > > > > > >> > The EMF is instatiated here:
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > Thanks again.
> > > > > > >> >
> > > > > > >> > Alejandro
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Thu, May 9, 2013 at 6:06 AM, Rick Curtis <
> > curtisr7@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >> >
> > > > > > >> > > Can we see the contents of your persistence.xml file?
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Wed, May 8, 2013 at 6:34 PM, Alejandro Abdelnur <
> > > > > > tucu00@gmail.com>
> > > > > > >> > > wrote:
> > > > > > >> > >
> > > > > > >> > > > We are seeing this using OpenJPA 2.1.0 in Oozie.
> > > > > > >> > > >
> > > > > > >> > > > A JDBCConfigurationImpl seems to be holding a large
> amount
> > > of
> > > > > > memory
> > > > > > >> > via
> > > > > > >> > > > its dataCacheTimeout (then listeners) instance variable
> > > > keeping
> > > > > > >> > > >
> > > > > > >> > > > Class # Objects Used Heap
> > > Retained
> > > > > > Heap
> > > > > > >> > > >
> > > > > >
> -------------------------------------------------------------------
> > > > > > >> > > >
> > > > > > >> > > > o.a.o.j.meta.MappingRepository 1,902 304,320
> > > > > 904,410,184
> > > > > > >> > > > o.a.o.j.meta.ClassMapping 3,824 1,223,680
> > > > > 105,073,624
> > > > > > >> > > >
> > > > > >
> -------------------------------------------------------------------
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > We are using a single EntityManagerFactory and we are
> > > closing
> > > > > all
> > > > > > >> > > > EntityManager instances correctly (via a command
> pattern).
> > > > > > >> > > >
> > > > > > >> > > > All caching settings are default ones.
> > > > > > >> > > >
> > > > > > >> > > > Any hint where to look would be appreciated it.
> > > > > > >> > > >
> > > > > > >> > > > Thanks
> > > > > > >> > > >
> > > > > > >> > > > Alejandro
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > --
> > > > > > >> > > *Rick Curtis*
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > *Rick Curtis*
> >
>
--
*Rick Curtis*
Re: OOM with JDBCConfigurationImpl holding 1+ GB of heap
Posted by Alejandro Abdelnur <tu...@gmail.com>.
Rick, by looking at the other thread memory analyzer image, the issues seem
to be different.
Thanks.
On Mon, May 20, 2013 at 9:47 AM, Rick Curtis <cu...@gmail.com> wrote:
> Alejandro --
>
> Is the problem that you're asking about the same as Hovan? If so, lets
> consolidate this conversation down to a single thread.
>
> http://openjpa.208410.n2.nabble.com/OOM-caused-by-OpenJPA-td7583869.html
>
> http://openjpa.208410.n2.nabble.com/OOM-with-JDBCConfigurationImpl-holding-1-GB-of-heap-td7583862.html
>
> Thanks,
> Rick
>
>
> On Mon, May 20, 2013 at 11:38 AM, Kevin Sutter <kw...@gmail.com> wrote:
>
> > Hi Alejandro,
> > It looks like something is holding onto references... How are you
> > enhancing your classes? At build time? Or, some other mechanism?
> >
> > Does this memory leak happen with any of your configured databases? Or,
> > just select ones?
> >
> > This shouldn't make a difference, but specifying your Entity classes via
> > both <class> and the openjpa.MetaDataFactory property is overkill. The
> > <class> elements are sufficient and is the standard, so I'd suggest
> > sticking with that mechanism and removing the openjpa.MetaDataFactory
> > property.
> >
> > We do considerable memory leak testing both as a standalone JPA provider
> as
> > well as the JPA provider for WebSphere Application Server. We could have
> > missed something due to your configuration requirements, but this looks
> to
> > be a pretty major leak...
> >
> > Can you narrow down or simplify your scenario to the point of sharing a
> > project that demonstrates the leak? Or, maybe there is memory data dumps
> > that can be shared to help diagnose the issue?
> >
> > Kevin
> >
> >
> > On Fri, May 17, 2013 at 4:50 PM, Alejandro Abdelnur <tucu00@gmail.com
> > >wrote:
> >
> > > And .... I'm back.....
> > >
> > > It seems that removing runtime enhancement has some effect, but the
> heap
> > > keeps building up in the same spot.
> > >
> > > The following link is show where is the memory sink
> > >
> > > http://people.apache.org/~tucu/tmp/OpenJPALeak.png
> > >
> > > I'd appreciate any ideas/suggestions.
> > >
> > > Thanks
> > >
> > > Alejandro
> > >
> > >
> > >
> > > On Fri, May 10, 2013 at 10:11 AM, Kevin Sutter <kw...@gmail.com>
> > wrote:
> > >
> > > > It shoudn't... But, with that flag set as "supported", we can't be
> > sure
> > > > that the entities were actually enhanced at build time. By setting
> > this
> > > > property to "unsupported" or removing it altogether, then if an
> > > unenhanced
> > > > entity class is loaded, we'll log a message and quit processing. So,
> > > let's
> > > > go this route first and then we'll deal with any potential memory
> leak.
> > > >
> > > > Thanks, Kevin
> > > >
> > > >
> > > > On Fri, May 10, 2013 at 12:05 PM, Alejandro Abdelnur <
> tucu00@gmail.com
> > > > >wrote:
> > > >
> > > > > I'm in the process of verifying if Kevin's suggestion fixes the
> > > problem.
> > > > >
> > > > > While making the necessary changes, I've notice we were already
> > > enhancing
> > > > > the classes at build time.
> > > > >
> > > > > If the classes were enhanced at build time, still would be we see
> the
> > > > leak
> > > > > if the runtime enhancement flag is enabled?
> > > > >
> > > > > Thanks
> > > > >
> > > > >
> > > > > On Thu, May 9, 2013 at 9:15 AM, Alejandro Abdelnur <
> tucu00@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Thanks Kevin, will try that.
> > > > > >
> > > > > >
> > > > > > On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter <kwsutter@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > >> Don't use this:
> > > > > >>
> > > > > >> <property name="openjpa.RuntimeUnenhancedClasses"
> > > value="supported"/>
> > > > > >>
> > > > > >> That is probably your issue. Reference this page for more
> > > > information:
> > > > > >> http://openjpa.apache.org/entity-enhancement.html
> > > > > >>
> > > > > >> Good luck, Kevin
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Thu, May 9, 2013 at 10:17 AM, Alejandro Abdelnur <
> > > tucu00@gmail.com
> > > > > >> >wrote:
> > > > > >>
> > > > > >> > Hi Rick,
> > > > > >> >
> > > > > >> > Thanks for looking into this, the persistence.xml file we are
> > > using
> > > > > is:
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml
> > > > > >> >
> > > > > >> > The EMF is instatiated here:
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156
> > > > > >> >
> > > > > >> >
> > > > > >> > Thanks again.
> > > > > >> >
> > > > > >> > Alejandro
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On Thu, May 9, 2013 at 6:06 AM, Rick Curtis <
> curtisr7@gmail.com
> > >
> > > > > wrote:
> > > > > >> >
> > > > > >> > > Can we see the contents of your persistence.xml file?
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Wed, May 8, 2013 at 6:34 PM, Alejandro Abdelnur <
> > > > > tucu00@gmail.com>
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > > We are seeing this using OpenJPA 2.1.0 in Oozie.
> > > > > >> > > >
> > > > > >> > > > A JDBCConfigurationImpl seems to be holding a large amount
> > of
> > > > > memory
> > > > > >> > via
> > > > > >> > > > its dataCacheTimeout (then listeners) instance variable
> > > keeping
> > > > > >> > > >
> > > > > >> > > > Class # Objects Used Heap
> > Retained
> > > > > Heap
> > > > > >> > > >
> > > > > -------------------------------------------------------------------
> > > > > >> > > >
> > > > > >> > > > o.a.o.j.meta.MappingRepository 1,902 304,320
> > > > 904,410,184
> > > > > >> > > > o.a.o.j.meta.ClassMapping 3,824 1,223,680
> > > > 105,073,624
> > > > > >> > > >
> > > > > -------------------------------------------------------------------
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > We are using a single EntityManagerFactory and we are
> > closing
> > > > all
> > > > > >> > > > EntityManager instances correctly (via a command pattern).
> > > > > >> > > >
> > > > > >> > > > All caching settings are default ones.
> > > > > >> > > >
> > > > > >> > > > Any hint where to look would be appreciated it.
> > > > > >> > > >
> > > > > >> > > > Thanks
> > > > > >> > > >
> > > > > >> > > > Alejandro
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > --
> > > > > >> > > *Rick Curtis*
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> *Rick Curtis*
>
Re: OOM with JDBCConfigurationImpl holding 1+ GB of heap
Posted by Rick Curtis <cu...@gmail.com>.
Alejandro --
Is the problem that you're asking about the same as Hovan? If so, lets
consolidate this conversation down to a single thread.
http://openjpa.208410.n2.nabble.com/OOM-caused-by-OpenJPA-td7583869.html
http://openjpa.208410.n2.nabble.com/OOM-with-JDBCConfigurationImpl-holding-1-GB-of-heap-td7583862.html
Thanks,
Rick
On Mon, May 20, 2013 at 11:38 AM, Kevin Sutter <kw...@gmail.com> wrote:
> Hi Alejandro,
> It looks like something is holding onto references... How are you
> enhancing your classes? At build time? Or, some other mechanism?
>
> Does this memory leak happen with any of your configured databases? Or,
> just select ones?
>
> This shouldn't make a difference, but specifying your Entity classes via
> both <class> and the openjpa.MetaDataFactory property is overkill. The
> <class> elements are sufficient and is the standard, so I'd suggest
> sticking with that mechanism and removing the openjpa.MetaDataFactory
> property.
>
> We do considerable memory leak testing both as a standalone JPA provider as
> well as the JPA provider for WebSphere Application Server. We could have
> missed something due to your configuration requirements, but this looks to
> be a pretty major leak...
>
> Can you narrow down or simplify your scenario to the point of sharing a
> project that demonstrates the leak? Or, maybe there is memory data dumps
> that can be shared to help diagnose the issue?
>
> Kevin
>
>
> On Fri, May 17, 2013 at 4:50 PM, Alejandro Abdelnur <tucu00@gmail.com
> >wrote:
>
> > And .... I'm back.....
> >
> > It seems that removing runtime enhancement has some effect, but the heap
> > keeps building up in the same spot.
> >
> > The following link is show where is the memory sink
> >
> > http://people.apache.org/~tucu/tmp/OpenJPALeak.png
> >
> > I'd appreciate any ideas/suggestions.
> >
> > Thanks
> >
> > Alejandro
> >
> >
> >
> > On Fri, May 10, 2013 at 10:11 AM, Kevin Sutter <kw...@gmail.com>
> wrote:
> >
> > > It shoudn't... But, with that flag set as "supported", we can't be
> sure
> > > that the entities were actually enhanced at build time. By setting
> this
> > > property to "unsupported" or removing it altogether, then if an
> > unenhanced
> > > entity class is loaded, we'll log a message and quit processing. So,
> > let's
> > > go this route first and then we'll deal with any potential memory leak.
> > >
> > > Thanks, Kevin
> > >
> > >
> > > On Fri, May 10, 2013 at 12:05 PM, Alejandro Abdelnur <tucu00@gmail.com
> > > >wrote:
> > >
> > > > I'm in the process of verifying if Kevin's suggestion fixes the
> > problem.
> > > >
> > > > While making the necessary changes, I've notice we were already
> > enhancing
> > > > the classes at build time.
> > > >
> > > > If the classes were enhanced at build time, still would be we see the
> > > leak
> > > > if the runtime enhancement flag is enabled?
> > > >
> > > > Thanks
> > > >
> > > >
> > > > On Thu, May 9, 2013 at 9:15 AM, Alejandro Abdelnur <tucu00@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Thanks Kevin, will try that.
> > > > >
> > > > >
> > > > > On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter <kw...@gmail.com>
> > > wrote:
> > > > >
> > > > >> Don't use this:
> > > > >>
> > > > >> <property name="openjpa.RuntimeUnenhancedClasses"
> > value="supported"/>
> > > > >>
> > > > >> That is probably your issue. Reference this page for more
> > > information:
> > > > >> http://openjpa.apache.org/entity-enhancement.html
> > > > >>
> > > > >> Good luck, Kevin
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Thu, May 9, 2013 at 10:17 AM, Alejandro Abdelnur <
> > tucu00@gmail.com
> > > > >> >wrote:
> > > > >>
> > > > >> > Hi Rick,
> > > > >> >
> > > > >> > Thanks for looking into this, the persistence.xml file we are
> > using
> > > > is:
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml
> > > > >> >
> > > > >> > The EMF is instatiated here:
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156
> > > > >> >
> > > > >> >
> > > > >> > Thanks again.
> > > > >> >
> > > > >> > Alejandro
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Thu, May 9, 2013 at 6:06 AM, Rick Curtis <curtisr7@gmail.com
> >
> > > > wrote:
> > > > >> >
> > > > >> > > Can we see the contents of your persistence.xml file?
> > > > >> > >
> > > > >> > >
> > > > >> > > On Wed, May 8, 2013 at 6:34 PM, Alejandro Abdelnur <
> > > > tucu00@gmail.com>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > We are seeing this using OpenJPA 2.1.0 in Oozie.
> > > > >> > > >
> > > > >> > > > A JDBCConfigurationImpl seems to be holding a large amount
> of
> > > > memory
> > > > >> > via
> > > > >> > > > its dataCacheTimeout (then listeners) instance variable
> > keeping
> > > > >> > > >
> > > > >> > > > Class # Objects Used Heap
> Retained
> > > > Heap
> > > > >> > > >
> > > > -------------------------------------------------------------------
> > > > >> > > >
> > > > >> > > > o.a.o.j.meta.MappingRepository 1,902 304,320
> > > 904,410,184
> > > > >> > > > o.a.o.j.meta.ClassMapping 3,824 1,223,680
> > > 105,073,624
> > > > >> > > >
> > > > -------------------------------------------------------------------
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > We are using a single EntityManagerFactory and we are
> closing
> > > all
> > > > >> > > > EntityManager instances correctly (via a command pattern).
> > > > >> > > >
> > > > >> > > > All caching settings are default ones.
> > > > >> > > >
> > > > >> > > > Any hint where to look would be appreciated it.
> > > > >> > > >
> > > > >> > > > Thanks
> > > > >> > > >
> > > > >> > > > Alejandro
> > > > >> > > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > --
> > > > >> > > *Rick Curtis*
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
--
*Rick Curtis*
Re: OOM with JDBCConfigurationImpl holding 1+ GB of heap
Posted by Kevin Sutter <kw...@gmail.com>.
Hi Alejandro,
It looks like something is holding onto references... How are you
enhancing your classes? At build time? Or, some other mechanism?
Does this memory leak happen with any of your configured databases? Or,
just select ones?
This shouldn't make a difference, but specifying your Entity classes via
both <class> and the openjpa.MetaDataFactory property is overkill. The
<class> elements are sufficient and is the standard, so I'd suggest
sticking with that mechanism and removing the openjpa.MetaDataFactory
property.
We do considerable memory leak testing both as a standalone JPA provider as
well as the JPA provider for WebSphere Application Server. We could have
missed something due to your configuration requirements, but this looks to
be a pretty major leak...
Can you narrow down or simplify your scenario to the point of sharing a
project that demonstrates the leak? Or, maybe there is memory data dumps
that can be shared to help diagnose the issue?
Kevin
On Fri, May 17, 2013 at 4:50 PM, Alejandro Abdelnur <tu...@gmail.com>wrote:
> And .... I'm back.....
>
> It seems that removing runtime enhancement has some effect, but the heap
> keeps building up in the same spot.
>
> The following link is show where is the memory sink
>
> http://people.apache.org/~tucu/tmp/OpenJPALeak.png
>
> I'd appreciate any ideas/suggestions.
>
> Thanks
>
> Alejandro
>
>
>
> On Fri, May 10, 2013 at 10:11 AM, Kevin Sutter <kw...@gmail.com> wrote:
>
> > It shoudn't... But, with that flag set as "supported", we can't be sure
> > that the entities were actually enhanced at build time. By setting this
> > property to "unsupported" or removing it altogether, then if an
> unenhanced
> > entity class is loaded, we'll log a message and quit processing. So,
> let's
> > go this route first and then we'll deal with any potential memory leak.
> >
> > Thanks, Kevin
> >
> >
> > On Fri, May 10, 2013 at 12:05 PM, Alejandro Abdelnur <tucu00@gmail.com
> > >wrote:
> >
> > > I'm in the process of verifying if Kevin's suggestion fixes the
> problem.
> > >
> > > While making the necessary changes, I've notice we were already
> enhancing
> > > the classes at build time.
> > >
> > > If the classes were enhanced at build time, still would be we see the
> > leak
> > > if the runtime enhancement flag is enabled?
> > >
> > > Thanks
> > >
> > >
> > > On Thu, May 9, 2013 at 9:15 AM, Alejandro Abdelnur <tu...@gmail.com>
> > > wrote:
> > >
> > > > Thanks Kevin, will try that.
> > > >
> > > >
> > > > On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter <kw...@gmail.com>
> > wrote:
> > > >
> > > >> Don't use this:
> > > >>
> > > >> <property name="openjpa.RuntimeUnenhancedClasses"
> value="supported"/>
> > > >>
> > > >> That is probably your issue. Reference this page for more
> > information:
> > > >> http://openjpa.apache.org/entity-enhancement.html
> > > >>
> > > >> Good luck, Kevin
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Thu, May 9, 2013 at 10:17 AM, Alejandro Abdelnur <
> tucu00@gmail.com
> > > >> >wrote:
> > > >>
> > > >> > Hi Rick,
> > > >> >
> > > >> > Thanks for looking into this, the persistence.xml file we are
> using
> > > is:
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml
> > > >> >
> > > >> > The EMF is instatiated here:
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156
> > > >> >
> > > >> >
> > > >> > Thanks again.
> > > >> >
> > > >> > Alejandro
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Thu, May 9, 2013 at 6:06 AM, Rick Curtis <cu...@gmail.com>
> > > wrote:
> > > >> >
> > > >> > > Can we see the contents of your persistence.xml file?
> > > >> > >
> > > >> > >
> > > >> > > On Wed, May 8, 2013 at 6:34 PM, Alejandro Abdelnur <
> > > tucu00@gmail.com>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > We are seeing this using OpenJPA 2.1.0 in Oozie.
> > > >> > > >
> > > >> > > > A JDBCConfigurationImpl seems to be holding a large amount of
> > > memory
> > > >> > via
> > > >> > > > its dataCacheTimeout (then listeners) instance variable
> keeping
> > > >> > > >
> > > >> > > > Class # Objects Used Heap Retained
> > > Heap
> > > >> > > >
> > > -------------------------------------------------------------------
> > > >> > > >
> > > >> > > > o.a.o.j.meta.MappingRepository 1,902 304,320
> > 904,410,184
> > > >> > > > o.a.o.j.meta.ClassMapping 3,824 1,223,680
> > 105,073,624
> > > >> > > >
> > > -------------------------------------------------------------------
> > > >> > > >
> > > >> > > >
> > > >> > > > We are using a single EntityManagerFactory and we are closing
> > all
> > > >> > > > EntityManager instances correctly (via a command pattern).
> > > >> > > >
> > > >> > > > All caching settings are default ones.
> > > >> > > >
> > > >> > > > Any hint where to look would be appreciated it.
> > > >> > > >
> > > >> > > > Thanks
> > > >> > > >
> > > >> > > > Alejandro
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > --
> > > >> > > *Rick Curtis*
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
Re: OOM with JDBCConfigurationImpl holding 1+ GB of heap
Posted by Alejandro Abdelnur <tu...@gmail.com>.
And .... I'm back.....
It seems that removing runtime enhancement has some effect, but the heap
keeps building up in the same spot.
The following link is show where is the memory sink
http://people.apache.org/~tucu/tmp/OpenJPALeak.png
I'd appreciate any ideas/suggestions.
Thanks
Alejandro
On Fri, May 10, 2013 at 10:11 AM, Kevin Sutter <kw...@gmail.com> wrote:
> It shoudn't... But, with that flag set as "supported", we can't be sure
> that the entities were actually enhanced at build time. By setting this
> property to "unsupported" or removing it altogether, then if an unenhanced
> entity class is loaded, we'll log a message and quit processing. So, let's
> go this route first and then we'll deal with any potential memory leak.
>
> Thanks, Kevin
>
>
> On Fri, May 10, 2013 at 12:05 PM, Alejandro Abdelnur <tucu00@gmail.com
> >wrote:
>
> > I'm in the process of verifying if Kevin's suggestion fixes the problem.
> >
> > While making the necessary changes, I've notice we were already enhancing
> > the classes at build time.
> >
> > If the classes were enhanced at build time, still would be we see the
> leak
> > if the runtime enhancement flag is enabled?
> >
> > Thanks
> >
> >
> > On Thu, May 9, 2013 at 9:15 AM, Alejandro Abdelnur <tu...@gmail.com>
> > wrote:
> >
> > > Thanks Kevin, will try that.
> > >
> > >
> > > On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter <kw...@gmail.com>
> wrote:
> > >
> > >> Don't use this:
> > >>
> > >> <property name="openjpa.RuntimeUnenhancedClasses" value="supported"/>
> > >>
> > >> That is probably your issue. Reference this page for more
> information:
> > >> http://openjpa.apache.org/entity-enhancement.html
> > >>
> > >> Good luck, Kevin
> > >>
> > >>
> > >>
> > >>
> > >> On Thu, May 9, 2013 at 10:17 AM, Alejandro Abdelnur <tucu00@gmail.com
> > >> >wrote:
> > >>
> > >> > Hi Rick,
> > >> >
> > >> > Thanks for looking into this, the persistence.xml file we are using
> > is:
> > >> >
> > >> >
> > >> >
> > >> >
> > >>
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml
> > >> >
> > >> > The EMF is instatiated here:
> > >> >
> > >> >
> > >> >
> > >> >
> > >>
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156
> > >> >
> > >> >
> > >> > Thanks again.
> > >> >
> > >> > Alejandro
> > >> >
> > >> >
> > >> >
> > >> > On Thu, May 9, 2013 at 6:06 AM, Rick Curtis <cu...@gmail.com>
> > wrote:
> > >> >
> > >> > > Can we see the contents of your persistence.xml file?
> > >> > >
> > >> > >
> > >> > > On Wed, May 8, 2013 at 6:34 PM, Alejandro Abdelnur <
> > tucu00@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > > > We are seeing this using OpenJPA 2.1.0 in Oozie.
> > >> > > >
> > >> > > > A JDBCConfigurationImpl seems to be holding a large amount of
> > memory
> > >> > via
> > >> > > > its dataCacheTimeout (then listeners) instance variable keeping
> > >> > > >
> > >> > > > Class # Objects Used Heap Retained
> > Heap
> > >> > > >
> > -------------------------------------------------------------------
> > >> > > >
> > >> > > > o.a.o.j.meta.MappingRepository 1,902 304,320
> 904,410,184
> > >> > > > o.a.o.j.meta.ClassMapping 3,824 1,223,680
> 105,073,624
> > >> > > >
> > -------------------------------------------------------------------
> > >> > > >
> > >> > > >
> > >> > > > We are using a single EntityManagerFactory and we are closing
> all
> > >> > > > EntityManager instances correctly (via a command pattern).
> > >> > > >
> > >> > > > All caching settings are default ones.
> > >> > > >
> > >> > > > Any hint where to look would be appreciated it.
> > >> > > >
> > >> > > > Thanks
> > >> > > >
> > >> > > > Alejandro
> > >> > > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > > *Rick Curtis*
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: OOM with JDBCConfigurationImpl holding 1+ GB of heap
Posted by Kevin Sutter <kw...@gmail.com>.
It shoudn't... But, with that flag set as "supported", we can't be sure
that the entities were actually enhanced at build time. By setting this
property to "unsupported" or removing it altogether, then if an unenhanced
entity class is loaded, we'll log a message and quit processing. So, let's
go this route first and then we'll deal with any potential memory leak.
Thanks, Kevin
On Fri, May 10, 2013 at 12:05 PM, Alejandro Abdelnur <tu...@gmail.com>wrote:
> I'm in the process of verifying if Kevin's suggestion fixes the problem.
>
> While making the necessary changes, I've notice we were already enhancing
> the classes at build time.
>
> If the classes were enhanced at build time, still would be we see the leak
> if the runtime enhancement flag is enabled?
>
> Thanks
>
>
> On Thu, May 9, 2013 at 9:15 AM, Alejandro Abdelnur <tu...@gmail.com>
> wrote:
>
> > Thanks Kevin, will try that.
> >
> >
> > On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter <kw...@gmail.com> wrote:
> >
> >> Don't use this:
> >>
> >> <property name="openjpa.RuntimeUnenhancedClasses" value="supported"/>
> >>
> >> That is probably your issue. Reference this page for more information:
> >> http://openjpa.apache.org/entity-enhancement.html
> >>
> >> Good luck, Kevin
> >>
> >>
> >>
> >>
> >> On Thu, May 9, 2013 at 10:17 AM, Alejandro Abdelnur <tucu00@gmail.com
> >> >wrote:
> >>
> >> > Hi Rick,
> >> >
> >> > Thanks for looking into this, the persistence.xml file we are using
> is:
> >> >
> >> >
> >> >
> >> >
> >>
> https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml
> >> >
> >> > The EMF is instatiated here:
> >> >
> >> >
> >> >
> >> >
> >>
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156
> >> >
> >> >
> >> > Thanks again.
> >> >
> >> > Alejandro
> >> >
> >> >
> >> >
> >> > On Thu, May 9, 2013 at 6:06 AM, Rick Curtis <cu...@gmail.com>
> wrote:
> >> >
> >> > > Can we see the contents of your persistence.xml file?
> >> > >
> >> > >
> >> > > On Wed, May 8, 2013 at 6:34 PM, Alejandro Abdelnur <
> tucu00@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > We are seeing this using OpenJPA 2.1.0 in Oozie.
> >> > > >
> >> > > > A JDBCConfigurationImpl seems to be holding a large amount of
> memory
> >> > via
> >> > > > its dataCacheTimeout (then listeners) instance variable keeping
> >> > > >
> >> > > > Class # Objects Used Heap Retained
> Heap
> >> > > >
> -------------------------------------------------------------------
> >> > > >
> >> > > > o.a.o.j.meta.MappingRepository 1,902 304,320 904,410,184
> >> > > > o.a.o.j.meta.ClassMapping 3,824 1,223,680 105,073,624
> >> > > >
> -------------------------------------------------------------------
> >> > > >
> >> > > >
> >> > > > We are using a single EntityManagerFactory and we are closing all
> >> > > > EntityManager instances correctly (via a command pattern).
> >> > > >
> >> > > > All caching settings are default ones.
> >> > > >
> >> > > > Any hint where to look would be appreciated it.
> >> > > >
> >> > > > Thanks
> >> > > >
> >> > > > Alejandro
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > *Rick Curtis*
> >> > >
> >> >
> >>
> >
> >
>
Re: OOM with JDBCConfigurationImpl holding 1+ GB of heap
Posted by Alejandro Abdelnur <tu...@gmail.com>.
I'm in the process of verifying if Kevin's suggestion fixes the problem.
While making the necessary changes, I've notice we were already enhancing
the classes at build time.
If the classes were enhanced at build time, still would be we see the leak
if the runtime enhancement flag is enabled?
Thanks
On Thu, May 9, 2013 at 9:15 AM, Alejandro Abdelnur <tu...@gmail.com> wrote:
> Thanks Kevin, will try that.
>
>
> On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter <kw...@gmail.com> wrote:
>
>> Don't use this:
>>
>> <property name="openjpa.RuntimeUnenhancedClasses" value="supported"/>
>>
>> That is probably your issue. Reference this page for more information:
>> http://openjpa.apache.org/entity-enhancement.html
>>
>> Good luck, Kevin
>>
>>
>>
>>
>> On Thu, May 9, 2013 at 10:17 AM, Alejandro Abdelnur <tucu00@gmail.com
>> >wrote:
>>
>> > Hi Rick,
>> >
>> > Thanks for looking into this, the persistence.xml file we are using is:
>> >
>> >
>> >
>> >
>> https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml
>> >
>> > The EMF is instatiated here:
>> >
>> >
>> >
>> >
>> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156
>> >
>> >
>> > Thanks again.
>> >
>> > Alejandro
>> >
>> >
>> >
>> > On Thu, May 9, 2013 at 6:06 AM, Rick Curtis <cu...@gmail.com> wrote:
>> >
>> > > Can we see the contents of your persistence.xml file?
>> > >
>> > >
>> > > On Wed, May 8, 2013 at 6:34 PM, Alejandro Abdelnur <tu...@gmail.com>
>> > > wrote:
>> > >
>> > > > We are seeing this using OpenJPA 2.1.0 in Oozie.
>> > > >
>> > > > A JDBCConfigurationImpl seems to be holding a large amount of memory
>> > via
>> > > > its dataCacheTimeout (then listeners) instance variable keeping
>> > > >
>> > > > Class # Objects Used Heap Retained Heap
>> > > > -------------------------------------------------------------------
>> > > >
>> > > > o.a.o.j.meta.MappingRepository 1,902 304,320 904,410,184
>> > > > o.a.o.j.meta.ClassMapping 3,824 1,223,680 105,073,624
>> > > > -------------------------------------------------------------------
>> > > >
>> > > >
>> > > > We are using a single EntityManagerFactory and we are closing all
>> > > > EntityManager instances correctly (via a command pattern).
>> > > >
>> > > > All caching settings are default ones.
>> > > >
>> > > > Any hint where to look would be appreciated it.
>> > > >
>> > > > Thanks
>> > > >
>> > > > Alejandro
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > *Rick Curtis*
>> > >
>> >
>>
>
>
Re: OOM with JDBCConfigurationImpl holding 1+ GB of heap
Posted by Alejandro Abdelnur <tu...@gmail.com>.
Thanks Kevin, will try that.
On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter <kw...@gmail.com> wrote:
> Don't use this:
>
> <property name="openjpa.RuntimeUnenhancedClasses" value="supported"/>
>
> That is probably your issue. Reference this page for more information:
> http://openjpa.apache.org/entity-enhancement.html
>
> Good luck, Kevin
>
>
>
>
> On Thu, May 9, 2013 at 10:17 AM, Alejandro Abdelnur <tucu00@gmail.com
> >wrote:
>
> > Hi Rick,
> >
> > Thanks for looking into this, the persistence.xml file we are using is:
> >
> >
> >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml
> >
> > The EMF is instatiated here:
> >
> >
> >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156
> >
> >
> > Thanks again.
> >
> > Alejandro
> >
> >
> >
> > On Thu, May 9, 2013 at 6:06 AM, Rick Curtis <cu...@gmail.com> wrote:
> >
> > > Can we see the contents of your persistence.xml file?
> > >
> > >
> > > On Wed, May 8, 2013 at 6:34 PM, Alejandro Abdelnur <tu...@gmail.com>
> > > wrote:
> > >
> > > > We are seeing this using OpenJPA 2.1.0 in Oozie.
> > > >
> > > > A JDBCConfigurationImpl seems to be holding a large amount of memory
> > via
> > > > its dataCacheTimeout (then listeners) instance variable keeping
> > > >
> > > > Class # Objects Used Heap Retained Heap
> > > > -------------------------------------------------------------------
> > > >
> > > > o.a.o.j.meta.MappingRepository 1,902 304,320 904,410,184
> > > > o.a.o.j.meta.ClassMapping 3,824 1,223,680 105,073,624
> > > > -------------------------------------------------------------------
> > > >
> > > >
> > > > We are using a single EntityManagerFactory and we are closing all
> > > > EntityManager instances correctly (via a command pattern).
> > > >
> > > > All caching settings are default ones.
> > > >
> > > > Any hint where to look would be appreciated it.
> > > >
> > > > Thanks
> > > >
> > > > Alejandro
> > > >
> > >
> > >
> > >
> > > --
> > > *Rick Curtis*
> > >
> >
>
Re: OOM with JDBCConfigurationImpl holding 1+ GB of heap
Posted by Kevin Sutter <kw...@gmail.com>.
Don't use this:
<property name="openjpa.RuntimeUnenhancedClasses" value="supported"/>
That is probably your issue. Reference this page for more information:
http://openjpa.apache.org/entity-enhancement.html
Good luck, Kevin
On Thu, May 9, 2013 at 10:17 AM, Alejandro Abdelnur <tu...@gmail.com>wrote:
> Hi Rick,
>
> Thanks for looking into this, the persistence.xml file we are using is:
>
>
>
> https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml
>
> The EMF is instatiated here:
>
>
>
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156
>
>
> Thanks again.
>
> Alejandro
>
>
>
> On Thu, May 9, 2013 at 6:06 AM, Rick Curtis <cu...@gmail.com> wrote:
>
> > Can we see the contents of your persistence.xml file?
> >
> >
> > On Wed, May 8, 2013 at 6:34 PM, Alejandro Abdelnur <tu...@gmail.com>
> > wrote:
> >
> > > We are seeing this using OpenJPA 2.1.0 in Oozie.
> > >
> > > A JDBCConfigurationImpl seems to be holding a large amount of memory
> via
> > > its dataCacheTimeout (then listeners) instance variable keeping
> > >
> > > Class # Objects Used Heap Retained Heap
> > > -------------------------------------------------------------------
> > >
> > > o.a.o.j.meta.MappingRepository 1,902 304,320 904,410,184
> > > o.a.o.j.meta.ClassMapping 3,824 1,223,680 105,073,624
> > > -------------------------------------------------------------------
> > >
> > >
> > > We are using a single EntityManagerFactory and we are closing all
> > > EntityManager instances correctly (via a command pattern).
> > >
> > > All caching settings are default ones.
> > >
> > > Any hint where to look would be appreciated it.
> > >
> > > Thanks
> > >
> > > Alejandro
> > >
> >
> >
> >
> > --
> > *Rick Curtis*
> >
>
Re: OOM with JDBCConfigurationImpl holding 1+ GB of heap
Posted by Alejandro Abdelnur <tu...@gmail.com>.
Hi Rick,
Thanks for looking into this, the persistence.xml file we are using is:
https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml
The EMF is instatiated here:
https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156
Thanks again.
Alejandro
On Thu, May 9, 2013 at 6:06 AM, Rick Curtis <cu...@gmail.com> wrote:
> Can we see the contents of your persistence.xml file?
>
>
> On Wed, May 8, 2013 at 6:34 PM, Alejandro Abdelnur <tu...@gmail.com>
> wrote:
>
> > We are seeing this using OpenJPA 2.1.0 in Oozie.
> >
> > A JDBCConfigurationImpl seems to be holding a large amount of memory via
> > its dataCacheTimeout (then listeners) instance variable keeping
> >
> > Class # Objects Used Heap Retained Heap
> > -------------------------------------------------------------------
> >
> > o.a.o.j.meta.MappingRepository 1,902 304,320 904,410,184
> > o.a.o.j.meta.ClassMapping 3,824 1,223,680 105,073,624
> > -------------------------------------------------------------------
> >
> >
> > We are using a single EntityManagerFactory and we are closing all
> > EntityManager instances correctly (via a command pattern).
> >
> > All caching settings are default ones.
> >
> > Any hint where to look would be appreciated it.
> >
> > Thanks
> >
> > Alejandro
> >
>
>
>
> --
> *Rick Curtis*
>
Re: OOM with JDBCConfigurationImpl holding 1+ GB of heap
Posted by Rick Curtis <cu...@gmail.com>.
Can we see the contents of your persistence.xml file?
On Wed, May 8, 2013 at 6:34 PM, Alejandro Abdelnur <tu...@gmail.com> wrote:
> We are seeing this using OpenJPA 2.1.0 in Oozie.
>
> A JDBCConfigurationImpl seems to be holding a large amount of memory via
> its dataCacheTimeout (then listeners) instance variable keeping
>
> Class # Objects Used Heap Retained Heap
> -------------------------------------------------------------------
>
> o.a.o.j.meta.MappingRepository 1,902 304,320 904,410,184
> o.a.o.j.meta.ClassMapping 3,824 1,223,680 105,073,624
> -------------------------------------------------------------------
>
>
> We are using a single EntityManagerFactory and we are closing all
> EntityManager instances correctly (via a command pattern).
>
> All caching settings are default ones.
>
> Any hint where to look would be appreciated it.
>
> Thanks
>
> Alejandro
>
--
*Rick Curtis*