You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@openjpa.apache.org by Darren Senel <da...@gmail.com> on 2009/06/27 17:54:09 UTC

Memory Usage

Hello,

We are using OpenJPA 1.2.0 in a WebSphere 6 web application.  We are not
using the EJB3 pack.  We are seeing a very large proportion of the heap
taken up by OpenJPA PCRegistry objects (as shown in image).  We use compile
time enhancement of our classes.  Eventually with enough requests of under
high load our servers fail with an OOM error as the heap is exhausted.  The
image below shows the heap analysis suspected leak subject.  Also, this heap
dump was taken when the heap was about 50% full.

We open, use, flush, close and set to null the EntityManager objects we
create.  

Would anyone have any ideas as to the cause of this growth of objects that
eventually leads to our OOM crash?

Many thanks,

Darren

http://n2.nabble.com/file/n3166851/HeapAnalysis.jpg 
-- 
View this message in context: http://n2.nabble.com/Memory-Usage-tp3166851p3166851.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.


Re: Memory Usage

Posted by Darren Senel <da...@gmail.com>.
Hello again,

We are still trying to trouble shoot our memory leak.  It seems that other
users have had the same issue.

https://issues.apache.org/jira/browse/OPENJPA-746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
JIRA-746 

Are there any updates on this issue that could help us resolve our problem.

Thanks in advance,

Darren


Pinaki Poddar wrote:
> 
> Hi,
>   1. JProfiler documentation provides useful guidelines to ascertain a
> memory leak and narrow its source [1]. Some of these guidelines may help
> to identify the cause. Our first (and possibly misguided) assumption was
> PCRegistry -- but double-checking that assumption is required.
> 
>   2. why does java.beans.BeanInfo appear in the memory heap profile?
> OpenJPA configuration elements implement BeanInfo interface, but the
> presented memory profile is about the persistent classes. Is any of the
> entities implementing/using BeanInfo?
> 
>   3. Can it be something specific to the data itself? You mentioned eager
> fetching of one-to-many relation from BaseContent to RepeatedAttribute and
> eager? Can cardinality of that relation grow huge under long running
> operations? 
> 
>   4. Somewhat off topic: you mentioned that the sessions are read-only.
> How is the application taking advantage of that fact.
> 
> [1]
> http://resources.ej-technologies.com/jprofiler/help/doc/helptopics/memory/memoryLeak.html
>   
> 
> Darren Senel wrote:
>> 
>> Good morning Pinaki,
>> 
>> So our site spider had the effect we were looking for and out of our
>> three servers 1 crashed with an OOM, the other two survived but heap
>> usage is around 95% on them!  I ran the debug code on these two servers
>> and the results are that there was only 13 persistent classes listed with
>> the same classloader.
>> 
>> Here is the output from server for your validation:
>> 
>> [6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
>> com.pwc.us.wcps.dto.BaseContentDTO loaded with classloader:
>> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>>    Delegation Mode: PARENT_LAST
>> [6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
>> com.pwc.us.wcps.dto.lists.RepeatedAttribute loaded with classloader:
>> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>>    Delegation Mode: PARENT_LAST
>> [6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
>> com.pwc.us.wcps.dto.DynamicCollectionDTO loaded with classloader:
>> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>>    Delegation Mode: PARENT_LAST
>> [6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
>> com.pwc.us.wcps.dto.PrimaryContentDTO loaded with classloader:
>> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>>    Delegation Mode: PARENT_LAST
>> [6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
>> com.pwc.us.wcps.dto.FriendlyURLDTO loaded with classloader:
>> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>>    Delegation Mode: PARENT_LAST
>> [6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
>> com.pwc.us.wcps.translations.TranslationDTO loaded with classloader:
>> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>>    Delegation Mode: PARENT_LAST
>> [6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
>> com.pwc.us.wcps.translations.VTranslationDTO loaded with classloader:
>> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>>    Delegation Mode: PARENT_LAST
>> [6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
>> com.pwc.us.wcps.dto.ConfigDTO loaded with classloader:
>> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>>    Delegation Mode: PARENT_LAST
>> [6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
>> com.pwc.us.wcps.dto.MigrationMasterDTO loaded with classloader:
>> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>>    Delegation Mode: PARENT_LAST
>> [6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
>> com.pwc.us.wcps.translations.SubgroupDTO loaded with classloader:
>> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>>    Delegation Mode: PARENT_LAST
>> [6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
>> com.pwc.us.wcps.translations.GroupDTO loaded with classloader:
>> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>>    Delegation Mode: PARENT_LAST
>> [6/30/09 8:06:52:538 EDT] 000022f1 SystemOut     O
>> com.pwc.us.wcps.taxonomy.TaxonomyDTO loaded with classloader:
>> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>>    Delegation Mode: PARENT_LAST
>> [6/30/09 8:06:52:538 EDT] 000022f1 SystemOut     O
>> com.pwc.us.wcps.dto.TerritoryDTO loaded with classloader:
>> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>>    Delegation Mode: PARENT_LAST
>> 
>> I am assuming now we have to look at an alternative root cause?
>> 
>> Any ideas where to start looking?
>> 
>> Cheers
>> 
>> Darren
>> 
>> 
>> Darren Senel wrote:
>>> 
>>> Hi Pinaki,
>>> 
>>> I can output the debug lines on demand, so I will wait until memory is
>>> constrained and issue the command.  I'll let you know if there are more
>>> than 13 persistent classes registered and if there are more than one
>>> classloader displayed.
>>> 
>>> We have a bot spider the site at around 5:00pm today, which expedites
>>> the issue, so I should be able to let you know later today or tomorrow
>>> the results of this test.
>>> 
>>> Cheers
>>> 
>>> Darren
>>> 
>> 
>> 
> 
> 

-- 
View this message in context: http://n2.nabble.com/Memory-Usage-tp3166851p3231862.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.

Re: Memory Usage

Posted by Pinaki Poddar <pp...@apache.org>.
Hi,
  1. JProfiler documentation provides useful guidelines to ascertain a
memory leak and narrow its source [1]. Some of these guidelines may help to
identify the cause. Our first (and possibly misguided) assumption was
PCRegistry -- but double-checking that assumption is required.

  2. why does java.beans.BeanInfo appear in the memory heap profile? OpenJPA
configuration elements implement BeanInfo interface, but the presented
memory profile is about the persistent classes. Is any of the entities
implementing/using BeanInfo?

  3. Can it be something specific to the data itself? You mentioned eager
fetching of one-to-many relation from BaseContent to RepeatedAttribute and
eager? Can cardinality of that relation grow huge under long running
operations? 

  4. Somewhat off topic: you mentioned that the sessions are read-only. How
is the application taking advantage of that fact.

[1]
http://resources.ej-technologies.com/jprofiler/help/doc/helptopics/memory/memoryLeak.html
  

Darren Senel wrote:
> 
> Good morning Pinaki,
> 
> So our site spider had the effect we were looking for and out of our three
> servers 1 crashed with an OOM, the other two survived but heap usage is
> around 95% on them!  I ran the debug code on these two servers and the
> results are that there was only 13 persistent classes listed with the same
> classloader.
> 
> Here is the output from server for your validation:
> 
> [6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
> com.pwc.us.wcps.dto.BaseContentDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>    Delegation Mode: PARENT_LAST
> [6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
> com.pwc.us.wcps.dto.lists.RepeatedAttribute loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>    Delegation Mode: PARENT_LAST
> [6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
> com.pwc.us.wcps.dto.DynamicCollectionDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>    Delegation Mode: PARENT_LAST
> [6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
> com.pwc.us.wcps.dto.PrimaryContentDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>    Delegation Mode: PARENT_LAST
> [6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
> com.pwc.us.wcps.dto.FriendlyURLDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>    Delegation Mode: PARENT_LAST
> [6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
> com.pwc.us.wcps.translations.TranslationDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>    Delegation Mode: PARENT_LAST
> [6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
> com.pwc.us.wcps.translations.VTranslationDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>    Delegation Mode: PARENT_LAST
> [6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
> com.pwc.us.wcps.dto.ConfigDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>    Delegation Mode: PARENT_LAST
> [6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
> com.pwc.us.wcps.dto.MigrationMasterDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>    Delegation Mode: PARENT_LAST
> [6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
> com.pwc.us.wcps.translations.SubgroupDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>    Delegation Mode: PARENT_LAST
> [6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
> com.pwc.us.wcps.translations.GroupDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>    Delegation Mode: PARENT_LAST
> [6/30/09 8:06:52:538 EDT] 000022f1 SystemOut     O
> com.pwc.us.wcps.taxonomy.TaxonomyDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>    Delegation Mode: PARENT_LAST
> [6/30/09 8:06:52:538 EDT] 000022f1 SystemOut     O
> com.pwc.us.wcps.dto.TerritoryDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@714a714a
>    Delegation Mode: PARENT_LAST
> 
> I am assuming now we have to look at an alternative root cause?
> 
> Any ideas where to start looking?
> 
> Cheers
> 
> Darren
> 
> 
> Darren Senel wrote:
>> 
>> Hi Pinaki,
>> 
>> I can output the debug lines on demand, so I will wait until memory is
>> constrained and issue the command.  I'll let you know if there are more
>> than 13 persistent classes registered and if there are more than one
>> classloader displayed.
>> 
>> We have a bot spider the site at around 5:00pm today, which expedites the
>> issue, so I should be able to let you know later today or tomorrow the
>> results of this test.
>> 
>> Cheers
>> 
>> Darren
>> 
> 
> 


-----
Pinaki 
-- 
View this message in context: http://n2.nabble.com/Memory-Usage-tp3166851p3183133.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.


Re: Memory Usage

Posted by Darren Senel <da...@gmail.com>.
Good morning Pinaki,

So our site spider had the effect we were looking for and out of our three
servers 1 crashed with an OOM, the other two survived but heap usage is
around 95% on them!  I ran the debug code on these two servers and the
results are that there was only 13 persistent classes listed with the same
classloader.

Here is the output from server for your validation:

[6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
com.pwc.us.wcps.dto.BaseContentDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@714a714a
   Delegation Mode: PARENT_LAST
[6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
com.pwc.us.wcps.dto.lists.RepeatedAttribute loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@714a714a
   Delegation Mode: PARENT_LAST
[6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
com.pwc.us.wcps.dto.DynamicCollectionDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@714a714a
   Delegation Mode: PARENT_LAST
[6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
com.pwc.us.wcps.dto.PrimaryContentDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@714a714a
   Delegation Mode: PARENT_LAST
[6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
com.pwc.us.wcps.dto.FriendlyURLDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@714a714a
   Delegation Mode: PARENT_LAST
[6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
com.pwc.us.wcps.translations.TranslationDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@714a714a
   Delegation Mode: PARENT_LAST
[6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
com.pwc.us.wcps.translations.VTranslationDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@714a714a
   Delegation Mode: PARENT_LAST
[6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
com.pwc.us.wcps.dto.ConfigDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@714a714a
   Delegation Mode: PARENT_LAST
[6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
com.pwc.us.wcps.dto.MigrationMasterDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@714a714a
   Delegation Mode: PARENT_LAST
[6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
com.pwc.us.wcps.translations.SubgroupDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@714a714a
   Delegation Mode: PARENT_LAST
[6/30/09 8:06:52:523 EDT] 000022f1 SystemOut     O
com.pwc.us.wcps.translations.GroupDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@714a714a
   Delegation Mode: PARENT_LAST
[6/30/09 8:06:52:538 EDT] 000022f1 SystemOut     O
com.pwc.us.wcps.taxonomy.TaxonomyDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@714a714a
   Delegation Mode: PARENT_LAST
[6/30/09 8:06:52:538 EDT] 000022f1 SystemOut     O
com.pwc.us.wcps.dto.TerritoryDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@714a714a
   Delegation Mode: PARENT_LAST

I am assuming now we have to look at an alternative root cause?

Any ideas where to start looking?

Cheers

Darren


Darren Senel wrote:
> 
> Hi Pinaki,
> 
> I can output the debug lines on demand, so I will wait until memory is
> constrained and issue the command.  I'll let you know if there are more
> than 13 persistent classes registered and if there are more than one
> classloader displayed.
> 
> We have a bot spider the site at around 5:00pm today, which expedites the
> issue, so I should be able to let you know later today or tomorrow the
> results of this test.
> 
> Cheers
> 
> Darren
> 

-- 
View this message in context: http://n2.nabble.com/Memory-Usage-tp3166851p3182164.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.


Re: Memory Usage

Posted by Darren Senel <da...@gmail.com>.
Hi Pinaki,

I can output the debug lines on demand, so I will wait until memory is
constrained and issue the command.  I'll let you know if there are more than
13 persistent classes registered and if there are more than one classloader
displayed.

We have a bot spider the site at around 5:00pm today, which expedites the
issue, so I should be able to let you know later today or tomorrow the
results of this test.

Cheers

Darren
-- 
View this message in context: http://n2.nabble.com/Memory-Usage-tp3166851p3176862.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.


Re: Memory Usage

Posted by Pinaki Poddar <pp...@apache.org>.
Hi Darren,
  So far the output looks what is expected.
  Instead of printing at regular interval, perhaps you can report only when
PCRegistry.getRegisteredTypes().size() gets larger than N where N is the
number of classes in persistence.xml? 
  If that trigger is never fired but still goes into OOM -- we have to look
somewhere else...



Darren Senel wrote:
> 
> Hello Pinaki,
> 
> Here is the first snapshot of the debug info from our Production
> environment.  I will post some more at various intervals.
> 
> [6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
> com.pwc.us.wcps.translations.TranslationDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@30e030e0
>    Delegation Mode: PARENT_LAST
> [6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
> com.pwc.us.wcps.dto.ConfigDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@30e030e0
>    Delegation Mode: PARENT_LAST
> [6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
> com.pwc.us.wcps.dto.lists.RepeatedAttribute loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@30e030e0
>    Delegation Mode: PARENT_LAST
> [6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
> com.pwc.us.wcps.dto.DynamicCollectionDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@30e030e0
>    Delegation Mode: PARENT_LAST
> [6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
> com.pwc.us.wcps.dto.PrimaryContentDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@30e030e0
>    Delegation Mode: PARENT_LAST
> [6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
> com.pwc.us.wcps.translations.SubgroupDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@30e030e0
>    Delegation Mode: PARENT_LAST
> [6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
> com.pwc.us.wcps.translations.VTranslationDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@30e030e0
>    Delegation Mode: PARENT_LAST
> [6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
> com.pwc.us.wcps.translations.GroupDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@30e030e0
>    Delegation Mode: PARENT_LAST
> [6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
> com.pwc.us.wcps.dto.BaseContentDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@30e030e0
>    Delegation Mode: PARENT_LAST
> [6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
> com.pwc.us.wcps.dto.FriendlyURLDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@30e030e0
>    Delegation Mode: PARENT_LAST
> [6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
> com.pwc.us.wcps.taxonomy.TaxonomyDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@30e030e0
>    Delegation Mode: PARENT_LAST
> [6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
> com.pwc.us.wcps.dto.TerritoryDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@30e030e0
>    Delegation Mode: PARENT_LAST
> [6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
> com.pwc.us.wcps.dto.MigrationMasterDTO loaded with classloader:
> com.ibm.ws.classloader.CompoundClassLoader@30e030e0
>    Delegation Mode: PARENT_LAST
> 
> Cheers
> 
> Darren
> 
> 


-----
Pinaki 
-- 
View this message in context: http://n2.nabble.com/Memory-Usage-tp3166851p3176743.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.


Re: Memory Usage

Posted by Darren Senel <da...@gmail.com>.
Hello Pinaki,

Here is the first snapshot of the debug info from our Production
environment.  I will post some more at various intervals.

[6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
com.pwc.us.wcps.translations.TranslationDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@30e030e0
   Delegation Mode: PARENT_LAST
[6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
com.pwc.us.wcps.dto.ConfigDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@30e030e0
   Delegation Mode: PARENT_LAST
[6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
com.pwc.us.wcps.dto.lists.RepeatedAttribute loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@30e030e0
   Delegation Mode: PARENT_LAST
[6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
com.pwc.us.wcps.dto.DynamicCollectionDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@30e030e0
   Delegation Mode: PARENT_LAST
[6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
com.pwc.us.wcps.dto.PrimaryContentDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@30e030e0
   Delegation Mode: PARENT_LAST
[6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
com.pwc.us.wcps.translations.SubgroupDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@30e030e0
   Delegation Mode: PARENT_LAST
[6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
com.pwc.us.wcps.translations.VTranslationDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@30e030e0
   Delegation Mode: PARENT_LAST
[6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
com.pwc.us.wcps.translations.GroupDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@30e030e0
   Delegation Mode: PARENT_LAST
[6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
com.pwc.us.wcps.dto.BaseContentDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@30e030e0
   Delegation Mode: PARENT_LAST
[6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
com.pwc.us.wcps.dto.FriendlyURLDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@30e030e0
   Delegation Mode: PARENT_LAST
[6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
com.pwc.us.wcps.taxonomy.TaxonomyDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@30e030e0
   Delegation Mode: PARENT_LAST
[6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
com.pwc.us.wcps.dto.TerritoryDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@30e030e0
   Delegation Mode: PARENT_LAST
[6/29/09 15:09:37:444 EDT] 00000589 SystemOut     O
com.pwc.us.wcps.dto.MigrationMasterDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@30e030e0
   Delegation Mode: PARENT_LAST

Cheers

Darren

-- 
View this message in context: http://n2.nabble.com/Memory-Usage-tp3166851p3176469.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.


Re: Memory Usage

Posted by Darren Senel <da...@gmail.com>.
Hello Pinaki,

Ok, I have added the code and currently have the application deployed to our
DEV environment.  I will deploy to our loaded environment a little later,
but I wanted to see if I am producing the data you require to remotely
debug.

Here is the output of the debug code.  I have hit it a few times at
different intervals and it seems to be outputing the same values each time.

[6/29/09 13:01:54:691 EDT] 00000051 SystemOut     O
com.pwc.us.wcps.translations.GroupDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@300c300c
   Delegation Mode: PARENT_FIRST
[6/29/09 13:01:54:691 EDT] 00000051 SystemOut     O
com.pwc.us.wcps.dto.ConfigDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@300c300c
   Delegation Mode: PARENT_FIRST
[6/29/09 13:01:54:691 EDT] 00000051 SystemOut     O
com.pwc.us.wcps.translations.SubgroupDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@300c300c
   Delegation Mode: PARENT_FIRST
[6/29/09 13:01:54:691 EDT] 00000051 SystemOut     O
com.pwc.us.wcps.dto.lists.RepeatedAttribute loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@300c300c
   Delegation Mode: PARENT_FIRST
[6/29/09 13:01:54:691 EDT] 00000051 SystemOut     O
com.pwc.us.wcps.dto.TerritoryDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@300c300c
   Delegation Mode: PARENT_FIRST
[6/29/09 13:01:54:691 EDT] 00000051 SystemOut     O
com.pwc.us.wcps.translations.VTranslationDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@300c300c
   Delegation Mode: PARENT_FIRST
[6/29/09 13:01:54:691 EDT] 00000051 SystemOut     O
com.pwc.us.wcps.translations.TranslationDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@300c300c
   Delegation Mode: PARENT_FIRST
[6/29/09 13:01:54:691 EDT] 00000051 SystemOut     O
com.pwc.us.wcps.dto.BaseContentDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@300c300c
   Delegation Mode: PARENT_FIRST
[6/29/09 13:01:54:707 EDT] 00000051 SystemOut     O
com.pwc.us.wcps.dto.PrimaryContentDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@300c300c
   Delegation Mode: PARENT_FIRST
[6/29/09 13:01:54:707 EDT] 00000051 SystemOut     O
com.pwc.us.wcps.dto.FriendlyURLDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@300c300c
   Delegation Mode: PARENT_FIRST
[6/29/09 13:01:54:707 EDT] 00000051 SystemOut     O
com.pwc.us.wcps.dto.DynamicCollectionDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@300c300c
   Delegation Mode: PARENT_FIRST
[6/29/09 13:01:54:707 EDT] 00000051 SystemOut     O
com.pwc.us.wcps.taxonomy.TaxonomyDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@300c300c
   Delegation Mode: PARENT_FIRST
[6/29/09 13:01:54:707 EDT] 00000051 SystemOut     O
com.pwc.us.wcps.dto.MigrationMasterDTO loaded with classloader:
com.ibm.ws.classloader.CompoundClassLoader@300c300c
   Delegation Mode: PARENT_FIRST

I have excluded the classpath output for each of these lines to shorten the
data in this post, but its identical for each entry:

   Local ClassPath: C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\classes;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\backport-util-concurrent.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\commons-beanutils-bean-collections.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\commons-beanutils.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\commons-collections-3.2.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\commons-digester-1.8.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\commons-logging-1.1.1.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\CountryHawk4j.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\dynacache.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\ehcache-1.5.0.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\javaee.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\junit-4.5.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\junit-dep-4.5.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\Krypt.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\log4j-1.2.14.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\ojdbc14.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\openjpa-1.2.0.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\PwC-Core.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\PWCLogging.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\serp-1.13.1.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\velocity-1.6-dep.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\velocity-tools-1.4.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\velocity-tools-generic-1.4.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\WCPS_base_library_1.0.0.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war\WEB-INF\lib\xstream-1.1.3.jar;C:\Program
Files\IBM\SDP70\runtimes\base_v61\profiles\AppSrv01\installedApps\DSENEL00240Node01Cell\WCPS.ear\WCPS_WIP.war


Please let me know if this is what you are after and I will run it in our
environment with higher load.

Cheers,

Darren

-- 
View this message in context: http://n2.nabble.com/Memory-Usage-tp3166851p3175812.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.


Re: Memory Usage

Posted by Pinaki Poddar <pp...@apache.org>.
Hi,
  Help us to debug it remotely. Call the following method periodically from
your application
  public static Collection PCRegistry.getRegisteredTypes();

  The returned collection elements are persistent domain classes. Print them
with their classloaders.
  


-----
Pinaki 
-- 
View this message in context: http://n2.nabble.com/Memory-Usage-tp3166851p3175298.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.


Re: Memory Usage

Posted by Darren Senel <da...@gmail.com>.
Hello Pinaki,

We declare the persistent domain classes in persistence.xml.  Here is our
persistence.xml content:

<persistence xmlns="http://java.sun.com/xml/ns/persistence" version="1.0">
  <persistence-unit name="active">
   
<provider>org.apache.openjpa.persistence.PersistenceProviderImpl</provider>  
    <jta-data-source>com.pwc.us.wcps.ds.WCPS_ACTIVE_DS</jta-data-source> 
   
<non-jta-data-source>com.pwc.us.wcps.ds.WCPS_ACTIVE_DS</non-jta-data-source>    
  	<class>com.pwc.us.wcps.dto.BaseContentDTO</class>	
  	<class>com.pwc.us.wcps.dto.ConfigDTO</class>	  	
  	<class>com.pwc.us.wcps.dto.PrimaryContentDTO</class>  
  	<class>com.pwc.us.wcps.dto.lists.RepeatedAttribute</class>
  	<class>com.pwc.us.wcps.dto.DynamicCollectionDTO</class>
  	<class>com.pwc.us.wcps.translations.VTranslationDTO</class>	    
  	<class>com.pwc.us.wcps.translations.TranslationDTO</class>	
  	<class>com.pwc.us.wcps.translations.SubgroupDTO</class>	
  	<class>com.pwc.us.wcps.translations.GroupDTO</class>
  	<class>com.pwc.us.wcps.dto.TerritoryDTO</class>	  
  	<class>com.pwc.us.wcps.dto.FriendlyURLDTO</class>	
  	<class>com.pwc.us.wcps.taxonomy.TaxonomyDTO</class>	 
  	<class>com.pwc.us.wcps.dto.MigrationMasterDTO</class>
    <properties>
	  <property name="openjpa.TransactionMode" value="managed"/>
	  <property name="openjpa.ConnectionFactoryMode" value="managed"/>      
      <property name="openjpa.jdbc.DBDictionary"
value="org.apache.openjpa.jdbc.sql.OracleDictionary"/>
      <property name="openjpa.Log" value="DefaultLevel=ERROR, Tool=FATAL,
SQL=FATAL"/>
      <property name="openjpa.jdbc.QuerySQLCache" value="false"/>
    </properties>
  </persistence-unit>    
</persistence>

For each request we create an instance of one of the persistent classes as
follows:

PrimaryContentDTO primaryContent = new PrimaryContentDTO();

And then we load from the database using a primary key:

                emf = WCPSUtils.getEMF();
		
                // Set primary key
		BaseContentDTOPk pk = new BaseContentDTOPk();
		pk.setA_webc_url(this.path);
		pk.setI_full_format("jhtml");
		pk.setLanguage_code(this.getLocaleOverride());

		// Load from database
		EntityManager em = emf.createEntityManager();
		try {
			PrimaryContentDTO temp = em.find(this.getClass(), pk);

			// Need to merge returned values with this object
			if (temp != null)
				merge(temp);

			if (log.isDebugEnabled()) {
				log.debug("Begin loadFromDB()");
			}
			
		} finally {
			em.close();
			em = null;
		}


The WCPSUtils class returns the singleton instance of the
EntityManagerFactory.

We have an inheritance hierarchy for most of the persistent classes. But the
one above follows this hierarchy:

BaseContentDTO -> ConfigDTO -> PrimaryContentDTO

BaseContentDTO has a one-to-many relationship to RepeatedAttribute
persistent class. We are eagerly fetching everything.  We only do reads from
the database, we have no inserts or updates.

I believe all the classes will be loaded using the same web application
class loader in WebSphere AppServer.

Hope this is the information you needed to help us analyze this issue.  Your
help is really appreciated.

Thanks,

Darren


Pinaki Poddar wrote:
> 
> Hi,
>   Having a single instance of EntityManagerFactory and making it live for
> the entire application lifetime is a good practice. Can you tell us how
> the persistent domain classes for are being specified (and loaded) for the
> application? 
>   For example, do you declare them in <class> clause of persistence.xml?
> Or in a orm.xml file? 
> Or are they not declared at all explicitly and loaded dynamically by the
> application? 
> Also is the application using multiple classloaders to load these classes?
> 
>   The container in PCRegistry that has occupied so much memory in the
> described profile is used to hold persistent domain class information.
> Knowing the classloading behavior of the application will help diagnosis.
>  
> 
> 

-- 
View this message in context: http://n2.nabble.com/Memory-Usage-tp3166851p3175091.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.


Re: Memory Usage

Posted by Pinaki Poddar <pp...@apache.org>.
Hi,
  Having a single instance of EntityManagerFactory and making it live for
the entire application lifetime is a good practice. Can you tell us how the
persistent domain classes for are being specified (and loaded) for the
application? 
  For example, do you declare them in <class> clause of persistence.xml? Or
in a orm.xml file? 
Or are they not declared at all explicitly and loaded dynamically by the
application? 
Also is the application using multiple classloaders to load these classes?

  The container in PCRegistry that has occupied so much memory in the
described profile is used to hold persistent domain class information.
Knowing the classloading behavior of the application will help diagnosis.
 


-----
Pinaki 
-- 
View this message in context: http://n2.nabble.com/Memory-Usage-tp3166851p3174914.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.


Re: Memory Usage

Posted by Darren Senel <da...@gmail.com>.
Hello Mike,

Thank you for your response.  The interesting thing is we use a single
EntityManagerFactory per JVM by using the Singleton pattern.  I double
checked the same heap dump and sure enough there only seems to be a single
factory object in the analysis.  We do keep this factory open for the
lifetime of the web application, but that was following guidance from the
OpenJPA documentation:

"Closing an EntityManagerFactory should not be taken lightly. It is much
better to keep a factory open for a long period of time than to repeatedly
create and close new factories. Thus, most applications will never close the
factory, or only close it when the application is exiting. Only applications
that require multiple factories with different configurations have an
obvious reason to create and close multiple EntityManagerFactory instances."

If we were to close the factory, when are where would you recommend we do it
in the web application? 

Any advice on this matter is very much appreciated.

Thanks,

Darren



Michael Dick wrote:
> 
> Hi Darren,
> 
> This is usually happens if the application creates a large number of
> EntityManager*Factories*, which aren't closed. Take a look at the
> application logic where you obtain the EMF, and make sure it gets closed
> (or
> cached appropriately).
> 
> hope this helps,
> -mike
> 
> On Sat, Jun 27, 2009 at 10:54 AM, Darren Senel
> <da...@gmail.com>wrote:
> 
>>
>> Hello,
>>
>> We are using OpenJPA 1.2.0 in a WebSphere 6 web application.  We are not
>> using the EJB3 pack.  We are seeing a very large proportion of the heap
>> taken up by OpenJPA PCRegistry objects (as shown in image).  We use
>> compile
>> time enhancement of our classes.  Eventually with enough requests of
>> under
>> high load our servers fail with an OOM error as the heap is exhausted. 
>> The
>> image below shows the heap analysis suspected leak subject.  Also, this
>> heap
>> dump was taken when the heap was about 50% full.
>>
>> We open, use, flush, close and set to null the EntityManager objects we
>> create.
>>
>> Would anyone have any ideas as to the cause of this growth of objects
>> that
>> eventually leads to our OOM crash?
>>
>> Many thanks,
>>
>> Darren
>>
>> http://n2.nabble.com/file/n3166851/HeapAnalysis.jpg
>> --
>> View this message in context:
>> http://n2.nabble.com/Memory-Usage-tp3166851p3166851.html
>> Sent from the OpenJPA Users mailing list archive at Nabble.com.
>>
>>
> 
> 

-- 
View this message in context: http://n2.nabble.com/Memory-Usage-tp3166851p3174602.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.


Re: Memory Usage

Posted by Michael Dick <mi...@gmail.com>.
Hi Darren,

This is usually happens if the application creates a large number of
EntityManager*Factories*, which aren't closed. Take a look at the
application logic where you obtain the EMF, and make sure it gets closed (or
cached appropriately).

hope this helps,
-mike

On Sat, Jun 27, 2009 at 10:54 AM, Darren Senel <da...@gmail.com>wrote:

>
> Hello,
>
> We are using OpenJPA 1.2.0 in a WebSphere 6 web application.  We are not
> using the EJB3 pack.  We are seeing a very large proportion of the heap
> taken up by OpenJPA PCRegistry objects (as shown in image).  We use compile
> time enhancement of our classes.  Eventually with enough requests of under
> high load our servers fail with an OOM error as the heap is exhausted.  The
> image below shows the heap analysis suspected leak subject.  Also, this
> heap
> dump was taken when the heap was about 50% full.
>
> We open, use, flush, close and set to null the EntityManager objects we
> create.
>
> Would anyone have any ideas as to the cause of this growth of objects that
> eventually leads to our OOM crash?
>
> Many thanks,
>
> Darren
>
> http://n2.nabble.com/file/n3166851/HeapAnalysis.jpg
> --
> View this message in context:
> http://n2.nabble.com/Memory-Usage-tp3166851p3166851.html
> Sent from the OpenJPA Users mailing list archive at Nabble.com.
>
>