You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@openjpa.apache.org by Adam Hardy <ad...@cyberspaceroad.com> on 2010/06/11 18:34:52 UTC
Performance with large lists
I have performance problems with large lists of beans due to the base
class I am using for my entities.
This is slightly non-OpenJPA specific, so I hope nobody minds, but it is
Friday afternoon so I'm hoping you give me a bit of slack here.
The problem arises when I start building lists with over 10,000 items on
a parent class.
The trouble is in the base class for the entities, which is quite clever
(but obviously not clever enough) and it has non-facile equals() and
hashcode() algorithms which make use of reflection. It's here that the
slow-down comes.
When I link the child with a parent that already has 10,000 children,
the equals() method is called by ArrayList before the new child is
placed in the index.
As far as I can tell I have a couple of options.
(1) ditch the reflection-based equals method and hard-code an equals method.
(2) don't use ArrayList but find a Collection-based class that uses
hashes or similar to identify items instead of equals. This is just
speculation - perhaps there is no such thing or it wouldn't help anyway:
- would a collection using hashes caches the hashes of the items
already indexed?
- would such a collection be persistable?
If anyone has been in this situation before, or has an idea about it ,
I'd really appreciate the help.
Regards
Adam
Re: Performance with large lists
Posted by Adam Hardy <ad...@cyberspaceroad.com>.
Thanks for the tip. I'll give it a try. What sort of performance do you get? Do
you do thousands in one submit?
C N Davies on 18/06/10 14:17, wrote:
> Not sure if you have the same issue, but I found breaking my commits down to
> 10 records per commit and calling em.clear() after each commit improved my
> performance and fixed a lot of dettached entity issues.
>
>
>
> -----Original Message----- From: Adam Hardy
> [mailto:adam.sql@cyberspaceroad.com] Sent: Friday, 18 June 2010 10:11 PM To:
> users@openjpa.apache.org Subject: Re: Performance with large lists
>
> I have a transaction that submits around 20K records, followed by another
> transaction in the user process which inserts the same into a related table.
>
> The performance issues I talked about below initially caused processing times
> of 45 mins, but I worked on the Java in the code and I tweaked the mySQL
> database and reduced it to 15 mins.
>
> This though is still a problem - I've been over the optimization guidelines
> in the documentation and there's nothing there that I can implement that I'm
> not already.
>
> The first transaction I mentioned takes 5 mins, but the second takes 15mins
> and is inserting child records of the records created in the first
> transaction. It looks like OpenJPA is fetching all of those first records
> again. Shouldn't they already be in memory?
>
>
> Thanks Adam
>
> Adam Hardy on 12/06/10 13:21, wrote:
>> I am trying to get a handle on what I should be able to achieve.
>>
>> Can someone give me some idea of the metrics I should be able to get
>> optimistically when persisting an object that has a child list with 20,000
>> child objects and 20,000 grandchildren? (one-to-one child -> grandchild)
>>
>> Can I reasonably expect to get this done in under a minute?
>>
>> I think that would work out at a rate of about 1.5 milliseconds per object.
>>
>>
>> Thanks Adam
>>
>> Adam Hardy on 11/06/10 17:34, wrote:
>>> I have performance problems with large lists of beans due to the base
>>> class I am using for my entities.
>>>
>>> This is slightly non-OpenJPA specific, so I hope nobody minds, but it is
>>> Friday afternoon so I'm hoping you give me a bit of slack here.
>>>
>>> The problem arises when I start building lists with over 10,000 items on
>>> a parent class.
>>>
>>> The trouble is in the base class for the entities, which is quite clever
>>> (but obviously not clever enough) and it has non-facile equals() and
>>> hashcode() algorithms which make use of reflection. It's here that the
>>> slow-down comes.
>>>
>>> When I link the child with a parent that already has 10,000 children, the
>>> equals() method is called by ArrayList before the new child is placed in
>>> the index.
>>>
>>> As far as I can tell I have a couple of options.
>>>
>>> (1) ditch the reflection-based equals method and hard-code an equals
>>> method.
>>>
>>> (2) don't use ArrayList but find a Collection-based class that uses
>>> hashes or similar to identify items instead of equals. This is just
>>> speculation - perhaps there is no such thing or it wouldn't help anyway:
>>>
>>> - would a collection using hashes caches the hashes of the items already
>>> indexed? - would such a collection be persistable?
>>>
>>> If anyone has been in this situation before, or has an idea about it ,
>>> I'd really appreciate the help.
>
>
RE: Performance with large lists
Posted by C N Davies <cn...@cndavies.com>.
Not sure if you have the same issue, but I found breaking my commits down to 10 records per commit and calling em.clear() after each commit improved my performance and fixed a lot of dettached entity issues.
-----Original Message-----
From: Adam Hardy [mailto:adam.sql@cyberspaceroad.com]
Sent: Friday, 18 June 2010 10:11 PM
To: users@openjpa.apache.org
Subject: Re: Performance with large lists
I have a transaction that submits around 20K records, followed by another
transaction in the user process which inserts the same into a related table.
The performance issues I talked about below initially caused processing times of
45 mins, but I worked on the Java in the code and I tweaked the mySQL database
and reduced it to 15 mins.
This though is still a problem - I've been over the optimization guidelines in
the documentation and there's nothing there that I can implement that I'm not
already.
The first transaction I mentioned takes 5 mins, but the second takes 15mins and
is inserting child records of the records created in the first transaction. It
looks like OpenJPA is fetching all of those first records again. Shouldn't they
already be in memory?
Thanks
Adam
Adam Hardy on 12/06/10 13:21, wrote:
> I am trying to get a handle on what I should be able to achieve.
>
> Can someone give me some idea of the metrics I should be able to get
> optimistically when persisting an object that has a child list with
> 20,000 child objects and 20,000 grandchildren? (one-to-one child ->
> grandchild)
>
> Can I reasonably expect to get this done in under a minute?
>
> I think that would work out at a rate of about 1.5 milliseconds per object.
>
> Thanks
> Adam
>
> Adam Hardy on 11/06/10 17:34, wrote:
>> I have performance problems with large lists of beans due to the base
>> class I am using for my entities.
>>
>> This is slightly non-OpenJPA specific, so I hope nobody minds, but it
>> is Friday afternoon so I'm hoping you give me a bit of slack here.
>>
>> The problem arises when I start building lists with over 10,000 items
>> on a parent class.
>>
>> The trouble is in the base class for the entities, which is quite
>> clever (but obviously not clever enough) and it has non-facile
>> equals() and hashcode() algorithms which make use of reflection. It's
>> here that the slow-down comes.
>>
>> When I link the child with a parent that already has 10,000 children,
>> the equals() method is called by ArrayList before the new child is
>> placed in the index.
>>
>> As far as I can tell I have a couple of options.
>>
>> (1) ditch the reflection-based equals method and hard-code an equals
>> method.
>>
>> (2) don't use ArrayList but find a Collection-based class that uses
>> hashes or similar to identify items instead of equals. This is just
>> speculation - perhaps there is no such thing or it wouldn't help anyway:
>>
>> - would a collection using hashes caches the hashes of the items
>> already indexed?
>> - would such a collection be persistable?
>>
>> If anyone has been in this situation before, or has an idea about it
>> , I'd really appreciate the help.
Re: Performance with large lists
Posted by Adam Hardy <ad...@cyberspaceroad.com>.
Donald Woods on 18/06/10 20:54, wrote:
>> 4) Actually no because in OpenJPA 1.2.2 there is a bug that throws NPEs
>> and means I can't run with it enabled. I just upgraded to 2.0.0 I think
>> I've got the same bug - albeit a different stack trace - but a quick
>> google didn't show up any immediately available work-around. I guess
>> this is the key issue, right?
>>
>
> Which JIRA and what does your 2.0.0 stack trace look like?
OK you caught me there. I did not find it in the Jira just now (searched on
either NullPointerException or DataCachPCDataImpl), I was referring to this
message here:
http://openjpa.208410.n2.nabble.com/NullPointerException-in-DataCache-td532523.html
The same stack trace as mine in 1.2.2 and this is my stacktrace with 2.0.0:
Caused by: java.lang.NullPointerException
at
org.apache.openjpa.datacache.DataCachePCDataImpl.clearInverseRelationCache(DataCachePCDataImpl.java:179)
at
org.apache.openjpa.datacache.DataCachePCDataImpl.storeField(DataCachePCDataImpl.java:159)
at org.apache.openjpa.kernel.PCDataImpl.store(PCDataImpl.java:235)
at
org.apache.openjpa.datacache.DataCachePCDataImpl.store(DataCachePCDataImpl.java:125)
at
org.apache.openjpa.datacache.DataCacheStoreManager.updateCaches(DataCacheStoreManager.java:179)
at
org.apache.openjpa.datacache.DataCacheStoreManager.commit(DataCacheStoreManager.java:90)
at
org.apache.openjpa.kernel.DelegatingStoreManager.commit(DelegatingStoreManager.java:95)
at
org.apache.openjpa.kernel.BrokerImpl.endStoreManagerTransaction(BrokerImpl.java:1436)
at
org.apache.openjpa.kernel.BrokerImpl.endTransaction(BrokerImpl.java:2316)
at
org.apache.openjpa.kernel.BrokerImpl.afterCompletion(BrokerImpl.java:1975)
... 69 more
Re: Performance with large lists
Posted by Donald Woods <dw...@apache.org>.
On 6/18/10 12:47 PM, Adam Hardy wrote:
> 1) I'm using tomcat, so no Java EE container. I'm using Spring to handle
> the transactions.
>
> 2) I'm using one EMF for the whole app from app start, and one EM for
> the transaction or sometimes 2, launched by a tomcat filter which opens
> it at the start of the HTTP request and closes it at the end. And the
> two transactions I'm talking about are 2 distinct user operations, so
> each gets one EM.
>
> 3) Build-time enhancement.
>
> 4) Actually no because in OpenJPA 1.2.2 there is a bug that throws NPEs
> and means I can't run with it enabled. I just upgraded to 2.0.0 I think
> I've got the same bug - albeit a different stack trace - but a quick
> google didn't show up any immediately available work-around. I guess
> this is the key issue, right?
>
Which JIRA and what does your 2.0.0 stack trace look like?
>
>
> Here's my persistence xml
> <persistence-unit name="OpenJpaJdbc">
> <description>Pattern Repo JPA Config with OpenJPA</description>
>
> <provider>org.apache.openjpa.persistence.PersistenceProviderImpl</provider>
>
> <mapping-file>org/permacode/atomic/domain/entity/AtomicEntity.xml</mapping-file>
>
>
> <mapping-file>org/permacode/patternrepo/orm/BrokerAccount.xml</mapping-file>
>
> <mapping-file>org/permacode/patternrepo/orm/Category.xml</mapping-file>
>
> <mapping-file>org/permacode/patternrepo/orm/DollarReturn.xml</mapping-file>
> <mapping-file>org/permacode/patternrepo/orm/Exchange.xml</mapping-file>
>
> <mapping-file>org/permacode/patternrepo/orm/ExchangeSymbol.xml</mapping-file>
>
> <mapping-file>org/permacode/patternrepo/orm/Fill.xml</mapping-file>
> <mapping-file>org/permacode/patternrepo/orm/Market.xml</mapping-file>
>
> <mapping-file>org/permacode/patternrepo/orm/MarketSymbol.xml</mapping-file>
>
> <mapping-file>org/permacode/patternrepo/orm/MarketSystem.xml</mapping-file>
> <mapping-file>org/permacode/patternrepo/orm/Order.xml</mapping-file>
> <mapping-file>org/permacode/patternrepo/orm/Pattern.xml</mapping-file>
>
> <mapping-file>org/permacode/patternrepo/orm/Portfolio.xml</mapping-file>
>
> <mapping-file>org/permacode/patternrepo/orm/PortfolioItem.xml</mapping-file>
>
>
> <mapping-file>org/permacode/patternrepo/orm/RequiredParam.xml</mapping-file>
>
>
> <mapping-file>org/permacode/patternrepo/orm/TestAnalysis.xml</mapping-file>
> <mapping-file>org/permacode/patternrepo/orm/TestRun.xml</mapping-file>
> <mapping-file>org/permacode/patternrepo/orm/Trade.xml</mapping-file>
>
> <mapping-file>org/permacode/patternrepo/orm/TradingParam.xml</mapping-file>
>
> <mapping-file>org/permacode/patternrepo/orm/TradingSystem.xml</mapping-file>
>
>
> <mapping-file>org/permacode/patternrepo/orm/Weighting.xml</mapping-file>
>
> <mapping-file>org/permacode/patternrepo/orm/WeightingGroup.xml</mapping-file>
>
>
> <mapping-file>org/permacode/patternrepo/orm/MiniAnalysis.xml</mapping-file>
>
> <mapping-file>org/permacode/patternrepo/orm/MiniResult.xml</mapping-file>
>
> <mapping-file>org/permacode/patternrepo/orm/TestQueries.xml</mapping-file>
> <properties>
> <property name="openjpa.RuntimeUnenhancedClasses"
> value="unsupported" />
> <property name="openjpa.Log" value="SQL=TRACE"/>
> <property name="openjpa.ConnectionFactoryProperties"
> value="PrettyPrint=true, PrettyPrintLineLength=72" />
> </properties>
> </persistence-unit>
>
>
>
>
> Donald Woods on 18/06/10 13:36, wrote:
>> Some quick initial questions -
>> 1) Are you using Java SE or a Java EE container (more of a transactions
>> and XA question)?
>> 2) Are you using one EMF and EM for both the parent, child and
>> grandchild operations? Are you calling flush() anywhere?
>> 3) Are you using build time or run time enhancement of entities?
>> 4) Have you enabled the DataCache or QueryCache?
>> http://openjpa.apache.org/builds/2.0.0/apache-openjpa-2.0.0/docs/manual/manual.html#ref_guide_cache_query
>>
>>
>> Also, what do the entities and your persistence.xml look like?
>>
>>
>> -Donald
>>
>>
>> On 6/18/10 8:11 AM, Adam Hardy wrote:
>>> I have a transaction that submits around 20K records, followed by
>>> another transaction in the user process which inserts the same into a
>>> related table.
>>>
>>> The performance issues I talked about below initially caused processing
>>> times of 45 mins, but I worked on the Java in the code and I tweaked the
>>> mySQL database and reduced it to 15 mins.
>>>
>>> This though is still a problem - I've been over the optimization
>>> guidelines in the documentation and there's nothing there that I can
>>> implement that I'm not already.
>>>
>>> The first transaction I mentioned takes 5 mins, but the second takes
>>> 15mins and is inserting child records of the records created in the
>>> first transaction. It looks like OpenJPA is fetching all of those first
>>> records again. Shouldn't they already be in memory?
>>>
>>>
>>> Thanks
>>> Adam
>>>
>>> Adam Hardy on 12/06/10 13:21, wrote:
>>>> I am trying to get a handle on what I should be able to achieve.
>>>>
>>>> Can someone give me some idea of the metrics I should be able to get
>>>> optimistically when persisting an object that has a child list with
>>>> 20,000 child objects and 20,000 grandchildren? (one-to-one child ->
>>>> grandchild)
>>>>
>>>> Can I reasonably expect to get this done in under a minute?
>>>>
>>>> I think that would work out at a rate of about 1.5 milliseconds per
>>>> object.
>>>>
>>>> Thanks
>>>> Adam
>>>>
>>>> Adam Hardy on 11/06/10 17:34, wrote:
>>>>> I have performance problems with large lists of beans due to the base
>>>>> class I am using for my entities.
>>>>>
>>>>> This is slightly non-OpenJPA specific, so I hope nobody minds, but it
>>>>> is Friday afternoon so I'm hoping you give me a bit of slack here.
>>>>>
>>>>> The problem arises when I start building lists with over 10,000 items
>>>>> on a parent class.
>>>>>
>>>>> The trouble is in the base class for the entities, which is quite
>>>>> clever (but obviously not clever enough) and it has non-facile
>>>>> equals() and hashcode() algorithms which make use of reflection. It's
>>>>> here that the slow-down comes.
>>>>>
>>>>> When I link the child with a parent that already has 10,000 children,
>>>>> the equals() method is called by ArrayList before the new child is
>>>>> placed in the index.
>>>>>
>>>>> As far as I can tell I have a couple of options.
>>>>>
>>>>> (1) ditch the reflection-based equals method and hard-code an equals
>>>>> method.
>>>>>
>>>>> (2) don't use ArrayList but find a Collection-based class that uses
>>>>> hashes or similar to identify items instead of equals. This is just
>>>>> speculation - perhaps there is no such thing or it wouldn't help
>>>>> anyway:
>>>>>
>>>>> - would a collection using hashes caches the hashes of the items
>>>>> already indexed?
>>>>> - would such a collection be persistable?
>>>>>
>>>>> If anyone has been in this situation before, or has an idea about it
>>>>> , I'd really appreciate the help.
>>>
>>
>
>
Re: Performance with large lists
Posted by Adam Hardy <ad...@cyberspaceroad.com>.
1) I'm using tomcat, so no Java EE container. I'm using Spring to handle the
transactions.
2) I'm using one EMF for the whole app from app start, and one EM for the
transaction or sometimes 2, launched by a tomcat filter which opens it at the
start of the HTTP request and closes it at the end. And the two transactions I'm
talking about are 2 distinct user operations, so each gets one EM.
3) Build-time enhancement.
4) Actually no because in OpenJPA 1.2.2 there is a bug that throws NPEs and
means I can't run with it enabled. I just upgraded to 2.0.0 I think I've got the
same bug - albeit a different stack trace - but a quick google didn't show up
any immediately available work-around. I guess this is the key issue, right?
Here's my persistence xml
<persistence-unit name="OpenJpaJdbc">
<description>Pattern Repo JPA Config with OpenJPA</description>
<provider>org.apache.openjpa.persistence.PersistenceProviderImpl</provider>
<mapping-file>org/permacode/atomic/domain/entity/AtomicEntity.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/BrokerAccount.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/Category.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/DollarReturn.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/Exchange.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/ExchangeSymbol.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/Fill.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/Market.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/MarketSymbol.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/MarketSystem.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/Order.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/Pattern.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/Portfolio.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/PortfolioItem.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/RequiredParam.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/TestAnalysis.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/TestRun.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/Trade.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/TradingParam.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/TradingSystem.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/Weighting.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/WeightingGroup.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/MiniAnalysis.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/MiniResult.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/TestQueries.xml</mapping-file>
<properties>
<property name="openjpa.RuntimeUnenhancedClasses" value="unsupported" />
<property name="openjpa.Log" value="SQL=TRACE"/>
<property name="openjpa.ConnectionFactoryProperties"
value="PrettyPrint=true, PrettyPrintLineLength=72" />
</properties>
</persistence-unit>
Donald Woods on 18/06/10 13:36, wrote:
> Some quick initial questions -
> 1) Are you using Java SE or a Java EE container (more of a transactions
> and XA question)?
> 2) Are you using one EMF and EM for both the parent, child and
> grandchild operations? Are you calling flush() anywhere?
> 3) Are you using build time or run time enhancement of entities?
> 4) Have you enabled the DataCache or QueryCache?
> http://openjpa.apache.org/builds/2.0.0/apache-openjpa-2.0.0/docs/manual/manual.html#ref_guide_cache_query
>
> Also, what do the entities and your persistence.xml look like?
>
>
> -Donald
>
>
> On 6/18/10 8:11 AM, Adam Hardy wrote:
>> I have a transaction that submits around 20K records, followed by
>> another transaction in the user process which inserts the same into a
>> related table.
>>
>> The performance issues I talked about below initially caused processing
>> times of 45 mins, but I worked on the Java in the code and I tweaked the
>> mySQL database and reduced it to 15 mins.
>>
>> This though is still a problem - I've been over the optimization
>> guidelines in the documentation and there's nothing there that I can
>> implement that I'm not already.
>>
>> The first transaction I mentioned takes 5 mins, but the second takes
>> 15mins and is inserting child records of the records created in the
>> first transaction. It looks like OpenJPA is fetching all of those first
>> records again. Shouldn't they already be in memory?
>>
>>
>> Thanks
>> Adam
>>
>> Adam Hardy on 12/06/10 13:21, wrote:
>>> I am trying to get a handle on what I should be able to achieve.
>>>
>>> Can someone give me some idea of the metrics I should be able to get
>>> optimistically when persisting an object that has a child list with
>>> 20,000 child objects and 20,000 grandchildren? (one-to-one child ->
>>> grandchild)
>>>
>>> Can I reasonably expect to get this done in under a minute?
>>>
>>> I think that would work out at a rate of about 1.5 milliseconds per
>>> object.
>>>
>>> Thanks
>>> Adam
>>>
>>> Adam Hardy on 11/06/10 17:34, wrote:
>>>> I have performance problems with large lists of beans due to the base
>>>> class I am using for my entities.
>>>>
>>>> This is slightly non-OpenJPA specific, so I hope nobody minds, but it
>>>> is Friday afternoon so I'm hoping you give me a bit of slack here.
>>>>
>>>> The problem arises when I start building lists with over 10,000 items
>>>> on a parent class.
>>>>
>>>> The trouble is in the base class for the entities, which is quite
>>>> clever (but obviously not clever enough) and it has non-facile
>>>> equals() and hashcode() algorithms which make use of reflection. It's
>>>> here that the slow-down comes.
>>>>
>>>> When I link the child with a parent that already has 10,000 children,
>>>> the equals() method is called by ArrayList before the new child is
>>>> placed in the index.
>>>>
>>>> As far as I can tell I have a couple of options.
>>>>
>>>> (1) ditch the reflection-based equals method and hard-code an equals
>>>> method.
>>>>
>>>> (2) don't use ArrayList but find a Collection-based class that uses
>>>> hashes or similar to identify items instead of equals. This is just
>>>> speculation - perhaps there is no such thing or it wouldn't help anyway:
>>>>
>>>> - would a collection using hashes caches the hashes of the items
>>>> already indexed?
>>>> - would such a collection be persistable?
>>>>
>>>> If anyone has been in this situation before, or has an idea about it
>>>> , I'd really appreciate the help.
>>
>
Re: Performance with large lists
Posted by Donald Woods <dw...@apache.org>.
Some quick initial questions -
1) Are you using Java SE or a Java EE container (more of a transactions
and XA question)?
2) Are you using one EMF and EM for both the parent, child and
grandchild operations? Are you calling flush() anywhere?
3) Are you using build time or run time enhancement of entities?
4) Have you enabled the DataCache or QueryCache?
http://openjpa.apache.org/builds/2.0.0/apache-openjpa-2.0.0/docs/manual/manual.html#ref_guide_cache_query
Also, what do the entities and your persistence.xml look like?
-Donald
On 6/18/10 8:11 AM, Adam Hardy wrote:
> I have a transaction that submits around 20K records, followed by
> another transaction in the user process which inserts the same into a
> related table.
>
> The performance issues I talked about below initially caused processing
> times of 45 mins, but I worked on the Java in the code and I tweaked the
> mySQL database and reduced it to 15 mins.
>
> This though is still a problem - I've been over the optimization
> guidelines in the documentation and there's nothing there that I can
> implement that I'm not already.
>
> The first transaction I mentioned takes 5 mins, but the second takes
> 15mins and is inserting child records of the records created in the
> first transaction. It looks like OpenJPA is fetching all of those first
> records again. Shouldn't they already be in memory?
>
>
> Thanks
> Adam
>
> Adam Hardy on 12/06/10 13:21, wrote:
>> I am trying to get a handle on what I should be able to achieve.
>>
>> Can someone give me some idea of the metrics I should be able to get
>> optimistically when persisting an object that has a child list with
>> 20,000 child objects and 20,000 grandchildren? (one-to-one child ->
>> grandchild)
>>
>> Can I reasonably expect to get this done in under a minute?
>>
>> I think that would work out at a rate of about 1.5 milliseconds per
>> object.
>>
>> Thanks
>> Adam
>>
>> Adam Hardy on 11/06/10 17:34, wrote:
>>> I have performance problems with large lists of beans due to the base
>>> class I am using for my entities.
>>>
>>> This is slightly non-OpenJPA specific, so I hope nobody minds, but it
>>> is Friday afternoon so I'm hoping you give me a bit of slack here.
>>>
>>> The problem arises when I start building lists with over 10,000 items
>>> on a parent class.
>>>
>>> The trouble is in the base class for the entities, which is quite
>>> clever (but obviously not clever enough) and it has non-facile
>>> equals() and hashcode() algorithms which make use of reflection. It's
>>> here that the slow-down comes.
>>>
>>> When I link the child with a parent that already has 10,000 children,
>>> the equals() method is called by ArrayList before the new child is
>>> placed in the index.
>>>
>>> As far as I can tell I have a couple of options.
>>>
>>> (1) ditch the reflection-based equals method and hard-code an equals
>>> method.
>>>
>>> (2) don't use ArrayList but find a Collection-based class that uses
>>> hashes or similar to identify items instead of equals. This is just
>>> speculation - perhaps there is no such thing or it wouldn't help anyway:
>>>
>>> - would a collection using hashes caches the hashes of the items
>>> already indexed?
>>> - would such a collection be persistable?
>>>
>>> If anyone has been in this situation before, or has an idea about it
>>> , I'd really appreciate the help.
>
>
Re: Performance with large lists
Posted by Adam Hardy <ad...@cyberspaceroad.com>.
I have a transaction that submits around 20K records, followed by another
transaction in the user process which inserts the same into a related table.
The performance issues I talked about below initially caused processing times of
45 mins, but I worked on the Java in the code and I tweaked the mySQL database
and reduced it to 15 mins.
This though is still a problem - I've been over the optimization guidelines in
the documentation and there's nothing there that I can implement that I'm not
already.
The first transaction I mentioned takes 5 mins, but the second takes 15mins and
is inserting child records of the records created in the first transaction. It
looks like OpenJPA is fetching all of those first records again. Shouldn't they
already be in memory?
Thanks
Adam
Adam Hardy on 12/06/10 13:21, wrote:
> I am trying to get a handle on what I should be able to achieve.
>
> Can someone give me some idea of the metrics I should be able to get
> optimistically when persisting an object that has a child list with
> 20,000 child objects and 20,000 grandchildren? (one-to-one child ->
> grandchild)
>
> Can I reasonably expect to get this done in under a minute?
>
> I think that would work out at a rate of about 1.5 milliseconds per object.
>
> Thanks
> Adam
>
> Adam Hardy on 11/06/10 17:34, wrote:
>> I have performance problems with large lists of beans due to the base
>> class I am using for my entities.
>>
>> This is slightly non-OpenJPA specific, so I hope nobody minds, but it
>> is Friday afternoon so I'm hoping you give me a bit of slack here.
>>
>> The problem arises when I start building lists with over 10,000 items
>> on a parent class.
>>
>> The trouble is in the base class for the entities, which is quite
>> clever (but obviously not clever enough) and it has non-facile
>> equals() and hashcode() algorithms which make use of reflection. It's
>> here that the slow-down comes.
>>
>> When I link the child with a parent that already has 10,000 children,
>> the equals() method is called by ArrayList before the new child is
>> placed in the index.
>>
>> As far as I can tell I have a couple of options.
>>
>> (1) ditch the reflection-based equals method and hard-code an equals
>> method.
>>
>> (2) don't use ArrayList but find a Collection-based class that uses
>> hashes or similar to identify items instead of equals. This is just
>> speculation - perhaps there is no such thing or it wouldn't help anyway:
>>
>> - would a collection using hashes caches the hashes of the items
>> already indexed?
>> - would such a collection be persistable?
>>
>> If anyone has been in this situation before, or has an idea about it
>> , I'd really appreciate the help.
Re: Performance with large lists
Posted by Adam Hardy <ad...@cyberspaceroad.com>.
I am trying to get a handle on what I should be able to achieve.
Can someone give me some idea of the metrics I should be able to get
optimistically when persisting an object that has a child list with 20,000 child
objects and 20,000 grandchildren? (one-to-one child -> grandchild)
Can I reasonably expect to get this done in under a minute?
I think that would work out at a rate of about 1.5 milliseconds per object.
Thanks
Adam
Adam Hardy on 11/06/10 17:34, wrote:
> I have performance problems with large lists of beans due to the base
> class I am using for my entities.
>
> This is slightly non-OpenJPA specific, so I hope nobody minds, but it is
> Friday afternoon so I'm hoping you give me a bit of slack here.
>
> The problem arises when I start building lists with over 10,000 items on
> a parent class.
>
> The trouble is in the base class for the entities, which is quite clever
> (but obviously not clever enough) and it has non-facile equals() and
> hashcode() algorithms which make use of reflection. It's here that the
> slow-down comes.
>
> When I link the child with a parent that already has 10,000 children,
> the equals() method is called by ArrayList before the new child is
> placed in the index.
>
> As far as I can tell I have a couple of options.
>
> (1) ditch the reflection-based equals method and hard-code an equals
> method.
>
> (2) don't use ArrayList but find a Collection-based class that uses
> hashes or similar to identify items instead of equals. This is just
> speculation - perhaps there is no such thing or it wouldn't help anyway:
>
> - would a collection using hashes caches the hashes of the items
> already indexed?
> - would such a collection be persistable?
>
> If anyone has been in this situation before, or has an idea about it ,
> I'd really appreciate the help.
>
> Regards
> Adam
>
>