You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@openjpa.apache.org by Michael Pflueger <Mi...@sma.de> on 2011/09/12 20:36:15 UTC
Speed of fetching simple entities using OpenJPA
Hi,
I compared JDBC and OpenJPA speed of fetching simple entities (consisting only of a single ID attribute). The speed difference seems quite severe.
With JDBC, I can read 2 million entities in about 3 seconds when OS/DB caches are warm.
With OpenJPA using slice (a single slice), it takes a bit above 30 seconds, so about ten times as long, more than I would expect.
Using a direct connection without slice reduces this time to about 24 seconds, ~8 times as much as the JDBC version.
In the JDBC test I also create objects from the rows, so both tests include object creation overhead of the entities (plus a sum calculation of the IDs).
I'm doing this test using PostgreSQL and an up to date OpenJPA 2.2 snapshot.
Entities are enhanced using javaagent enhancement.
I tried optimizing OpenJPA performance with:
<property name="openjpa.ConnectionRetainMode" value="always"></property>
<property name="openjpa.ProxyManager" value="TrackChanges=false"/>
<property name="openjpa.IgnoreChanges" value="true"/>
<property name="openjpa.LockManager" value="none"/>
<property name="openjpa.DetachState" value="fetch-groups(DetachedStateField=false)"/>
But those didn't seem to affect performance in any significant way.
Now, is such a large performance hit expected, and what would be the reason?
It would be nice if some performance numbers like these would be in the documentation, by the way, to be better able to judge for what applications OpenJPA can be used/what performance penalties one has to expect, depending on the situation.
Regards,
Michael
___________________________________________________
SMA Solar Technology AG
Aufsichtsrat: Guenther Cramer (Vorsitzender)
Vorstand: Juergen Dolle, Roland Grebe, Uwe Hertel, Pierre-Pascal Urbon, Marko Werner
Handelsregister: Amtsgericht Kassel HRB 3972
Sitz der Gesellschaft: 34266 Niestetal
USt-ID-Nr. DE 113 08 59 54
WEEE-Reg.-Nr. DE 95881150
___________________________________________________
AW: Speed of fetching simple entities using OpenJPA
Posted by Michael Pflueger <Mi...@sma.de>.
I think this test also demonstrates that PostgreSQL Cursors aren't working with OpenJPA.
When I fetch 5 million entries using JDBC and iterate over them with cursors, heap memory consumption stays below 4 MB.
When I fetch 2 million entries using OpenJPA with FetchBatchSize=2000, heap memory increases to 125MB.
When I do it without FetchBatchSize set, memory usage is even higher, it hits 100% heap memory usage at 250MB.
So, what I think might happen is that with FetchBatchSIze set, OpenJPA streams through the results of the JDBC driver, but the driver itself loads all result rows into memory at once and does not use cursors.
With FetchBatchSize unset, OpenJPA tries to instantiate all entities at once and thus memory usage is even higher.
So, something is probably wrong in the communication between the OpenJPA and the Postgre driver?
Michael
-----Ursprüngliche Nachricht-----
Von: Michael Pflueger [mailto:Michael.Pflueger@sma.de]
Gesendet: Dienstag, 13. September 2011 15:29
An: users@openjpa.apache.org
Betreff: AW: Speed of fetching simple entities using OpenJPA
Hi,
attached is a zip containing both the OpenJPA and JDBC tests to read the entities, and a small prog to fill the DB. I'm doing entytymanager.clear() every 2000 records so memory does not fill up, and have tried to do it only every 20k records, but that seemed to make it a bit slower, not faster.
Michael
___________________________________________________
SMA Solar Technology AG
Aufsichtsrat: Guenther Cramer (Vorsitzender)
Vorstand: Juergen Dolle, Roland Grebe, Uwe Hertel, Pierre-Pascal Urbon, Marko Werner
Handelsregister: Amtsgericht Kassel HRB 3972
Sitz der Gesellschaft: 34266 Niestetal
USt-ID-Nr. DE 113 08 59 54
WEEE-Reg.-Nr. DE 95881150
___________________________________________________
AW: Speed of fetching simple entities using OpenJPA
Posted by Michael Pflueger <Mi...@sma.de>.
Hi,
attached is a zip containing both the OpenJPA and JDBC tests to read the entities, and a small prog to fill the DB. I'm doing entytymanager.clear() every 2000 records so memory does not fill up, and have tried to do it only every 20k records, but that seemed to make it a bit slower, not faster.
Michael
-----Ursprüngliche Nachricht-----
Von: Rick Curtis [mailto:curtisr7@gmail.com]
Gesendet: Dienstag, 13. September 2011 00:00
An: users@openjpa.apache.org
Betreff: Re: Speed of fetching simple entities using OpenJPA
> I could submit a zip including the tests and a persistence.xml tomorrow, i
guess.
That would probably be the most expedient way to figure out what is going
on.
> Does anyone on the openjpa dev team run such performance tests?
We don't have anything (that I'm aware of) that runs as part of our builds.
You are correct that it would be easy enough write a smallish benchmark root
out potential regressions ... if someone invested the time. That being said,
we're continually assessing the performance of OpenJPA to ensure that it is
competitive with other providers.
*Rick Curtis*
___________________________________________________
SMA Solar Technology AG
Aufsichtsrat: Guenther Cramer (Vorsitzender)
Vorstand: Juergen Dolle, Roland Grebe, Uwe Hertel, Pierre-Pascal Urbon, Marko Werner
Handelsregister: Amtsgericht Kassel HRB 3972
Sitz der Gesellschaft: 34266 Niestetal
USt-ID-Nr. DE 113 08 59 54
WEEE-Reg.-Nr. DE 95881150
___________________________________________________
Re: Speed of fetching simple entities using OpenJPA
Posted by Rick Curtis <cu...@gmail.com>.
> I could submit a zip including the tests and a persistence.xml tomorrow, i
guess.
That would probably be the most expedient way to figure out what is going
on.
> Does anyone on the openjpa dev team run such performance tests?
We don't have anything (that I'm aware of) that runs as part of our builds.
You are correct that it would be easy enough write a smallish benchmark root
out potential regressions ... if someone invested the time. That being said,
we're continually assessing the performance of OpenJPA to ensure that it is
competitive with other providers.
On Mon, Sep 12, 2011 at 2:12 PM, Michael Pflueger
<Mi...@sma.de>wrote:
> Hi, here's the code:
>
> em = emf.createEntityManager();
> CriteriaBuilder cb = em.getCriteriaBuilder();
> CriteriaQuery<TestEntity> cqte =
> cb.createQuery(TestEntity.class);
> Root<TestEntity> rte = cqte.from(TestEntity.class);
> TypedQuery<TestEntity> qte =
> em.createQuery(cqte).setMaxResults(2000000);
> long queryTime = System.currentTimeMillis();
> List<TestEntity> entities = qte.getResultList();
> int cnt = 0;
> long sum = 0;
> for(TestEntity te : entities) {
> cnt++;
> sum += te.getId();
> if(cnt % 2000 == 0) {
> em.clear();
> }
> }
> //<code to output query time>
>
> TestEntity just has that single ID attribute.
> I could submit a zip including the tests and a persistence.xml tomorrow, i
> guess.
>
> Does anyone on the openjpa dev team run such performance tests?
> I guess those could be automated and run regularly against trunk so
> regressions are spotted quickly.
>
> Michael
>
> -----Ursprüngliche Nachricht-----
> Von: Rick Curtis [mailto:curtisr7@gmail.com]
> Gesendet: Montag, 12. September 2011 20:44
> An: users@openjpa.apache.org
> Betreff: Re: Speed of fetching simple entities using OpenJPA
>
> Michael -
>
> I wouldn't expect to see that large of a difference of JPA vs JDBC. Can you
> describe your scenario a little better? Are you issuing just a single JPQL
> to select all of your Entities?
>
> Thanks,
> Rick
>
> On Mon, Sep 12, 2011 at 1:36 PM, Michael Pflueger
> <Mi...@sma.de>wrote:
>
> > Hi,
> > I compared JDBC and OpenJPA speed of fetching simple entities (consisting
> > only of a single ID attribute). The speed difference seems quite severe.
> >
> > With JDBC, I can read 2 million entities in about 3 seconds when OS/DB
> > caches are warm.
> > With OpenJPA using slice (a single slice), it takes a bit above 30
> seconds,
> > so about ten times as long, more than I would expect.
> > Using a direct connection without slice reduces this time to about 24
> > seconds, ~8 times as much as the JDBC version.
> >
> > In the JDBC test I also create objects from the rows, so both tests
> include
> > object creation overhead of the entities (plus a sum calculation of the
> > IDs).
> >
> > I'm doing this test using PostgreSQL and an up to date OpenJPA 2.2
> > snapshot.
> > Entities are enhanced using javaagent enhancement.
> >
> > I tried optimizing OpenJPA performance with:
> > <property name="openjpa.ConnectionRetainMode"
> > value="always"></property>
> > <property name="openjpa.ProxyManager" value="TrackChanges=false"/>
> > <property name="openjpa.IgnoreChanges" value="true"/>
> > <property name="openjpa.LockManager" value="none"/>
> > <property name="openjpa.DetachState"
> > value="fetch-groups(DetachedStateField=false)"/>
> >
> > But those didn't seem to affect performance in any significant way.
> >
> > Now, is such a large performance hit expected, and what would be the
> > reason?
> > It would be nice if some performance numbers like these would be in the
> > documentation, by the way, to be better able to judge for what
> applications
> > OpenJPA can be used/what performance penalties one has to expect,
> depending
> > on the situation.
> >
> > Regards,
> > Michael
> > ___________________________________________________
> >
> > SMA Solar Technology AG
> > Aufsichtsrat: Guenther Cramer (Vorsitzender)
> > Vorstand: Juergen Dolle, Roland Grebe, Uwe Hertel, Pierre-Pascal Urbon,
> > Marko Werner
> > Handelsregister: Amtsgericht Kassel HRB 3972
> > Sitz der Gesellschaft: 34266 Niestetal
> > USt-ID-Nr. DE 113 08 59 54
> > WEEE-Reg.-Nr. DE 95881150
> > ___________________________________________________
> >
> >
>
>
> --
> *Rick Curtis*
>
--
*Rick Curtis*
AW: Speed of fetching simple entities using OpenJPA
Posted by Michael Pflueger <Mi...@sma.de>.
Hi, here's the code:
em = emf.createEntityManager();
CriteriaBuilder cb = em.getCriteriaBuilder();
CriteriaQuery<TestEntity> cqte = cb.createQuery(TestEntity.class);
Root<TestEntity> rte = cqte.from(TestEntity.class);
TypedQuery<TestEntity> qte = em.createQuery(cqte).setMaxResults(2000000);
long queryTime = System.currentTimeMillis();
List<TestEntity> entities = qte.getResultList();
int cnt = 0;
long sum = 0;
for(TestEntity te : entities) {
cnt++;
sum += te.getId();
if(cnt % 2000 == 0) {
em.clear();
}
}
//<code to output query time>
TestEntity just has that single ID attribute.
I could submit a zip including the tests and a persistence.xml tomorrow, i guess.
Does anyone on the openjpa dev team run such performance tests?
I guess those could be automated and run regularly against trunk so regressions are spotted quickly.
Michael
-----Ursprüngliche Nachricht-----
Von: Rick Curtis [mailto:curtisr7@gmail.com]
Gesendet: Montag, 12. September 2011 20:44
An: users@openjpa.apache.org
Betreff: Re: Speed of fetching simple entities using OpenJPA
Michael -
I wouldn't expect to see that large of a difference of JPA vs JDBC. Can you
describe your scenario a little better? Are you issuing just a single JPQL
to select all of your Entities?
Thanks,
Rick
On Mon, Sep 12, 2011 at 1:36 PM, Michael Pflueger
<Mi...@sma.de>wrote:
> Hi,
> I compared JDBC and OpenJPA speed of fetching simple entities (consisting
> only of a single ID attribute). The speed difference seems quite severe.
>
> With JDBC, I can read 2 million entities in about 3 seconds when OS/DB
> caches are warm.
> With OpenJPA using slice (a single slice), it takes a bit above 30 seconds,
> so about ten times as long, more than I would expect.
> Using a direct connection without slice reduces this time to about 24
> seconds, ~8 times as much as the JDBC version.
>
> In the JDBC test I also create objects from the rows, so both tests include
> object creation overhead of the entities (plus a sum calculation of the
> IDs).
>
> I'm doing this test using PostgreSQL and an up to date OpenJPA 2.2
> snapshot.
> Entities are enhanced using javaagent enhancement.
>
> I tried optimizing OpenJPA performance with:
> <property name="openjpa.ConnectionRetainMode"
> value="always"></property>
> <property name="openjpa.ProxyManager" value="TrackChanges=false"/>
> <property name="openjpa.IgnoreChanges" value="true"/>
> <property name="openjpa.LockManager" value="none"/>
> <property name="openjpa.DetachState"
> value="fetch-groups(DetachedStateField=false)"/>
>
> But those didn't seem to affect performance in any significant way.
>
> Now, is such a large performance hit expected, and what would be the
> reason?
> It would be nice if some performance numbers like these would be in the
> documentation, by the way, to be better able to judge for what applications
> OpenJPA can be used/what performance penalties one has to expect, depending
> on the situation.
>
> Regards,
> Michael
> ___________________________________________________
>
> SMA Solar Technology AG
> Aufsichtsrat: Guenther Cramer (Vorsitzender)
> Vorstand: Juergen Dolle, Roland Grebe, Uwe Hertel, Pierre-Pascal Urbon,
> Marko Werner
> Handelsregister: Amtsgericht Kassel HRB 3972
> Sitz der Gesellschaft: 34266 Niestetal
> USt-ID-Nr. DE 113 08 59 54
> WEEE-Reg.-Nr. DE 95881150
> ___________________________________________________
>
>
--
*Rick Curtis*
Re: Speed of fetching simple entities using OpenJPA
Posted by Rick Curtis <cu...@gmail.com>.
Michael -
I wouldn't expect to see that large of a difference of JPA vs JDBC. Can you
describe your scenario a little better? Are you issuing just a single JPQL
to select all of your Entities?
Thanks,
Rick
On Mon, Sep 12, 2011 at 1:36 PM, Michael Pflueger
<Mi...@sma.de>wrote:
> Hi,
> I compared JDBC and OpenJPA speed of fetching simple entities (consisting
> only of a single ID attribute). The speed difference seems quite severe.
>
> With JDBC, I can read 2 million entities in about 3 seconds when OS/DB
> caches are warm.
> With OpenJPA using slice (a single slice), it takes a bit above 30 seconds,
> so about ten times as long, more than I would expect.
> Using a direct connection without slice reduces this time to about 24
> seconds, ~8 times as much as the JDBC version.
>
> In the JDBC test I also create objects from the rows, so both tests include
> object creation overhead of the entities (plus a sum calculation of the
> IDs).
>
> I'm doing this test using PostgreSQL and an up to date OpenJPA 2.2
> snapshot.
> Entities are enhanced using javaagent enhancement.
>
> I tried optimizing OpenJPA performance with:
> <property name="openjpa.ConnectionRetainMode"
> value="always"></property>
> <property name="openjpa.ProxyManager" value="TrackChanges=false"/>
> <property name="openjpa.IgnoreChanges" value="true"/>
> <property name="openjpa.LockManager" value="none"/>
> <property name="openjpa.DetachState"
> value="fetch-groups(DetachedStateField=false)"/>
>
> But those didn't seem to affect performance in any significant way.
>
> Now, is such a large performance hit expected, and what would be the
> reason?
> It would be nice if some performance numbers like these would be in the
> documentation, by the way, to be better able to judge for what applications
> OpenJPA can be used/what performance penalties one has to expect, depending
> on the situation.
>
> Regards,
> Michael
> ___________________________________________________
>
> SMA Solar Technology AG
> Aufsichtsrat: Guenther Cramer (Vorsitzender)
> Vorstand: Juergen Dolle, Roland Grebe, Uwe Hertel, Pierre-Pascal Urbon,
> Marko Werner
> Handelsregister: Amtsgericht Kassel HRB 3972
> Sitz der Gesellschaft: 34266 Niestetal
> USt-ID-Nr. DE 113 08 59 54
> WEEE-Reg.-Nr. DE 95881150
> ___________________________________________________
>
>
--
*Rick Curtis*
Re: AW: Speed of fetching simple entities using OpenJPA
Posted by "M. Walter" <ma...@sbb.ch>.
In the meantime we were able to speed up things by only selecting the fields
from the database we really need in the application at application start.
You can do this with constructor expressions. You probably know that. We
approximately had 50 seconds to read and deliver 60'000 entities to the
remote client. Now this is done in 2 seconds with only reading two fields
instead of everything and the use of setBatchFetchSize() feature!
Michael Pflueger wrote:
>
> Hi,
> Thanks for your comment.
> Well, in your case it seems to be a DB problem though?
> Have you tried reading without cursors or using a different database?
>
--
View this message in context: http://openjpa.208410.n2.nabble.com/Speed-of-fetching-simple-entities-using-OpenJPA-tp6784781p6824183.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.
AW: Speed of fetching simple entities using OpenJPA
Posted by Michael Pflueger <Mi...@sma.de>.
Hi,
Thanks for your comment.
Well, in your case it seems to be a DB problem though?
Have you tried reading without cursors or using a different database?
-----Ursprüngliche Nachricht-----
Von: M. Walter [mailto:marc.walter@sbb.ch]
Gesendet: Dienstag, 13. September 2011 11:50
An: users@openjpa.apache.org
Betreff: Re: Speed of fetching simple entities using OpenJPA
Hi Michael,
I had a similar question about this topic. Reading 60000 rows and converting
them into entities needs 24 seconds (using OpenJPA 1.2.3). This is quite a
long time in my opinion.
Take a look:
http://openjpa.208410.n2.nabble.com/Entity-generation-is-very-time-consuming-tp6579389p6579389.html
Entity generation is very time consuming
Unfortunately I haven't found a solution to speed up things yet...
--
View this message in context: http://openjpa.208410.n2.nabble.com/Speed-of-fetching-simple-entities-using-OpenJPA-tp6784781p6786834.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.
___________________________________________________
SMA Solar Technology AG
Aufsichtsrat: Guenther Cramer (Vorsitzender)
Vorstand: Juergen Dolle, Roland Grebe, Uwe Hertel, Pierre-Pascal Urbon, Marko Werner
Handelsregister: Amtsgericht Kassel HRB 3972
Sitz der Gesellschaft: 34266 Niestetal
USt-ID-Nr. DE 113 08 59 54
WEEE-Reg.-Nr. DE 95881150
___________________________________________________
Re: Speed of fetching simple entities using OpenJPA
Posted by "M. Walter" <ma...@sbb.ch>.
Hi Michael,
I had a similar question about this topic. Reading 60000 rows and converting
them into entities needs 24 seconds (using OpenJPA 1.2.3). This is quite a
long time in my opinion.
Take a look:
http://openjpa.208410.n2.nabble.com/Entity-generation-is-very-time-consuming-tp6579389p6579389.html
Entity generation is very time consuming
Unfortunately I haven't found a solution to speed up things yet...
--
View this message in context: http://openjpa.208410.n2.nabble.com/Speed-of-fetching-simple-entities-using-OpenJPA-tp6784781p6786834.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.