You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tapestry.apache.org by James Sherwood <js...@rgisolutions.com> on 2009/01/15 19:16:04 UTC
T5 and hibernate slowing with more records
Hello,
I am using POI to take an excel file and put it into a Mysql database using
hibernate.
I put a counter on the records going into the database and it starts off
fast but then slows to over 15 seconds a record and I cannot figure out why.
My code is basically this:
Public class DatabaseDump{
@Inject
Private Session _session;
Public void onAction(){ //they hit the go button
Loop rows{
_session.beginTransaction();
Loop cells{
Fill the
object from the cells
_session.save(Object);
}
_session.getTransaction().commit();
}
}
Everything works fine as in the database gets filled for the rows it has
completed but gradually slows to a crawl and at around 2700 records of 10500
records it's a record every 15 seconds.
I run the project under eclipse using Tomcat. The memory starts at around
80mb and only goes up to around 110mb.
What am I doing wrong?
--James
RE: T5 and hibernate slowing with more records
Posted by James Sherwood <js...@rgisolutions.com>.
Hello,
I suppose I could restrict the number of records allowed however I do not
believe that I need to. I believe there is a problem somewhere, maybe in
how I am doing the transactions or something.
Before using t5/hibernate we were using t3/torque and if that system would
have no problem with this I cannot see the newer technology having any
problem.
I guess I will ask the hibernate forums but I thought it would have
something to do with the way I use the session/transactions.
--James
-----Original Message-----
From: Szemere Szemere [mailto:szemereszemere@googlemail.com]
Sent: January-15-09 6:30 PM
To: Tapestry users
Subject: Re: T5 and hibernate slowing with more records
Feels like a Hibernate question, not Tapestry. Does your table have the
right index, do you need to have 10k transactions in a single web-request?
Szemere
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
RE: T5 and hibernate slowing with more records
Posted by James Sherwood <js...@rgisolutions.com>.
Hello,
Sorry I forgot to mention that I do not believe indexing is an issue as it
is only an 'over time' issue that it gets slow not a total number in the db
that causes the slowness. I can have 0 records or 40k records in the db and
the transactions are still blazing fast at the start.
--James
-----Original Message-----
From: Szemere Szemere [mailto:szemereszemere@googlemail.com]
Sent: January-15-09 6:30 PM
To: Tapestry users
Subject: Re: T5 and hibernate slowing with more records
Feels like a Hibernate question, not Tapestry. Does your table have the
right index, do you need to have 10k transactions in a single web-request?
Szemere
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
Re: T5 and hibernate slowing with more records
Posted by Szemere Szemere <sz...@googlemail.com>.
Feels like a Hibernate question, not Tapestry. Does your table have the
right index, do you need to have 10k transactions in a single web-request?
Szemere
RE: T5 and hibernate slowing with more records
Posted by James Sherwood <js...@rgisolutions.com>.
Hello,
Thank you for everyone's input.
I had originally taken out the sub queries from the main method and moved
them to another method(IE check if city is there if not add it and link the
foreign key) and then moved it to a map held in memory(still putting unknown
ones in the db) so there were less calls to the db. This didn't help but
then I moved back to the @CommitAfter(which was the original way I had
started) and it worked fine.
I like your clear idea and creating my own transactions. I think I'll give
that a go as well.
--James
-----Original Message-----
From: Jonathan Barker [mailto:jonathan.theitguy@gmail.com]
Sent: January-16-09 12:31 AM
To: 'Tapestry users'
Subject: RE: T5 and hibernate slowing with more records
Been there, hit that.
You need to clear() the session. Or, you can also evict() as you go.
Basically, every time Hibernate needs to commit something, it needs to flush
to the database, and it looks at all of the Hibernate objects that it knows
about - not just those to which you have current references. For every
object you handle, you add one more that it needs to check to see if it
needs to have changes flushed.
The clear() gets rid of anything you are not holding a reference to.
I tend to do it every ten records or so, but I've never tried to optimize
it.
The other option, as someone suggested, is to commit a batch rather than
single records, but that can be nasty for large sets. I'm working with an
import of about 250,000 records as I write this.
Jonathan
> -----Original Message-----
> From: James Sherwood [mailto:jsherwood@rgisolutions.com]
> Sent: Thursday, January 15, 2009 13:16
> To: 'Tapestry users'
> Subject: T5 and hibernate slowing with more records
>
> Hello,
>
>
>
> I am using POI to take an excel file and put it into a Mysql database
> using
> hibernate.
>
>
>
> I put a counter on the records going into the database and it starts off
> fast but then slows to over 15 seconds a record and I cannot figure out
> why.
>
>
>
> My code is basically this:
>
>
>
> Public class DatabaseDump{
>
>
>
> @Inject
>
> Private Session _session;
>
>
>
> Public void onAction(){ //they hit the go button
>
> Loop rows{
>
>
> _session.beginTransaction();
>
> Loop cells{
>
> Fill the
> object from the cells
>
>
> _session.save(Object);
>
> }
>
>
> _session.getTransaction().commit();
>
> }
>
> }
>
>
>
> Everything works fine as in the database gets filled for the rows it has
> completed but gradually slows to a crawl and at around 2700 records of
> 10500
> records it's a record every 15 seconds.
>
>
>
> I run the project under eclipse using Tomcat. The memory starts at around
> 80mb and only goes up to around 110mb.
>
>
>
> What am I doing wrong?
>
>
>
> --James
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
RE: T5 and hibernate slowing with more records
Posted by Jonathan Barker <jo...@gmail.com>.
Been there, hit that.
You need to clear() the session. Or, you can also evict() as you go.
Basically, every time Hibernate needs to commit something, it needs to flush
to the database, and it looks at all of the Hibernate objects that it knows
about - not just those to which you have current references. For every
object you handle, you add one more that it needs to check to see if it
needs to have changes flushed.
The clear() gets rid of anything you are not holding a reference to.
I tend to do it every ten records or so, but I've never tried to optimize
it.
The other option, as someone suggested, is to commit a batch rather than
single records, but that can be nasty for large sets. I'm working with an
import of about 250,000 records as I write this.
Jonathan
> -----Original Message-----
> From: James Sherwood [mailto:jsherwood@rgisolutions.com]
> Sent: Thursday, January 15, 2009 13:16
> To: 'Tapestry users'
> Subject: T5 and hibernate slowing with more records
>
> Hello,
>
>
>
> I am using POI to take an excel file and put it into a Mysql database
> using
> hibernate.
>
>
>
> I put a counter on the records going into the database and it starts off
> fast but then slows to over 15 seconds a record and I cannot figure out
> why.
>
>
>
> My code is basically this:
>
>
>
> Public class DatabaseDump{
>
>
>
> @Inject
>
> Private Session _session;
>
>
>
> Public void onAction(){ //they hit the go button
>
> Loop rows{
>
>
> _session.beginTransaction();
>
> Loop cells{
>
> Fill the
> object from the cells
>
>
> _session.save(Object);
>
> }
>
>
> _session.getTransaction().commit();
>
> }
>
> }
>
>
>
> Everything works fine as in the database gets filled for the rows it has
> completed but gradually slows to a crawl and at around 2700 records of
> 10500
> records it's a record every 15 seconds.
>
>
>
> I run the project under eclipse using Tomcat. The memory starts at around
> 80mb and only goes up to around 110mb.
>
>
>
> What am I doing wrong?
>
>
>
> --James
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
RE: Re: T5 and hibernate slowing with more records
Posted by Angelo Chen <an...@yahoo.com.hk>.
a quick way is:
instead of creating a service, make something like this
@CommitAfter
private void saveThem(List<Myobjt> lst) {
// put code here to save objects, don't beging a transaction as it has been
started by Tapestry-Hibernate
}
how long?
James Sherwood wrote:
>
> Hello,
>
> It took 13 seconds to create the List with 14000 objects.
>
> At 1 hour 45 minutes I was at 8600 records committing each one.
>
> For the service, I haven't went that far in T5 yet so I will have to look
> it
> up.
>
> If there was a way to save the whole list at once using the session that
> might be faster as well.
>
> --James
>
> -----Original Message-----
> From: Angelo Chen [mailto:angelochen960@yahoo.com.hk]
> Sent: January-15-09 9:33 PM
> To: users@tapestry.apache.org
> Subject: [Spam Rating:3.18] Re: T5 and hibernate slowing with more records
>
>
> Hi James,
>
> I do not see any reason why it is slow, but would suggest to do following:
>
> 1) try to put everything into a list of objects without saving to the
> database, slow?
> 2) use a service to accept the list from #1, beging a transaction, save,
> commit, slow?
>
>
> Angelo
>
>
> James Sherwood wrote:
>>
>> Hello,
>>
>>
>>
>> I am using POI to take an excel file and put it into a Mysql database
>> using
>> hibernate.
>>
>>
>>
>> I put a counter on the records going into the database and it starts off
>> fast but then slows to over 15 seconds a record and I cannot figure out
>> why.
>>
>>
>>
>> My code is basically this:
>>
>>
>>
>> Public class DatabaseDump{
>>
>>
>>
>> @Inject
>>
>> Private Session _session;
>>
>>
>>
>> Public void onAction(){ //they hit the go button
>>
>> Loop rows{
>>
>>
>> _session.beginTransaction();
>>
>> Loop cells{
>>
>> Fill the
>> object from the cells
>>
>>
>> _session.save(Object);
>>
>> }
>>
>>
>> _session.getTransaction().commit();
>>
>> }
>>
>> }
>>
>>
>>
>> Everything works fine as in the database gets filled for the rows it has
>> completed but gradually slows to a crawl and at around 2700 records of
>> 10500
>> records it's a record every 15 seconds.
>>
>>
>>
>> I run the project under eclipse using Tomcat. The memory starts at
>> around
>> 80mb and only goes up to around 110mb.
>>
>>
>>
>> What am I doing wrong?
>>
>>
>>
>> --James
>>
>>
>>
>>
>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/T5-and-hibernate-slowing-with-more-records-tp21483684p
> 21490838.html
> Sent from the Tapestry - User mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
> For additional commands, e-mail: users-help@tapestry.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
> For additional commands, e-mail: users-help@tapestry.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/T5-and-hibernate-slowing-with-more-records-tp21483684p21491243.html
Sent from the Tapestry - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
Re: T5 and hibernate slowing with more records
Posted by Christian Edward Gruber <ch...@gmail.com>.
Can you avoid committing the transaction on every item? Committing
takes more database effort than the initial update/insert, so batching
these within one transaction is highly advisable if it's possible.
Christian.
On 15-Jan-09, at 21:02 , James Sherwood wrote:
> Hello,
>
> It took 13 seconds to create the List with 14000 objects.
>
> At 1 hour 45 minutes I was at 8600 records committing each one.
>
> For the service, I haven't went that far in T5 yet so I will have to
> look it
> up.
>
> If there was a way to save the whole list at once using the session
> that
> might be faster as well.
>
> --James
>
> -----Original Message-----
> From: Angelo Chen [mailto:angelochen960@yahoo.com.hk]
> Sent: January-15-09 9:33 PM
> To: users@tapestry.apache.org
> Subject: [Spam Rating:3.18] Re: T5 and hibernate slowing with more
> records
>
>
> Hi James,
>
> I do not see any reason why it is slow, but would suggest to do
> following:
>
> 1) try to put everything into a list of objects without saving to the
> database, slow?
> 2) use a service to accept the list from #1, beging a transaction,
> save,
> commit, slow?
>
>
> Angelo
>
>
> James Sherwood wrote:
>>
>> Hello,
>>
>>
>>
>> I am using POI to take an excel file and put it into a Mysql database
>> using
>> hibernate.
>>
>>
>>
>> I put a counter on the records going into the database and it
>> starts off
>> fast but then slows to over 15 seconds a record and I cannot figure
>> out
>> why.
>>
>>
>>
>> My code is basically this:
>>
>>
>>
>> Public class DatabaseDump{
>>
>>
>>
>> @Inject
>>
>> Private Session _session;
>>
>>
>>
>> Public void onAction(){ //they hit the go button
>>
>> Loop rows{
>>
>>
>> _session.beginTransaction();
>>
>> Loop cells{
>>
>> Fill
>> the
>> object from the cells
>>
>>
>> _session.save(Object);
>>
>> }
>>
>>
>> _session.getTransaction().commit();
>>
>> }
>>
>> }
>>
>>
>>
>> Everything works fine as in the database gets filled for the rows
>> it has
>> completed but gradually slows to a crawl and at around 2700 records
>> of
>> 10500
>> records it's a record every 15 seconds.
>>
>>
>>
>> I run the project under eclipse using Tomcat. The memory starts at
>> around
>> 80mb and only goes up to around 110mb.
>>
>>
>>
>> What am I doing wrong?
>>
>>
>>
>> --James
>>
>>
>>
>>
>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/T5-and-hibernate-slowing-with-more-records-tp21483684p
> 21490838.html
> Sent from the Tapestry - User mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
> For additional commands, e-mail: users-help@tapestry.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
> For additional commands, e-mail: users-help@tapestry.apache.org
>
Christian Edward Gruber
christianedwardgruber@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
RE: Re: T5 and hibernate slowing with more records
Posted by James Sherwood <js...@rgisolutions.com>.
Hello,
It took 13 seconds to create the List with 14000 objects.
At 1 hour 45 minutes I was at 8600 records committing each one.
For the service, I haven't went that far in T5 yet so I will have to look it
up.
If there was a way to save the whole list at once using the session that
might be faster as well.
--James
-----Original Message-----
From: Angelo Chen [mailto:angelochen960@yahoo.com.hk]
Sent: January-15-09 9:33 PM
To: users@tapestry.apache.org
Subject: [Spam Rating:3.18] Re: T5 and hibernate slowing with more records
Hi James,
I do not see any reason why it is slow, but would suggest to do following:
1) try to put everything into a list of objects without saving to the
database, slow?
2) use a service to accept the list from #1, beging a transaction, save,
commit, slow?
Angelo
James Sherwood wrote:
>
> Hello,
>
>
>
> I am using POI to take an excel file and put it into a Mysql database
> using
> hibernate.
>
>
>
> I put a counter on the records going into the database and it starts off
> fast but then slows to over 15 seconds a record and I cannot figure out
> why.
>
>
>
> My code is basically this:
>
>
>
> Public class DatabaseDump{
>
>
>
> @Inject
>
> Private Session _session;
>
>
>
> Public void onAction(){ //they hit the go button
>
> Loop rows{
>
>
> _session.beginTransaction();
>
> Loop cells{
>
> Fill the
> object from the cells
>
>
> _session.save(Object);
>
> }
>
>
> _session.getTransaction().commit();
>
> }
>
> }
>
>
>
> Everything works fine as in the database gets filled for the rows it has
> completed but gradually slows to a crawl and at around 2700 records of
> 10500
> records it's a record every 15 seconds.
>
>
>
> I run the project under eclipse using Tomcat. The memory starts at around
> 80mb and only goes up to around 110mb.
>
>
>
> What am I doing wrong?
>
>
>
> --James
>
>
>
>
>
>
>
--
View this message in context:
http://www.nabble.com/T5-and-hibernate-slowing-with-more-records-tp21483684p
21490838.html
Sent from the Tapestry - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
Re: T5 and hibernate slowing with more records
Posted by Angelo Chen <an...@yahoo.com.hk>.
Hi James,
I do not see any reason why it is slow, but would suggest to do following:
1) try to put everything into a list of objects without saving to the
database, slow?
2) use a service to accept the list from #1, beging a transaction, save,
commit, slow?
Angelo
James Sherwood wrote:
>
> Hello,
>
>
>
> I am using POI to take an excel file and put it into a Mysql database
> using
> hibernate.
>
>
>
> I put a counter on the records going into the database and it starts off
> fast but then slows to over 15 seconds a record and I cannot figure out
> why.
>
>
>
> My code is basically this:
>
>
>
> Public class DatabaseDump{
>
>
>
> @Inject
>
> Private Session _session;
>
>
>
> Public void onAction(){ //they hit the go button
>
> Loop rows{
>
>
> _session.beginTransaction();
>
> Loop cells{
>
> Fill the
> object from the cells
>
>
> _session.save(Object);
>
> }
>
>
> _session.getTransaction().commit();
>
> }
>
> }
>
>
>
> Everything works fine as in the database gets filled for the rows it has
> completed but gradually slows to a crawl and at around 2700 records of
> 10500
> records it's a record every 15 seconds.
>
>
>
> I run the project under eclipse using Tomcat. The memory starts at around
> 80mb and only goes up to around 110mb.
>
>
>
> What am I doing wrong?
>
>
>
> --James
>
>
>
>
>
>
>
--
View this message in context: http://www.nabble.com/T5-and-hibernate-slowing-with-more-records-tp21483684p21490838.html
Sent from the Tapestry - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org