Out Of Memory - String Operation

[Viks <vi...@yahoo.co.in>]

Hi,
   
  My app is running out of app.
   
  Code does a lot of string opeartion.It involves fetching records from DB
   
  Initially i was using string objects- Program used to finish in about 40-45 min with ~95 cpu usage 
   
  We did some optimization on the code like changing the query from String to StringBuffer.
   
  This has improved the execution time and cpu usage.
   
  For 60k records the prog ran fine ....execution time improved from 30 mins to around 4 min and cpu usage from 95 % to 15%.
   
  For around 100k records my apps goes down on memory.
   
  Wondering if the app is able to sustain on String Obj why it is going down with StringBuffer Objects.
   
   


Viks
 				
---------------------------------
 Here’s a new way to find what you're looking for - Yahoo! Answers 

Re: Out Of Memory - String Operation

[Mikolaj Rydzewski <mi...@ceti.pl>]

Viks wrote:
>    
>   My app is running out of app.
>   
Why do you think it is tomcat specific problem?

-- 
Mikolaj Rydzewski <mi...@ceti.pl>

[OT] Re: Out Of Memory - String Operation

[Christopher Schultz <ch...@christopherschultz.net>]

Viks,

> My app is running out of [memory].
> 
> Code does a lot of string opeartion. It involves fetching records
> from DB.

[snip]

> For around 100k records my apps goes down on memory.
> 
> Wondering if the app is able to sustain on String Obj why it is going
> down with StringBuffer Objects.

Do you have everything in one big method? If so, you might be collecting
local variables that cannot be GC'd, and so you're essentially leaking
memory.

I assume your program is something of the form:

conn = getConnection
ps = conn.prepareSatement
rs = ps.executeQuery

while(rs.next)
{
    // Do some stuff
}

Do you retain any data across each of the loop iterations? For example,
are you keeping an array/map/list/whatever of information around that
includes data from each row? If so, you may be simply running out of
memory. Do you know what the memory settings are for your JVM?

-chris

Re: Out Of Memory - String Operation

[David Delbecq <da...@oma.be>]

Viks wrote:
> Hi,
>    
>   My app is running out of app.
>    
>   Code does a lot of string opeartion.It involves fetching records from DB
>    
>   Initially i was using string objects- Program used to finish in about 40-45 min with ~95 cpu usage 
>   
String concatenation like this String c = a+b; is done internally like 
this in java:
String c = new StringBuffer(a).append(b).toString();
This explains why String is slower than StringBuffer.
>    
>   We did some optimization on the code like changing the query from String to StringBuffer.
>   
If you go java 1.5, prefer StringBuilder, it's not synchronized and as 
such works way faster on multi CPU computers (a synchronization is a 
costy operation on multi core / multi CPU systems, even if you only have 
one thread)
>    
>   This has improved the execution time and cpu usage.
>    
>   For 60k records the prog ran fine ....execution time improved from 30 mins to around 4 min and cpu usage from 95 % to 15%.
>    
>   For around 100k records my apps goes down on memory.
>    
>   Wondering if the app is able to sustain on String Obj why it is going down with StringBuffer Objects.
>   
String buffer, for performance reasons, have a somehow particular way of 
handling their growable array. Because a StringBuffer could be subject 
to 3 or 4 append() calls in a row (like when you do "This a message from 
"+sender+" to "+receiver), it increase it's internal array a bit more 
then needed to prevent multiple useless array expansion which is 
something costy. StringBuffer always attempts in sun implementation, to 
double it's internal array when it has become too short. If it's still 
not enough, it increase it to requested size. Here is code of 
'ensureCapacity' from the sun implementation (subject to sun licence):

char value[];
....
void expandCapacity(int minimumCapacity) {
int newCapacity = (value.length + 1) * 2;
if (newCapacity < 0) {
newCapacity = Integer.MAX_VALUE;
} else if (minimumCapacity > newCapacity) {
newCapacity = minimumCapacity;
}
char newValue[] = new char[newCapacity];
System.arraycopy(value, 0, newValue, 0, count);
value = newValue;
}
As a result, a stringbuffer can take up to about 2 times the size of 
corresponding String (worst case)

I suggest, when you are sure a StringBuffer will not be modified anymore 
in near future, convert it back to String

btw, simply increase the jvm allocated size is a good way to go away 
with out of memory if you don't want to spend lots of time optimizing 
your code.

Last but not least, what does it have to do with tomcat??
>    
>    
>
>
> Viks
>  				
> ---------------------------------
>  Here’s a new way to find what you're looking for - Yahoo! Answers 
>   


---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org