Where are ByteBuffers allocated?

[Paul Mallach <pm...@ariva.de>]

Hi!

I'm debugging a memory leak in my MINA application. Of course the leak only
happens after a few hours of real world usage and not in my development
environment. :-) I need to understand where the actual memory for ByteBuffers 
comes from.

My problem is memory that seems to be allocated outside of the normal java
memory space. Normally my program uses about 400MB of heap memory, 30MB of
other memory and the java process (sun jre 1.5.0) uses 1.1GB of memory. But
after a few hours the process starts to grow rapidly until it dies, while
heap and other memory usage (as reported inside the jvm and in my profiler)
stays about the some.

I suspect the memory leak to be related to either MySQL (as an external
library) or ByteBuffers.

I double-checked that at every occurence of SQL statements all ResultSets and
Statements are always properly closed. So unless I'm hitting a bug in the
driver I think that everything should be fine here.

The other possibility is that I'm leaving some ByteBuffers hanging around.

I'm still using MINA 0.8.2, so all my ByteBuffers will probably be direct
buffers. Are they included in the number returned by
Runtime.getRuntime().totalMemory() (this number doesn't grow at all)? Would
the size of the allocated ByteBuffers be reflected in the heap size or in the
number of object on the heap?

Any other idea on how to find out where this alarming (non-heap) memory usage
is coming from?

bye, Paul.

--
ARIVA.DE AG, Paul Mallach, Produktentwicklung
Walkerdamm 17, 24103 Kiel
Tel. (+49) 0431 97108-23, Fax. (+49) 0431 97108-29
E-Mail: pm@ariva.de, Web: www.ariva.de <http://www.ariva.de/>

-------------------------------------------------------

-- 
ARIVA.DE AG, Paul Mallach, Produktentwicklung
Walkerdamm 17, 24103 Kiel
Tel. (+49) 0431 97108-23, Fax. (+49) 0431 97108-29 
E-Mail: pm@ariva.de, Web: www.ariva.de <http://www.ariva.de/>

Re: Where are ByteBuffers allocated?

[peter royal <pr...@apache.org>]

On May 11, 2006, at 5:01 PM, pm wrote:
> Im currently working on that anyway... But ByteBuffers are already  
> being
> reused with MINA 0.8, aren't they?

re-used yes, but once allocated, they will never go away, which can  
lead to memory problems.

> But at least debugging the ByteBuffers on the heap might be  
> easier. :-)

Certainly. Also, I highly recommend tools such as YourKit profiler,  
<http://www.yourkit.com/home/index.jsp>. You can take heap snapshots  
(or have it automatically take a snapshot once X% of the heap is  
used), and then peer in to see where the memory actually is.
-pete

-- 
proyal@apache.org - http://fotap.org/~osi

Re: Where are ByteBuffers allocated?

[pm <pm...@ariva.de>]

Hi, Peter.

Thanks for your quick reply.

> Direct ByteBuffer's won't show up in the heap. Try the latest release
> of MINA, you'll have to upgrade your app with the API changes, but
> the ByteBuffer pool now expires unused buffers, preventing memory
> growths like that.
Im currently working on that anyway... But ByteBuffers are already being 
reused with MINA 0.8, aren't they? 

Then everything seems to indicate, that - for some reason - my application  
sometimes needs a few hundred megabytes of ByteBuffers at the same time (and 
even though the memory might get reused afterwards, it isn't released, so the 
process doesn't shrink)? If my application really creates so many ByteBuffers 
at the same time that the process dies, then having MINA release unused 
ByteBuffers wouldn't help. Does that make sense?

But at least debugging the ByteBuffers on the heap might be easier. :-)

bye, Paul.

-- 
Paul Mallach
ARIVA.DE AG
Ostseekai 2
D - 24103 Kiel

Tel: +49 (0)431/97108-23               E-Mail: pm@ariva.de
Fax: +49 (0)431/97108-29               Internet: http://www.ariva.de

Re: Where are ByteBuffers allocated?

[peter royal <pr...@apache.org>]

On May 11, 2006, at 6:13 AM, Paul Mallach wrote:
> My problem is memory that seems to be allocated outside of the  
> normal java
> memory space. Normally my program uses about 400MB of heap memory,  
> 30MB of
> other memory and the java process (sun jre 1.5.0) uses 1.1GB of  
> memory. But
> after a few hours the process starts to grow rapidly until it dies,  
> while
> heap and other memory usage (as reported inside the jvm and in my  
> profiler)
> stays about the some.
>
> I suspect the memory leak to be related to either MySQL (as an  
> external
> library) or ByteBuffers.
>
> I double-checked that at every occurence of SQL statements all  
> ResultSets and
> Statements are always properly closed. So unless I'm hitting a bug  
> in the
> driver I think that everything should be fine here.
>
> The other possibility is that I'm leaving some ByteBuffers hanging  
> around.
>
> I'm still using MINA 0.8.2, so all my ByteBuffers will probably be  
> direct
> buffers. Are they included in the number returned by
> Runtime.getRuntime().totalMemory() (this number doesn't grow at  
> all)? Would
> the size of the allocated ByteBuffers be reflected in the heap size  
> or in the
> number of object on the heap?

Direct ByteBuffer's won't show up in the heap. Try the latest release  
of MINA, you'll have to upgrade your app with the API changes, but  
the ByteBuffer pool now expires unused buffers, preventing memory  
growths like that.
-pete

-- 
proyal@apache.org - http://fotap.org/~osi