You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@mina.apache.org by Gerrit Grobbelaar <ge...@jeasy.net> on 2008/07/26 09:33:25 UTC
byte[] usage - not GCd
Hi,
I'm having an issue with OOM - here is my story:
Criteria:
--------
- concurrent clients connecting to a chatserver, need to broadcast the
messages
- clients keep same connection (each one having its own IoSession during
lifetime)
Symptoms:
--------
- massive amounts of byte[] allocations that just grows and grows, until OOM.
- still OOM even if i have 100 bots connected, sending one message each every
5 minutes
When I have 1 - 5 clients connected (all simulated with bots) that send one
message per bot every 2 - 5 seconds, heap space stays under control with GC.
Configs/Tools:
-------------
- JVM: 1.6u7
- VM arguments: -Xms128m -Xmx384m
- Operating System: Linux 2.6.15.7
- Architecture: i386
- Number of processors: 2
- Model: Intel(R) Pentium(R) 4 CPU 2.80GHz
- Committed virtual memory: 681,564 kbytes
- Heap buffer allocation (Direct is very insufficient)
- SimpleByteBufferAllocator (do note I have same result with
PooledByteBufferAllocator)
- YourKit and JProfiler
Both YourKit and JProfiler point fingers at:
byte[] allocation due to org.apache.mina.common.IoSession.write() calls
What could I possibly be doing wrong that I will have the above scenario where
the byte[] allocations never GC properly?
Thanks,
Gerrit
Re: byte[] usage - not GCd
Posted by Sangjin Lee <sj...@gmail.com>.
OK thanks. Just to eliminate the obvious, one logical possibility that the
buffers are not garbage collected might be if your request objects have
somehow retained references to the buffers (direct or indirect). I assume
your request objects will be in scope as long as they are outstanding so
they are long-lived. So any objects that are connected to the request
objects will not be collected... My 2 cents.
Sangjin
On Sat, Jul 26, 2008 at 9:15 AM, Gerrit Grobbelaar <ge...@jeasy.net> wrote:
> Hi there,
>
> I will package the problem-simulation soon, but in the meantime I also want
> to
> port the bots to be MINA based as well, instead of the current own-rolled
> blocking IO architecture.
>
> The reason why I'm saying that is because I find it weird that my Flash
> clients stay connected during this disconnection process and only the Java
> bots die.
>
> Thanks,
> Gerrit
>
> > -----Original Message-----
> > From: "Sangjin Lee" <sj...@gmail.com>
> > Sent: Saturday 26 July 2008 18:04
> > To: users@mina.apache.org
> > CC:
> > Subject: Re: byte[] usage - not GCd
> >
> >
> > Would you be able to create simple code that demonstrates the memory leak
> > behavior? I haven't played extensively with server-side load, but in my
> > experience at least with the client-side load shows no such memory
> > behavior, and we push it pretty hard. A code sample would be very
> helpful.
> > Thanks,
> > Sangjin
> >
> > On Sat, Jul 26, 2008 at 8:04 AM, Gerrit Grobbelaar <ge...@jeasy.net>
> wrote:
> > > What I get from this is that MINA (out of the box) might not be the
> > > solution
> > > for me in a production environment. 2.0.0 looks to perform a bit
> better,
> > > but
> > > the 50 clients, 1 message per client every 2 seconds, also get
> > > disconnected after a while.
> > >
> > > Granted, all my clients are simulated from one Host machine on a LAN,
> but
> > > surely that can't make a difference?
> > >
> > > I've checked src out there on the net, and my IoSession/IoBuffer
> handling
> > > seems to be correct. The area where my problem is is situated outside
> of
> > > my
> > > domain in anyway, that is the NIO handling of the Mina library.
> > >
> > > I've been reading such good reviews about this framework, but just
> > > frustrating
> > > that I can't get the same out of it due to the buffer handling (byte[]
> > > allocations).
> > >
> > > I've stumbled onto this:
> > > =========>
> > > However, there's a case that allocation of heap buffers becomes
> > > bottleneck if
> > > the allocation occurs too fast. It's because any allocation of byte
> > > arrays requires filling up the content of the array with '0', which
> > > consumes memory
> > > bandwidth. CachedBufferAllocator is provided to address this concern,
> but
> > > you
> > > won't need to change the default SimpleBufferAllocator in most cases.
> > > <=========
> > > Which confirms to me that if I'm reading and writing on these buffers
> at
> > > ridiculous amounts, then MINA will not be of any benefit to me out of
> the
> > > box.
> > >
> > >
> > > I'll appreciate any 2 cents added.
> > >
> > > Thanks,
> > > Gerrit
> > >
> > > > -----Original Message-----
> > > > From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > > Sent: Saturday 26 July 2008 16:33
> > > > To: users@mina.apache.org
> > > > CC:
> > > > Subject: Re: byte[] usage - not GCd
> > > >
> > > >
> > > > I have MUCH better results with MINA 2.0.0 M2, but after a short
> while
> > >
> > > all
> > >
> > > > clients/bots disconnect with:
> > > >
> > > > An error occurred : null
> > > > org.apache.mina.core.write.WriteTimeoutException
> > > >
> > > > Any ideas anyone?
> > > >
> > > > Thanks,
> > > > Gerrit
> > > >
> > > > > -----Original Message-----
> > > > > From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > > > Sent: Saturday 26 July 2008 14:14
> > > > > To: users@mina.apache.org
> > > > > CC:
> > > > > Subject: Re: byte[] usage - not GCd
> > > > >
> > > > >
> > > > > MINA 1.1.4 performs even better than 1.1.5.
> > > > >
> > > > > I still believe I'm doing something wrong in my code though, as I
> > > > > read only good things about MINA in the reviews, but for me it just
> > > > > seems to crumble underneath heavy load, the buffers not clearing
> > > > > quickly enough
> > >
> > > to
> > >
> > > > > be GCd.
> > > > >
> > > > > ==============>
> > > > > MINA 1.1.4 - 30 bots
> > > > > 1. 30 bots - each sending one message every 2 seconds
> > > > > 1.1. after 10 minutes
> > > > > 1.1.1. Heap size - 7MB
> > > > > 1.1.2. byte[] allocation:
> > > > > 1.1.2.1. no. of objects - 3,791:
> > > > > 1.1.2.2. shallow size - 996,320
> > > > > 1.2. after killing 30 bots
> > > > > 1.2.1. Heap size - 2,7MB
> > > > > 1.2.2. byte[] allocation:
> > > > > 1.2.2.1. no. of objects - 3,051:
> > > > > 1.2.2.2. shallow size - 659,344
> > > > > <==============
> > > > >
> > > > > ==============>
> > > > > MINA 1.1.4 - 100 bots
> > > > > 2. 100 bots - each sending one message every 2 seconds
> > > > > 2.1. after 2 minutes
> > > > > 2.1.1. Heap size - 329MB - JVM unresponsive
> > > > > 2.1.2. byte[] allocation:
> > > > > 2.1.2.1. no. of objects - 423,512:
> > > > > 2.1.2.2. shallow size - 201,053,744
> > > > >
> > > > > 2.2. after killing 100 bots
> > > > > 2.2.1. no effect - JVM unresponsive
> > > > >
> > > > > Interesting part: after about 1 minute this time, the server
> starts
> > > > > spewing out logs again, seems to have some action on the buffers
> > > > > again, processing messages still sent a while back, obviously
> > > > > printing out
> > >
> > > that
> > >
> > > > > the session are closed already. Very soon all bots disconnected,
> > >
> > > freeing
> > >
> > > > > up all the heap mem again.
> > > > > <=============
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > > > > Sent: Saturday 26 July 2008 12:03
> > > > > > To: users@mina.apache.org
> > > > > > CC:
> > > > > > Subject: Re: byte[] usage - not GCd
> > > > > >
> > > > > >
> > > > > > Some more test results (I definitely have better results with
> MINA
> > > > > > 1.1.5, but still not ideal under load). Seeing that I get
> > > > > > different, and somewhat better, result with an older version, I'm
> > > > > > going to try even older versions than 1.1.5 now.
> > > > > >
> > > > > > In the meantime: howcome these different results though, and why
> > >
> > > will
> > >
> > > > > > 1.1.5 perform better with Buffer (byte[])
> > >
> > > handling/allocations/flushing
> > >
> > > > > > than 1.1.6 and 1.1.7? Could be that I'm definitely doing
> something
> > > > > > wrong with my MINA integration, but then I should get consistent
> > > > > > bad performances across all versions?
> > > > > >
> > > > > > MINA 1.1.5 results as follows:
> > > > > >
> > > > > > ==============>
> > > > > > MINA 1.1.5 - 30 bots
> > > > > > 1. 30 bots - each sending one message every 2 seconds
> > > > > > 1.1. after 10 minutes
> > > > > > 1.1.1. Heap size - 20MB
> > > > > > 1.1.2. byte[] allocation:
> > > > > > 1.1.2.1. no. of objects - 12,104:
> > > > > > 1.1.2.2. shallow size - 4,207,440
> > > > > > 1.2. after killing 30 bots
> > > > > > 1.2.1. Heap size - 1,5MB
> > > > > > 1.2.2. byte[] allocation:
> > > > > > 1.2.2.1. no. of objects - 3,794:
> > > > > > 1.2.2.2. shallow size - 722,696
> > > > > > <==============
> > > > > >
> > > > > > ==============>
> > > > > > MINA 1.1.5 - 100 bots
> > > > > > 2. 100 bots - each sending one message every 2 seconds
> > > > > > 2.1. after 3 minutes
> > > > > > 2.1.1. Heap size - 374MB
> > > > > > 2.1.2. byte[] allocation:
> > > > > > 2.1.2.1. no. of objects - 455,736:
> > > > > > 2.1.2.2. shallow size - 216,055,400
> > > > > >
> > > > > > 2.2. after killing 30 bots
> > > > > > 2.2.1. no effect - JVM unresponsive
> > > > > >
> > > > > > Interesting part: after about 2 - 3 minutes, the server starts
> > >
> > > spewing
> > >
> > > > > > out logs again, seems to have some action on the buffers again,
> > > > > > processing messages still sent a while back, obviously printing
> out
> > > > > > that the session are closed already. Only after about 5 minutes
> it
> > >
> > > is
> > >
> > > > > > starting to detect the client disconnections. And then hangs
> > > > > > again, after disconnecting only 14 of the 100 bots. There it
> > > > > > continues
> > >
> > > again,
> > >
> > > > > > still trying to send chat messages that was still in the buffers.
> > >
> > > Now
> > >
> > > > > > detected 40 disconnections after another 2 minutes. Do note,
> Heap
> > >
> > > mem
> > >
> > > > > > down to 40MB from 374MB. There we go: all disconnected now,
> > > > > > perform GC, 2MB heap used.
> > > > > >
> > > > > > What could I be doing wrong that it seems that these buffers do
> not
> > > > > > flush quickly enough, getting bogged down with the data?
> > > > > > <=============
> > > > > >
> > > > > > ==============>
> > > > > > MINA 1.1.5 - 100 bots
> > > > > > 2. 100 bots - each sending one message every 5 minutes, and flash
> > > > > > client sending message every 1 - 3 seconds
> > > > > > 2.1. after 10minutes
> > > > > > 2.1.1. Heap size - 165MB
> > > > > > 2.1.2. byte[] allocation:
> > > > > > 2.1.2.1. no. of objects - 180,352
> > > > > > 2.1.2.2. shallow size - 103,458,904
> > > > > >
> > > > > > 1.2. after killing 100 bots
> > > > > > 1.2.1. Heap size - 1,5MB
> > > > > > 1.2.2. byte[] allocation:
> > > > > > 1.2.2.1. no. of objects - 3,890:
> > > > > > 1.2.2.2. shallow size - 792,536
> > > > > > <=============
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > > > > > Sent: Saturday 26 July 2008 10:32
> > > > > > > To: users@mina.apache.org
> > > > > > > CC:
> > > > > > > Subject: Re: byte[] usage - not GCd
> > > > > > >
> > > > > > >
> > > > > > > Yes, 1.1.7 but 1.1.6 as well.
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Emmanuel Lecharny <el...@gmail.com>
> > > > > > > > Sent: Saturday 26 July 2008 10:20
> > > > > > > > To: users@mina.apache.org
> > > > > > > > CC:
> > > > > > > > Subject: Re: byte[] usage - not GCd
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > MINA 1.1.7 ?
> > > > > > > >
> > > > > > > > Gerrit Grobbelaar wrote:
> > > > > > > > > YourKit shows the Class list as follows:
> > > > > > > > > - java.nio.HeapByteBuffer
> > > > > > > > > -
> > >
> > > org.apache.mina.common.SimpleByteBufferAllocator$SimpleByteBuffer
> > >
> > > > > > > > > -
> > >
> > > org.apache.mina.filter.codec.ProtocolCodecFilter$HiddenByteBuffer
> > >
> > > > > > > > > - org.apache.mina.common.IoFilter$WriteRequest
> > > > > > > > > - java.util.concurrent.ConcurrentLinkedQueue$Node
> > > > > > > > >
> > > > > > > > > Note the above contributes 99%+ for the byte[] allocations,
> > > > > > > > > and the byte[] allocations are causing the OOM.
> > > > > > > > >
> > > > > > > > >> -----Original Message-----
> > > > > > > > >> From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > > > > > > >> Sent: Saturday 26 July 2008 09:48
> > > > > > > > >> To: users@mina.apache.org
> > > > > > > > >> CC:
> > > > > > > > >> Subject: Re: byte[] usage - not GCd
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> Another symptom:
> > > > > > > > >> - 100 Java client bots connected, sending NO messages.
> > > > > > > > >> - 1 Flash client, typing in a message every 2 seconds
> > > > > > > > >> - memory consumption (byte[] allocations) increases and a
> > >
> > > Mark
> > >
> > > > > > > > >> Sweep with JConsole doesn't do anything (stays on 100s of
> > >
> > > MBs).
> > >
> > > > > > > > >> - kill off all bots
> > > > > > > > >> - Mark sweep
> > > > > > > > >> - Heap mem usage drops to about 2 - 5MB
> > > > > > > > >>
> > > > > > > > >>> -----Original Message-----
> > > > > > > > >>> From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > > > > > > >>> Sent: Saturday 26 July 2008 09:33
> > > > > > > > >>> To: users@mina.apache.org
> > > > > > > > >>> CC:
> > > > > > > > >>> Subject: byte[] usage - not GCd
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> Hi,
> > > > > > > > >>>
> > > > > > > > >>> I'm having an issue with OOM - here is my story:
> > > > > > > > >>>
> > > > > > > > >>> Criteria:
> > > > > > > > >>> --------
> > > > > > > > >>> - concurrent clients connecting to a chatserver, need to
> > > > > > > > >>> broadcast the messages
> > > > > > > > >>> - clients keep same connection (each one having its own
> > > > > > > > >>> IoSession during lifetime)
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> Symptoms:
> > > > > > > > >>> --------
> > > > > > > > >>> - massive amounts of byte[] allocations that just grows
> > > > > > > > >>> and grows, until OOM. - still OOM even if i have 100 bots
> > > > > > > > >>> connected, sending one message each every 5 minutes
> > > > > > > > >>>
> > > > > > > > >>> When I have 1 - 5 clients connected (all simulated with
> > > > > > > > >>> bots) that send one message per bot every 2 - 5 seconds,
> > > > > > > > >>> heap space stays under control with GC.
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> Configs/Tools:
> > > > > > > > >>> -------------
> > > > > > > > >>> - JVM: 1.6u7
> > > > > > > > >>> - VM arguments: -Xms128m -Xmx384m
> > > > > > > > >>> - Operating System: Linux 2.6.15.7
> > > > > > > > >>> - Architecture: i386
> > > > > > > > >>> - Number of processors: 2
> > > > > > > > >>> - Model: Intel(R) Pentium(R) 4 CPU 2.80GHz
> > > > > > > > >>> - Committed virtual memory: 681,564 kbytes
> > > > > > > > >>> - Heap buffer allocation (Direct is very insufficient)
> > > > > > > > >>> - SimpleByteBufferAllocator (do note I have same result
> > > > > > > > >>> with PooledByteBufferAllocator)
> > > > > > > > >>> - YourKit and JProfiler
> > > > > > > > >>>
> > > > > > > > >>> Both YourKit and JProfiler point fingers at:
> > > > > > > > >>> byte[] allocation due to
> > > > > > > > >>> org.apache.mina.common.IoSession.write() calls
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> What could I possibly be doing wrong that I will have the
> > >
> > > above
> > >
> > > > > > > > >>> scenario where the byte[] allocations never GC properly?
> > > > > > > > >>>
> > > > > > > > >>> Thanks,
> > > > > > > > >>> Gerrit
>
Re: byte[] usage - not GCd
Posted by Gerrit Grobbelaar <ge...@jeasy.net>.
Hi there,
I will package the problem-simulation soon, but in the meantime I also want to
port the bots to be MINA based as well, instead of the current own-rolled
blocking IO architecture.
The reason why I'm saying that is because I find it weird that my Flash
clients stay connected during this disconnection process and only the Java
bots die.
Thanks,
Gerrit
> -----Original Message-----
> From: "Sangjin Lee" <sj...@gmail.com>
> Sent: Saturday 26 July 2008 18:04
> To: users@mina.apache.org
> CC:
> Subject: Re: byte[] usage - not GCd
>
>
> Would you be able to create simple code that demonstrates the memory leak
> behavior? I haven't played extensively with server-side load, but in my
> experience at least with the client-side load shows no such memory
> behavior, and we push it pretty hard. A code sample would be very helpful.
> Thanks,
> Sangjin
>
> On Sat, Jul 26, 2008 at 8:04 AM, Gerrit Grobbelaar <ge...@jeasy.net> wrote:
> > What I get from this is that MINA (out of the box) might not be the
> > solution
> > for me in a production environment. 2.0.0 looks to perform a bit better,
> > but
> > the 50 clients, 1 message per client every 2 seconds, also get
> > disconnected after a while.
> >
> > Granted, all my clients are simulated from one Host machine on a LAN, but
> > surely that can't make a difference?
> >
> > I've checked src out there on the net, and my IoSession/IoBuffer handling
> > seems to be correct. The area where my problem is is situated outside of
> > my
> > domain in anyway, that is the NIO handling of the Mina library.
> >
> > I've been reading such good reviews about this framework, but just
> > frustrating
> > that I can't get the same out of it due to the buffer handling (byte[]
> > allocations).
> >
> > I've stumbled onto this:
> > =========>
> > However, there's a case that allocation of heap buffers becomes
> > bottleneck if
> > the allocation occurs too fast. It's because any allocation of byte
> > arrays requires filling up the content of the array with '0', which
> > consumes memory
> > bandwidth. CachedBufferAllocator is provided to address this concern, but
> > you
> > won't need to change the default SimpleBufferAllocator in most cases.
> > <=========
> > Which confirms to me that if I'm reading and writing on these buffers at
> > ridiculous amounts, then MINA will not be of any benefit to me out of the
> > box.
> >
> >
> > I'll appreciate any 2 cents added.
> >
> > Thanks,
> > Gerrit
> >
> > > -----Original Message-----
> > > From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > Sent: Saturday 26 July 2008 16:33
> > > To: users@mina.apache.org
> > > CC:
> > > Subject: Re: byte[] usage - not GCd
> > >
> > >
> > > I have MUCH better results with MINA 2.0.0 M2, but after a short while
> >
> > all
> >
> > > clients/bots disconnect with:
> > >
> > > An error occurred : null
> > > org.apache.mina.core.write.WriteTimeoutException
> > >
> > > Any ideas anyone?
> > >
> > > Thanks,
> > > Gerrit
> > >
> > > > -----Original Message-----
> > > > From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > > Sent: Saturday 26 July 2008 14:14
> > > > To: users@mina.apache.org
> > > > CC:
> > > > Subject: Re: byte[] usage - not GCd
> > > >
> > > >
> > > > MINA 1.1.4 performs even better than 1.1.5.
> > > >
> > > > I still believe I'm doing something wrong in my code though, as I
> > > > read only good things about MINA in the reviews, but for me it just
> > > > seems to crumble underneath heavy load, the buffers not clearing
> > > > quickly enough
> >
> > to
> >
> > > > be GCd.
> > > >
> > > > ==============>
> > > > MINA 1.1.4 - 30 bots
> > > > 1. 30 bots - each sending one message every 2 seconds
> > > > 1.1. after 10 minutes
> > > > 1.1.1. Heap size - 7MB
> > > > 1.1.2. byte[] allocation:
> > > > 1.1.2.1. no. of objects - 3,791:
> > > > 1.1.2.2. shallow size - 996,320
> > > > 1.2. after killing 30 bots
> > > > 1.2.1. Heap size - 2,7MB
> > > > 1.2.2. byte[] allocation:
> > > > 1.2.2.1. no. of objects - 3,051:
> > > > 1.2.2.2. shallow size - 659,344
> > > > <==============
> > > >
> > > > ==============>
> > > > MINA 1.1.4 - 100 bots
> > > > 2. 100 bots - each sending one message every 2 seconds
> > > > 2.1. after 2 minutes
> > > > 2.1.1. Heap size - 329MB - JVM unresponsive
> > > > 2.1.2. byte[] allocation:
> > > > 2.1.2.1. no. of objects - 423,512:
> > > > 2.1.2.2. shallow size - 201,053,744
> > > >
> > > > 2.2. after killing 100 bots
> > > > 2.2.1. no effect - JVM unresponsive
> > > >
> > > > Interesting part: after about 1 minute this time, the server starts
> > > > spewing out logs again, seems to have some action on the buffers
> > > > again, processing messages still sent a while back, obviously
> > > > printing out
> >
> > that
> >
> > > > the session are closed already. Very soon all bots disconnected,
> >
> > freeing
> >
> > > > up all the heap mem again.
> > > > <=============
> > > >
> > > > > -----Original Message-----
> > > > > From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > > > Sent: Saturday 26 July 2008 12:03
> > > > > To: users@mina.apache.org
> > > > > CC:
> > > > > Subject: Re: byte[] usage - not GCd
> > > > >
> > > > >
> > > > > Some more test results (I definitely have better results with MINA
> > > > > 1.1.5, but still not ideal under load). Seeing that I get
> > > > > different, and somewhat better, result with an older version, I'm
> > > > > going to try even older versions than 1.1.5 now.
> > > > >
> > > > > In the meantime: howcome these different results though, and why
> >
> > will
> >
> > > > > 1.1.5 perform better with Buffer (byte[])
> >
> > handling/allocations/flushing
> >
> > > > > than 1.1.6 and 1.1.7? Could be that I'm definitely doing something
> > > > > wrong with my MINA integration, but then I should get consistent
> > > > > bad performances across all versions?
> > > > >
> > > > > MINA 1.1.5 results as follows:
> > > > >
> > > > > ==============>
> > > > > MINA 1.1.5 - 30 bots
> > > > > 1. 30 bots - each sending one message every 2 seconds
> > > > > 1.1. after 10 minutes
> > > > > 1.1.1. Heap size - 20MB
> > > > > 1.1.2. byte[] allocation:
> > > > > 1.1.2.1. no. of objects - 12,104:
> > > > > 1.1.2.2. shallow size - 4,207,440
> > > > > 1.2. after killing 30 bots
> > > > > 1.2.1. Heap size - 1,5MB
> > > > > 1.2.2. byte[] allocation:
> > > > > 1.2.2.1. no. of objects - 3,794:
> > > > > 1.2.2.2. shallow size - 722,696
> > > > > <==============
> > > > >
> > > > > ==============>
> > > > > MINA 1.1.5 - 100 bots
> > > > > 2. 100 bots - each sending one message every 2 seconds
> > > > > 2.1. after 3 minutes
> > > > > 2.1.1. Heap size - 374MB
> > > > > 2.1.2. byte[] allocation:
> > > > > 2.1.2.1. no. of objects - 455,736:
> > > > > 2.1.2.2. shallow size - 216,055,400
> > > > >
> > > > > 2.2. after killing 30 bots
> > > > > 2.2.1. no effect - JVM unresponsive
> > > > >
> > > > > Interesting part: after about 2 - 3 minutes, the server starts
> >
> > spewing
> >
> > > > > out logs again, seems to have some action on the buffers again,
> > > > > processing messages still sent a while back, obviously printing out
> > > > > that the session are closed already. Only after about 5 minutes it
> >
> > is
> >
> > > > > starting to detect the client disconnections. And then hangs
> > > > > again, after disconnecting only 14 of the 100 bots. There it
> > > > > continues
> >
> > again,
> >
> > > > > still trying to send chat messages that was still in the buffers.
> >
> > Now
> >
> > > > > detected 40 disconnections after another 2 minutes. Do note, Heap
> >
> > mem
> >
> > > > > down to 40MB from 374MB. There we go: all disconnected now,
> > > > > perform GC, 2MB heap used.
> > > > >
> > > > > What could I be doing wrong that it seems that these buffers do not
> > > > > flush quickly enough, getting bogged down with the data?
> > > > > <=============
> > > > >
> > > > > ==============>
> > > > > MINA 1.1.5 - 100 bots
> > > > > 2. 100 bots - each sending one message every 5 minutes, and flash
> > > > > client sending message every 1 - 3 seconds
> > > > > 2.1. after 10minutes
> > > > > 2.1.1. Heap size - 165MB
> > > > > 2.1.2. byte[] allocation:
> > > > > 2.1.2.1. no. of objects - 180,352
> > > > > 2.1.2.2. shallow size - 103,458,904
> > > > >
> > > > > 1.2. after killing 100 bots
> > > > > 1.2.1. Heap size - 1,5MB
> > > > > 1.2.2. byte[] allocation:
> > > > > 1.2.2.1. no. of objects - 3,890:
> > > > > 1.2.2.2. shallow size - 792,536
> > > > > <=============
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > > > > Sent: Saturday 26 July 2008 10:32
> > > > > > To: users@mina.apache.org
> > > > > > CC:
> > > > > > Subject: Re: byte[] usage - not GCd
> > > > > >
> > > > > >
> > > > > > Yes, 1.1.7 but 1.1.6 as well.
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Emmanuel Lecharny <el...@gmail.com>
> > > > > > > Sent: Saturday 26 July 2008 10:20
> > > > > > > To: users@mina.apache.org
> > > > > > > CC:
> > > > > > > Subject: Re: byte[] usage - not GCd
> > > > > > >
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > MINA 1.1.7 ?
> > > > > > >
> > > > > > > Gerrit Grobbelaar wrote:
> > > > > > > > YourKit shows the Class list as follows:
> > > > > > > > - java.nio.HeapByteBuffer
> > > > > > > > -
> >
> > org.apache.mina.common.SimpleByteBufferAllocator$SimpleByteBuffer
> >
> > > > > > > > -
> >
> > org.apache.mina.filter.codec.ProtocolCodecFilter$HiddenByteBuffer
> >
> > > > > > > > - org.apache.mina.common.IoFilter$WriteRequest
> > > > > > > > - java.util.concurrent.ConcurrentLinkedQueue$Node
> > > > > > > >
> > > > > > > > Note the above contributes 99%+ for the byte[] allocations,
> > > > > > > > and the byte[] allocations are causing the OOM.
> > > > > > > >
> > > > > > > >> -----Original Message-----
> > > > > > > >> From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > > > > > >> Sent: Saturday 26 July 2008 09:48
> > > > > > > >> To: users@mina.apache.org
> > > > > > > >> CC:
> > > > > > > >> Subject: Re: byte[] usage - not GCd
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> Another symptom:
> > > > > > > >> - 100 Java client bots connected, sending NO messages.
> > > > > > > >> - 1 Flash client, typing in a message every 2 seconds
> > > > > > > >> - memory consumption (byte[] allocations) increases and a
> >
> > Mark
> >
> > > > > > > >> Sweep with JConsole doesn't do anything (stays on 100s of
> >
> > MBs).
> >
> > > > > > > >> - kill off all bots
> > > > > > > >> - Mark sweep
> > > > > > > >> - Heap mem usage drops to about 2 - 5MB
> > > > > > > >>
> > > > > > > >>> -----Original Message-----
> > > > > > > >>> From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > > > > > >>> Sent: Saturday 26 July 2008 09:33
> > > > > > > >>> To: users@mina.apache.org
> > > > > > > >>> CC:
> > > > > > > >>> Subject: byte[] usage - not GCd
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> Hi,
> > > > > > > >>>
> > > > > > > >>> I'm having an issue with OOM - here is my story:
> > > > > > > >>>
> > > > > > > >>> Criteria:
> > > > > > > >>> --------
> > > > > > > >>> - concurrent clients connecting to a chatserver, need to
> > > > > > > >>> broadcast the messages
> > > > > > > >>> - clients keep same connection (each one having its own
> > > > > > > >>> IoSession during lifetime)
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> Symptoms:
> > > > > > > >>> --------
> > > > > > > >>> - massive amounts of byte[] allocations that just grows
> > > > > > > >>> and grows, until OOM. - still OOM even if i have 100 bots
> > > > > > > >>> connected, sending one message each every 5 minutes
> > > > > > > >>>
> > > > > > > >>> When I have 1 - 5 clients connected (all simulated with
> > > > > > > >>> bots) that send one message per bot every 2 - 5 seconds,
> > > > > > > >>> heap space stays under control with GC.
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> Configs/Tools:
> > > > > > > >>> -------------
> > > > > > > >>> - JVM: 1.6u7
> > > > > > > >>> - VM arguments: -Xms128m -Xmx384m
> > > > > > > >>> - Operating System: Linux 2.6.15.7
> > > > > > > >>> - Architecture: i386
> > > > > > > >>> - Number of processors: 2
> > > > > > > >>> - Model: Intel(R) Pentium(R) 4 CPU 2.80GHz
> > > > > > > >>> - Committed virtual memory: 681,564 kbytes
> > > > > > > >>> - Heap buffer allocation (Direct is very insufficient)
> > > > > > > >>> - SimpleByteBufferAllocator (do note I have same result
> > > > > > > >>> with PooledByteBufferAllocator)
> > > > > > > >>> - YourKit and JProfiler
> > > > > > > >>>
> > > > > > > >>> Both YourKit and JProfiler point fingers at:
> > > > > > > >>> byte[] allocation due to
> > > > > > > >>> org.apache.mina.common.IoSession.write() calls
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> What could I possibly be doing wrong that I will have the
> >
> > above
> >
> > > > > > > >>> scenario where the byte[] allocations never GC properly?
> > > > > > > >>>
> > > > > > > >>> Thanks,
> > > > > > > >>> Gerrit
Re: byte[] usage - not GCd
Posted by Sangjin Lee <sj...@gmail.com>.
Would you be able to create simple code that demonstrates the memory leak
behavior? I haven't played extensively with server-side load, but in my
experience at least with the client-side load shows no such memory behavior,
and we push it pretty hard. A code sample would be very helpful.
Thanks,
Sangjin
On Sat, Jul 26, 2008 at 8:04 AM, Gerrit Grobbelaar <ge...@jeasy.net> wrote:
> What I get from this is that MINA (out of the box) might not be the
> solution
> for me in a production environment. 2.0.0 looks to perform a bit better,
> but
> the 50 clients, 1 message per client every 2 seconds, also get disconnected
> after a while.
>
> Granted, all my clients are simulated from one Host machine on a LAN, but
> surely that can't make a difference?
>
> I've checked src out there on the net, and my IoSession/IoBuffer handling
> seems to be correct. The area where my problem is is situated outside of
> my
> domain in anyway, that is the NIO handling of the Mina library.
>
> I've been reading such good reviews about this framework, but just
> frustrating
> that I can't get the same out of it due to the buffer handling (byte[]
> allocations).
>
> I've stumbled onto this:
> =========>
> However, there's a case that allocation of heap buffers becomes bottleneck
> if
> the allocation occurs too fast. It's because any allocation of byte arrays
> requires filling up the content of the array with '0', which consumes
> memory
> bandwidth. CachedBufferAllocator is provided to address this concern, but
> you
> won't need to change the default SimpleBufferAllocator in most cases.
> <=========
> Which confirms to me that if I'm reading and writing on these buffers at
> ridiculous amounts, then MINA will not be of any benefit to me out of the
> box.
>
>
> I'll appreciate any 2 cents added.
>
> Thanks,
> Gerrit
>
> > -----Original Message-----
> > From: Gerrit Grobbelaar <ge...@jeasy.net>
> > Sent: Saturday 26 July 2008 16:33
> > To: users@mina.apache.org
> > CC:
> > Subject: Re: byte[] usage - not GCd
> >
> >
> > I have MUCH better results with MINA 2.0.0 M2, but after a short while
> all
> > clients/bots disconnect with:
> >
> > An error occurred : null
> > org.apache.mina.core.write.WriteTimeoutException
> >
> > Any ideas anyone?
> >
> > Thanks,
> > Gerrit
> >
> > > -----Original Message-----
> > > From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > Sent: Saturday 26 July 2008 14:14
> > > To: users@mina.apache.org
> > > CC:
> > > Subject: Re: byte[] usage - not GCd
> > >
> > >
> > > MINA 1.1.4 performs even better than 1.1.5.
> > >
> > > I still believe I'm doing something wrong in my code though, as I read
> > > only good things about MINA in the reviews, but for me it just seems to
> > > crumble underneath heavy load, the buffers not clearing quickly enough
> to
> > > be GCd.
> > >
> > > ==============>
> > > MINA 1.1.4 - 30 bots
> > > 1. 30 bots - each sending one message every 2 seconds
> > > 1.1. after 10 minutes
> > > 1.1.1. Heap size - 7MB
> > > 1.1.2. byte[] allocation:
> > > 1.1.2.1. no. of objects - 3,791:
> > > 1.1.2.2. shallow size - 996,320
> > > 1.2. after killing 30 bots
> > > 1.2.1. Heap size - 2,7MB
> > > 1.2.2. byte[] allocation:
> > > 1.2.2.1. no. of objects - 3,051:
> > > 1.2.2.2. shallow size - 659,344
> > > <==============
> > >
> > > ==============>
> > > MINA 1.1.4 - 100 bots
> > > 2. 100 bots - each sending one message every 2 seconds
> > > 2.1. after 2 minutes
> > > 2.1.1. Heap size - 329MB - JVM unresponsive
> > > 2.1.2. byte[] allocation:
> > > 2.1.2.1. no. of objects - 423,512:
> > > 2.1.2.2. shallow size - 201,053,744
> > >
> > > 2.2. after killing 100 bots
> > > 2.2.1. no effect - JVM unresponsive
> > >
> > > Interesting part: after about 1 minute this time, the server starts
> > > spewing out logs again, seems to have some action on the buffers again,
> > > processing messages still sent a while back, obviously printing out
> that
> > > the session are closed already. Very soon all bots disconnected,
> freeing
> > > up all the heap mem again.
> > > <=============
> > >
> > > > -----Original Message-----
> > > > From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > > Sent: Saturday 26 July 2008 12:03
> > > > To: users@mina.apache.org
> > > > CC:
> > > > Subject: Re: byte[] usage - not GCd
> > > >
> > > >
> > > > Some more test results (I definitely have better results with MINA
> > > > 1.1.5, but still not ideal under load). Seeing that I get different,
> > > > and somewhat better, result with an older version, I'm going to try
> > > > even older versions than 1.1.5 now.
> > > >
> > > > In the meantime: howcome these different results though, and why
> will
> > > > 1.1.5 perform better with Buffer (byte[])
> handling/allocations/flushing
> > > > than 1.1.6 and 1.1.7? Could be that I'm definitely doing something
> > > > wrong with my MINA integration, but then I should get consistent bad
> > > > performances across all versions?
> > > >
> > > > MINA 1.1.5 results as follows:
> > > >
> > > > ==============>
> > > > MINA 1.1.5 - 30 bots
> > > > 1. 30 bots - each sending one message every 2 seconds
> > > > 1.1. after 10 minutes
> > > > 1.1.1. Heap size - 20MB
> > > > 1.1.2. byte[] allocation:
> > > > 1.1.2.1. no. of objects - 12,104:
> > > > 1.1.2.2. shallow size - 4,207,440
> > > > 1.2. after killing 30 bots
> > > > 1.2.1. Heap size - 1,5MB
> > > > 1.2.2. byte[] allocation:
> > > > 1.2.2.1. no. of objects - 3,794:
> > > > 1.2.2.2. shallow size - 722,696
> > > > <==============
> > > >
> > > > ==============>
> > > > MINA 1.1.5 - 100 bots
> > > > 2. 100 bots - each sending one message every 2 seconds
> > > > 2.1. after 3 minutes
> > > > 2.1.1. Heap size - 374MB
> > > > 2.1.2. byte[] allocation:
> > > > 2.1.2.1. no. of objects - 455,736:
> > > > 2.1.2.2. shallow size - 216,055,400
> > > >
> > > > 2.2. after killing 30 bots
> > > > 2.2.1. no effect - JVM unresponsive
> > > >
> > > > Interesting part: after about 2 - 3 minutes, the server starts
> spewing
> > > > out logs again, seems to have some action on the buffers again,
> > > > processing messages still sent a while back, obviously printing out
> > > > that the session are closed already. Only after about 5 minutes it
> is
> > > > starting to detect the client disconnections. And then hangs again,
> > > > after disconnecting only 14 of the 100 bots. There it continues
> again,
> > > > still trying to send chat messages that was still in the buffers.
> Now
> > > > detected 40 disconnections after another 2 minutes. Do note, Heap
> mem
> > > > down to 40MB from 374MB. There we go: all disconnected now, perform
> > > > GC, 2MB heap used.
> > > >
> > > > What could I be doing wrong that it seems that these buffers do not
> > > > flush quickly enough, getting bogged down with the data?
> > > > <=============
> > > >
> > > > ==============>
> > > > MINA 1.1.5 - 100 bots
> > > > 2. 100 bots - each sending one message every 5 minutes, and flash
> > > > client sending message every 1 - 3 seconds
> > > > 2.1. after 10minutes
> > > > 2.1.1. Heap size - 165MB
> > > > 2.1.2. byte[] allocation:
> > > > 2.1.2.1. no. of objects - 180,352
> > > > 2.1.2.2. shallow size - 103,458,904
> > > >
> > > > 1.2. after killing 100 bots
> > > > 1.2.1. Heap size - 1,5MB
> > > > 1.2.2. byte[] allocation:
> > > > 1.2.2.1. no. of objects - 3,890:
> > > > 1.2.2.2. shallow size - 792,536
> > > > <=============
> > > >
> > > > > -----Original Message-----
> > > > > From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > > > Sent: Saturday 26 July 2008 10:32
> > > > > To: users@mina.apache.org
> > > > > CC:
> > > > > Subject: Re: byte[] usage - not GCd
> > > > >
> > > > >
> > > > > Yes, 1.1.7 but 1.1.6 as well.
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Emmanuel Lecharny <el...@gmail.com>
> > > > > > Sent: Saturday 26 July 2008 10:20
> > > > > > To: users@mina.apache.org
> > > > > > CC:
> > > > > > Subject: Re: byte[] usage - not GCd
> > > > > >
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > MINA 1.1.7 ?
> > > > > >
> > > > > > Gerrit Grobbelaar wrote:
> > > > > > > YourKit shows the Class list as follows:
> > > > > > > - java.nio.HeapByteBuffer
> > > > > > > -
> > > > > > >
> org.apache.mina.common.SimpleByteBufferAllocator$SimpleByteBuffer
> > > > > > > -
> > > > > > >
> org.apache.mina.filter.codec.ProtocolCodecFilter$HiddenByteBuffer
> > > > > > > - org.apache.mina.common.IoFilter$WriteRequest
> > > > > > > - java.util.concurrent.ConcurrentLinkedQueue$Node
> > > > > > >
> > > > > > > Note the above contributes 99%+ for the byte[] allocations, and
> > > > > > > the byte[] allocations are causing the OOM.
> > > > > > >
> > > > > > >> -----Original Message-----
> > > > > > >> From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > > > > >> Sent: Saturday 26 July 2008 09:48
> > > > > > >> To: users@mina.apache.org
> > > > > > >> CC:
> > > > > > >> Subject: Re: byte[] usage - not GCd
> > > > > > >>
> > > > > > >>
> > > > > > >> Another symptom:
> > > > > > >> - 100 Java client bots connected, sending NO messages.
> > > > > > >> - 1 Flash client, typing in a message every 2 seconds
> > > > > > >> - memory consumption (byte[] allocations) increases and a
> Mark
> > > > > > >> Sweep with JConsole doesn't do anything (stays on 100s of
> MBs).
> > > > > > >> - kill off all bots
> > > > > > >> - Mark sweep
> > > > > > >> - Heap mem usage drops to about 2 - 5MB
> > > > > > >>
> > > > > > >>> -----Original Message-----
> > > > > > >>> From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > > > > >>> Sent: Saturday 26 July 2008 09:33
> > > > > > >>> To: users@mina.apache.org
> > > > > > >>> CC:
> > > > > > >>> Subject: byte[] usage - not GCd
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> Hi,
> > > > > > >>>
> > > > > > >>> I'm having an issue with OOM - here is my story:
> > > > > > >>>
> > > > > > >>> Criteria:
> > > > > > >>> --------
> > > > > > >>> - concurrent clients connecting to a chatserver, need to
> > > > > > >>> broadcast the messages
> > > > > > >>> - clients keep same connection (each one having its own
> > > > > > >>> IoSession during lifetime)
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> Symptoms:
> > > > > > >>> --------
> > > > > > >>> - massive amounts of byte[] allocations that just grows and
> > > > > > >>> grows, until OOM. - still OOM even if i have 100 bots
> > > > > > >>> connected, sending one message each every 5 minutes
> > > > > > >>>
> > > > > > >>> When I have 1 - 5 clients connected (all simulated with bots)
> > > > > > >>> that send one message per bot every 2 - 5 seconds, heap space
> > > > > > >>> stays under control with GC.
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> Configs/Tools:
> > > > > > >>> -------------
> > > > > > >>> - JVM: 1.6u7
> > > > > > >>> - VM arguments: -Xms128m -Xmx384m
> > > > > > >>> - Operating System: Linux 2.6.15.7
> > > > > > >>> - Architecture: i386
> > > > > > >>> - Number of processors: 2
> > > > > > >>> - Model: Intel(R) Pentium(R) 4 CPU 2.80GHz
> > > > > > >>> - Committed virtual memory: 681,564 kbytes
> > > > > > >>> - Heap buffer allocation (Direct is very insufficient)
> > > > > > >>> - SimpleByteBufferAllocator (do note I have same result with
> > > > > > >>> PooledByteBufferAllocator)
> > > > > > >>> - YourKit and JProfiler
> > > > > > >>>
> > > > > > >>> Both YourKit and JProfiler point fingers at:
> > > > > > >>> byte[] allocation due to
> > > > > > >>> org.apache.mina.common.IoSession.write() calls
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> What could I possibly be doing wrong that I will have the
> above
> > > > > > >>> scenario where the byte[] allocations never GC properly?
> > > > > > >>>
> > > > > > >>> Thanks,
> > > > > > >>> Gerrit
>
Re: byte[] usage - not GCd
Posted by Gerrit Grobbelaar <ge...@jeasy.net>.
What I get from this is that MINA (out of the box) might not be the solution
for me in a production environment. 2.0.0 looks to perform a bit better, but
the 50 clients, 1 message per client every 2 seconds, also get disconnected
after a while.
Granted, all my clients are simulated from one Host machine on a LAN, but
surely that can't make a difference?
I've checked src out there on the net, and my IoSession/IoBuffer handling
seems to be correct. The area where my problem is is situated outside of my
domain in anyway, that is the NIO handling of the Mina library.
I've been reading such good reviews about this framework, but just frustrating
that I can't get the same out of it due to the buffer handling (byte[]
allocations).
I've stumbled onto this:
=========>
However, there's a case that allocation of heap buffers becomes bottleneck if
the allocation occurs too fast. It's because any allocation of byte arrays
requires filling up the content of the array with '0', which consumes memory
bandwidth. CachedBufferAllocator is provided to address this concern, but you
won't need to change the default SimpleBufferAllocator in most cases.
<=========
Which confirms to me that if I'm reading and writing on these buffers at
ridiculous amounts, then MINA will not be of any benefit to me out of the
box.
I'll appreciate any 2 cents added.
Thanks,
Gerrit
> -----Original Message-----
> From: Gerrit Grobbelaar <ge...@jeasy.net>
> Sent: Saturday 26 July 2008 16:33
> To: users@mina.apache.org
> CC:
> Subject: Re: byte[] usage - not GCd
>
>
> I have MUCH better results with MINA 2.0.0 M2, but after a short while all
> clients/bots disconnect with:
>
> An error occurred : null
> org.apache.mina.core.write.WriteTimeoutException
>
> Any ideas anyone?
>
> Thanks,
> Gerrit
>
> > -----Original Message-----
> > From: Gerrit Grobbelaar <ge...@jeasy.net>
> > Sent: Saturday 26 July 2008 14:14
> > To: users@mina.apache.org
> > CC:
> > Subject: Re: byte[] usage - not GCd
> >
> >
> > MINA 1.1.4 performs even better than 1.1.5.
> >
> > I still believe I'm doing something wrong in my code though, as I read
> > only good things about MINA in the reviews, but for me it just seems to
> > crumble underneath heavy load, the buffers not clearing quickly enough to
> > be GCd.
> >
> > ==============>
> > MINA 1.1.4 - 30 bots
> > 1. 30 bots - each sending one message every 2 seconds
> > 1.1. after 10 minutes
> > 1.1.1. Heap size - 7MB
> > 1.1.2. byte[] allocation:
> > 1.1.2.1. no. of objects - 3,791:
> > 1.1.2.2. shallow size - 996,320
> > 1.2. after killing 30 bots
> > 1.2.1. Heap size - 2,7MB
> > 1.2.2. byte[] allocation:
> > 1.2.2.1. no. of objects - 3,051:
> > 1.2.2.2. shallow size - 659,344
> > <==============
> >
> > ==============>
> > MINA 1.1.4 - 100 bots
> > 2. 100 bots - each sending one message every 2 seconds
> > 2.1. after 2 minutes
> > 2.1.1. Heap size - 329MB - JVM unresponsive
> > 2.1.2. byte[] allocation:
> > 2.1.2.1. no. of objects - 423,512:
> > 2.1.2.2. shallow size - 201,053,744
> >
> > 2.2. after killing 100 bots
> > 2.2.1. no effect - JVM unresponsive
> >
> > Interesting part: after about 1 minute this time, the server starts
> > spewing out logs again, seems to have some action on the buffers again,
> > processing messages still sent a while back, obviously printing out that
> > the session are closed already. Very soon all bots disconnected, freeing
> > up all the heap mem again.
> > <=============
> >
> > > -----Original Message-----
> > > From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > Sent: Saturday 26 July 2008 12:03
> > > To: users@mina.apache.org
> > > CC:
> > > Subject: Re: byte[] usage - not GCd
> > >
> > >
> > > Some more test results (I definitely have better results with MINA
> > > 1.1.5, but still not ideal under load). Seeing that I get different,
> > > and somewhat better, result with an older version, I'm going to try
> > > even older versions than 1.1.5 now.
> > >
> > > In the meantime: howcome these different results though, and why will
> > > 1.1.5 perform better with Buffer (byte[]) handling/allocations/flushing
> > > than 1.1.6 and 1.1.7? Could be that I'm definitely doing something
> > > wrong with my MINA integration, but then I should get consistent bad
> > > performances across all versions?
> > >
> > > MINA 1.1.5 results as follows:
> > >
> > > ==============>
> > > MINA 1.1.5 - 30 bots
> > > 1. 30 bots - each sending one message every 2 seconds
> > > 1.1. after 10 minutes
> > > 1.1.1. Heap size - 20MB
> > > 1.1.2. byte[] allocation:
> > > 1.1.2.1. no. of objects - 12,104:
> > > 1.1.2.2. shallow size - 4,207,440
> > > 1.2. after killing 30 bots
> > > 1.2.1. Heap size - 1,5MB
> > > 1.2.2. byte[] allocation:
> > > 1.2.2.1. no. of objects - 3,794:
> > > 1.2.2.2. shallow size - 722,696
> > > <==============
> > >
> > > ==============>
> > > MINA 1.1.5 - 100 bots
> > > 2. 100 bots - each sending one message every 2 seconds
> > > 2.1. after 3 minutes
> > > 2.1.1. Heap size - 374MB
> > > 2.1.2. byte[] allocation:
> > > 2.1.2.1. no. of objects - 455,736:
> > > 2.1.2.2. shallow size - 216,055,400
> > >
> > > 2.2. after killing 30 bots
> > > 2.2.1. no effect - JVM unresponsive
> > >
> > > Interesting part: after about 2 - 3 minutes, the server starts spewing
> > > out logs again, seems to have some action on the buffers again,
> > > processing messages still sent a while back, obviously printing out
> > > that the session are closed already. Only after about 5 minutes it is
> > > starting to detect the client disconnections. And then hangs again,
> > > after disconnecting only 14 of the 100 bots. There it continues again,
> > > still trying to send chat messages that was still in the buffers. Now
> > > detected 40 disconnections after another 2 minutes. Do note, Heap mem
> > > down to 40MB from 374MB. There we go: all disconnected now, perform
> > > GC, 2MB heap used.
> > >
> > > What could I be doing wrong that it seems that these buffers do not
> > > flush quickly enough, getting bogged down with the data?
> > > <=============
> > >
> > > ==============>
> > > MINA 1.1.5 - 100 bots
> > > 2. 100 bots - each sending one message every 5 minutes, and flash
> > > client sending message every 1 - 3 seconds
> > > 2.1. after 10minutes
> > > 2.1.1. Heap size - 165MB
> > > 2.1.2. byte[] allocation:
> > > 2.1.2.1. no. of objects - 180,352
> > > 2.1.2.2. shallow size - 103,458,904
> > >
> > > 1.2. after killing 100 bots
> > > 1.2.1. Heap size - 1,5MB
> > > 1.2.2. byte[] allocation:
> > > 1.2.2.1. no. of objects - 3,890:
> > > 1.2.2.2. shallow size - 792,536
> > > <=============
> > >
> > > > -----Original Message-----
> > > > From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > > Sent: Saturday 26 July 2008 10:32
> > > > To: users@mina.apache.org
> > > > CC:
> > > > Subject: Re: byte[] usage - not GCd
> > > >
> > > >
> > > > Yes, 1.1.7 but 1.1.6 as well.
> > > >
> > > > > -----Original Message-----
> > > > > From: Emmanuel Lecharny <el...@gmail.com>
> > > > > Sent: Saturday 26 July 2008 10:20
> > > > > To: users@mina.apache.org
> > > > > CC:
> > > > > Subject: Re: byte[] usage - not GCd
> > > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > MINA 1.1.7 ?
> > > > >
> > > > > Gerrit Grobbelaar wrote:
> > > > > > YourKit shows the Class list as follows:
> > > > > > - java.nio.HeapByteBuffer
> > > > > > -
> > > > > > org.apache.mina.common.SimpleByteBufferAllocator$SimpleByteBuffer
> > > > > > -
> > > > > > org.apache.mina.filter.codec.ProtocolCodecFilter$HiddenByteBuffer
> > > > > > - org.apache.mina.common.IoFilter$WriteRequest
> > > > > > - java.util.concurrent.ConcurrentLinkedQueue$Node
> > > > > >
> > > > > > Note the above contributes 99%+ for the byte[] allocations, and
> > > > > > the byte[] allocations are causing the OOM.
> > > > > >
> > > > > >> -----Original Message-----
> > > > > >> From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > > > >> Sent: Saturday 26 July 2008 09:48
> > > > > >> To: users@mina.apache.org
> > > > > >> CC:
> > > > > >> Subject: Re: byte[] usage - not GCd
> > > > > >>
> > > > > >>
> > > > > >> Another symptom:
> > > > > >> - 100 Java client bots connected, sending NO messages.
> > > > > >> - 1 Flash client, typing in a message every 2 seconds
> > > > > >> - memory consumption (byte[] allocations) increases and a Mark
> > > > > >> Sweep with JConsole doesn't do anything (stays on 100s of MBs).
> > > > > >> - kill off all bots
> > > > > >> - Mark sweep
> > > > > >> - Heap mem usage drops to about 2 - 5MB
> > > > > >>
> > > > > >>> -----Original Message-----
> > > > > >>> From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > > > >>> Sent: Saturday 26 July 2008 09:33
> > > > > >>> To: users@mina.apache.org
> > > > > >>> CC:
> > > > > >>> Subject: byte[] usage - not GCd
> > > > > >>>
> > > > > >>>
> > > > > >>> Hi,
> > > > > >>>
> > > > > >>> I'm having an issue with OOM - here is my story:
> > > > > >>>
> > > > > >>> Criteria:
> > > > > >>> --------
> > > > > >>> - concurrent clients connecting to a chatserver, need to
> > > > > >>> broadcast the messages
> > > > > >>> - clients keep same connection (each one having its own
> > > > > >>> IoSession during lifetime)
> > > > > >>>
> > > > > >>>
> > > > > >>> Symptoms:
> > > > > >>> --------
> > > > > >>> - massive amounts of byte[] allocations that just grows and
> > > > > >>> grows, until OOM. - still OOM even if i have 100 bots
> > > > > >>> connected, sending one message each every 5 minutes
> > > > > >>>
> > > > > >>> When I have 1 - 5 clients connected (all simulated with bots)
> > > > > >>> that send one message per bot every 2 - 5 seconds, heap space
> > > > > >>> stays under control with GC.
> > > > > >>>
> > > > > >>>
> > > > > >>> Configs/Tools:
> > > > > >>> -------------
> > > > > >>> - JVM: 1.6u7
> > > > > >>> - VM arguments: -Xms128m -Xmx384m
> > > > > >>> - Operating System: Linux 2.6.15.7
> > > > > >>> - Architecture: i386
> > > > > >>> - Number of processors: 2
> > > > > >>> - Model: Intel(R) Pentium(R) 4 CPU 2.80GHz
> > > > > >>> - Committed virtual memory: 681,564 kbytes
> > > > > >>> - Heap buffer allocation (Direct is very insufficient)
> > > > > >>> - SimpleByteBufferAllocator (do note I have same result with
> > > > > >>> PooledByteBufferAllocator)
> > > > > >>> - YourKit and JProfiler
> > > > > >>>
> > > > > >>> Both YourKit and JProfiler point fingers at:
> > > > > >>> byte[] allocation due to
> > > > > >>> org.apache.mina.common.IoSession.write() calls
> > > > > >>>
> > > > > >>>
> > > > > >>> What could I possibly be doing wrong that I will have the above
> > > > > >>> scenario where the byte[] allocations never GC properly?
> > > > > >>>
> > > > > >>> Thanks,
> > > > > >>> Gerrit
Re: byte[] usage - not GCd
Posted by Gerrit Grobbelaar <ge...@jeasy.net>.
I have MUCH better results with MINA 2.0.0 M2, but after a short while all
clients/bots disconnect with:
An error occurred : null
org.apache.mina.core.write.WriteTimeoutException
Any ideas anyone?
Thanks,
Gerrit
> -----Original Message-----
> From: Gerrit Grobbelaar <ge...@jeasy.net>
> Sent: Saturday 26 July 2008 14:14
> To: users@mina.apache.org
> CC:
> Subject: Re: byte[] usage - not GCd
>
>
> MINA 1.1.4 performs even better than 1.1.5.
>
> I still believe I'm doing something wrong in my code though, as I read only
> good things about MINA in the reviews, but for me it just seems to crumble
> underneath heavy load, the buffers not clearing quickly enough to be GCd.
>
> ==============>
> MINA 1.1.4 - 30 bots
> 1. 30 bots - each sending one message every 2 seconds
> 1.1. after 10 minutes
> 1.1.1. Heap size - 7MB
> 1.1.2. byte[] allocation:
> 1.1.2.1. no. of objects - 3,791:
> 1.1.2.2. shallow size - 996,320
> 1.2. after killing 30 bots
> 1.2.1. Heap size - 2,7MB
> 1.2.2. byte[] allocation:
> 1.2.2.1. no. of objects - 3,051:
> 1.2.2.2. shallow size - 659,344
> <==============
>
> ==============>
> MINA 1.1.4 - 100 bots
> 2. 100 bots - each sending one message every 2 seconds
> 2.1. after 2 minutes
> 2.1.1. Heap size - 329MB - JVM unresponsive
> 2.1.2. byte[] allocation:
> 2.1.2.1. no. of objects - 423,512:
> 2.1.2.2. shallow size - 201,053,744
>
> 2.2. after killing 100 bots
> 2.2.1. no effect - JVM unresponsive
>
> Interesting part: after about 1 minute this time, the server starts
> spewing out logs again, seems to have some action on the buffers again,
> processing messages still sent a while back, obviously printing out that
> the session are closed already. Very soon all bots disconnected, freeing
> up all the heap mem again.
> <=============
>
> > -----Original Message-----
> > From: Gerrit Grobbelaar <ge...@jeasy.net>
> > Sent: Saturday 26 July 2008 12:03
> > To: users@mina.apache.org
> > CC:
> > Subject: Re: byte[] usage - not GCd
> >
> >
> > Some more test results (I definitely have better results with MINA 1.1.5,
> > but still not ideal under load). Seeing that I get different, and
> > somewhat better, result with an older version, I'm going to try even
> > older versions than 1.1.5 now.
> >
> > In the meantime: howcome these different results though, and why will
> > 1.1.5 perform better with Buffer (byte[]) handling/allocations/flushing
> > than 1.1.6 and 1.1.7? Could be that I'm definitely doing something wrong
> > with my MINA integration, but then I should get consistent bad
> > performances across all versions?
> >
> > MINA 1.1.5 results as follows:
> >
> > ==============>
> > MINA 1.1.5 - 30 bots
> > 1. 30 bots - each sending one message every 2 seconds
> > 1.1. after 10 minutes
> > 1.1.1. Heap size - 20MB
> > 1.1.2. byte[] allocation:
> > 1.1.2.1. no. of objects - 12,104:
> > 1.1.2.2. shallow size - 4,207,440
> > 1.2. after killing 30 bots
> > 1.2.1. Heap size - 1,5MB
> > 1.2.2. byte[] allocation:
> > 1.2.2.1. no. of objects - 3,794:
> > 1.2.2.2. shallow size - 722,696
> > <==============
> >
> > ==============>
> > MINA 1.1.5 - 100 bots
> > 2. 100 bots - each sending one message every 2 seconds
> > 2.1. after 3 minutes
> > 2.1.1. Heap size - 374MB
> > 2.1.2. byte[] allocation:
> > 2.1.2.1. no. of objects - 455,736:
> > 2.1.2.2. shallow size - 216,055,400
> >
> > 2.2. after killing 30 bots
> > 2.2.1. no effect - JVM unresponsive
> >
> > Interesting part: after about 2 - 3 minutes, the server starts spewing
> > out logs again, seems to have some action on the buffers again,
> > processing messages still sent a while back, obviously printing out that
> > the session are closed already. Only after about 5 minutes it is
> > starting to detect the client disconnections. And then hangs again,
> > after disconnecting only 14 of the 100 bots. There it continues again,
> > still trying to send chat messages that was still in the buffers. Now
> > detected 40 disconnections after another 2 minutes. Do note, Heap mem
> > down to 40MB from 374MB. There we go: all disconnected now, perform GC,
> > 2MB heap used.
> >
> > What could I be doing wrong that it seems that these buffers do not flush
> > quickly enough, getting bogged down with the data?
> > <=============
> >
> > ==============>
> > MINA 1.1.5 - 100 bots
> > 2. 100 bots - each sending one message every 5 minutes, and flash client
> > sending message every 1 - 3 seconds
> > 2.1. after 10minutes
> > 2.1.1. Heap size - 165MB
> > 2.1.2. byte[] allocation:
> > 2.1.2.1. no. of objects - 180,352
> > 2.1.2.2. shallow size - 103,458,904
> >
> > 1.2. after killing 100 bots
> > 1.2.1. Heap size - 1,5MB
> > 1.2.2. byte[] allocation:
> > 1.2.2.1. no. of objects - 3,890:
> > 1.2.2.2. shallow size - 792,536
> > <=============
> >
> > > -----Original Message-----
> > > From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > Sent: Saturday 26 July 2008 10:32
> > > To: users@mina.apache.org
> > > CC:
> > > Subject: Re: byte[] usage - not GCd
> > >
> > >
> > > Yes, 1.1.7 but 1.1.6 as well.
> > >
> > > > -----Original Message-----
> > > > From: Emmanuel Lecharny <el...@gmail.com>
> > > > Sent: Saturday 26 July 2008 10:20
> > > > To: users@mina.apache.org
> > > > CC:
> > > > Subject: Re: byte[] usage - not GCd
> > > >
> > > >
> > > > Hi,
> > > >
> > > > MINA 1.1.7 ?
> > > >
> > > > Gerrit Grobbelaar wrote:
> > > > > YourKit shows the Class list as follows:
> > > > > - java.nio.HeapByteBuffer
> > > > > -
> > > > > org.apache.mina.common.SimpleByteBufferAllocator$SimpleByteBuffer -
> > > > > org.apache.mina.filter.codec.ProtocolCodecFilter$HiddenByteBuffer -
> > > > > org.apache.mina.common.IoFilter$WriteRequest
> > > > > - java.util.concurrent.ConcurrentLinkedQueue$Node
> > > > >
> > > > > Note the above contributes 99%+ for the byte[] allocations, and the
> > > > > byte[] allocations are causing the OOM.
> > > > >
> > > > >> -----Original Message-----
> > > > >> From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > > >> Sent: Saturday 26 July 2008 09:48
> > > > >> To: users@mina.apache.org
> > > > >> CC:
> > > > >> Subject: Re: byte[] usage - not GCd
> > > > >>
> > > > >>
> > > > >> Another symptom:
> > > > >> - 100 Java client bots connected, sending NO messages.
> > > > >> - 1 Flash client, typing in a message every 2 seconds
> > > > >> - memory consumption (byte[] allocations) increases and a Mark
> > > > >> Sweep with JConsole doesn't do anything (stays on 100s of MBs).
> > > > >> - kill off all bots
> > > > >> - Mark sweep
> > > > >> - Heap mem usage drops to about 2 - 5MB
> > > > >>
> > > > >>> -----Original Message-----
> > > > >>> From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > > >>> Sent: Saturday 26 July 2008 09:33
> > > > >>> To: users@mina.apache.org
> > > > >>> CC:
> > > > >>> Subject: byte[] usage - not GCd
> > > > >>>
> > > > >>>
> > > > >>> Hi,
> > > > >>>
> > > > >>> I'm having an issue with OOM - here is my story:
> > > > >>>
> > > > >>> Criteria:
> > > > >>> --------
> > > > >>> - concurrent clients connecting to a chatserver, need to
> > > > >>> broadcast the messages
> > > > >>> - clients keep same connection (each one having its own
> > > > >>> IoSession during lifetime)
> > > > >>>
> > > > >>>
> > > > >>> Symptoms:
> > > > >>> --------
> > > > >>> - massive amounts of byte[] allocations that just grows and
> > > > >>> grows, until OOM. - still OOM even if i have 100 bots connected,
> > > > >>> sending one message each every 5 minutes
> > > > >>>
> > > > >>> When I have 1 - 5 clients connected (all simulated with bots)
> > > > >>> that send one message per bot every 2 - 5 seconds, heap space
> > > > >>> stays under control with GC.
> > > > >>>
> > > > >>>
> > > > >>> Configs/Tools:
> > > > >>> -------------
> > > > >>> - JVM: 1.6u7
> > > > >>> - VM arguments: -Xms128m -Xmx384m
> > > > >>> - Operating System: Linux 2.6.15.7
> > > > >>> - Architecture: i386
> > > > >>> - Number of processors: 2
> > > > >>> - Model: Intel(R) Pentium(R) 4 CPU 2.80GHz
> > > > >>> - Committed virtual memory: 681,564 kbytes
> > > > >>> - Heap buffer allocation (Direct is very insufficient)
> > > > >>> - SimpleByteBufferAllocator (do note I have same result with
> > > > >>> PooledByteBufferAllocator)
> > > > >>> - YourKit and JProfiler
> > > > >>>
> > > > >>> Both YourKit and JProfiler point fingers at:
> > > > >>> byte[] allocation due to org.apache.mina.common.IoSession.write()
> > > > >>> calls
> > > > >>>
> > > > >>>
> > > > >>> What could I possibly be doing wrong that I will have the above
> > > > >>> scenario where the byte[] allocations never GC properly?
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Gerrit
Re: byte[] usage - not GCd
Posted by Gerrit Grobbelaar <ge...@jeasy.net>.
MINA 1.1.4 performs even better than 1.1.5.
I still believe I'm doing something wrong in my code though, as I read only
good things about MINA in the reviews, but for me it just seems to crumble
underneath heavy load, the buffers not clearing quickly enough to be GCd.
==============>
MINA 1.1.4 - 30 bots
1. 30 bots - each sending one message every 2 seconds
1.1. after 10 minutes
1.1.1. Heap size - 7MB
1.1.2. byte[] allocation:
1.1.2.1. no. of objects - 3,791:
1.1.2.2. shallow size - 996,320
1.2. after killing 30 bots
1.2.1. Heap size - 2,7MB
1.2.2. byte[] allocation:
1.2.2.1. no. of objects - 3,051:
1.2.2.2. shallow size - 659,344
<==============
==============>
MINA 1.1.4 - 100 bots
2. 100 bots - each sending one message every 2 seconds
2.1. after 2 minutes
2.1.1. Heap size - 329MB - JVM unresponsive
2.1.2. byte[] allocation:
2.1.2.1. no. of objects - 423,512:
2.1.2.2. shallow size - 201,053,744
2.2. after killing 100 bots
2.2.1. no effect - JVM unresponsive
Interesting part: after about 1 minute this time, the server starts spewing
out logs again, seems to have some action on the buffers again, processing
messages still sent a while back, obviously printing out that the session are
closed already. Very soon all bots disconnected, freeing up all the heap mem
again.
<=============
> -----Original Message-----
> From: Gerrit Grobbelaar <ge...@jeasy.net>
> Sent: Saturday 26 July 2008 12:03
> To: users@mina.apache.org
> CC:
> Subject: Re: byte[] usage - not GCd
>
>
> Some more test results (I definitely have better results with MINA 1.1.5,
> but still not ideal under load). Seeing that I get different, and somewhat
> better, result with an older version, I'm going to try even older versions
> than 1.1.5 now.
>
> In the meantime: howcome these different results though, and why will
> 1.1.5 perform better with Buffer (byte[]) handling/allocations/flushing
> than 1.1.6 and 1.1.7? Could be that I'm definitely doing something wrong
> with my MINA integration, but then I should get consistent bad performances
> across all versions?
>
> MINA 1.1.5 results as follows:
>
> ==============>
> MINA 1.1.5 - 30 bots
> 1. 30 bots - each sending one message every 2 seconds
> 1.1. after 10 minutes
> 1.1.1. Heap size - 20MB
> 1.1.2. byte[] allocation:
> 1.1.2.1. no. of objects - 12,104:
> 1.1.2.2. shallow size - 4,207,440
> 1.2. after killing 30 bots
> 1.2.1. Heap size - 1,5MB
> 1.2.2. byte[] allocation:
> 1.2.2.1. no. of objects - 3,794:
> 1.2.2.2. shallow size - 722,696
> <==============
>
> ==============>
> MINA 1.1.5 - 100 bots
> 2. 100 bots - each sending one message every 2 seconds
> 2.1. after 3 minutes
> 2.1.1. Heap size - 374MB
> 2.1.2. byte[] allocation:
> 2.1.2.1. no. of objects - 455,736:
> 2.1.2.2. shallow size - 216,055,400
>
> 2.2. after killing 30 bots
> 2.2.1. no effect - JVM unresponsive
>
> Interesting part: after about 2 - 3 minutes, the server starts spewing out
> logs again, seems to have some action on the buffers again, processing
> messages still sent a while back, obviously printing out that the session
> are closed already. Only after about 5 minutes it is starting to detect
> the client disconnections. And then hangs again, after disconnecting only
> 14 of the 100 bots. There it continues again, still trying to send chat
> messages that was still in the buffers. Now detected 40 disconnections
> after another 2 minutes. Do note, Heap mem down to 40MB from 374MB. There
> we go: all disconnected now, perform GC, 2MB heap used.
>
> What could I be doing wrong that it seems that these buffers do not flush
> quickly enough, getting bogged down with the data?
> <=============
>
> ==============>
> MINA 1.1.5 - 100 bots
> 2. 100 bots - each sending one message every 5 minutes, and flash client
> sending message every 1 - 3 seconds
> 2.1. after 10minutes
> 2.1.1. Heap size - 165MB
> 2.1.2. byte[] allocation:
> 2.1.2.1. no. of objects - 180,352
> 2.1.2.2. shallow size - 103,458,904
>
> 1.2. after killing 100 bots
> 1.2.1. Heap size - 1,5MB
> 1.2.2. byte[] allocation:
> 1.2.2.1. no. of objects - 3,890:
> 1.2.2.2. shallow size - 792,536
> <=============
>
> > -----Original Message-----
> > From: Gerrit Grobbelaar <ge...@jeasy.net>
> > Sent: Saturday 26 July 2008 10:32
> > To: users@mina.apache.org
> > CC:
> > Subject: Re: byte[] usage - not GCd
> >
> >
> > Yes, 1.1.7 but 1.1.6 as well.
> >
> > > -----Original Message-----
> > > From: Emmanuel Lecharny <el...@gmail.com>
> > > Sent: Saturday 26 July 2008 10:20
> > > To: users@mina.apache.org
> > > CC:
> > > Subject: Re: byte[] usage - not GCd
> > >
> > >
> > > Hi,
> > >
> > > MINA 1.1.7 ?
> > >
> > > Gerrit Grobbelaar wrote:
> > > > YourKit shows the Class list as follows:
> > > > - java.nio.HeapByteBuffer
> > > > - org.apache.mina.common.SimpleByteBufferAllocator$SimpleByteBuffer
> > > > - org.apache.mina.filter.codec.ProtocolCodecFilter$HiddenByteBuffer
> > > > - org.apache.mina.common.IoFilter$WriteRequest
> > > > - java.util.concurrent.ConcurrentLinkedQueue$Node
> > > >
> > > > Note the above contributes 99%+ for the byte[] allocations, and the
> > > > byte[] allocations are causing the OOM.
> > > >
> > > >> -----Original Message-----
> > > >> From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > >> Sent: Saturday 26 July 2008 09:48
> > > >> To: users@mina.apache.org
> > > >> CC:
> > > >> Subject: Re: byte[] usage - not GCd
> > > >>
> > > >>
> > > >> Another symptom:
> > > >> - 100 Java client bots connected, sending NO messages.
> > > >> - 1 Flash client, typing in a message every 2 seconds
> > > >> - memory consumption (byte[] allocations) increases and a Mark
> > > >> Sweep with JConsole doesn't do anything (stays on 100s of MBs).
> > > >> - kill off all bots
> > > >> - Mark sweep
> > > >> - Heap mem usage drops to about 2 - 5MB
> > > >>
> > > >>> -----Original Message-----
> > > >>> From: Gerrit Grobbelaar <ge...@jeasy.net>
> > > >>> Sent: Saturday 26 July 2008 09:33
> > > >>> To: users@mina.apache.org
> > > >>> CC:
> > > >>> Subject: byte[] usage - not GCd
> > > >>>
> > > >>>
> > > >>> Hi,
> > > >>>
> > > >>> I'm having an issue with OOM - here is my story:
> > > >>>
> > > >>> Criteria:
> > > >>> --------
> > > >>> - concurrent clients connecting to a chatserver, need to broadcast
> > > >>> the messages
> > > >>> - clients keep same connection (each one having its own IoSession
> > > >>> during lifetime)
> > > >>>
> > > >>>
> > > >>> Symptoms:
> > > >>> --------
> > > >>> - massive amounts of byte[] allocations that just grows and grows,
> > > >>> until OOM. - still OOM even if i have 100 bots connected, sending
> > > >>> one message each every 5 minutes
> > > >>>
> > > >>> When I have 1 - 5 clients connected (all simulated with bots) that
> > > >>> send one message per bot every 2 - 5 seconds, heap space stays
> > > >>> under control with GC.
> > > >>>
> > > >>>
> > > >>> Configs/Tools:
> > > >>> -------------
> > > >>> - JVM: 1.6u7
> > > >>> - VM arguments: -Xms128m -Xmx384m
> > > >>> - Operating System: Linux 2.6.15.7
> > > >>> - Architecture: i386
> > > >>> - Number of processors: 2
> > > >>> - Model: Intel(R) Pentium(R) 4 CPU 2.80GHz
> > > >>> - Committed virtual memory: 681,564 kbytes
> > > >>> - Heap buffer allocation (Direct is very insufficient)
> > > >>> - SimpleByteBufferAllocator (do note I have same result with
> > > >>> PooledByteBufferAllocator)
> > > >>> - YourKit and JProfiler
> > > >>>
> > > >>> Both YourKit and JProfiler point fingers at:
> > > >>> byte[] allocation due to org.apache.mina.common.IoSession.write()
> > > >>> calls
> > > >>>
> > > >>>
> > > >>> What could I possibly be doing wrong that I will have the above
> > > >>> scenario where the byte[] allocations never GC properly?
> > > >>>
> > > >>> Thanks,
> > > >>> Gerrit
Re: byte[] usage - not GCd
Posted by Gerrit Grobbelaar <ge...@jeasy.net>.
Some more test results (I definitely have better results with MINA 1.1.5, but
still not ideal under load). Seeing that I get different, and somewhat
better, result with an older version, I'm going to try even older versions
than 1.1.5 now.
In the meantime: howcome these different results though, and why will 1.1.5
perform better with Buffer (byte[]) handling/allocations/flushing than 1.1.6
and 1.1.7? Could be that I'm definitely doing something wrong with my MINA
integration, but then I should get consistent bad performances across all
versions?
MINA 1.1.5 results as follows:
==============>
MINA 1.1.5 - 30 bots
1. 30 bots - each sending one message every 2 seconds
1.1. after 10 minutes
1.1.1. Heap size - 20MB
1.1.2. byte[] allocation:
1.1.2.1. no. of objects - 12,104:
1.1.2.2. shallow size - 4,207,440
1.2. after killing 30 bots
1.2.1. Heap size - 1,5MB
1.2.2. byte[] allocation:
1.2.2.1. no. of objects - 3,794:
1.2.2.2. shallow size - 722,696
<==============
==============>
MINA 1.1.5 - 100 bots
2. 100 bots - each sending one message every 2 seconds
2.1. after 3 minutes
2.1.1. Heap size - 374MB
2.1.2. byte[] allocation:
2.1.2.1. no. of objects - 455,736:
2.1.2.2. shallow size - 216,055,400
2.2. after killing 30 bots
2.2.1. no effect - JVM unresponsive
Interesting part: after about 2 - 3 minutes, the server starts spewing out
logs again, seems to have some action on the buffers again, processing
messages still sent a while back, obviously printing out that the session are
closed already. Only after about 5 minutes it is starting to detect the
client disconnections. And then hangs again, after disconnecting only 14 of
the 100 bots. There it continues again, still trying to send chat messages
that was still in the buffers. Now detected 40 disconnections after another
2 minutes. Do note, Heap mem down to 40MB from 374MB. There we go: all
disconnected now, perform GC, 2MB heap used.
What could I be doing wrong that it seems that these buffers do not flush
quickly enough, getting bogged down with the data?
<=============
==============>
MINA 1.1.5 - 100 bots
2. 100 bots - each sending one message every 5 minutes, and flash client
sending message every 1 - 3 seconds
2.1. after 10minutes
2.1.1. Heap size - 165MB
2.1.2. byte[] allocation:
2.1.2.1. no. of objects - 180,352
2.1.2.2. shallow size - 103,458,904
1.2. after killing 100 bots
1.2.1. Heap size - 1,5MB
1.2.2. byte[] allocation:
1.2.2.1. no. of objects - 3,890:
1.2.2.2. shallow size - 792,536
<=============
> -----Original Message-----
> From: Gerrit Grobbelaar <ge...@jeasy.net>
> Sent: Saturday 26 July 2008 10:32
> To: users@mina.apache.org
> CC:
> Subject: Re: byte[] usage - not GCd
>
>
> Yes, 1.1.7 but 1.1.6 as well.
>
> > -----Original Message-----
> > From: Emmanuel Lecharny <el...@gmail.com>
> > Sent: Saturday 26 July 2008 10:20
> > To: users@mina.apache.org
> > CC:
> > Subject: Re: byte[] usage - not GCd
> >
> >
> > Hi,
> >
> > MINA 1.1.7 ?
> >
> > Gerrit Grobbelaar wrote:
> > > YourKit shows the Class list as follows:
> > > - java.nio.HeapByteBuffer
> > > - org.apache.mina.common.SimpleByteBufferAllocator$SimpleByteBuffer
> > > - org.apache.mina.filter.codec.ProtocolCodecFilter$HiddenByteBuffer
> > > - org.apache.mina.common.IoFilter$WriteRequest
> > > - java.util.concurrent.ConcurrentLinkedQueue$Node
> > >
> > > Note the above contributes 99%+ for the byte[] allocations, and the
> > > byte[] allocations are causing the OOM.
> > >
> > >> -----Original Message-----
> > >> From: Gerrit Grobbelaar <ge...@jeasy.net>
> > >> Sent: Saturday 26 July 2008 09:48
> > >> To: users@mina.apache.org
> > >> CC:
> > >> Subject: Re: byte[] usage - not GCd
> > >>
> > >>
> > >> Another symptom:
> > >> - 100 Java client bots connected, sending NO messages.
> > >> - 1 Flash client, typing in a message every 2 seconds
> > >> - memory consumption (byte[] allocations) increases and a Mark Sweep
> > >> with JConsole doesn't do anything (stays on 100s of MBs).
> > >> - kill off all bots
> > >> - Mark sweep
> > >> - Heap mem usage drops to about 2 - 5MB
> > >>
> > >>> -----Original Message-----
> > >>> From: Gerrit Grobbelaar <ge...@jeasy.net>
> > >>> Sent: Saturday 26 July 2008 09:33
> > >>> To: users@mina.apache.org
> > >>> CC:
> > >>> Subject: byte[] usage - not GCd
> > >>>
> > >>>
> > >>> Hi,
> > >>>
> > >>> I'm having an issue with OOM - here is my story:
> > >>>
> > >>> Criteria:
> > >>> --------
> > >>> - concurrent clients connecting to a chatserver, need to broadcast
> > >>> the messages
> > >>> - clients keep same connection (each one having its own IoSession
> > >>> during lifetime)
> > >>>
> > >>>
> > >>> Symptoms:
> > >>> --------
> > >>> - massive amounts of byte[] allocations that just grows and grows,
> > >>> until OOM. - still OOM even if i have 100 bots connected, sending one
> > >>> message each every 5 minutes
> > >>>
> > >>> When I have 1 - 5 clients connected (all simulated with bots) that
> > >>> send one message per bot every 2 - 5 seconds, heap space stays under
> > >>> control with GC.
> > >>>
> > >>>
> > >>> Configs/Tools:
> > >>> -------------
> > >>> - JVM: 1.6u7
> > >>> - VM arguments: -Xms128m -Xmx384m
> > >>> - Operating System: Linux 2.6.15.7
> > >>> - Architecture: i386
> > >>> - Number of processors: 2
> > >>> - Model: Intel(R) Pentium(R) 4 CPU 2.80GHz
> > >>> - Committed virtual memory: 681,564 kbytes
> > >>> - Heap buffer allocation (Direct is very insufficient)
> > >>> - SimpleByteBufferAllocator (do note I have same result with
> > >>> PooledByteBufferAllocator)
> > >>> - YourKit and JProfiler
> > >>>
> > >>> Both YourKit and JProfiler point fingers at:
> > >>> byte[] allocation due to org.apache.mina.common.IoSession.write()
> > >>> calls
> > >>>
> > >>>
> > >>> What could I possibly be doing wrong that I will have the above
> > >>> scenario where the byte[] allocations never GC properly?
> > >>>
> > >>> Thanks,
> > >>> Gerrit
Re: byte[] usage - not GCd
Posted by Gerrit Grobbelaar <ge...@jeasy.net>.
Yes, 1.1.7 but 1.1.6 as well.
> -----Original Message-----
> From: Emmanuel Lecharny <el...@gmail.com>
> Sent: Saturday 26 July 2008 10:20
> To: users@mina.apache.org
> CC:
> Subject: Re: byte[] usage - not GCd
>
>
> Hi,
>
> MINA 1.1.7 ?
>
> Gerrit Grobbelaar wrote:
> > YourKit shows the Class list as follows:
> > - java.nio.HeapByteBuffer
> > - org.apache.mina.common.SimpleByteBufferAllocator$SimpleByteBuffer
> > - org.apache.mina.filter.codec.ProtocolCodecFilter$HiddenByteBuffer
> > - org.apache.mina.common.IoFilter$WriteRequest
> > - java.util.concurrent.ConcurrentLinkedQueue$Node
> >
> > Note the above contributes 99%+ for the byte[] allocations, and the
> > byte[] allocations are causing the OOM.
> >
> >> -----Original Message-----
> >> From: Gerrit Grobbelaar <ge...@jeasy.net>
> >> Sent: Saturday 26 July 2008 09:48
> >> To: users@mina.apache.org
> >> CC:
> >> Subject: Re: byte[] usage - not GCd
> >>
> >>
> >> Another symptom:
> >> - 100 Java client bots connected, sending NO messages.
> >> - 1 Flash client, typing in a message every 2 seconds
> >> - memory consumption (byte[] allocations) increases and a Mark Sweep
> >> with JConsole doesn't do anything (stays on 100s of MBs).
> >> - kill off all bots
> >> - Mark sweep
> >> - Heap mem usage drops to about 2 - 5MB
> >>
> >>> -----Original Message-----
> >>> From: Gerrit Grobbelaar <ge...@jeasy.net>
> >>> Sent: Saturday 26 July 2008 09:33
> >>> To: users@mina.apache.org
> >>> CC:
> >>> Subject: byte[] usage - not GCd
> >>>
> >>>
> >>> Hi,
> >>>
> >>> I'm having an issue with OOM - here is my story:
> >>>
> >>> Criteria:
> >>> --------
> >>> - concurrent clients connecting to a chatserver, need to broadcast the
> >>> messages
> >>> - clients keep same connection (each one having its own IoSession
> >>> during lifetime)
> >>>
> >>>
> >>> Symptoms:
> >>> --------
> >>> - massive amounts of byte[] allocations that just grows and grows,
> >>> until OOM. - still OOM even if i have 100 bots connected, sending one
> >>> message each every 5 minutes
> >>>
> >>> When I have 1 - 5 clients connected (all simulated with bots) that send
> >>> one message per bot every 2 - 5 seconds, heap space stays under control
> >>> with GC.
> >>>
> >>>
> >>> Configs/Tools:
> >>> -------------
> >>> - JVM: 1.6u7
> >>> - VM arguments: -Xms128m -Xmx384m
> >>> - Operating System: Linux 2.6.15.7
> >>> - Architecture: i386
> >>> - Number of processors: 2
> >>> - Model: Intel(R) Pentium(R) 4 CPU 2.80GHz
> >>> - Committed virtual memory: 681,564 kbytes
> >>> - Heap buffer allocation (Direct is very insufficient)
> >>> - SimpleByteBufferAllocator (do note I have same result with
> >>> PooledByteBufferAllocator)
> >>> - YourKit and JProfiler
> >>>
> >>> Both YourKit and JProfiler point fingers at:
> >>> byte[] allocation due to org.apache.mina.common.IoSession.write() calls
> >>>
> >>>
> >>> What could I possibly be doing wrong that I will have the above
> >>> scenario where the byte[] allocations never GC properly?
> >>>
> >>> Thanks,
> >>> Gerrit
Re: byte[] usage - not GCd
Posted by Emmanuel Lecharny <el...@gmail.com>.
Hi,
MINA 1.1.7 ?
Gerrit Grobbelaar wrote:
> YourKit shows the Class list as follows:
> - java.nio.HeapByteBuffer
> - org.apache.mina.common.SimpleByteBufferAllocator$SimpleByteBuffer
> - org.apache.mina.filter.codec.ProtocolCodecFilter$HiddenByteBuffer
> - org.apache.mina.common.IoFilter$WriteRequest
> - java.util.concurrent.ConcurrentLinkedQueue$Node
>
> Note the above contributes 99%+ for the byte[] allocations, and the byte[]
> allocations are causing the OOM.
>
>
>> -----Original Message-----
>> From: Gerrit Grobbelaar <ge...@jeasy.net>
>> Sent: Saturday 26 July 2008 09:48
>> To: users@mina.apache.org
>> CC:
>> Subject: Re: byte[] usage - not GCd
>>
>>
>> Another symptom:
>> - 100 Java client bots connected, sending NO messages.
>> - 1 Flash client, typing in a message every 2 seconds
>> - memory consumption (byte[] allocations) increases and a Mark Sweep with
>> JConsole doesn't do anything (stays on 100s of MBs).
>> - kill off all bots
>> - Mark sweep
>> - Heap mem usage drops to about 2 - 5MB
>>
>>
>>> -----Original Message-----
>>> From: Gerrit Grobbelaar <ge...@jeasy.net>
>>> Sent: Saturday 26 July 2008 09:33
>>> To: users@mina.apache.org
>>> CC:
>>> Subject: byte[] usage - not GCd
>>>
>>>
>>> Hi,
>>>
>>> I'm having an issue with OOM - here is my story:
>>>
>>> Criteria:
>>> --------
>>> - concurrent clients connecting to a chatserver, need to broadcast the
>>> messages
>>> - clients keep same connection (each one having its own IoSession during
>>> lifetime)
>>>
>>>
>>> Symptoms:
>>> --------
>>> - massive amounts of byte[] allocations that just grows and grows, until
>>> OOM. - still OOM even if i have 100 bots connected, sending one message
>>> each every 5 minutes
>>>
>>> When I have 1 - 5 clients connected (all simulated with bots) that send
>>> one message per bot every 2 - 5 seconds, heap space stays under control
>>> with GC.
>>>
>>>
>>> Configs/Tools:
>>> -------------
>>> - JVM: 1.6u7
>>> - VM arguments: -Xms128m -Xmx384m
>>> - Operating System: Linux 2.6.15.7
>>> - Architecture: i386
>>> - Number of processors: 2
>>> - Model: Intel(R) Pentium(R) 4 CPU 2.80GHz
>>> - Committed virtual memory: 681,564 kbytes
>>> - Heap buffer allocation (Direct is very insufficient)
>>> - SimpleByteBufferAllocator (do note I have same result with
>>> PooledByteBufferAllocator)
>>> - YourKit and JProfiler
>>>
>>> Both YourKit and JProfiler point fingers at:
>>> byte[] allocation due to org.apache.mina.common.IoSession.write() calls
>>>
>>>
>>> What could I possibly be doing wrong that I will have the above scenario
>>> where the byte[] allocations never GC properly?
>>>
>>> Thanks,
>>> Gerrit
>>>
>
>
--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org
Re: byte[] usage - not GCd
Posted by Gerrit Grobbelaar <ge...@jeasy.net>.
On 27 July 2008 04:47, Tuure Laurinolli wrote:
> Can you ensure
> that the client is not possible cause of congestion (is it doing
> something other than blocking in a read(), recv() or select() call when
> waiting for data?).
I realized that my clients are not even reading the data from the socket input
stream, thus congesting the byte[] in the server VM that is trying to write
to these clients. This is implemented in the _run_ method for each of the
clients (implementing java.lang.Runnable), but the _start_ method was not
invoked on any of them.
This bug on my side was due to the fact that this code worked 100% before, but
was refactored a few times in the meantime, and obviously getting rid of this
socket inputstream handling (starting of the threads) in the process.
However, no excuse and sincere apologies for this. I'm glad I found it
because as I said earlier:
On 26 July 2008 14:14, Gerrit Grobbelaar wrote:
> > > > I still believe I'm doing something wrong in my code though, as I
> > > > read only good things about MINA in the reviews,
Back to some stats:
Simulated clients: 150 (each 1 msg every 3 seconds)
JVM uptime: 1h30mins
Total heap mem: 16MB
byte[] heap mem: 1.1MB
This is GREAT seeing that before with the above criteria byte[] allocations
caused the VM to crash (OOM) within 2 minutes.
Thanks,
Gerrit
> -----Original Message-----
> From: Tuure Laurinolli <tu...@indagon.com>
> Sent: Sunday 27 July 2008 04:47
> To: users@mina.apache.org
> CC:
> Subject: Re: byte[] usage - not GCd
>
>
> Gerrit Grobbelaar wrote:
> > YourKit shows the Class list as follows:
> > - java.nio.HeapByteBuffer
> > - org.apache.mina.common.SimpleByteBufferAllocator$SimpleByteBuffer
> > - org.apache.mina.filter.codec.ProtocolCodecFilter$HiddenByteBuffer
> > - org.apache.mina.common.IoFilter$WriteRequest
> > - java.util.concurrent.ConcurrentLinkedQueue$Node
>
> Is this the reference chain that keeps the byte[]s alive? Also, should
> it be interpreted so that there are ConcurrentLinkedQueue$Nodes that are
> alive (presumably because they are in a queue which is alive) that have
> references to WriteRequests that have references to HiddenByteBuffers
> that through some other objects refer to the actual byte[]s?
>
> If this is the case, it would seem that write requests never get garbage
> collected for some reason. In a later message you mention that you get
> WriteTimeoutExceptions on MINA 2. Given these two clues I would think
> that your writes are not completing in timely manner. Are you sure that
> there is no source of congestion between the simulated clients and the
> server?
>
> Do note that if you are using TCP (and since you are talking about
> connections in the first place, you most likely are), the receiving end
> directly affects the sending end's ability to send data. I understand
> from your later messages that you have some custom network code deployed
> on the clients that are supposed to be receiving data. Can you ensure
> that the client is not possible cause of congestion (is it doing
> something other than blocking in a read(), recv() or select() call when
> waiting for data?).
>
> A server-side remedy might be to reduce the write timeout, and to limit
> the maximum amount of data buffered for writing per connection. The
> former is done via IoSessionConfig in MINA 2 at least, but I think you
> have to actively check for the latter in your protocol code. There has
> been some discussion about it earlier on this mailing list, or the -dev
> list.
Re: byte[] usage - not GCd
Posted by Tuure Laurinolli <tu...@indagon.com>.
Gerrit Grobbelaar wrote:
> YourKit shows the Class list as follows:
> - java.nio.HeapByteBuffer
> - org.apache.mina.common.SimpleByteBufferAllocator$SimpleByteBuffer
> - org.apache.mina.filter.codec.ProtocolCodecFilter$HiddenByteBuffer
> - org.apache.mina.common.IoFilter$WriteRequest
> - java.util.concurrent.ConcurrentLinkedQueue$Node
Is this the reference chain that keeps the byte[]s alive? Also, should
it be interpreted so that there are ConcurrentLinkedQueue$Nodes that are
alive (presumably because they are in a queue which is alive) that have
references to WriteRequests that have references to HiddenByteBuffers
that through some other objects refer to the actual byte[]s?
If this is the case, it would seem that write requests never get garbage
collected for some reason. In a later message you mention that you get
WriteTimeoutExceptions on MINA 2. Given these two clues I would think
that your writes are not completing in timely manner. Are you sure that
there is no source of congestion between the simulated clients and the
server?
Do note that if you are using TCP (and since you are talking about
connections in the first place, you most likely are), the receiving end
directly affects the sending end's ability to send data. I understand
from your later messages that you have some custom network code deployed
on the clients that are supposed to be receiving data. Can you ensure
that the client is not possible cause of congestion (is it doing
something other than blocking in a read(), recv() or select() call when
waiting for data?).
A server-side remedy might be to reduce the write timeout, and to limit
the maximum amount of data buffered for writing per connection. The
former is done via IoSessionConfig in MINA 2 at least, but I think you
have to actively check for the latter in your protocol code. There has
been some discussion about it earlier on this mailing list, or the -dev
list.
Re: byte[] usage - not GCd
Posted by Gerrit Grobbelaar <ge...@jeasy.net>.
YourKit shows the Class list as follows:
- java.nio.HeapByteBuffer
- org.apache.mina.common.SimpleByteBufferAllocator$SimpleByteBuffer
- org.apache.mina.filter.codec.ProtocolCodecFilter$HiddenByteBuffer
- org.apache.mina.common.IoFilter$WriteRequest
- java.util.concurrent.ConcurrentLinkedQueue$Node
Note the above contributes 99%+ for the byte[] allocations, and the byte[]
allocations are causing the OOM.
> -----Original Message-----
> From: Gerrit Grobbelaar <ge...@jeasy.net>
> Sent: Saturday 26 July 2008 09:48
> To: users@mina.apache.org
> CC:
> Subject: Re: byte[] usage - not GCd
>
>
> Another symptom:
> - 100 Java client bots connected, sending NO messages.
> - 1 Flash client, typing in a message every 2 seconds
> - memory consumption (byte[] allocations) increases and a Mark Sweep with
> JConsole doesn't do anything (stays on 100s of MBs).
> - kill off all bots
> - Mark sweep
> - Heap mem usage drops to about 2 - 5MB
>
> > -----Original Message-----
> > From: Gerrit Grobbelaar <ge...@jeasy.net>
> > Sent: Saturday 26 July 2008 09:33
> > To: users@mina.apache.org
> > CC:
> > Subject: byte[] usage - not GCd
> >
> >
> > Hi,
> >
> > I'm having an issue with OOM - here is my story:
> >
> > Criteria:
> > --------
> > - concurrent clients connecting to a chatserver, need to broadcast the
> > messages
> > - clients keep same connection (each one having its own IoSession during
> > lifetime)
> >
> >
> > Symptoms:
> > --------
> > - massive amounts of byte[] allocations that just grows and grows, until
> > OOM. - still OOM even if i have 100 bots connected, sending one message
> > each every 5 minutes
> >
> > When I have 1 - 5 clients connected (all simulated with bots) that send
> > one message per bot every 2 - 5 seconds, heap space stays under control
> > with GC.
> >
> >
> > Configs/Tools:
> > -------------
> > - JVM: 1.6u7
> > - VM arguments: -Xms128m -Xmx384m
> > - Operating System: Linux 2.6.15.7
> > - Architecture: i386
> > - Number of processors: 2
> > - Model: Intel(R) Pentium(R) 4 CPU 2.80GHz
> > - Committed virtual memory: 681,564 kbytes
> > - Heap buffer allocation (Direct is very insufficient)
> > - SimpleByteBufferAllocator (do note I have same result with
> > PooledByteBufferAllocator)
> > - YourKit and JProfiler
> >
> > Both YourKit and JProfiler point fingers at:
> > byte[] allocation due to org.apache.mina.common.IoSession.write() calls
> >
> >
> > What could I possibly be doing wrong that I will have the above scenario
> > where the byte[] allocations never GC properly?
> >
> > Thanks,
> > Gerrit
Re: byte[] usage - not GCd
Posted by Gerrit Grobbelaar <ge...@jeasy.net>.
Another symptom:
- 100 Java client bots connected, sending NO messages.
- 1 Flash client, typing in a message every 2 seconds
- memory consumption (byte[] allocations) increases and a Mark Sweep with
JConsole doesn't do anything (stays on 100s of MBs).
- kill off all bots
- Mark sweep
- Heap mem usage drops to about 2 - 5MB
> -----Original Message-----
> From: Gerrit Grobbelaar <ge...@jeasy.net>
> Sent: Saturday 26 July 2008 09:33
> To: users@mina.apache.org
> CC:
> Subject: byte[] usage - not GCd
>
>
> Hi,
>
> I'm having an issue with OOM - here is my story:
>
> Criteria:
> --------
> - concurrent clients connecting to a chatserver, need to broadcast the
> messages
> - clients keep same connection (each one having its own IoSession during
> lifetime)
>
>
> Symptoms:
> --------
> - massive amounts of byte[] allocations that just grows and grows, until
> OOM. - still OOM even if i have 100 bots connected, sending one message
> each every 5 minutes
>
> When I have 1 - 5 clients connected (all simulated with bots) that send one
> message per bot every 2 - 5 seconds, heap space stays under control with
> GC.
>
>
> Configs/Tools:
> -------------
> - JVM: 1.6u7
> - VM arguments: -Xms128m -Xmx384m
> - Operating System: Linux 2.6.15.7
> - Architecture: i386
> - Number of processors: 2
> - Model: Intel(R) Pentium(R) 4 CPU 2.80GHz
> - Committed virtual memory: 681,564 kbytes
> - Heap buffer allocation (Direct is very insufficient)
> - SimpleByteBufferAllocator (do note I have same result with
> PooledByteBufferAllocator)
> - YourKit and JProfiler
>
> Both YourKit and JProfiler point fingers at:
> byte[] allocation due to org.apache.mina.common.IoSession.write() calls
>
>
> What could I possibly be doing wrong that I will have the above scenario
> where the byte[] allocations never GC properly?
>
> Thanks,
> Gerrit