Help needed with OutOfMemory error and/or GC Issues/Dead Locks

[Sai Pullabhotla <sa...@jmethods.com>]

Guys,

I've been fighting with various issues using Mina over the past couple
of weeks, and a couple of them are -

1) OutOfMemory on Heap
2) Possible dead lock (it could just be a GC issue). Application does
not make any progress nor any error is reported.
3) When running under Java 6, I often am getting
java.lang.OutOfMemoryError: GC overhead limit exceeded, which tells me
that GC is not able to clean up the stuff.

The version of MINA I'm using is 2.0-RC1.

Just to give you a background - I'm trying to create what I would like
to call "Broker". A Broker basically connects to two different
servers, and forwards the data received from one server to the other.
Sounds simple, but I'm not having much luck using the Nio
implementation of MINA. Things seem to work fine when there is not
much data flows through the broker, but, when large amounts of data is
flowing through, I'm running out of memory.

I did see the MINA documentation on the OOM, and am using
IoEventQueueThrottle, but did not get any better results. I hope I'm
using it the right way, as I did not see much documentation on that
class.

I decided to stop working on the main project, and decided to put a
simple test case as follows:

1. Create a MINA server using NioSocketAcceptor. I called this the
MinaSenderServer. All this does is as soon as a client connects to it,
it starts sending a bunch of random data, upto the specified size, and
closes the session. Basically, the IoHandler.sessionOpened, goes into
a loop and calls IoSession.write until the desired amount of bytes are
written, and the calls the IoSession.close(false).

2. I Created another MINA server using the very same
NioSocketAcceptor. I called this the MinaReceiverServer. All this does
is - accepts connections from clients, and dumps any data received
from the client into a temp file.

3. I also created my Broker, which uses NioSocketConnector. It
connects to to MinaReceiverServer first, and then to MinaSenderServer.
Creates a tunnel using both the sessions. It then forwards the random
data received from MinaSenderServer to the MinaReceiverServer.

This appears to be working for small amounts of data (< 1MB), but get
either a OOM on heap, or GC error in Java 6 or app appears to have
locked up making no progress. Just so you know, I'm running these 3
classes in 3 different JVMs using default memory settings for Java. My
understanding is that not much of data should be kept in memory, but
some how it is retained in memory, or GC is not able to reclaim it. My
requirement is that I would have server concurrent Broker sessions at
any given time, which could be carrying any amount of data ranging
from a few KB to a few GB.

I've tried various configuration settings on the Acceptors and
Connectors such as -

1. Using thread pool executor/Not using
2. Using IoEventQueue Throttle/Not using
3. When using a Thread pool, waiting for WriteFutures to complete
4. Setting various session configuration options such as Send and
Receive buffers of sockets

None of these gave me any positive result. Obviously I'm missing something.

I can send the test classes that I wrote if some one wants to take a
look at it and what I'm doing wrong. Hopefully, it is something simple
that I'm unknown of, or overlooking. Let me know the preferred way for
posting this test code.

Re: Help needed with OutOfMemory error and/or GC Issues/Dead Locks

[Sai Pullabhotla <sa...@jmethods.com>]

I've attached a couple of simple classes, one server, and one client
to reproduce this lockup/OOM, at
https://issues.apache.org/jira/browse/DIRMINA-789. I appreciate your
time and help looking into this.


On Thu, Jun 10, 2010 at 6:32 PM, Emmanuel Lecharny <el...@gmail.com> wrote:
> On 6/11/10 1:10 AM, Sai Pullabhotla wrote:
>>
>> Guys,
>>
>> I've been fighting with various issues using Mina over the past couple
>> of weeks, and a couple of them are -
>>
>> 1) OutOfMemory on Heap
>> 2) Possible dead lock (it could just be a GC issue). Application does
>> not make any progress nor any error is reported.
>> 3) When running under Java 6, I often am getting
>> java.lang.OutOfMemoryError: GC overhead limit exceeded, which tells me
>> that GC is not able to clean up the stuff.
>>
>> The version of MINA I'm using is 2.0-RC1.
>>
>> Just to give you a background - I'm trying to create what I would like
>> to call "Broker". A Broker basically connects to two different
>> servers, and forwards the data received from one server to the other.
>> Sounds simple, but I'm not having much luck using the Nio
>> implementation of MINA. Things seem to work fine when there is not
>> much data flows through the broker, but, when large amounts of data is
>> flowing through, I'm running out of memory.
>>
>> I did see the MINA documentation on the OOM, and am using
>> IoEventQueueThrottle, but did not get any better results. I hope I'm
>> using it the right way, as I did not see much documentation on that
>> class.
>>
>> I decided to stop working on the main project, and decided to put a
>> simple test case as follows:
>>
>> 1. Create a MINA server using NioSocketAcceptor. I called this the
>> MinaSenderServer. All this does is as soon as a client connects to it,
>> it starts sending a bunch of random data, upto the specified size, and
>> closes the session. Basically, the IoHandler.sessionOpened, goes into
>> a loop and calls IoSession.write until the desired amount of bytes are
>> written, and the calls the IoSession.close(false).
>>
>> 2. I Created another MINA server using the very same
>> NioSocketAcceptor. I called this the MinaReceiverServer. All this does
>> is - accepts connections from clients, and dumps any data received
>> from the client into a temp file.
>>
>> 3. I also created my Broker, which uses NioSocketConnector. It
>> connects to to MinaReceiverServer first, and then to MinaSenderServer.
>> Creates a tunnel using both the sessions. It then forwards the random
>> data received from MinaSenderServer to the MinaReceiverServer.
>>
>> This appears to be working for small amounts of data (<  1MB), but get
>> either a OOM on heap, or GC error in Java 6 or app appears to have
>> locked up making no progress. Just so you know, I'm running these 3
>> classes in 3 different JVMs using default memory settings for Java. My
>> understanding is that not much of data should be kept in memory, but
>> some how it is retained in memory, or GC is not able to reclaim it. My
>> requirement is that I would have server concurrent Broker sessions at
>> any given time, which could be carrying any amount of data ranging
>> from a few KB to a few GB.
>>
>> I've tried various configuration settings on the Acceptors and
>> Connectors such as -
>>
>> 1. Using thread pool executor/Not using
>> 2. Using IoEventQueue Throttle/Not using
>> 3. When using a Thread pool, waiting for WriteFutures to complete
>> 4. Setting various session configuration options such as Send and
>> Receive buffers of sockets
>>
>> None of these gave me any positive result. Obviously I'm missing
>> something.
>>
>> I can send the test classes that I wrote if some one wants to take a
>> look at it and what I'm doing wrong. Hopefully, it is something simple
>> that I'm unknown of, or overlooking. Let me know the preferred way for
>> posting this test code.
>>
>
> Can you create a JIRA and attach those files into it ?
>
> Thanks !
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.nextury.com
>
>
>

Re: Help needed with OutOfMemory error and/or GC Issues/Dead Locks

[Emmanuel Lecharny <el...@gmail.com>]

On 6/11/10 1:10 AM, Sai Pullabhotla wrote:
> Guys,
>
> I've been fighting with various issues using Mina over the past couple
> of weeks, and a couple of them are -
>
> 1) OutOfMemory on Heap
> 2) Possible dead lock (it could just be a GC issue). Application does
> not make any progress nor any error is reported.
> 3) When running under Java 6, I often am getting
> java.lang.OutOfMemoryError: GC overhead limit exceeded, which tells me
> that GC is not able to clean up the stuff.
>
> The version of MINA I'm using is 2.0-RC1.
>
> Just to give you a background - I'm trying to create what I would like
> to call "Broker". A Broker basically connects to two different
> servers, and forwards the data received from one server to the other.
> Sounds simple, but I'm not having much luck using the Nio
> implementation of MINA. Things seem to work fine when there is not
> much data flows through the broker, but, when large amounts of data is
> flowing through, I'm running out of memory.
>
> I did see the MINA documentation on the OOM, and am using
> IoEventQueueThrottle, but did not get any better results. I hope I'm
> using it the right way, as I did not see much documentation on that
> class.
>
> I decided to stop working on the main project, and decided to put a
> simple test case as follows:
>
> 1. Create a MINA server using NioSocketAcceptor. I called this the
> MinaSenderServer. All this does is as soon as a client connects to it,
> it starts sending a bunch of random data, upto the specified size, and
> closes the session. Basically, the IoHandler.sessionOpened, goes into
> a loop and calls IoSession.write until the desired amount of bytes are
> written, and the calls the IoSession.close(false).
>
> 2. I Created another MINA server using the very same
> NioSocketAcceptor. I called this the MinaReceiverServer. All this does
> is - accepts connections from clients, and dumps any data received
> from the client into a temp file.
>
> 3. I also created my Broker, which uses NioSocketConnector. It
> connects to to MinaReceiverServer first, and then to MinaSenderServer.
> Creates a tunnel using both the sessions. It then forwards the random
> data received from MinaSenderServer to the MinaReceiverServer.
>
> This appears to be working for small amounts of data (<  1MB), but get
> either a OOM on heap, or GC error in Java 6 or app appears to have
> locked up making no progress. Just so you know, I'm running these 3
> classes in 3 different JVMs using default memory settings for Java. My
> understanding is that not much of data should be kept in memory, but
> some how it is retained in memory, or GC is not able to reclaim it. My
> requirement is that I would have server concurrent Broker sessions at
> any given time, which could be carrying any amount of data ranging
> from a few KB to a few GB.
>
> I've tried various configuration settings on the Acceptors and
> Connectors such as -
>
> 1. Using thread pool executor/Not using
> 2. Using IoEventQueue Throttle/Not using
> 3. When using a Thread pool, waiting for WriteFutures to complete
> 4. Setting various session configuration options such as Send and
> Receive buffers of sockets
>
> None of these gave me any positive result. Obviously I'm missing something.
>
> I can send the test classes that I wrote if some one wants to take a
> look at it and what I'm doing wrong. Hopefully, it is something simple
> that I'm unknown of, or overlooking. Let me know the preferred way for
> posting this test code.
>    
Can you create a JIRA and attach those files into it ?

Thanks !


-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.nextury.com