Re: OutOfMemory when using ReadThrottleFilter

[ma...@swenmoc.de]

Trustin Lee wrote:
> Hi Swen,

Hi Trustin,

sorry for the late response...

> 
> On 1/31/07, Swen Moczarski <ma...@swenmoc.de> wrote:
>>
>> Hi Trustin,
>>
>> thanks a lot for your response.
>>
>> When I'm using the remaining() calls, I still get an out-of-memory error.
>> But when I'm using capacity() it looks ok.
>> Maybe, this becomes only a problem when a lot of small messages are 
>> stored
>> in bigger ByteBuffers. But I think it would
>> be safer to prevent overloading when the ReadThrottleFilter tracks the
>> memory allocated by the ByteBuffers (capacity())
>> instead of the received bytes. WDYT?
> 
> 
> The root cause of this problem is that the default receive buffer size is
> too big for smaller sockets.  MINA allocates a buffer whose size is same
> with the default receive buffer size because it doesn't know how large 
> chunk
> of data will come in.  I didn't thought the default receive buffer size is
> big, but it is turned out to be bigger than 40k.  This means you allocate
> 40k buffer even if the size of the packet you exchange is much smaller than
> that.  In this case, using remaining() might cause unexpected
> OutOfMemoryError.  Configuring the size of the receive buffer with the
> following code will do the trick:
> 
>      ((SocketSessionConfig) acceptor.getDefaultConfig
> ().getSessionConfig()).setReceiveBufferSize(512);
> 
> If we replace remaining() with capacity(), we won't need to do that, but we
> will observe unexpected slowdown in low throughput communication.

Ok, I see your point.

> 
> Here is my simple stess-test:
>>
>> public class Test {
>>
>>     private final static int PORT = 8056;
>>
>>     public static void main(String[] args) throws Exception {
>>        IoHandler handler = new IoHandlerAdapter() {
>>
>>           public void exceptionCaught(IoSession session, Throwable cause)
>> throws Exception {
>>              cause.printStackTrace();
>>           }
>>           public void sessionCreated(IoSession session) throws 
>> Exception {
>>              ReadThrottleFilterBuilder throttleFilter = new
>> ReadThrottleFilterBuilder();
>>              throttleFilter.attach(session.getFilterChain());
>>           }
>>        };
>>        SocketAcceptor acceptor = new SocketAcceptor();
>>        acceptor.bind(new InetSocketAddress(PORT), handler);
>>
>>        Socket socket = new Socket("localhost", PORT);
>>        Writer writer = new OutputStreamWriter(socket.getOutputStream());
>>
>>        while (true) {
>>           writer.write("test");
>>           writer.flush();
>>        }
>>     }
>> }
>>
>> I have tried the test with version 1.0.1 and the current trunk (with the
>> current trunk it takes longer until the memory
>> error occures).
>> Should I attach the test to
>> https://issues.apache.org/jira/browse/DIRMINA-338 and reopen the issue?
> 
> 
> It takes very long for me to reproduce the problem.  I guess you can get 
> rid
> of this problem trying these options:
> 
> 1) Increase max total direct buffer size using 
> -XX:MaxDirectMemorySizeoption (
> e.g. -XX:MaxDirectMemorySize=128M)

That we have already tried, but the error will occure only later.

> 2) Change the allocator to SimpleByteBufferAllocator (i.e.
> ByteBuffer.setAllocator(new SimpleByteBufferAllocator());)

This option solve our problem :-) 

> 3) Allocate heap buffers instead of direct buffers (i.e.
> ByteBuffer.setUseDirectBuffers(false);)

This option only brings an out-of-memory error for the heap.

> 
> Please try each step not reverting back the previous step.  I believe
> OutOfMemoryError will be gone once you reach to the step 3, but please let
> me know the result.

Again, the SimpleByteBufferAllocator seems to solve the problem. Thanks a lot for your help! 
The mina framework help us a lot in the development of our network layer. Thanks to the mina team!

> 
> HTH,
> Trustin

Re: OutOfMemory when using ReadThrottleFilter

[Trustin Lee <tr...@gmail.com>]

2007-02-20 (화), 23:49 +0100, mail@swenmoc.de 쓰시길:
> > 1) Increase max total direct buffer size using 
> > -XX:MaxDirectMemorySizeoption (
> > e.g. -XX:MaxDirectMemorySize=128M)
> 
> That we have already tried, but the error will occure only later.
> 
> > 2) Change the allocator to SimpleByteBufferAllocator (i.e.
> > ByteBuffer.setAllocator(new SimpleByteBufferAllocator());)
> 
> This option solve our problem :-) 
> 
> > 3) Allocate heap buffers instead of direct buffers (i.e.
> > ByteBuffer.setUseDirectBuffers(false);)
> 
> This option only brings an out-of-memory error for the heap.
> 
> > 
> > Please try each step not reverting back the previous step.  I believe
> > OutOfMemoryError will be gone once you reach to the step 3, but please let
> > me know the result.
> 
> Again, the SimpleByteBufferAllocator seems to solve the problem. Thanks a lot for your help! 
> The mina framework help us a lot in the development of our network layer. Thanks to the mina team!

Employing both 2) and 3) will be the best in performance for current
VMs.

HTH,
Trustin
-- 
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6