You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by mat <fo...@gmail.com> on 2007/07/20 09:38:28 UTC
Out Of Memory Problem Again
My server sometimes faced "OOM" problem. (I couldn't profile it since TPTP
crashed my server before OOM occured). I didn't see major memory leak when
profiling. Therefore, I believe OOM happens when READ or WRITE operation
can't handle the incoming messages or outgoing messages. (However my
incoming messages normally 20 * 512bytes/sec, NOT too fast, right?). Last
time i saw my server memory usage is over 600MB in windows XP.
Re: Out Of Memory Problem Again
Posted by mat <fo...@gmail.com>.
Normally the OOM happens randomly. However it happens more often recently. I
will deploy 2 servers with 1.5 and 1.6. See what will happen. Thanks.
On 7/21/07, 向秦贤 <fy...@gmail.com> wrote:
>
> No, I dont think it is a bug of vm.
> Running in jdk6, just try to expect jdk6 maybe smart enough to handle
> direct
> buffer reference.
> I done a quick check of mina code, and not found "release direct buffer"
> code.
>
> Is there a result when running jdk6?
>
> 2007/7/21, mat <fo...@gmail.com>:
> >
> > Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_08-b03)
> > Java HotSpot(TM) Client VM (build 1.5.0_08-b03, mixed mode, sharing)
> > Windows 2003
> >
> > So you think it is bug of JVM?
> >
> > On 7/21/07, 向秦贤 <fy...@gmail.com> wrote:
> > >
> > > Not sure your jdk version.
> > > http://issues.apache.org/jira/browse/DIRMINA-320 not running in jdk6.
> > > Try run in jdk6 to check it out whether happen again.
> > > There a sure thing, some direct buffer reference hold by application.
> vm
> > > cannot clean it.
> > >
> > > 2007/7/21, mat <fo...@gmail.com>:
> > > >
> > > > http://issues.apache.org/jira/browse/DIRMINA-320
> > > >
> > > > Please check this out. It seems that i am not the only one who faces
> > > this
> > > > problem and it happens in mina core. I quote something written by
> > > Trustin.
> > > > "Other session might have allocated huge memory block and other
> > session
> > > > might be being affected by its side-effect. " However, I only had
> one
> > > > client
> > > > connecting to my server when OOM occurs.
> > > >
> > > > On 7/21/07, 向秦贤 <fy...@gmail.com> wrote:
> > > > >
> > > > > Maybe Direct buffer not released.
> > > > > Direct buffer must explicit release.
> > > > > so somewhere may check if direct buffer and set to null.
> > > > >
> > > > > 2007/7/21, mat <fo...@gmail.com>:
> > > > > >
> > > > > > I captured the exception message.
> > > > > >
> > > > > >
> > org.apache.mina.common.support.DefaultExceptionMonitorexceptionCaught
> > > > > > Unexpected exception.
> > > > > > java.lang.OutOfMemoryError: Direct buffer memory
> > > > > > at java.nio.Bits.reserveMemory
> > > > > > at java.nio.DirectByteBuffer.<init>
> > > > > > at java.nio.ByteBuffer.allocateDirect
> > > > > > at sun.nio.ch.Util.getTemporaryDirectBuffer
> > > > > > at sun.nio.ch.IOUtil.write
> > > > > > at sun.nio.ch.SocketChannelImpl.write
> > > > > > at
> org.apache.mina.transport.socket.nio.SocketIOProcessor.doFlush<
> > > > > > SocketIoProcessor.java:428>
> > > > > > at
> org.apache.mina.transport.socket.nio.SocketIOProcessor.doFlush<
> > > > > > SocketIoProcessor.java:366>
> > > > > > at
> > org.apache.mina.transport.socket.nio.SocketIOProcessor.access$600
> > > <
> > > > > > SocketIoProcessor.java:44>
> > > > > > at
> > org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run
> > > <
> > > > > > SocketIoProcessor.java:509>
> > > > > > at org.apache.mina.util.NamePreservingRunnable.run<
> > > > > > NamePreservingRunnable.java:43>
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> > > > > > at java.util.concurrect.ThreadPoolExecutor$Worker.run
> > > > > > at java.lang.Thread.run
> > > > > >
> > > > > >
> > > > > > On 7/20/07, Luis Neves <lu...@co.sapo.pt> wrote:
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > mat wrote:
> > > > > > > > My server sometimes faced "OOM" problem. (I couldn't profile
> > it
> > > > > since
> > > > > > > TPTP
> > > > > > > > crashed my server before OOM occured). I didn't see major
> > memory
> > > > > leak
> > > > > > > when
> > > > > > > > profiling. Therefore, I believe OOM happens when READ or
> WRITE
> > > > > > operation
> > > > > > > > can't handle the incoming messages or outgoing messages.
> > > (However
> > > > my
> > > > > > > > incoming messages normally 20 * 512bytes/sec, NOT too fast,
> > > > right?).
> > > > > > > Last
> > > > > > > > time i saw my server memory usage is over 600MB in windows
> XP.
> > > > > > >
> > > > > > > Your code is broken ... the question is where. Mina can handle
> > > that
> > > > > > amount
> > > > > > > of
> > > > > > > messages without breaking a sweat.
> > > > > > > Do you have some kind of heavy processing in the receiving end
> > > that
> > > > > > delays
> > > > > > > the
> > > > > > > acceptance of messages?
> > > > > > >
> > > > > > > Did you try to use the ReadThrottleFilter?
> > > > > > > How are you doing you writes?
> > > > > > > A simple "iosession.write()" ?
> > > > > > > Did you try something like:
> > > > > > > WriteFuture wf = iosession.write();
> > > > > > > wf.join();
> > > > > > >
> > > > > > > Can we see the code of your Encoder/Decoder?
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Luis Neves
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > 向秦贤
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > 向秦贤
> > >
> >
>
>
>
> --
> 向秦贤
>
Re: Out Of Memory Problem Again
Posted by 向秦贤 <fy...@gmail.com>.
No, I dont think it is a bug of vm.
Running in jdk6, just try to expect jdk6 maybe smart enough to handle direct
buffer reference.
I done a quick check of mina code, and not found "release direct buffer"
code.
Is there a result when running jdk6?
2007/7/21, mat <fo...@gmail.com>:
>
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_08-b03)
> Java HotSpot(TM) Client VM (build 1.5.0_08-b03, mixed mode, sharing)
> Windows 2003
>
> So you think it is bug of JVM?
>
> On 7/21/07, 向秦贤 <fy...@gmail.com> wrote:
> >
> > Not sure your jdk version.
> > http://issues.apache.org/jira/browse/DIRMINA-320 not running in jdk6.
> > Try run in jdk6 to check it out whether happen again.
> > There a sure thing, some direct buffer reference hold by application. vm
> > cannot clean it.
> >
> > 2007/7/21, mat <fo...@gmail.com>:
> > >
> > > http://issues.apache.org/jira/browse/DIRMINA-320
> > >
> > > Please check this out. It seems that i am not the only one who faces
> > this
> > > problem and it happens in mina core. I quote something written by
> > Trustin.
> > > "Other session might have allocated huge memory block and other
> session
> > > might be being affected by its side-effect. " However, I only had one
> > > client
> > > connecting to my server when OOM occurs.
> > >
> > > On 7/21/07, 向秦贤 <fy...@gmail.com> wrote:
> > > >
> > > > Maybe Direct buffer not released.
> > > > Direct buffer must explicit release.
> > > > so somewhere may check if direct buffer and set to null.
> > > >
> > > > 2007/7/21, mat <fo...@gmail.com>:
> > > > >
> > > > > I captured the exception message.
> > > > >
> > > > >
> org.apache.mina.common.support.DefaultExceptionMonitorexceptionCaught
> > > > > Unexpected exception.
> > > > > java.lang.OutOfMemoryError: Direct buffer memory
> > > > > at java.nio.Bits.reserveMemory
> > > > > at java.nio.DirectByteBuffer.<init>
> > > > > at java.nio.ByteBuffer.allocateDirect
> > > > > at sun.nio.ch.Util.getTemporaryDirectBuffer
> > > > > at sun.nio.ch.IOUtil.write
> > > > > at sun.nio.ch.SocketChannelImpl.write
> > > > > at org.apache.mina.transport.socket.nio.SocketIOProcessor.doFlush<
> > > > > SocketIoProcessor.java:428>
> > > > > at org.apache.mina.transport.socket.nio.SocketIOProcessor.doFlush<
> > > > > SocketIoProcessor.java:366>
> > > > > at
> org.apache.mina.transport.socket.nio.SocketIOProcessor.access$600
> > <
> > > > > SocketIoProcessor.java:44>
> > > > > at
> org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run
> > <
> > > > > SocketIoProcessor.java:509>
> > > > > at org.apache.mina.util.NamePreservingRunnable.run<
> > > > > NamePreservingRunnable.java:43>
> > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> > > > > at java.util.concurrect.ThreadPoolExecutor$Worker.run
> > > > > at java.lang.Thread.run
> > > > >
> > > > >
> > > > > On 7/20/07, Luis Neves <lu...@co.sapo.pt> wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > mat wrote:
> > > > > > > My server sometimes faced "OOM" problem. (I couldn't profile
> it
> > > > since
> > > > > > TPTP
> > > > > > > crashed my server before OOM occured). I didn't see major
> memory
> > > > leak
> > > > > > when
> > > > > > > profiling. Therefore, I believe OOM happens when READ or WRITE
> > > > > operation
> > > > > > > can't handle the incoming messages or outgoing messages.
> > (However
> > > my
> > > > > > > incoming messages normally 20 * 512bytes/sec, NOT too fast,
> > > right?).
> > > > > > Last
> > > > > > > time i saw my server memory usage is over 600MB in windows XP.
> > > > > >
> > > > > > Your code is broken ... the question is where. Mina can handle
> > that
> > > > > amount
> > > > > > of
> > > > > > messages without breaking a sweat.
> > > > > > Do you have some kind of heavy processing in the receiving end
> > that
> > > > > delays
> > > > > > the
> > > > > > acceptance of messages?
> > > > > >
> > > > > > Did you try to use the ReadThrottleFilter?
> > > > > > How are you doing you writes?
> > > > > > A simple "iosession.write()" ?
> > > > > > Did you try something like:
> > > > > > WriteFuture wf = iosession.write();
> > > > > > wf.join();
> > > > > >
> > > > > > Can we see the code of your Encoder/Decoder?
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Luis Neves
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > 向秦贤
> > > >
> > >
> >
> >
> >
> > --
> > 向秦贤
> >
>
--
向秦贤
Re: Out Of Memory Problem Again
Posted by mat <fo...@gmail.com>.
I already used heap buffers and disable direct buffers.
ByteBuffer.setUseDirectBuffers(false); // Direct buffers perform bad.
ByteBuffer.setAllocator(new SimpleByteBufferAllocator()); // Disables
pooling.
On 7/21/07, Trustin Lee <tr...@gmail.com> wrote:
>
> On 7/21/07, mat <fo...@gmail.com> wrote:
> > Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_08-b03)
> > Java HotSpot(TM) Client VM (build 1.5.0_08-b03, mixed mode, sharing)
> > Windows 2003
> >
> > So you think it is bug of JVM?
>
> I'd suggest you to use heap buffers instead of direct buffers:
>
>
> http://mina.apache.org/faq.html#FAQ-Iget%7B%7BOutOfMemoryError%7D%7Dorresponsetimeoutandconnectionresetunderheavyload
> .
>
> HTH,
> Trustin
> --
> what we call human nature is actually human habit
> --
> http://gleamynode.net/
> --
> PGP Key ID: 0x0255ECA6
>
Re: Out Of Memory Problem Again
Posted by Trustin Lee <tr...@gmail.com>.
On 8/8/07, mat <fo...@gmail.com> wrote:
> After I use -XX:+HeapDumpOnOutOfMemoryError, finally I got "a
> java_pid2264.hprof" file(76M). How should I examine this file? Thanks.
You can open the file in your profiler application such as YourKit.
Netbeans have free CPU/heap profiler, so you could give it a try if
you don't have a commercial profiler.
If you are in trouble analyzing the dump file, you could upload it
somewhere so people here can take a look into.
HTH,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
Re: Out Of Memory Problem Again
Posted by mat <fo...@gmail.com>.
After I use -XX:+HeapDumpOnOutOfMemoryError, finally I got "a
java_pid2264.hprof" file(76M). How should I examine this file? Thanks.
On 8/7/07, Trustin Lee <tr...@gmail.com> wrote:
>
> On 8/7/07, mat <fo...@gmail.com> wrote:
> > In ClientSessionHandler
> >
> > public void messageReceived(IoSession session, Object message) {
> > // server only sends ResultMessage. otherwise, we will have to
> > identify
> > // its type using instanceof operator.
> > ResultMessage rm = (ResultMessage) message;
> > if (rm.isOk()) {
> > // server returned OK code.
> > // if received the result message which has the last sequence
> > // number,
> > if (rm.getSequence() == values.length - 1) {
> > // print the sum and disconnect.
> > //System.out.println("The sum: " + rm.getValue());
> >
> > for (int i = 0; i < values.length; i++) {
> > AddMessage m = new AddMessage();
> > m.setSequence(i);
> > m.setValue(values[i]);
> > session.write(m);
> > }
> >
> > try
> > {
> > Thread.sleep(500);
> > } catch (InterruptedException e)
> > {
> > // TODO Auto-generated catch block
> > e.printStackTrace();
> > }
> >
> > //session.close();
> > //finished = true;
> > }
> > } else {
> > // seever returned error code because of overflow, etc.
> > SessionLog.warn(session, "Server error, disconnecting...");
> > session.close();
> > finished = true;
> > }
> > }
>
> Are you sure OOM is thrown on the server side? If so, please provide
> HEAP DUMP generated by 'jmap' command.
>
> Your modification will lead to OOM on the client side, because any
> received 'RESULT' message will not be processed until the current
> messageReceived() event handler method returns. You write many
> messages in messageReceived() and sleep 500 ms. MINA ExecutorFilter
> will queue all received 'RESULT' messages in the internal queue, and
> that will lead to OOM. To avoid this problem, please try to add
> ReadThrottleFilter.
>
> If you didn't add any ExecutorFilter and set the thread model to
> MANUAL, your client can also throw OOM because of too many write
> request items in the internal write queue. But it's less risky than
> when ExecutorFilter is employed.
>
> Trustin
> --
> what we call human nature is actually human habit
> --
> http://gleamynode.net/
> --
> PGP Key ID: 0x0255ECA6
>
Re: Out Of Memory Problem Again
Posted by Trustin Lee <tr...@gmail.com>.
On 8/7/07, mat <fo...@gmail.com> wrote:
> In ClientSessionHandler
>
> public void messageReceived(IoSession session, Object message) {
> // server only sends ResultMessage. otherwise, we will have to
> identify
> // its type using instanceof operator.
> ResultMessage rm = (ResultMessage) message;
> if (rm.isOk()) {
> // server returned OK code.
> // if received the result message which has the last sequence
> // number,
> if (rm.getSequence() == values.length - 1) {
> // print the sum and disconnect.
> //System.out.println("The sum: " + rm.getValue());
>
> for (int i = 0; i < values.length; i++) {
> AddMessage m = new AddMessage();
> m.setSequence(i);
> m.setValue(values[i]);
> session.write(m);
> }
>
> try
> {
> Thread.sleep(500);
> } catch (InterruptedException e)
> {
> // TODO Auto-generated catch block
> e.printStackTrace();
> }
>
> //session.close();
> //finished = true;
> }
> } else {
> // seever returned error code because of overflow, etc.
> SessionLog.warn(session, "Server error, disconnecting...");
> session.close();
> finished = true;
> }
> }
Are you sure OOM is thrown on the server side? If so, please provide
HEAP DUMP generated by 'jmap' command.
Your modification will lead to OOM on the client side, because any
received 'RESULT' message will not be processed until the current
messageReceived() event handler method returns. You write many
messages in messageReceived() and sleep 500 ms. MINA ExecutorFilter
will queue all received 'RESULT' messages in the internal queue, and
that will lead to OOM. To avoid this problem, please try to add
ReadThrottleFilter.
If you didn't add any ExecutorFilter and set the thread model to
MANUAL, your client can also throw OOM because of too many write
request items in the internal write queue. But it's less risky than
when ExecutorFilter is employed.
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
Re: Out Of Memory Problem Again
Posted by mat <fo...@gmail.com>.
In ClientSessionHandler
public void messageReceived(IoSession session, Object message) {
// server only sends ResultMessage. otherwise, we will have to
identify
// its type using instanceof operator.
ResultMessage rm = (ResultMessage) message;
if (rm.isOk()) {
// server returned OK code.
// if received the result message which has the last sequence
// number,
if (rm.getSequence() == values.length - 1) {
// print the sum and disconnect.
//System.out.println("The sum: " + rm.getValue());
for (int i = 0; i < values.length; i++) {
AddMessage m = new AddMessage();
m.setSequence(i);
m.setValue(values[i]);
session.write(m);
}
try
{
Thread.sleep(500);
} catch (InterruptedException e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
//session.close();
//finished = true;
}
} else {
// seever returned error code because of overflow, etc.
SessionLog.warn(session, "Server error, disconnecting...");
session.close();
finished = true;
}
}
On 8/7/07, Trustin Lee <tr...@gmail.com> wrote:
>
> On 8/7/07, mat <fo...@gmail.com> wrote:
> > It normally takes 1 or 2 days to get OOM on my server. However, memory
> leak
> > does happen as you can see. Since I can't wait OOM until it happens, I
> just
> > simply modify the Sumupserver example to prove the memory leak may
> caused by
> > mina core. (I assume sumupserver has no memory leak itself)
>
> What modification did you make? Please provide us full patch so we
> can reproduce the problem by ourselves.
>
> Trustin
> --
> what we call human nature is actually human habit
> --
> http://gleamynode.net/
> --
> PGP Key ID: 0x0255ECA6
>
Re: Out Of Memory Problem Again
Posted by Trustin Lee <tr...@gmail.com>.
On 8/7/07, mat <fo...@gmail.com> wrote:
> It normally takes 1 or 2 days to get OOM on my server. However, memory leak
> does happen as you can see. Since I can't wait OOM until it happens, I just
> simply modify the Sumupserver example to prove the memory leak may caused by
> mina core. (I assume sumupserver has no memory leak itself)
What modification did you make? Please provide us full patch so we
can reproduce the problem by ourselves.
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
Re: Out Of Memory Problem Again
Posted by mat <fo...@gmail.com>.
It normally takes 1 or 2 days to get OOM on my server. However, memory leak
does happen as you can see. Since I can't wait OOM until it happens, I just
simply modify the Sumupserver example to prove the memory leak may caused by
mina core. (I assume sumupserver has no memory leak itself)
On 8/7/07, peter royal <pr...@apache.org> wrote:
>
> On Aug 5, 2007, at 4:56 PM, mat wrote:
> > The following are the GC dump: Can you see someting here?
>
> Did you set the JVM parameter to have it generate a heap snapshot
> upon OOM?
>
> Read http://blogs.sun.com/alanb/entry/
> heapdumponoutofmemoryerror_option_in_5_0u7
>
> Use -XX:+HeapDumpOnOutOfMemoryError, and the resulting snapshot can
> be opened by hprof as well as YourKit's profiler.
>
> -pete
>
>
> --
> proyal@apache.org - http://fotap.org/~osi
>
>
>
>
>
Re: Out Of Memory Problem Again
Posted by peter royal <pr...@apache.org>.
On Aug 5, 2007, at 4:56 PM, mat wrote:
> The following are the GC dump: Can you see someting here?
Did you set the JVM parameter to have it generate a heap snapshot
upon OOM?
Read http://blogs.sun.com/alanb/entry/
heapdumponoutofmemoryerror_option_in_5_0u7
Use -XX:+HeapDumpOnOutOfMemoryError, and the resulting snapshot can
be opened by hprof as well as YourKit's profiler.
-pete
--
proyal@apache.org - http://fotap.org/~osi
Re: Out Of Memory Problem Again
Posted by mat <fo...@gmail.com>.
[GC 512K->175K(1984K), 0.0029448 secs]
[GC 687K->203K(1984K), 0.0016242 secs]
[GC 715K->215K(1984K), 0.0012507 secs]
[GC 727K->253K(1984K), 0.0019461 secs]
[GC 765K->292K(1984K), 0.0014474 secs]
[GC 804K->329K(1984K), 0.0011993 secs]
[GC 841K->342K(1984K), 0.0008107 secs]
I think the above is also a typical GC with memory leak, right? Actually I
got this by modifing the Sumupserver example. After client receives the
result, it keeps sending new requests every 2 seconds. I believe after a
LONG run, it will eventually reach OOM. Does this indicate mina have a
memory leak?
On 8/6/07, Trustin Lee <tr...@gmail.com> wrote:
>
> On 8/6/07, mat <fo...@gmail.com> wrote:
> > The following are the GC dump: Can you see someting here?
>
> It's a typical GC log of a program with memory leak. Heap dump is
> required to analyze the problem.
>
> Trustin
> --
> what we call human nature is actually human habit
> --
> http://gleamynode.net/
> --
> PGP Key ID: 0x0255ECA6
>
Re: Out Of Memory Problem Again
Posted by mat <fo...@gmail.com>.
Heap dump is able to manitest the leak exists in which class?
On 8/6/07, Trustin Lee <tr...@gmail.com> wrote:
>
> On 8/6/07, mat <fo...@gmail.com> wrote:
> > The following are the GC dump: Can you see someting here?
>
> It's a typical GC log of a program with memory leak. Heap dump is
> required to analyze the problem.
>
> Trustin
> --
> what we call human nature is actually human habit
> --
> http://gleamynode.net/
> --
> PGP Key ID: 0x0255ECA6
>
Re: Out Of Memory Problem Again
Posted by Trustin Lee <tr...@gmail.com>.
On 8/6/07, mat <fo...@gmail.com> wrote:
> The following are the GC dump: Can you see someting here?
It's a typical GC log of a program with memory leak. Heap dump is
required to analyze the problem.
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
Re: Out Of Memory Problem Again
Posted by mat <fo...@gmail.com>.
The following are the GC dump: Can you see someting here?
[GC 3358K->3017K(3840K), 0.0021926 secs]
[GC 3529K->3185K(3840K), 0.0023076 secs]
[GC 3697K->3327K(3840K), 0.0025027 secs]
[Full GC 3327K->2337K(3840K), 0.0590504 secs]
[GC 2849K->2505K(4476K), 0.0018559 secs]
...
[Full GC 4240K->3566K(4860K), 0.0845137 secs]
[GC 4078K->3736K(6524K), 0.0019117 secs]
...
[Full GC 6011K->5357K(6524K), 0.0915253 secs]
[GC 6061K->5596K(9700K), 0.0024451 secs]
...
[Full GC 9197K->8298K(9956K), 0.1164336 secs]
[GC 9386K->8650K(14984K), 0.0087182 secs]
...
[Full GC 13998K->13710K(15112K), 0.1377688 secs]
[GC 15310K->14237K(24644K), 0.0051665 secs]
...
[Full GC 23365K->22634K(25028K), 0.2012658 secs]
[GC 25194K->23542K(40608K), 0.0074439 secs]
...
[Full GC 38689K->37952K(41376K), 0.3009403 secs]
[GC 42048K->39387K(65088K), 0.0110980 secs]
...
[Full GC 60556K->53999K(65088K), 0.4842984 secs]
[GC 58095K->55418K(65088K), 0.0116043 secs]
[GC 59514K->56852K(65088K), 0.0154620 secs]
[Full GC 60948K->58168K(65088K), 0.4342759 secs]
[Full GC 62264K->59506K(65088K), 0.4431218 secs]
[Full GC 63602K->60841K(65088K), 0.4561872 secs]
[Full GC 64640K->61673K(65088K), 0.6459586 secs]
[Full GC 65087K->62667K(65088K), 0.4649776 secs]
[Full GC 65087K->63446K(65088K), 0.5155173 secs]
[Full GC 65087K->63631K(65088K), 0.4685915 secs]
[Full GC 65087K->63584K(65088K), 0.5764973 secs]
[Full GC 65087K->63754K(65088K), 0.4789820 secs]
[Full GC 65087K->63909K(65088K), 0.4758840 secs]
[Full GC 65087K->64067K(65088K), 0.4728703 secs]
[Full GC 65087K->64237K(65088K), 0.4927398 secs]
[Full GC 65087K->64403K(65088K), 0.4745337 secs]
[Full GC 65087K->64570K(65088K), 0.5550482 secs]
[Full GC 65087K->64737K(65088K), 0.4732689 secs]
[Full GC 64737K->64737K(65088K), 0.4797642 secs]
[Full GC 65087K->64855K(65088K), 0.4759919 secs]
[Full GC 64855K->64855K(65088K), 0.4814910 secs]
[Full GC 65088K->64859K(65088K), 0.4749250 secs]
[Full GC 65088K->64856K(65088K), 0.5837997 secs]
[Full GC 64856K->64856K(65088K), 0.4743265 secs]
[Full GC 65087K->64982K(65088K), 0.4723679 secs]
[Full GC 64982K->64982K(65088K), 0.4733373 secs]
[Full GC 65087K->65044K(65088K), 0.4893652 secs]
[Full GC 65044K->65044K(65088K), 0.4683774 secs]
[Full GC 65084K->65045K(65088K), 0.4796948 secs]
[Full GC 65045K->65044K(65088K), 0.4726712 secs]
Exception in thread "Thread-4" java.lang.OutOfMemoryError: Java heap space
[Full GC 65087K->65042K(65088K), 0.5791150 secs]
[Full GC 65042K->65042K(65088K), 0.4764032 secs]
[Full GC 65087K->65042K(65088K), 0.4735241 secs]
[Full GC 65042K->65042K(65088K), 0.4718914 secs]
[Full GC 65087K->65042K(65088K), 0.4771627 secs]
[Full GC 65042K->65042K(65088K), 0.4818546 secs]
[Full GC 65087K->65042K(65088K), 0.4767027 secs]
[Full GC 65042K->65042K(65088K), 0.4658126 secs]
[Full GC 65087K->65042K(65088K), 0.4728113 secs]
[Full GC 65042K->65042K(65088K), 0.4803292 secs]
[Full GC 65087K->65043K(65088K), 0.4820415 secs]
[Full GC 65043K->65043K(65088K), 0.4711774 secs]
[Full GC 65087K->65043K(65088K), 0.4838518 secs]
[Full GC 65043K->65043K(65088K), 0.4732720 secs]
[Full GC 65087K->65043K(65088K), 0.4888645 secs]
[Full GC 65043K->65019K(65088K), 0.5766317 secs]
Exception in thread "AnonymousIoService-9" java.lang.OutOfMemoryError: Java
heap
space
On 7/22/07, peter royal <pr...@apache.org> wrote:
>
> On Jul 21, 2007, at 10:41 AM, Mark Webb wrote:
> > This is a difficult problem and I wish I had a good suggestion.
> > The best I
> > can give you is to profile your application and pass along
> > information you
> > find so that we can help you.
>
> I second this.
>
> Enable http://blogs.sun.com/alanb/entry/
> heapdumponoutofmemoryerror_option_in_5_0u7
>
> And look at the dump to see what is taking up your memory.
>
> -pete
>
>
> --
> proyal@apache.org - http://fotap.org/~osi
>
>
>
>
>
Re: Out Of Memory Problem Again
Posted by peter royal <pr...@apache.org>.
On Jul 21, 2007, at 10:41 AM, Mark Webb wrote:
> This is a difficult problem and I wish I had a good suggestion.
> The best I
> can give you is to profile your application and pass along
> information you
> find so that we can help you.
I second this.
Enable http://blogs.sun.com/alanb/entry/
heapdumponoutofmemoryerror_option_in_5_0u7
And look at the dump to see what is taking up your memory.
-pete
--
proyal@apache.org - http://fotap.org/~osi
Re: Out Of Memory Problem Again
Posted by Mark Webb <el...@gmail.com>.
This is a difficult problem and I wish I had a good suggestion. The best I
can give you is to profile your application and pass along information you
find so that we can help you.
On 7/21/07, Raman Gupta <ro...@fastmail.fm> wrote:
>
> Trustin Lee wrote:
> > On 7/21/07, mat <fo...@gmail.com> wrote:
> >> Hi, Trustin
> >>
> >> I read this "The problem was the SocketChannel.write
> >> implementation. Indeed, even if you don't use DirectByteBuffer in
> >> your code, SocketChannel.writedoes it for you. This means that
> >> every time you call the write method, a new DirectByteBuffer is
> >> created. And since there is no real and effective GC on this type
> >> of objects, on heavy load, your application simply crash."
> >
> > Yes, it's true, but such a situation doesn't happen easily AFAIK.
> >
> >> However, my case seems not under heaving loading. I guess this
> >> problem should happen to most of peoples using Mina. No??
> >
> > At least not for me. Other people could shed some light here.
>
> I hesitate to post this because I don't have any factual data here.
> However, I have been having memory problems in my production
> application recently, and everything seems to have started right
> around the time the Mina code started to be used.
>
> Unfortunately, I have NOT been able to reproduce any memory leak
> whatsoever in a test environment, with or without MINA. The production
> problem only manifests over the course of about 5 days, and it seems
> to be associated with load. When the server is under load, memory is
> used up (and never collected), and when the server is not under load
> (even though transactions are still occurring) memory use stabilizes
> (with the normal gc peaks/valleys).
>
> My next step is to try and get a profiler on our running production
> app, but due to the performance overhead and the downtime while taking
> snapshots, this may very well be impossible. I know this may not even
> be MINA related but any suggestions are welcome :-)
>
> Cheers,
> Raman
>
>
--
..Cheers
Mark
Re: Out Of Memory Problem Again
Posted by Raman Gupta <ro...@fastmail.fm>.
Trustin Lee wrote:
> On 7/21/07, mat <fo...@gmail.com> wrote:
>> Hi, Trustin
>>
>> I read this "The problem was the SocketChannel.write
>> implementation. Indeed, even if you don't use DirectByteBuffer in
>> your code, SocketChannel.writedoes it for you. This means that
>> every time you call the write method, a new DirectByteBuffer is
>> created. And since there is no real and effective GC on this type
>> of objects, on heavy load, your application simply crash."
>
> Yes, it's true, but such a situation doesn't happen easily AFAIK.
>
>> However, my case seems not under heaving loading. I guess this
>> problem should happen to most of peoples using Mina. No??
>
> At least not for me. Other people could shed some light here.
I hesitate to post this because I don't have any factual data here.
However, I have been having memory problems in my production
application recently, and everything seems to have started right
around the time the Mina code started to be used.
Unfortunately, I have NOT been able to reproduce any memory leak
whatsoever in a test environment, with or without MINA. The production
problem only manifests over the course of about 5 days, and it seems
to be associated with load. When the server is under load, memory is
used up (and never collected), and when the server is not under load
(even though transactions are still occurring) memory use stabilizes
(with the normal gc peaks/valleys).
My next step is to try and get a profiler on our running production
app, but due to the performance overhead and the downtime while taking
snapshots, this may very well be impossible. I know this may not even
be MINA related but any suggestions are welcome :-)
Cheers,
Raman
Re: Out Of Memory Problem Again
Posted by Mark Webb <el...@gmail.com>.
I have not experienced this OOM problem, but if you could provide a running
sample program that reproduces the problem, we would be able to help you out
better.
Thanks.
On 7/21/07, mat <fo...@gmail.com> wrote:
>
> On 7/21/07, Trustin Lee <tr...@gmail.com> wrote:
> >
> > On 7/21/07, mat <fo...@gmail.com> wrote:
> > > Hi, Trustin
> > >
> > > I read this "The problem was the SocketChannel.write implementation.
> > Indeed,
> > > even if you don't use DirectByteBuffer in your code,
> > > SocketChannel.writedoes it for you. This means that every time you
> > > call the write method, a new
> > > DirectByteBuffer is created. And since there is no real and effective
> GC
> > on
> > > this type of objects, on heavy load, your application simply crash."
> >
> > Yes, it's true, but such a situation doesn't happen easily AFAIK.
>
>
> Thanks Trustin.
>
> Sure. It normally happens after 10 hours my server starts. However, my
> server needs to be running 24 hours * 5 days. I really don't know how to
> solve this problem.
>
> 1) BTW, even I configure MaxDirectMemorySize, won't help with this issue,
> right?
> 2)Since the loading seems not heavy, WriteThrottleFilter won't help
> either?
> 3)Any other possible suggestions for my case?
>
> > However, my case seems not under heaving loading.
> > > I guess this problem should happen to most of peoples using Mina. No??
> >
> > At least not for me. Other people could shed some light here.
> >
> > Trustin
> > --
> > what we call human nature is actually human habit
> > --
> > http://gleamynode.net/
> > --
> > PGP Key ID: 0x0255ECA6
> >
>
--
..Cheers
Mark
Re: Out Of Memory Problem Again
Posted by mat <fo...@gmail.com>.
On 7/21/07, Trustin Lee <tr...@gmail.com> wrote:
>
> On 7/21/07, mat <fo...@gmail.com> wrote:
> > Hi, Trustin
> >
> > I read this "The problem was the SocketChannel.write implementation.
> Indeed,
> > even if you don't use DirectByteBuffer in your code,
> > SocketChannel.writedoes it for you. This means that every time you
> > call the write method, a new
> > DirectByteBuffer is created. And since there is no real and effective GC
> on
> > this type of objects, on heavy load, your application simply crash."
>
> Yes, it's true, but such a situation doesn't happen easily AFAIK.
Thanks Trustin.
Sure. It normally happens after 10 hours my server starts. However, my
server needs to be running 24 hours * 5 days. I really don't know how to
solve this problem.
1) BTW, even I configure MaxDirectMemorySize, won't help with this issue,
right?
2)Since the loading seems not heavy, WriteThrottleFilter won't help either?
3)Any other possible suggestions for my case?
> However, my case seems not under heaving loading.
> > I guess this problem should happen to most of peoples using Mina. No??
>
> At least not for me. Other people could shed some light here.
>
> Trustin
> --
> what we call human nature is actually human habit
> --
> http://gleamynode.net/
> --
> PGP Key ID: 0x0255ECA6
>
Re: Out Of Memory Problem Again
Posted by Trustin Lee <tr...@gmail.com>.
On 7/21/07, mat <fo...@gmail.com> wrote:
> Hi, Trustin
>
> I read this "The problem was the SocketChannel.write implementation. Indeed,
> even if you don't use DirectByteBuffer in your code,
> SocketChannel.writedoes it for you. This means that every time you
> call the write method, a new
> DirectByteBuffer is created. And since there is no real and effective GC on
> this type of objects, on heavy load, your application simply crash."
Yes, it's true, but such a situation doesn't happen easily AFAIK.
> However, my case seems not under heaving loading.
> I guess this problem should happen to most of peoples using Mina. No??
At least not for me. Other people could shed some light here.
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
Re: Out Of Memory Problem Again
Posted by mat <fo...@gmail.com>.
Hi, Trustin
I read this "The problem was the SocketChannel.write implementation. Indeed,
even if you don't use DirectByteBuffer in your code,
SocketChannel.writedoes it for you. This means that every time you
call the write method, a new
DirectByteBuffer is created. And since there is no real and effective GC on
this type of objects, on heavy load, your application simply crash."
However, my case seems not under heaving loading.
I guess this problem should happen to most of peoples using Mina. No??
On 7/21/07, Trustin Lee <tr...@gmail.com> wrote:
>
> On 7/21/07, mat <fo...@gmail.com> wrote:
> > Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_08-b03)
> > Java HotSpot(TM) Client VM (build 1.5.0_08-b03, mixed mode, sharing)
> > Windows 2003
> >
> > So you think it is bug of JVM?
>
> I'd suggest you to use heap buffers instead of direct buffers:
>
>
> http://mina.apache.org/faq.html#FAQ-Iget%7B%7BOutOfMemoryError%7D%7Dorresponsetimeoutandconnectionresetunderheavyload
> .
>
> HTH,
> Trustin
> --
> what we call human nature is actually human habit
> --
> http://gleamynode.net/
> --
> PGP Key ID: 0x0255ECA6
>
Re: Out Of Memory Problem Again
Posted by Trustin Lee <tr...@gmail.com>.
On 7/21/07, mat <fo...@gmail.com> wrote:
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_08-b03)
> Java HotSpot(TM) Client VM (build 1.5.0_08-b03, mixed mode, sharing)
> Windows 2003
>
> So you think it is bug of JVM?
I'd suggest you to use heap buffers instead of direct buffers:
http://mina.apache.org/faq.html#FAQ-Iget%7B%7BOutOfMemoryError%7D%7Dorresponsetimeoutandconnectionresetunderheavyload.
HTH,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
Re: Out Of Memory Problem Again
Posted by mat <fo...@gmail.com>.
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_08-b03)
Java HotSpot(TM) Client VM (build 1.5.0_08-b03, mixed mode, sharing)
Windows 2003
So you think it is bug of JVM?
On 7/21/07, 向秦贤 <fy...@gmail.com> wrote:
>
> Not sure your jdk version.
> http://issues.apache.org/jira/browse/DIRMINA-320 not running in jdk6.
> Try run in jdk6 to check it out whether happen again.
> There a sure thing, some direct buffer reference hold by application. vm
> cannot clean it.
>
> 2007/7/21, mat <fo...@gmail.com>:
> >
> > http://issues.apache.org/jira/browse/DIRMINA-320
> >
> > Please check this out. It seems that i am not the only one who faces
> this
> > problem and it happens in mina core. I quote something written by
> Trustin.
> > "Other session might have allocated huge memory block and other session
> > might be being affected by its side-effect. " However, I only had one
> > client
> > connecting to my server when OOM occurs.
> >
> > On 7/21/07, 向秦贤 <fy...@gmail.com> wrote:
> > >
> > > Maybe Direct buffer not released.
> > > Direct buffer must explicit release.
> > > so somewhere may check if direct buffer and set to null.
> > >
> > > 2007/7/21, mat <fo...@gmail.com>:
> > > >
> > > > I captured the exception message.
> > > >
> > > > org.apache.mina.common.support.DefaultExceptionMonitorexceptionCaught
> > > > Unexpected exception.
> > > > java.lang.OutOfMemoryError: Direct buffer memory
> > > > at java.nio.Bits.reserveMemory
> > > > at java.nio.DirectByteBuffer.<init>
> > > > at java.nio.ByteBuffer.allocateDirect
> > > > at sun.nio.ch.Util.getTemporaryDirectBuffer
> > > > at sun.nio.ch.IOUtil.write
> > > > at sun.nio.ch.SocketChannelImpl.write
> > > > at org.apache.mina.transport.socket.nio.SocketIOProcessor.doFlush<
> > > > SocketIoProcessor.java:428>
> > > > at org.apache.mina.transport.socket.nio.SocketIOProcessor.doFlush<
> > > > SocketIoProcessor.java:366>
> > > > at org.apache.mina.transport.socket.nio.SocketIOProcessor.access$600
> <
> > > > SocketIoProcessor.java:44>
> > > > at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run
> <
> > > > SocketIoProcessor.java:509>
> > > > at org.apache.mina.util.NamePreservingRunnable.run<
> > > > NamePreservingRunnable.java:43>
> > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> > > > at java.util.concurrect.ThreadPoolExecutor$Worker.run
> > > > at java.lang.Thread.run
> > > >
> > > >
> > > > On 7/20/07, Luis Neves <lu...@co.sapo.pt> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > mat wrote:
> > > > > > My server sometimes faced "OOM" problem. (I couldn't profile it
> > > since
> > > > > TPTP
> > > > > > crashed my server before OOM occured). I didn't see major memory
> > > leak
> > > > > when
> > > > > > profiling. Therefore, I believe OOM happens when READ or WRITE
> > > > operation
> > > > > > can't handle the incoming messages or outgoing messages.
> (However
> > my
> > > > > > incoming messages normally 20 * 512bytes/sec, NOT too fast,
> > right?).
> > > > > Last
> > > > > > time i saw my server memory usage is over 600MB in windows XP.
> > > > >
> > > > > Your code is broken ... the question is where. Mina can handle
> that
> > > > amount
> > > > > of
> > > > > messages without breaking a sweat.
> > > > > Do you have some kind of heavy processing in the receiving end
> that
> > > > delays
> > > > > the
> > > > > acceptance of messages?
> > > > >
> > > > > Did you try to use the ReadThrottleFilter?
> > > > > How are you doing you writes?
> > > > > A simple "iosession.write()" ?
> > > > > Did you try something like:
> > > > > WriteFuture wf = iosession.write();
> > > > > wf.join();
> > > > >
> > > > > Can we see the code of your Encoder/Decoder?
> > > > >
> > > > >
> > > > > --
> > > > > Luis Neves
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > 向秦贤
> > >
> >
>
>
>
> --
> 向秦贤
>
Re: Out Of Memory Problem Again
Posted by 向秦贤 <fy...@gmail.com>.
Not sure your jdk version.
http://issues.apache.org/jira/browse/DIRMINA-320 not running in jdk6.
Try run in jdk6 to check it out whether happen again.
There a sure thing, some direct buffer reference hold by application. vm
cannot clean it.
2007/7/21, mat <fo...@gmail.com>:
>
> http://issues.apache.org/jira/browse/DIRMINA-320
>
> Please check this out. It seems that i am not the only one who faces this
> problem and it happens in mina core. I quote something written by Trustin.
> "Other session might have allocated huge memory block and other session
> might be being affected by its side-effect. " However, I only had one
> client
> connecting to my server when OOM occurs.
>
> On 7/21/07, 向秦贤 <fy...@gmail.com> wrote:
> >
> > Maybe Direct buffer not released.
> > Direct buffer must explicit release.
> > so somewhere may check if direct buffer and set to null.
> >
> > 2007/7/21, mat <fo...@gmail.com>:
> > >
> > > I captured the exception message.
> > >
> > > org.apache.mina.common.support.DefaultExceptionMonitor exceptionCaught
> > > Unexpected exception.
> > > java.lang.OutOfMemoryError: Direct buffer memory
> > > at java.nio.Bits.reserveMemory
> > > at java.nio.DirectByteBuffer.<init>
> > > at java.nio.ByteBuffer.allocateDirect
> > > at sun.nio.ch.Util.getTemporaryDirectBuffer
> > > at sun.nio.ch.IOUtil.write
> > > at sun.nio.ch.SocketChannelImpl.write
> > > at org.apache.mina.transport.socket.nio.SocketIOProcessor.doFlush<
> > > SocketIoProcessor.java:428>
> > > at org.apache.mina.transport.socket.nio.SocketIOProcessor.doFlush<
> > > SocketIoProcessor.java:366>
> > > at org.apache.mina.transport.socket.nio.SocketIOProcessor.access$600<
> > > SocketIoProcessor.java:44>
> > > at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run<
> > > SocketIoProcessor.java:509>
> > > at org.apache.mina.util.NamePreservingRunnable.run<
> > > NamePreservingRunnable.java:43>
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> > > at java.util.concurrect.ThreadPoolExecutor$Worker.run
> > > at java.lang.Thread.run
> > >
> > >
> > > On 7/20/07, Luis Neves <lu...@co.sapo.pt> wrote:
> > > >
> > > > Hi,
> > > >
> > > > mat wrote:
> > > > > My server sometimes faced "OOM" problem. (I couldn't profile it
> > since
> > > > TPTP
> > > > > crashed my server before OOM occured). I didn't see major memory
> > leak
> > > > when
> > > > > profiling. Therefore, I believe OOM happens when READ or WRITE
> > > operation
> > > > > can't handle the incoming messages or outgoing messages. (However
> my
> > > > > incoming messages normally 20 * 512bytes/sec, NOT too fast,
> right?).
> > > > Last
> > > > > time i saw my server memory usage is over 600MB in windows XP.
> > > >
> > > > Your code is broken ... the question is where. Mina can handle that
> > > amount
> > > > of
> > > > messages without breaking a sweat.
> > > > Do you have some kind of heavy processing in the receiving end that
> > > delays
> > > > the
> > > > acceptance of messages?
> > > >
> > > > Did you try to use the ReadThrottleFilter?
> > > > How are you doing you writes?
> > > > A simple "iosession.write()" ?
> > > > Did you try something like:
> > > > WriteFuture wf = iosession.write();
> > > > wf.join();
> > > >
> > > > Can we see the code of your Encoder/Decoder?
> > > >
> > > >
> > > > --
> > > > Luis Neves
> > > >
> > >
> >
> >
> >
> > --
> > 向秦贤
> >
>
--
向秦贤
Re: Out Of Memory Problem Again
Posted by mat <fo...@gmail.com>.
http://issues.apache.org/jira/browse/DIRMINA-320
Please check this out. It seems that i am not the only one who faces this
problem and it happens in mina core. I quote something written by Trustin.
"Other session might have allocated huge memory block and other session
might be being affected by its side-effect. " However, I only had one client
connecting to my server when OOM occurs.
On 7/21/07, 向秦贤 <fy...@gmail.com> wrote:
>
> Maybe Direct buffer not released.
> Direct buffer must explicit release.
> so somewhere may check if direct buffer and set to null.
>
> 2007/7/21, mat <fo...@gmail.com>:
> >
> > I captured the exception message.
> >
> > org.apache.mina.common.support.DefaultExceptionMonitor exceptionCaught
> > Unexpected exception.
> > java.lang.OutOfMemoryError: Direct buffer memory
> > at java.nio.Bits.reserveMemory
> > at java.nio.DirectByteBuffer.<init>
> > at java.nio.ByteBuffer.allocateDirect
> > at sun.nio.ch.Util.getTemporaryDirectBuffer
> > at sun.nio.ch.IOUtil.write
> > at sun.nio.ch.SocketChannelImpl.write
> > at org.apache.mina.transport.socket.nio.SocketIOProcessor.doFlush<
> > SocketIoProcessor.java:428>
> > at org.apache.mina.transport.socket.nio.SocketIOProcessor.doFlush<
> > SocketIoProcessor.java:366>
> > at org.apache.mina.transport.socket.nio.SocketIOProcessor.access$600<
> > SocketIoProcessor.java:44>
> > at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run<
> > SocketIoProcessor.java:509>
> > at org.apache.mina.util.NamePreservingRunnable.run<
> > NamePreservingRunnable.java:43>
> > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> > at java.util.concurrect.ThreadPoolExecutor$Worker.run
> > at java.lang.Thread.run
> >
> >
> > On 7/20/07, Luis Neves <lu...@co.sapo.pt> wrote:
> > >
> > > Hi,
> > >
> > > mat wrote:
> > > > My server sometimes faced "OOM" problem. (I couldn't profile it
> since
> > > TPTP
> > > > crashed my server before OOM occured). I didn't see major memory
> leak
> > > when
> > > > profiling. Therefore, I believe OOM happens when READ or WRITE
> > operation
> > > > can't handle the incoming messages or outgoing messages. (However my
> > > > incoming messages normally 20 * 512bytes/sec, NOT too fast, right?).
> > > Last
> > > > time i saw my server memory usage is over 600MB in windows XP.
> > >
> > > Your code is broken ... the question is where. Mina can handle that
> > amount
> > > of
> > > messages without breaking a sweat.
> > > Do you have some kind of heavy processing in the receiving end that
> > delays
> > > the
> > > acceptance of messages?
> > >
> > > Did you try to use the ReadThrottleFilter?
> > > How are you doing you writes?
> > > A simple "iosession.write()" ?
> > > Did you try something like:
> > > WriteFuture wf = iosession.write();
> > > wf.join();
> > >
> > > Can we see the code of your Encoder/Decoder?
> > >
> > >
> > > --
> > > Luis Neves
> > >
> >
>
>
>
> --
> 向秦贤
>
Re: Out Of Memory Problem Again
Posted by 向秦贤 <fy...@gmail.com>.
Maybe Direct buffer not released.
Direct buffer must explicit release.
so somewhere may check if direct buffer and set to null.
2007/7/21, mat <fo...@gmail.com>:
>
> I captured the exception message.
>
> org.apache.mina.common.support.DefaultExceptionMonitor exceptionCaught
> Unexpected exception.
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory
> at java.nio.DirectByteBuffer.<init>
> at java.nio.ByteBuffer.allocateDirect
> at sun.nio.ch.Util.getTemporaryDirectBuffer
> at sun.nio.ch.IOUtil.write
> at sun.nio.ch.SocketChannelImpl.write
> at org.apache.mina.transport.socket.nio.SocketIOProcessor.doFlush<
> SocketIoProcessor.java:428>
> at org.apache.mina.transport.socket.nio.SocketIOProcessor.doFlush<
> SocketIoProcessor.java:366>
> at org.apache.mina.transport.socket.nio.SocketIOProcessor.access$600<
> SocketIoProcessor.java:44>
> at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run<
> SocketIoProcessor.java:509>
> at org.apache.mina.util.NamePreservingRunnable.run<
> NamePreservingRunnable.java:43>
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> at java.util.concurrect.ThreadPoolExecutor$Worker.run
> at java.lang.Thread.run
>
>
> On 7/20/07, Luis Neves <lu...@co.sapo.pt> wrote:
> >
> > Hi,
> >
> > mat wrote:
> > > My server sometimes faced "OOM" problem. (I couldn't profile it since
> > TPTP
> > > crashed my server before OOM occured). I didn't see major memory leak
> > when
> > > profiling. Therefore, I believe OOM happens when READ or WRITE
> operation
> > > can't handle the incoming messages or outgoing messages. (However my
> > > incoming messages normally 20 * 512bytes/sec, NOT too fast, right?).
> > Last
> > > time i saw my server memory usage is over 600MB in windows XP.
> >
> > Your code is broken ... the question is where. Mina can handle that
> amount
> > of
> > messages without breaking a sweat.
> > Do you have some kind of heavy processing in the receiving end that
> delays
> > the
> > acceptance of messages?
> >
> > Did you try to use the ReadThrottleFilter?
> > How are you doing you writes?
> > A simple "iosession.write()" ?
> > Did you try something like:
> > WriteFuture wf = iosession.write();
> > wf.join();
> >
> > Can we see the code of your Encoder/Decoder?
> >
> >
> > --
> > Luis Neves
> >
>
--
向秦贤
Re: Out Of Memory Problem Again
Posted by mat <fo...@gmail.com>.
I captured the exception message.
org.apache.mina.common.support.DefaultExceptionMonitor exceptionCaught
Unexpected exception.
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory
at java.nio.DirectByteBuffer.<init>
at java.nio.ByteBuffer.allocateDirect
at sun.nio.ch.Util.getTemporaryDirectBuffer
at sun.nio.ch.IOUtil.write
at sun.nio.ch.SocketChannelImpl.write
at org.apache.mina.transport.socket.nio.SocketIOProcessor.doFlush<
SocketIoProcessor.java:428>
at org.apache.mina.transport.socket.nio.SocketIOProcessor.doFlush<
SocketIoProcessor.java:366>
at org.apache.mina.transport.socket.nio.SocketIOProcessor.access$600<
SocketIoProcessor.java:44>
at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run<
SocketIoProcessor.java:509>
at org.apache.mina.util.NamePreservingRunnable.run<
NamePreservingRunnable.java:43>
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
at java.util.concurrect.ThreadPoolExecutor$Worker.run
at java.lang.Thread.run
On 7/20/07, Luis Neves <lu...@co.sapo.pt> wrote:
>
> Hi,
>
> mat wrote:
> > My server sometimes faced "OOM" problem. (I couldn't profile it since
> TPTP
> > crashed my server before OOM occured). I didn't see major memory leak
> when
> > profiling. Therefore, I believe OOM happens when READ or WRITE operation
> > can't handle the incoming messages or outgoing messages. (However my
> > incoming messages normally 20 * 512bytes/sec, NOT too fast, right?).
> Last
> > time i saw my server memory usage is over 600MB in windows XP.
>
> Your code is broken ... the question is where. Mina can handle that amount
> of
> messages without breaking a sweat.
> Do you have some kind of heavy processing in the receiving end that delays
> the
> acceptance of messages?
>
> Did you try to use the ReadThrottleFilter?
> How are you doing you writes?
> A simple "iosession.write()" ?
> Did you try something like:
> WriteFuture wf = iosession.write();
> wf.join();
>
> Can we see the code of your Encoder/Decoder?
>
>
> --
> Luis Neves
>
Re: Out Of Memory Problem Again
Posted by mat <fo...@gmail.com>.
I wonder whether Trustin or Peter can give more comments on this one? Thanks
a lot.
On 7/23/07, Paddy O'Neill <ct...@mobilecohesion.com> wrote:
>
> Hi
>
> This sounds similar to a problem that we encountered where mina was
> serving multiple sessions and forwarding to a single connection, similar
> to your environment. We found that the thread management in mina on the
> single connection side would not work efficiently and end up only
> allocating jobs to a single thread. This effectively caused the jobs to
> be executed sequentially, which in turn caused the backlog and memory
> buildup. I suspect that the issue is related to there only being a
> single connection on the IoConnector as we found that the jobs were
> allocated correctly across threads on the IoAcceptor side.
>
> We worked around it by removing the thread pool from the mina filter
> chain and implementing our own thread management just outside the mina
> interface.
>
> Paddy
>
> mat wrote:
> > Thanks a lot.
> >
> > My server has one client (using mina too) connecting to a data
> > source(20 -
> > 30 /sec incoming messages) and the other side is server listening on one
> > port to broadcasting messages to end users(I wish Mina could handle
> 1000+
> > concurrent users). No other heaving loading process.
> >
> > If the incoming messages is not fast, I don't think i need a
> > ReadThrottleFilter, right?
> >
> >
> > The following are the part of write method:(I am sure not the write
> > problem
> > since normally no user connected OOM also happened, however you can
> point
> > out why Future.join helps here. ^^)
> >
> > synchronized(sessions)
> > {
> > ByteBuffer buf = ByteBuffer.allocate(512);
> > buf.setAutoExpand(true);
> >
> > Object message = serviceMessage.getMessage();
> >
> > if (!(message instanceof byte[]))
> > return;
> >
> > buf.put((byte[])message);
> > buf.flip();
> >
> > Iterator iter = sessions.values().iterator();
> >
> > while(iter.hasNext())
> > {
> > IoSession session = (IoSession) iter.next();
> > session.write(buf.duplicate());
> > }
> > }
> >
> >
> > On 7/20/07, Luis Neves <lu...@co.sapo.pt> wrote:
> >>
> >> Hi,
> >>
> >> mat wrote:
> >> > My server sometimes faced "OOM" problem. (I couldn't profile it since
> >> TPTP
> >> > crashed my server before OOM occured). I didn't see major memory leak
> >> when
> >> > profiling. Therefore, I believe OOM happens when READ or WRITE
> >> operation
> >> > can't handle the incoming messages or outgoing messages. (However my
> >> > incoming messages normally 20 * 512bytes/sec, NOT too fast, right?).
> >> Last
> >> > time i saw my server memory usage is over 600MB in windows XP.
> >>
> >> Your code is broken ... the question is where. Mina can handle that
> >> amount
> >> of
> >> messages without breaking a sweat.
> >> Do you have some kind of heavy processing in the receiving end that
> >> delays
> >> the
> >> acceptance of messages?
> >>
> >> Did you try to use the ReadThrottleFilter?
> >> How are you doing you writes?
> >> A simple "iosession.write()" ?
> >> Did you try something like:
> >> WriteFuture wf = iosession.write();
> >> wf.join();
> >>
> >> Can we see the code of your Encoder/Decoder?
> >>
> >>
> >> --
> >> Luis Neves
> >>
> >
>
>
>
Re: Out Of Memory Problem Again
Posted by Paddy O'Neill <ct...@mobilecohesion.com>.
Sorry, don't have the code as it was all changed to do our own thread
management. It was all pretty standard mina configuration anyway. We
were able to see the problem by adding a %t to the log4j line template
which shows the name of the executing thread in the log. It shows that
multiple requests were all being executed by the same thread.
It may not even be the same problem. Just thought it sounded familiar
from your description. Hope it helps.
Paddy
Mark Webb wrote:
> Great information! Do you have any code snippets or maybe even an
> example
> you could share?
>
>
> On 7/23/07, Paddy O'Neill <ct...@mobilecohesion.com> wrote:
>>
>> Hi
>>
>> This sounds similar to a problem that we encountered where mina was
>> serving multiple sessions and forwarding to a single connection, similar
>> to your environment. We found that the thread management in mina on the
>> single connection side would not work efficiently and end up only
>> allocating jobs to a single thread. This effectively caused the jobs to
>> be executed sequentially, which in turn caused the backlog and memory
>> buildup. I suspect that the issue is related to there only being a
>> single connection on the IoConnector as we found that the jobs were
>> allocated correctly across threads on the IoAcceptor side.
>>
>> We worked around it by removing the thread pool from the mina filter
>> chain and implementing our own thread management just outside the mina
>> interface.
>>
>> Paddy
>>
>> mat wrote:
>> > Thanks a lot.
>> >
>> > My server has one client (using mina too) connecting to a data
>> > source(20 -
>> > 30 /sec incoming messages) and the other side is server listening
>> on one
>> > port to broadcasting messages to end users(I wish Mina could handle
>> 1000+
>> > concurrent users). No other heaving loading process.
>> >
>> > If the incoming messages is not fast, I don't think i need a
>> > ReadThrottleFilter, right?
>> >
>> >
>> > The following are the part of write method:(I am sure not the write
>> > problem
>> > since normally no user connected OOM also happened, however you can
>> point
>> > out why Future.join helps here. ^^)
>> >
>> > synchronized(sessions)
>> > {
>> > ByteBuffer buf = ByteBuffer.allocate(512);
>> > buf.setAutoExpand(true);
>> >
>> > Object message = serviceMessage.getMessage();
>> >
>> > if (!(message instanceof byte[]))
>> > return;
>> >
>> > buf.put((byte[])message);
>> > buf.flip();
>> >
>> > Iterator iter = sessions.values().iterator();
>> >
>> > while(iter.hasNext())
>> > {
>> > IoSession session = (IoSession) iter.next();
>> > session.write(buf.duplicate());
>> > }
>> > }
>> >
>> >
>> > On 7/20/07, Luis Neves <lu...@co.sapo.pt> wrote:
>> >>
>> >> Hi,
>> >>
>> >> mat wrote:
>> >> > My server sometimes faced "OOM" problem. (I couldn't profile it
>> since
>> >> TPTP
>> >> > crashed my server before OOM occured). I didn't see major memory
>> leak
>> >> when
>> >> > profiling. Therefore, I believe OOM happens when READ or WRITE
>> >> operation
>> >> > can't handle the incoming messages or outgoing messages.
>> (However my
>> >> > incoming messages normally 20 * 512bytes/sec, NOT too fast,
>> right?).
>> >> Last
>> >> > time i saw my server memory usage is over 600MB in windows XP.
>> >>
>> >> Your code is broken ... the question is where. Mina can handle that
>> >> amount
>> >> of
>> >> messages without breaking a sweat.
>> >> Do you have some kind of heavy processing in the receiving end that
>> >> delays
>> >> the
>> >> acceptance of messages?
>> >>
>> >> Did you try to use the ReadThrottleFilter?
>> >> How are you doing you writes?
>> >> A simple "iosession.write()" ?
>> >> Did you try something like:
>> >> WriteFuture wf = iosession.write();
>> >> wf.join();
>> >>
>> >> Can we see the code of your Encoder/Decoder?
>> >>
>> >>
>> >> --
>> >> Luis Neves
>> >>
>> >
>>
>>
>>
>
>
Re: Out Of Memory Problem Again
Posted by Mark Webb <el...@gmail.com>.
Great information! Do you have any code snippets or maybe even an example
you could share?
On 7/23/07, Paddy O'Neill <ct...@mobilecohesion.com> wrote:
>
> Hi
>
> This sounds similar to a problem that we encountered where mina was
> serving multiple sessions and forwarding to a single connection, similar
> to your environment. We found that the thread management in mina on the
> single connection side would not work efficiently and end up only
> allocating jobs to a single thread. This effectively caused the jobs to
> be executed sequentially, which in turn caused the backlog and memory
> buildup. I suspect that the issue is related to there only being a
> single connection on the IoConnector as we found that the jobs were
> allocated correctly across threads on the IoAcceptor side.
>
> We worked around it by removing the thread pool from the mina filter
> chain and implementing our own thread management just outside the mina
> interface.
>
> Paddy
>
> mat wrote:
> > Thanks a lot.
> >
> > My server has one client (using mina too) connecting to a data
> > source(20 -
> > 30 /sec incoming messages) and the other side is server listening on one
> > port to broadcasting messages to end users(I wish Mina could handle
> 1000+
> > concurrent users). No other heaving loading process.
> >
> > If the incoming messages is not fast, I don't think i need a
> > ReadThrottleFilter, right?
> >
> >
> > The following are the part of write method:(I am sure not the write
> > problem
> > since normally no user connected OOM also happened, however you can
> point
> > out why Future.join helps here. ^^)
> >
> > synchronized(sessions)
> > {
> > ByteBuffer buf = ByteBuffer.allocate(512);
> > buf.setAutoExpand(true);
> >
> > Object message = serviceMessage.getMessage();
> >
> > if (!(message instanceof byte[]))
> > return;
> >
> > buf.put((byte[])message);
> > buf.flip();
> >
> > Iterator iter = sessions.values().iterator();
> >
> > while(iter.hasNext())
> > {
> > IoSession session = (IoSession) iter.next();
> > session.write(buf.duplicate());
> > }
> > }
> >
> >
> > On 7/20/07, Luis Neves <lu...@co.sapo.pt> wrote:
> >>
> >> Hi,
> >>
> >> mat wrote:
> >> > My server sometimes faced "OOM" problem. (I couldn't profile it since
> >> TPTP
> >> > crashed my server before OOM occured). I didn't see major memory leak
> >> when
> >> > profiling. Therefore, I believe OOM happens when READ or WRITE
> >> operation
> >> > can't handle the incoming messages or outgoing messages. (However my
> >> > incoming messages normally 20 * 512bytes/sec, NOT too fast, right?).
> >> Last
> >> > time i saw my server memory usage is over 600MB in windows XP.
> >>
> >> Your code is broken ... the question is where. Mina can handle that
> >> amount
> >> of
> >> messages without breaking a sweat.
> >> Do you have some kind of heavy processing in the receiving end that
> >> delays
> >> the
> >> acceptance of messages?
> >>
> >> Did you try to use the ReadThrottleFilter?
> >> How are you doing you writes?
> >> A simple "iosession.write()" ?
> >> Did you try something like:
> >> WriteFuture wf = iosession.write();
> >> wf.join();
> >>
> >> Can we see the code of your Encoder/Decoder?
> >>
> >>
> >> --
> >> Luis Neves
> >>
> >
>
>
>
--
..Cheers
Mark
Re: Out Of Memory Problem Again
Posted by Trustin Lee <tr...@gmail.com>.
Hi mat,
On 7/28/07, mat <fo...@gmail.com> wrote:
> On 7/28/07, Gaston Dombiak <ga...@jivesoftware.com> wrote:
> >
> > Hey Trustin,
> >
> > I noticed many times (specially when behind a NAT device) that sometimes
> > when a socket connection was lost/closed it's like the TCP layer does not
> > inform the JVM about the event. So the JVM thinks that the socket is fine
> > and even after writing some data (before the write queue gets full) the JVM
> > will not get any IO exception so it will still think that the socket is
> > alive. The only workarounds we applied (at the application level) were 1)
> > add heartbeats that will eventually (hopefully) will fill up the write queue
> > and get an IO exception or 2) close socket that have not "sent" any data to
> > the server. I think that #1 is not 100% safe so #2 will eventually close the
> > "assumed" dead connection.
>
>
> Will setWriteTimeout solve uncaught IO exception problem? Or a
> setIdleTime for Read and Write and send ping-pong?
Yep, both will work. I'd use both parameters for maximum safety.
> > ----- Original Message -----
> > From: "Trustin Lee" <tr...@gmail.com>
> > To: dev@mina.apache.org
> > Sent: Friday, July 27, 2007 10:11:14 AM (GMT-0800) America/Los_Angeles
> > Subject: Re: Out Of Memory Problem Again
> >
> > On 7/28/07, mat <fo...@gmail.com> wrote:
> > > After further research, I found out if one client disconnection was not
> > > caught and session.write() kept got called. Which causes the OOM? Is
> > that
> > > possible?
> >
> > Yes. You have to make sure if the connection is alive and write
> > operation was successful. You can check if the connection is alive by
> > calling IoSession.isClosing(), and WriteFuture.isWritten() for write
> > operations.
>
>
> Should that be session.isConnected()? However, if disconnection doesn't get
> caught, session.isConnected() will never work since you never do
> session.close(), right? SetWriteTimeout works for this issue, right? Why
> WriteFuture.isWritten()??
Right, it was IoSession.isConnected(). Sorry for the confusion. :)
WriteFuture.isWritten() returns false if the connection is closed
already or there was an I/O exception.
HTH,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
Re: Out Of Memory Problem Again
Posted by mat <fo...@gmail.com>.
On 7/28/07, Gaston Dombiak <ga...@jivesoftware.com> wrote:
>
> Hey Trustin,
>
> I noticed many times (specially when behind a NAT device) that sometimes
> when a socket connection was lost/closed it's like the TCP layer does not
> inform the JVM about the event. So the JVM thinks that the socket is fine
> and even after writing some data (before the write queue gets full) the JVM
> will not get any IO exception so it will still think that the socket is
> alive. The only workarounds we applied (at the application level) were 1)
> add heartbeats that will eventually (hopefully) will fill up the write queue
> and get an IO exception or 2) close socket that have not "sent" any data to
> the server. I think that #1 is not 100% safe so #2 will eventually close the
> "assumed" dead connection.
Will setWriteTimeout solve uncaught IO exception problem? Or a
setIdleTime for Read and Write and send ping-pong?
Regards,
>
> -- Gato
>
> ----- Original Message -----
> From: "Trustin Lee" <tr...@gmail.com>
> To: dev@mina.apache.org
> Sent: Friday, July 27, 2007 10:11:14 AM (GMT-0800) America/Los_Angeles
> Subject: Re: Out Of Memory Problem Again
>
> On 7/28/07, mat <fo...@gmail.com> wrote:
> > After further research, I found out if one client disconnection was not
> > caught and session.write() kept got called. Which causes the OOM? Is
> that
> > possible?
>
> Yes. You have to make sure if the connection is alive and write
> operation was successful. You can check if the connection is alive by
> calling IoSession.isClosing(), and WriteFuture.isWritten() for write
> operations.
Should that be session.isConnected()? However, if disconnection doesn't get
caught, session.isConnected() will never work since you never do
session.close(), right? SetWriteTimeout works for this issue, right? Why
WriteFuture.isWritten()??
HTH,
> Trustin
> --
> what we call human nature is actually human habit
> --
> http://gleamynode.net/
> --
> PGP Key ID: 0x0255ECA6
>
>
Re: Out Of Memory Problem Again
Posted by Gaston Dombiak <ga...@jivesoftware.com>.
Hey Trustin,
I noticed many times (specially when behind a NAT device) that sometimes when a socket connection was lost/closed it's like the TCP layer does not inform the JVM about the event. So the JVM thinks that the socket is fine and even after writing some data (before the write queue gets full) the JVM will not get any IO exception so it will still think that the socket is alive. The only workarounds we applied (at the application level) were 1) add heartbeats that will eventually (hopefully) will fill up the write queue and get an IO exception or 2) close socket that have not "sent" any data to the server. I think that #1 is not 100% safe so #2 will eventually close the "assumed" dead connection.
Regards,
-- Gato
----- Original Message -----
From: "Trustin Lee" <tr...@gmail.com>
To: dev@mina.apache.org
Sent: Friday, July 27, 2007 10:11:14 AM (GMT-0800) America/Los_Angeles
Subject: Re: Out Of Memory Problem Again
On 7/28/07, mat <fo...@gmail.com> wrote:
> After further research, I found out if one client disconnection was not
> caught and session.write() kept got called. Which causes the OOM? Is that
> possible?
Yes. You have to make sure if the connection is alive and write
operation was successful. You can check if the connection is alive by
calling IoSession.isClosing(), and WriteFuture.isWritten() for write
operations.
HTH,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
Re: Out Of Memory Problem Again
Posted by Trustin Lee <tr...@gmail.com>.
On 7/28/07, mat <fo...@gmail.com> wrote:
> After further research, I found out if one client disconnection was not
> caught and session.write() kept got called. Which causes the OOM? Is that
> possible?
Yes. You have to make sure if the connection is alive and write
operation was successful. You can check if the connection is alive by
calling IoSession.isClosing(), and WriteFuture.isWritten() for write
operations.
HTH,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
Re: Out Of Memory Problem Again
Posted by mat <fo...@gmail.com>.
After further research, I found out if one client disconnection was not
caught and session.write() kept got called. Which causes the OOM? Is that
possible?
On 7/27/07, mat <fo...@gmail.com> wrote:
>
> Let me conclude this thread.
> After serveral days's observation, I found out my OOM was nothing to do
> with either VM tuning or memory leak. Actually I got the hints from this
> issue "permanent solution for OOM"(See Paul Chen's comment). The OOM happens
> when writeRequestQueue blows up in my case. I found out because when I put
> my client on the same machine as server, OOM doesn't happen after 2 days.
> However, when I move my client to different network. OOM happens sometimes
> due to "slow traffice"? That's all I can conclude at this moment. And i want
> to share with all of you. Maybe a Write throttle filter is needed.
> Finally dear commitee, please consider point out this to all of Mina users
> and I believe Read/Write throttle filter is really helpful.
>
>
> On 7/25/07, Mark Webb <el...@gmail.com> wrote:
> >
> > Not sure if this article will help, but check this out:
> > http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html
> >
> > Maybe if you choose a different garbage collector, your memory might get
> > cleaned up more efficiently.
> >
> >
> > On 7/24/07, mat < forum.maillist@gmail.com> wrote:
> > >
> > > I got the following error message again after 40 hours starting my
> > server.
> > > It seems the Direct buffer memory happens only the heap memory usage
> > high.
> > > I
> > > also record the memory usage history in windows task manager. It seems
> > GC
> > > doesn't collect until heavy loading is gone. However after every GC,
> > my
> > > memory still increases. Eventually OOM happens.
> > >
> > > 2007-7-25 4:45:51
> > > org.apache.mina.common.support.DefaultExceptionMonitorexcepti
> > > onCaught
> > > warning: Unexpected exception.
> > > java.lang.OutOfMemoryError: Direct buffer memory
> > > at java.nio.Bits.reserveMemory(Unknown Source)
> > > at java.nio.DirectByteBuffer.<init>(Unknown Source)
> > > at java.nio.ByteBuffer.allocateDirect(Unknown Source)
> > > at sun.nio.ch.Util.getTemporaryDirectBuffer(Unknown Source)
> > > at sun.nio.ch.IOUtil.write(Unknown Source)
> > > at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
> > > at
> > org.apache.mina.transport.socket.nio.SocketIoProcessor.doFlush
> > > (Socket
> > > IoProcessor.java:428)
> > > at
> > org.apache.mina.transport.socket.nio.SocketIoProcessor.doFlush
> > > (Socket
> > > IoProcessor.java:366)
> > > at
> > > org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600
> > > (Soc
> > > ketIoProcessor.java:44)
> > > at
> > > org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run
> > > (Soc
> > > ketIoProcessor.java:509)
> > > at org.apache.mina.util.NamePreservingRunnable.run
> > > (NamePreservingRunnabl
> > > e.java:43)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> > (Unknown
> > > Source
> > > )
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run (Unknown
> > > Source)
> > > at java.lang.Thread.run(Unknown Source)
> > > 2007-7-25 5:16:53
> > > org.apache.mina.common.support.DefaultExceptionMonitorexcepti
> > > onCaught
> > > warning: Unexpected exception.
> > > java.lang.OutOfMemoryError: Direct buffer memory
> > > at java.nio.Bits.reserveMemory(Unknown Source)
> > > at java.nio.DirectByteBuffer.<init>(Unknown Source)
> > > at java.nio.ByteBuffer.allocateDirect (Unknown Source)
> > > at sun.nio.ch.Util.getTemporaryDirectBuffer(Unknown Source)
> > > at sun.nio.ch.IOUtil.write(Unknown Source)
> > > at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
> > > at sun.nio.ch.PipeImpl$Initializer.run(Unknown Source)
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at sun.nio.ch.PipeImpl.<init>(Unknown Source)
> > > at sun.nio.ch.SelectorProviderImpl.openPipe(Unknown Source)
> > > at java.nio.channels.Pipe.open(Unknown Source)
> > > at sun.nio.ch.WindowsSelectorImpl.<init>(Unknown Source)
> > > at sun.nio.ch.WindowsSelectorProvider.openSelector(Unknown
> > Source)
> > > at java.nio.channels.Selector.open(Unknown Source)
> > > at
> > > org.apache.mina.transport.socket.nio.SocketIoProcessor.startupWorker (
> > > SocketIoProcessor.java:84)
> > > at
> > org.apache.mina.transport.socket.nio.SocketIoProcessor.addNew
> > > (SocketI
> > > oProcessor.java:69)
> > > at
> > > org.apache.mina.transport.socket.nio.SocketAcceptor$Worker.processSes
> > > sions(SocketAcceptor.java:319)
> > > at
> > org.apache.mina.transport.socket.nio.SocketAcceptor$Worker.run
> > > (Socket
> > > Acceptor.java:239)
> > > at org.apache.mina.util.NamePreservingRunnable.run
> > > (NamePreservingRunnabl
> > > e.java:43)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> > (Unknown
> > > Source
> > > )
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run (Unknown
> > > Source)
> > > at java.lang.Thread.run(Unknown Source)
> > > Exception in thread "pool-7-thread-6" java.lang.OutOfMemoryError: Java
> > > heap
> > > spac
> > > e
> > > Exception in thread "Thread-5"
> > >
> > >
> > > On 7/23/07, peter royal <pr...@apache.org> wrote:
> > > >
> > > > On Jul 23, 2007, at 3:45 AM, Paddy O'Neill wrote:
> > > > > This sounds similar to a problem that we encountered where mina
> > was
> > > > > serving multiple sessions and forwarding to a single connection,
> > > > > similar to your environment. We found that the thread management
> > > > > in mina on the single connection side would not work efficiently
> > > > > and end up only allocating jobs to a single thread. This
> > > > > effectively caused the jobs to be executed sequentially, which in
> > > > > turn caused the backlog and memory buildup. I suspect that the
> > > > > issue is related to there only being a single connection on the
> > > > > IoConnector as we found that the jobs were allocated correctly
> > > > > across threads on the IoAcceptor side.
> > > >
> > > > sure.. not unsurprising. that's how the ExecutorFilter works. what
> > > > you did was the right thing.. recognizing you have some
> > > > parallelization in processing of messages for your app and managing
> > > > it yourself.
> > > >
> > > > -pete
> > > >
> > > >
> > > >
> > > > --
> > > > proyal@apache.org - http://fotap.org/~osi
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > ..Cheers
> > Mark
> >
>
>
Re: Out Of Memory Problem Again
Posted by mat <fo...@gmail.com>.
Let me conclude this thread.
After serveral days's observation, I found out my OOM was nothing to do with
either VM tuning or memory leak. Actually I got the hints from this issue
"permanent solution for OOM"(See Paul Chen's comment). The OOM happens when
writeRequestQueue blows up in my case. I found out because when I put my
client on the same machine as server, OOM doesn't happen after 2 days.
However, when I move my client to different network. OOM happens sometimes
due to "slow traffice"? That's all I can conclude at this moment. And i want
to share with all of you. Maybe a Write throttle filter is needed.
Finally dear commitee, please consider point out this to all of Mina users
and I believe Read/Write throttle filter is really helpful.
On 7/25/07, Mark Webb <el...@gmail.com> wrote:
>
> Not sure if this article will help, but check this out:
> http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html
>
> Maybe if you choose a different garbage collector, your memory might get
> cleaned up more efficiently.
>
>
> On 7/24/07, mat <fo...@gmail.com> wrote:
> >
> > I got the following error message again after 40 hours starting my
> server.
> > It seems the Direct buffer memory happens only the heap memory usage
> high.
> > I
> > also record the memory usage history in windows task manager. It seems
> GC
> > doesn't collect until heavy loading is gone. However after every GC, my
> > memory still increases. Eventually OOM happens.
> >
> > 2007-7-25 4:45:51
> > org.apache.mina.common.support.DefaultExceptionMonitorexcepti
> > onCaught
> > warning: Unexpected exception.
> > java.lang.OutOfMemoryError: Direct buffer memory
> > at java.nio.Bits.reserveMemory(Unknown Source)
> > at java.nio.DirectByteBuffer.<init>(Unknown Source)
> > at java.nio.ByteBuffer.allocateDirect(Unknown Source)
> > at sun.nio.ch.Util.getTemporaryDirectBuffer(Unknown Source)
> > at sun.nio.ch.IOUtil.write(Unknown Source)
> > at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
> > at
> org.apache.mina.transport.socket.nio.SocketIoProcessor.doFlush
> > (Socket
> > IoProcessor.java:428)
> > at
> org.apache.mina.transport.socket.nio.SocketIoProcessor.doFlush
> > (Socket
> > IoProcessor.java:366)
> > at
> > org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600
> > (Soc
> > ketIoProcessor.java:44)
> > at
> > org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run
> > (Soc
> > ketIoProcessor.java:509)
> > at org.apache.mina.util.NamePreservingRunnable.run
> > (NamePreservingRunnabl
> > e.java:43)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> (Unknown
> > Source
> > )
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> > Source)
> > at java.lang.Thread.run(Unknown Source)
> > 2007-7-25 5:16:53
> > org.apache.mina.common.support.DefaultExceptionMonitorexcepti
> > onCaught
> > warning: Unexpected exception.
> > java.lang.OutOfMemoryError: Direct buffer memory
> > at java.nio.Bits.reserveMemory(Unknown Source)
> > at java.nio.DirectByteBuffer.<init>(Unknown Source)
> > at java.nio.ByteBuffer.allocateDirect(Unknown Source)
> > at sun.nio.ch.Util.getTemporaryDirectBuffer(Unknown Source)
> > at sun.nio.ch.IOUtil.write(Unknown Source)
> > at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
> > at sun.nio.ch.PipeImpl$Initializer.run(Unknown Source)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at sun.nio.ch.PipeImpl.<init>(Unknown Source)
> > at sun.nio.ch.SelectorProviderImpl.openPipe(Unknown Source)
> > at java.nio.channels.Pipe.open(Unknown Source)
> > at sun.nio.ch.WindowsSelectorImpl.<init>(Unknown Source)
> > at sun.nio.ch.WindowsSelectorProvider.openSelector(Unknown
> Source)
> > at java.nio.channels.Selector.open(Unknown Source)
> > at
> > org.apache.mina.transport.socket.nio.SocketIoProcessor.startupWorker(
> > SocketIoProcessor.java:84)
> > at org.apache.mina.transport.socket.nio.SocketIoProcessor.addNew
> > (SocketI
> > oProcessor.java:69)
> > at
> > org.apache.mina.transport.socket.nio.SocketAcceptor$Worker.processSes
> > sions(SocketAcceptor.java:319)
> > at
> org.apache.mina.transport.socket.nio.SocketAcceptor$Worker.run
> > (Socket
> > Acceptor.java:239)
> > at org.apache.mina.util.NamePreservingRunnable.run
> > (NamePreservingRunnabl
> > e.java:43)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> (Unknown
> > Source
> > )
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> > Source)
> > at java.lang.Thread.run(Unknown Source)
> > Exception in thread "pool-7-thread-6" java.lang.OutOfMemoryError: Java
> > heap
> > spac
> > e
> > Exception in thread "Thread-5"
> >
> >
> > On 7/23/07, peter royal <pr...@apache.org> wrote:
> > >
> > > On Jul 23, 2007, at 3:45 AM, Paddy O'Neill wrote:
> > > > This sounds similar to a problem that we encountered where mina was
> > > > serving multiple sessions and forwarding to a single connection,
> > > > similar to your environment. We found that the thread management
> > > > in mina on the single connection side would not work efficiently
> > > > and end up only allocating jobs to a single thread. This
> > > > effectively caused the jobs to be executed sequentially, which in
> > > > turn caused the backlog and memory buildup. I suspect that the
> > > > issue is related to there only being a single connection on the
> > > > IoConnector as we found that the jobs were allocated correctly
> > > > across threads on the IoAcceptor side.
> > >
> > > sure.. not unsurprising. that's how the ExecutorFilter works. what
> > > you did was the right thing.. recognizing you have some
> > > parallelization in processing of messages for your app and managing
> > > it yourself.
> > >
> > > -pete
> > >
> > >
> > >
> > > --
> > > proyal@apache.org - http://fotap.org/~osi
> > >
> > >
> > >
> > >
> > >
> >
>
>
>
> --
> ..Cheers
> Mark
>
Re: Out Of Memory Problem Again
Posted by Mark Webb <el...@gmail.com>.
Not sure if this article will help, but check this out:
http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html
Maybe if you choose a different garbage collector, your memory might get
cleaned up more efficiently.
On 7/24/07, mat <fo...@gmail.com> wrote:
>
> I got the following error message again after 40 hours starting my server.
> It seems the Direct buffer memory happens only the heap memory usage high.
> I
> also record the memory usage history in windows task manager. It seems GC
> doesn't collect until heavy loading is gone. However after every GC, my
> memory still increases. Eventually OOM happens.
>
> 2007-7-25 4:45:51
> org.apache.mina.common.support.DefaultExceptionMonitorexcepti
> onCaught
> warning: Unexpected exception.
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Unknown Source)
> at java.nio.DirectByteBuffer.<init>(Unknown Source)
> at java.nio.ByteBuffer.allocateDirect(Unknown Source)
> at sun.nio.ch.Util.getTemporaryDirectBuffer(Unknown Source)
> at sun.nio.ch.IOUtil.write(Unknown Source)
> at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
> at org.apache.mina.transport.socket.nio.SocketIoProcessor.doFlush
> (Socket
> IoProcessor.java:428)
> at org.apache.mina.transport.socket.nio.SocketIoProcessor.doFlush
> (Socket
> IoProcessor.java:366)
> at
> org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600
> (Soc
> ketIoProcessor.java:44)
> at
> org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run
> (Soc
> ketIoProcessor.java:509)
> at org.apache.mina.util.NamePreservingRunnable.run
> (NamePreservingRunnabl
> e.java:43)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source
> )
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at java.lang.Thread.run(Unknown Source)
> 2007-7-25 5:16:53
> org.apache.mina.common.support.DefaultExceptionMonitorexcepti
> onCaught
> warning: Unexpected exception.
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Unknown Source)
> at java.nio.DirectByteBuffer.<init>(Unknown Source)
> at java.nio.ByteBuffer.allocateDirect(Unknown Source)
> at sun.nio.ch.Util.getTemporaryDirectBuffer(Unknown Source)
> at sun.nio.ch.IOUtil.write(Unknown Source)
> at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
> at sun.nio.ch.PipeImpl$Initializer.run(Unknown Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.nio.ch.PipeImpl.<init>(Unknown Source)
> at sun.nio.ch.SelectorProviderImpl.openPipe(Unknown Source)
> at java.nio.channels.Pipe.open(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl.<init>(Unknown Source)
> at sun.nio.ch.WindowsSelectorProvider.openSelector(Unknown Source)
> at java.nio.channels.Selector.open(Unknown Source)
> at
> org.apache.mina.transport.socket.nio.SocketIoProcessor.startupWorker(
> SocketIoProcessor.java:84)
> at org.apache.mina.transport.socket.nio.SocketIoProcessor.addNew
> (SocketI
> oProcessor.java:69)
> at
> org.apache.mina.transport.socket.nio.SocketAcceptor$Worker.processSes
> sions(SocketAcceptor.java:319)
> at org.apache.mina.transport.socket.nio.SocketAcceptor$Worker.run
> (Socket
> Acceptor.java:239)
> at org.apache.mina.util.NamePreservingRunnable.run
> (NamePreservingRunnabl
> e.java:43)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source
> )
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at java.lang.Thread.run(Unknown Source)
> Exception in thread "pool-7-thread-6" java.lang.OutOfMemoryError: Java
> heap
> spac
> e
> Exception in thread "Thread-5"
>
>
> On 7/23/07, peter royal <pr...@apache.org> wrote:
> >
> > On Jul 23, 2007, at 3:45 AM, Paddy O'Neill wrote:
> > > This sounds similar to a problem that we encountered where mina was
> > > serving multiple sessions and forwarding to a single connection,
> > > similar to your environment. We found that the thread management
> > > in mina on the single connection side would not work efficiently
> > > and end up only allocating jobs to a single thread. This
> > > effectively caused the jobs to be executed sequentially, which in
> > > turn caused the backlog and memory buildup. I suspect that the
> > > issue is related to there only being a single connection on the
> > > IoConnector as we found that the jobs were allocated correctly
> > > across threads on the IoAcceptor side.
> >
> > sure.. not unsurprising. that's how the ExecutorFilter works. what
> > you did was the right thing.. recognizing you have some
> > parallelization in processing of messages for your app and managing
> > it yourself.
> >
> > -pete
> >
> >
> >
> > --
> > proyal@apache.org - http://fotap.org/~osi
> >
> >
> >
> >
> >
>
--
..Cheers
Mark
Re: Out Of Memory Problem Again
Posted by mat <fo...@gmail.com>.
I got the following error message again after 40 hours starting my server.
It seems the Direct buffer memory happens only the heap memory usage high. I
also record the memory usage history in windows task manager. It seems GC
doesn't collect until heavy loading is gone. However after every GC, my
memory still increases. Eventually OOM happens.
2007-7-25 4:45:51 org.apache.mina.common.support.DefaultExceptionMonitorexcepti
onCaught
warning: Unexpected exception.
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Unknown Source)
at java.nio.DirectByteBuffer.<init>(Unknown Source)
at java.nio.ByteBuffer.allocateDirect(Unknown Source)
at sun.nio.ch.Util.getTemporaryDirectBuffer(Unknown Source)
at sun.nio.ch.IOUtil.write(Unknown Source)
at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.doFlush
(Socket
IoProcessor.java:428)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.doFlush
(Socket
IoProcessor.java:366)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600
(Soc
ketIoProcessor.java:44)
at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run
(Soc
ketIoProcessor.java:509)
at org.apache.mina.util.NamePreservingRunnable.run
(NamePreservingRunnabl
e.java:43)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source
)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
2007-7-25 5:16:53 org.apache.mina.common.support.DefaultExceptionMonitorexcepti
onCaught
warning: Unexpected exception.
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Unknown Source)
at java.nio.DirectByteBuffer.<init>(Unknown Source)
at java.nio.ByteBuffer.allocateDirect(Unknown Source)
at sun.nio.ch.Util.getTemporaryDirectBuffer(Unknown Source)
at sun.nio.ch.IOUtil.write(Unknown Source)
at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
at sun.nio.ch.PipeImpl$Initializer.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.nio.ch.PipeImpl.<init>(Unknown Source)
at sun.nio.ch.SelectorProviderImpl.openPipe(Unknown Source)
at java.nio.channels.Pipe.open(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.<init>(Unknown Source)
at sun.nio.ch.WindowsSelectorProvider.openSelector(Unknown Source)
at java.nio.channels.Selector.open(Unknown Source)
at
org.apache.mina.transport.socket.nio.SocketIoProcessor.startupWorker(
SocketIoProcessor.java:84)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.addNew
(SocketI
oProcessor.java:69)
at
org.apache.mina.transport.socket.nio.SocketAcceptor$Worker.processSes
sions(SocketAcceptor.java:319)
at org.apache.mina.transport.socket.nio.SocketAcceptor$Worker.run
(Socket
Acceptor.java:239)
at org.apache.mina.util.NamePreservingRunnable.run
(NamePreservingRunnabl
e.java:43)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source
)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
Exception in thread "pool-7-thread-6" java.lang.OutOfMemoryError: Java heap
spac
e
Exception in thread "Thread-5"
On 7/23/07, peter royal <pr...@apache.org> wrote:
>
> On Jul 23, 2007, at 3:45 AM, Paddy O'Neill wrote:
> > This sounds similar to a problem that we encountered where mina was
> > serving multiple sessions and forwarding to a single connection,
> > similar to your environment. We found that the thread management
> > in mina on the single connection side would not work efficiently
> > and end up only allocating jobs to a single thread. This
> > effectively caused the jobs to be executed sequentially, which in
> > turn caused the backlog and memory buildup. I suspect that the
> > issue is related to there only being a single connection on the
> > IoConnector as we found that the jobs were allocated correctly
> > across threads on the IoAcceptor side.
>
> sure.. not unsurprising. that's how the ExecutorFilter works. what
> you did was the right thing.. recognizing you have some
> parallelization in processing of messages for your app and managing
> it yourself.
>
> -pete
>
>
>
> --
> proyal@apache.org - http://fotap.org/~osi
>
>
>
>
>
Re: Out Of Memory Problem Again
Posted by peter royal <pr...@apache.org>.
On Jul 23, 2007, at 3:45 AM, Paddy O'Neill wrote:
> This sounds similar to a problem that we encountered where mina was
> serving multiple sessions and forwarding to a single connection,
> similar to your environment. We found that the thread management
> in mina on the single connection side would not work efficiently
> and end up only allocating jobs to a single thread. This
> effectively caused the jobs to be executed sequentially, which in
> turn caused the backlog and memory buildup. I suspect that the
> issue is related to there only being a single connection on the
> IoConnector as we found that the jobs were allocated correctly
> across threads on the IoAcceptor side.
sure.. not unsurprising. that's how the ExecutorFilter works. what
you did was the right thing.. recognizing you have some
parallelization in processing of messages for your app and managing
it yourself.
-pete
--
proyal@apache.org - http://fotap.org/~osi
Re: Out Of Memory Problem Again
Posted by Paddy O'Neill <ct...@mobilecohesion.com>.
Hi
This sounds similar to a problem that we encountered where mina was
serving multiple sessions and forwarding to a single connection, similar
to your environment. We found that the thread management in mina on the
single connection side would not work efficiently and end up only
allocating jobs to a single thread. This effectively caused the jobs to
be executed sequentially, which in turn caused the backlog and memory
buildup. I suspect that the issue is related to there only being a
single connection on the IoConnector as we found that the jobs were
allocated correctly across threads on the IoAcceptor side.
We worked around it by removing the thread pool from the mina filter
chain and implementing our own thread management just outside the mina
interface.
Paddy
mat wrote:
> Thanks a lot.
>
> My server has one client (using mina too) connecting to a data
> source(20 -
> 30 /sec incoming messages) and the other side is server listening on one
> port to broadcasting messages to end users(I wish Mina could handle 1000+
> concurrent users). No other heaving loading process.
>
> If the incoming messages is not fast, I don't think i need a
> ReadThrottleFilter, right?
>
>
> The following are the part of write method:(I am sure not the write
> problem
> since normally no user connected OOM also happened, however you can point
> out why Future.join helps here. ^^)
>
> synchronized(sessions)
> {
> ByteBuffer buf = ByteBuffer.allocate(512);
> buf.setAutoExpand(true);
>
> Object message = serviceMessage.getMessage();
>
> if (!(message instanceof byte[]))
> return;
>
> buf.put((byte[])message);
> buf.flip();
>
> Iterator iter = sessions.values().iterator();
>
> while(iter.hasNext())
> {
> IoSession session = (IoSession) iter.next();
> session.write(buf.duplicate());
> }
> }
>
>
> On 7/20/07, Luis Neves <lu...@co.sapo.pt> wrote:
>>
>> Hi,
>>
>> mat wrote:
>> > My server sometimes faced "OOM" problem. (I couldn't profile it since
>> TPTP
>> > crashed my server before OOM occured). I didn't see major memory leak
>> when
>> > profiling. Therefore, I believe OOM happens when READ or WRITE
>> operation
>> > can't handle the incoming messages or outgoing messages. (However my
>> > incoming messages normally 20 * 512bytes/sec, NOT too fast, right?).
>> Last
>> > time i saw my server memory usage is over 600MB in windows XP.
>>
>> Your code is broken ... the question is where. Mina can handle that
>> amount
>> of
>> messages without breaking a sweat.
>> Do you have some kind of heavy processing in the receiving end that
>> delays
>> the
>> acceptance of messages?
>>
>> Did you try to use the ReadThrottleFilter?
>> How are you doing you writes?
>> A simple "iosession.write()" ?
>> Did you try something like:
>> WriteFuture wf = iosession.write();
>> wf.join();
>>
>> Can we see the code of your Encoder/Decoder?
>>
>>
>> --
>> Luis Neves
>>
>
Re: Out Of Memory Problem Again
Posted by mat <fo...@gmail.com>.
Thanks a lot.
My server has one client (using mina too) connecting to a data source(20 -
30 /sec incoming messages) and the other side is server listening on one
port to broadcasting messages to end users(I wish Mina could handle 1000+
concurrent users). No other heaving loading process.
If the incoming messages is not fast, I don't think i need a
ReadThrottleFilter, right?
The following are the part of write method:(I am sure not the write problem
since normally no user connected OOM also happened, however you can point
out why Future.join helps here. ^^)
synchronized(sessions)
{
ByteBuffer buf = ByteBuffer.allocate(512);
buf.setAutoExpand(true);
Object message = serviceMessage.getMessage();
if (!(message instanceof byte[]))
return;
buf.put((byte[])message);
buf.flip();
Iterator iter = sessions.values().iterator();
while(iter.hasNext())
{
IoSession session = (IoSession) iter.next();
session.write(buf.duplicate());
}
}
On 7/20/07, Luis Neves <lu...@co.sapo.pt> wrote:
>
> Hi,
>
> mat wrote:
> > My server sometimes faced "OOM" problem. (I couldn't profile it since
> TPTP
> > crashed my server before OOM occured). I didn't see major memory leak
> when
> > profiling. Therefore, I believe OOM happens when READ or WRITE operation
> > can't handle the incoming messages or outgoing messages. (However my
> > incoming messages normally 20 * 512bytes/sec, NOT too fast, right?).
> Last
> > time i saw my server memory usage is over 600MB in windows XP.
>
> Your code is broken ... the question is where. Mina can handle that amount
> of
> messages without breaking a sweat.
> Do you have some kind of heavy processing in the receiving end that delays
> the
> acceptance of messages?
>
> Did you try to use the ReadThrottleFilter?
> How are you doing you writes?
> A simple "iosession.write()" ?
> Did you try something like:
> WriteFuture wf = iosession.write();
> wf.join();
>
> Can we see the code of your Encoder/Decoder?
>
>
> --
> Luis Neves
>
Re: Out Of Memory Problem Again
Posted by Luis Neves <lu...@co.sapo.pt>.
Hi,
mat wrote:
> My server sometimes faced "OOM" problem. (I couldn't profile it since TPTP
> crashed my server before OOM occured). I didn't see major memory leak when
> profiling. Therefore, I believe OOM happens when READ or WRITE operation
> can't handle the incoming messages or outgoing messages. (However my
> incoming messages normally 20 * 512bytes/sec, NOT too fast, right?). Last
> time i saw my server memory usage is over 600MB in windows XP.
Your code is broken ... the question is where. Mina can handle that amount of
messages without breaking a sweat.
Do you have some kind of heavy processing in the receiving end that delays the
acceptance of messages?
Did you try to use the ReadThrottleFilter?
How are you doing you writes?
A simple "iosession.write()" ?
Did you try something like:
WriteFuture wf = iosession.write();
wf.join();
Can we see the code of your Encoder/Decoder?
--
Luis Neves