You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by Scott Carter <Sc...@conexant.com> on 2007/10/21 01:07:54 UTC
Out of Memory issues with our mina proxy
We have written a server proxy layer using the mina library, and
everything works as it should except we are running into "Out of Memory"
exceptions. We did the obvious things such as increasing the memory for
the proxy and still always had the same issue (just took a little longer
to produce it). We analyzed the application with a profiler and
discovered that a thread from mina was creating LinkedBlockingQueue$Node
objects that were not being garbage collected. I read a couple of
previous posts on the issue and could not find a good solution. Based
on the posts I did verify that these objects were not being garbage
collected even after the server had been idol for a long time and all
sessions had been closed.
Our proxy binds to the outside port of a version control server and
passes data back and fourth between clients and the version control
server. I have attached our code, which is pretty simple, to this post.
You can see that we have some methods that take a few bites off of the
messages and verify the client for us, but other than that this is just
a very straight forward proxy application.
I really need some help to figure out how to get the
LinkedBlockingQueue$Node objects garbage collected. I have posted below
the stack trace that creates the LinkedBlockingQueue$Node objects from
the mina thread:
TRACE 319549: (thread=200037)
java.util.concurrent.LinkedBlockingQueue$Node.<init>(LinkedBlockingQueue
.java:70)
java.util.concurrent.LinkedBlockingQueue.insert(LinkedBlockingQueue.java
:129)
java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:
335)
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:
653)
org.apache.mina.filter.executor.ExecutorFilter.fireEvent(ExecutorFilter.
java:101)
org.apache.mina.filter.executor.ExecutorFilter.messageSent(ExecutorFilte
r.java:214)
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageSent
(AbstractIoFilterChain.java:320)
org.apache.mina.common.support.AbstractIoFilterChain.access$1200(Abstrac
tIoFilterChain.java:53)
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.message
Sent(AbstractIoFilterChain.java:652)
org.apache.mina.common.support.AbstractIoFilterChain$HeadFilter.messageS
ent(AbstractIoFilterChain.java:503)
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageSent
(AbstractIoFilterChain.java:320)
org.apache.mina.common.support.AbstractIoFilterChain.fireMessageSent(Abs
tractIoFilterChain.java:314)
org.apache.mina.transport.socket.nio.SocketIoProcessor.doFlush(SocketIoP
rocessor.java:358)
org.apache.mina.transport.socket.nio.SocketIoProcessor.doFlush(SocketIoP
rocessor.java:314)
org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600(Socket
IoProcessor.java:44)
org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(Socket
IoProcessor.java:431)
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.j
ava:39)
java.lang.Thread.run(Thread.java:619)
Thank you in advance for helping a Newbie. This is a production
application that I need to get working ASAP. Thanks - Scott
Conexant E-mail Firewall (Conexant.Com) made the following annotations
---------------------------------------------------------------------
********************** Legal Disclaimer ****************************
"This email may contain confidential and privileged material for the sole use of the intended recipient. Any unauthorized review, use or distribution by others is strictly prohibited. If you have received the message in error, please advise the sender by reply email and delete the message. Thank you."
**********************************************************************
---------------------------------------------------------------------
RE: Out of Memory issues with our mina proxy
Posted by Scott Carter <Sc...@conexant.com>.
Thanks for the response. Can you explain how using heap buffers fixes
the issue? Seems that they would still use a memory foot print. Are
they just garbage collected differently? Did this change fix the issue
for you?
-----Original Message-----
From: Scott Peters [mailto:speters@visto.com]
Sent: Saturday, October 20, 2007 7:02 PM
To: dev@mina.apache.org
Subject: RE: Out of Memory issues with our mina proxy
I have run into similar issues with out of memory problems and moved the
code to use heap buffers instead of direct buffers.
See this page.
http://mina.apache.org/faq.html#FAQ-Iget%7B%7BOutOfMemoryError%7D%7Dorre
sponsetimeoutandconnectionresetunderheavyload.
If you use the SSL filter is not controlled in the same way and you need
to set an environment variable either at the java start line or
programmatically by:
System.setProperty("mina.sslfilter.directbuffer", "false");
Hope that helps,
Scott
________________________________
From: Scott Carter [mailto:Scott.Carter@conexant.com]
Sent: Saturday, October 20, 2007 4:08 PM
To: dev@mina.apache.org
Subject: Out of Memory issues with our mina proxy
We have written a server proxy layer using the mina library, and
everything works as it should except we are running into "Out of Memory"
exceptions. We did the obvious things such as increasing the memory for
the proxy and still always had the same issue (just took a little longer
to produce it). We analyzed the application with a profiler and
discovered that a thread from mina was creating LinkedBlockingQueue$Node
objects that were not being garbage collected. I read a couple of
previous posts on the issue and could not find a good solution. Based
on the posts I did verify that these objects were not being garbage
collected even after the server had been idol for a long time and all
sessions had been closed.
Our proxy binds to the outside port of a version control server and
passes data back and fourth between clients and the version control
server. I have attached our code, which is pretty simple, to this post.
You can see that we have some methods that take a few bites off of the
messages and verify the client for us, but other than that this is just
a very straight forward proxy application.
I really need some help to figure out how to get the
LinkedBlockingQueue$Node objects garbage collected. I have posted below
the stack trace that creates the LinkedBlockingQueue$Node objects from
the mina thread:
TRACE 319549: (thread=200037)
java.util.concurrent.LinkedBlockingQueue$Node.<init>(LinkedBlockingQueue
.java:70)
java.util.concurrent.LinkedBlockingQueue.insert(LinkedBlockingQueue.java
:129)
java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:
335)
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:
653)
org.apache.mina.filter.executor.ExecutorFilter.fireEvent(ExecutorFilter.
java:101)
org.apache.mina.filter.executor.ExecutorFilter.messageSent(ExecutorFilte
r.java:214)
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageSent
(AbstractIoFilterChain.java:320)
org.apache.mina.common.support.AbstractIoFilterChain.access$1200(Abstrac
tIoFilterChain.java:53)
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.message
Sent(AbstractIoFilterChain.java:652)
org.apache.mina.common.support.AbstractIoFilterChain$HeadFilter.messageS
ent(AbstractIoFilterChain.java:503)
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageSent
(AbstractIoFilterChain.java:320)
org.apache.mina.common.support.AbstractIoFilterChain.fireMessageSent(Abs
tractIoFilterChain.java:314)
org.apache.mina.transport.socket.nio.SocketIoProcessor.doFlush(SocketIoP
rocessor.java:358)
org.apache.mina.transport.socket.nio.SocketIoProcessor.doFlush(SocketIoP
rocessor.java:314)
org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600(Socket
IoProcessor.java:44)
org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(Socket
IoProcessor.java:431)
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.j
ava:39)
java.lang.Thread.run(Thread.java:619)
Thank you in advance for helping a Newbie. This is a production
application that I need to get working ASAP. Thanks - Scott
Conexant E-mail Firewall (Conexant.Com) made the following annotations
---------------------------------------------------------------------
********************** Legal Disclaimer ****************************
"This email may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or
distribution by others is strictly prohibited. If you have received the
message in error, please advise the sender by reply email and delete the
message. Thank you."
**********************************************************************
---------------------------------------------------------------------
RE: Out of Memory issues with our mina proxy
Posted by Scott Peters <sp...@visto.com>.
I have run into similar issues with out of memory problems and moved the
code to use heap buffers instead of direct buffers.
See this page.
http://mina.apache.org/faq.html#FAQ-Iget%7B%7BOutOfMemoryError%7D%7Dorre
sponsetimeoutandconnectionresetunderheavyload.
If you use the SSL filter is not controlled in the same way and you need
to set an environment variable either at the java start line or
programmatically by:
System.setProperty("mina.sslfilter.directbuffer", "false");
Hope that helps,
Scott
________________________________
From: Scott Carter [mailto:Scott.Carter@conexant.com]
Sent: Saturday, October 20, 2007 4:08 PM
To: dev@mina.apache.org
Subject: Out of Memory issues with our mina proxy
We have written a server proxy layer using the mina library, and
everything works as it should except we are running into "Out of Memory"
exceptions. We did the obvious things such as increasing the memory for
the proxy and still always had the same issue (just took a little longer
to produce it). We analyzed the application with a profiler and
discovered that a thread from mina was creating LinkedBlockingQueue$Node
objects that were not being garbage collected. I read a couple of
previous posts on the issue and could not find a good solution. Based
on the posts I did verify that these objects were not being garbage
collected even after the server had been idol for a long time and all
sessions had been closed.
Our proxy binds to the outside port of a version control server and
passes data back and fourth between clients and the version control
server. I have attached our code, which is pretty simple, to this post.
You can see that we have some methods that take a few bites off of the
messages and verify the client for us, but other than that this is just
a very straight forward proxy application.
I really need some help to figure out how to get the
LinkedBlockingQueue$Node objects garbage collected. I have posted below
the stack trace that creates the LinkedBlockingQueue$Node objects from
the mina thread:
TRACE 319549: (thread=200037)
java.util.concurrent.LinkedBlockingQueue$Node.<init>(LinkedBlockingQueue
.java:70)
java.util.concurrent.LinkedBlockingQueue.insert(LinkedBlockingQueue.java
:129)
java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:
335)
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:
653)
org.apache.mina.filter.executor.ExecutorFilter.fireEvent(ExecutorFilter.
java:101)
org.apache.mina.filter.executor.ExecutorFilter.messageSent(ExecutorFilte
r.java:214)
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageSent
(AbstractIoFilterChain.java:320)
org.apache.mina.common.support.AbstractIoFilterChain.access$1200(Abstrac
tIoFilterChain.java:53)
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.message
Sent(AbstractIoFilterChain.java:652)
org.apache.mina.common.support.AbstractIoFilterChain$HeadFilter.messageS
ent(AbstractIoFilterChain.java:503)
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageSent
(AbstractIoFilterChain.java:320)
org.apache.mina.common.support.AbstractIoFilterChain.fireMessageSent(Abs
tractIoFilterChain.java:314)
org.apache.mina.transport.socket.nio.SocketIoProcessor.doFlush(SocketIoP
rocessor.java:358)
org.apache.mina.transport.socket.nio.SocketIoProcessor.doFlush(SocketIoP
rocessor.java:314)
org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600(Socket
IoProcessor.java:44)
org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(Socket
IoProcessor.java:431)
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.j
ava:39)
java.lang.Thread.run(Thread.java:619)
Thank you in advance for helping a Newbie. This is a production
application that I need to get working ASAP. Thanks - Scott
Conexant E-mail Firewall (Conexant.Com) made the following annotations
---------------------------------------------------------------------
********************** Legal Disclaimer ****************************
"This email may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or
distribution by others is strictly prohibited. If you have received the
message in error, please advise the sender by reply email and delete the
message. Thank you."
**********************************************************************
---------------------------------------------------------------------
Re: Out of Memory issues with our mina proxy
Posted by Michael Matovsky <mi...@gmail.com>.
In our testing we also found an additional opportunity for improvement. The
improvement comes from the recognizing a known limitation in Sun's
implementation of SSLEngine. The SSLEngine will return BUFFER_OVERFLOW
unless the buffer size is at least SSLSession.getPacketBufferSize(), which
is 16K. Consequently, regardless of how much data is produced, SSLEngine
always requires 16K buffers. To improve memory consumption, packet buffer
pool can be introduced. The pooled buffers will be used to decode/encode
data from/to SSLEngine. The produced data is then copied into smaller
private buffers and the common buffer is released to the pool.
-----
Thanks,
Michael
michael.matovsky@gmail.com
--
View this message in context: http://www.nabble.com/Out-of-Memory-issues-with-our-mina-proxy-tf4664266s16868.html#a13888281
Sent from the Apache MINA Support Forum mailing list archive at Nabble.com.
Re: Out of Memory issues with our mina proxy
Posted by Michael Matovsky <mi...@gmail.com>.
JIRA issue https://issues.apache.org/jira/browse/DIRMINA-471 DIRMINA-471
was opened.
Thanks,
Michael
Trustin Lee wrote:
>
> I think we can resolve this issue by dynamically re-adjusting the
> buffer capacity. (or allocation-on-demand?) Could you open a JIRA
> issue?
>
--
View this message in context: http://www.nabble.com/Out-of-Memory-issues-with-our-mina-proxy-tf4664266s16868.html#a13633204
Sent from the Apache MINA Support Forum mailing list archive at Nabble.com.
Re: Out of Memory issues with our mina proxy
Posted by Trustin Lee <tr...@gmail.com>.
On 11/7/07, Michael Matovsky <mi...@gmail.com> wrote:
>
> We are observing a significant overhead for SSL connections.
> From our investigation, every SSL connection requires an SSLHandler, which
> contains 3 buffers:
> - Two buffers for encrypted inbound and outbound packets, respectively.
> - One buffer for decrypted application data
>
> The buffer sizes are computed as follows:
> Packet buffer size = the current size of the largest SSL/TLS packet that is
> expected when using this session (SSLSession.packetBufferSize())
> Application buffer size = 2 x packet buffer size
>
> In Sun JSSE the SSLSession.packetBufferSize() equals to 16K, which implies a
> total of 64K SSL buffer space per connection. For 10K active this means that
> 640MB of memory is required just for SSL buffers.
>
> Such overhead really limits the number of concurrent connections an
> application can support. Is there any way to reduce this overhead? Or is
> this a known limitation?
I think we can resolve this issue by dynamically re-adjusting the
buffer capacity. (or allocation-on-demand?) Could you open a JIRA
issue?
http://issues.apache.org/jira/browse/DIRMINA
Thanks,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
RE: Out of Memory issues with our mina proxy
Posted by Michael Matovsky <mi...@gmail.com>.
We are observing a significant overhead for SSL connections.
>From our investigation, every SSL connection requires an SSLHandler, which
contains 3 buffers:
- Two buffers for encrypted inbound and outbound packets, respectively.
- One buffer for decrypted application data
The buffer sizes are computed as follows:
Packet buffer size = the current size of the largest SSL/TLS packet that is
expected when using this session (SSLSession.packetBufferSize())
Application buffer size = 2 x packet buffer size
In Sun JSSE the SSLSession.packetBufferSize() equals to 16K, which implies a
total of 64K SSL buffer space per connection. For 10K active this means that
640MB of memory is required just for SSL buffers.
Such overhead really limits the number of concurrent connections an
application can support. Is there any way to reduce this overhead? Or is
this a known limitation?
Thanks in advance,
Michael
Scott Peters-3 wrote:
>
> ...
> Our program uses 10-15k connections going at the same time (10 with the
> added overhead of SSL). With direct buffers the JVM just used too much
> memory..
> ...
>
--
View this message in context: http://www.nabble.com/Out-of-Memory-issues-with-our-mina-proxy-tf4664266s16868.html#a13616082
Sent from the Apache MINA Support Forum mailing list archive at Nabble.com.