You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by James Gould <jg...@verisign.com> on 2007/07/27 21:50:38 UTC
Deadlock with Mina-based SSL Gateway
I've deployed a Mina-based SSL Gateway that uses a remote client IoAcceptor
with the SSLFilter, called the source channel, linked to an IoConnector that
does not include the SSLFilter, called the destination channel, to TCP
communicate to a back-end Application Server. The IoSession of the source
and destination channels are held in a PipedSession where the source can
push packets to the destination and vice-versa. We're coming across a
synchronization issue, where the Gateway will run for over an hour and than
will lock up. We incorporated the ExecutorFilter for both the source and
destination to reduce the likelihood of the lock up, but it eventually will
lock up the threads in the source and destination thread pools.
We've taken a thread dump of the Gateway in the bad state and the following
is a typical scenario, where the client might be pipelining. In pipelining,
the client could send packets without waiting for the response of each
packet. What we see is the source decrypting the request packet and the
destination attempting to get a lock on the SSLHandler to encrypt the
response back to the client. The issue is that the source is waiting on a
lock in the SSLEngineImpl.unwrap of the Sun JDK 6.0 U2. The thread dump
does not show who has the SSLEngineImpl.unwrap lock, but I believe that
whoever has the lock can't release it due to a deadlock scenario. An
example of the thread dump entries of the scenario are included. I'm
reviewing the SSLFilter and SSLHandler, I'm not sure what code path could
cause such a scenario.
To note, we're running Mina 1.1.0 with
mina-filter-ssl-1.1.1-20070611.145558-5.jar, since using the
mina-filter-1.1.0.jar caused lockup issues with the SSL handshake. We
deployed mina-filter-ssl-1.1.1-20070611.145558-5.jar to address the SSL
handshake induced lockup.
Is anyone else have similar SSL synchronization/deadlock issues? Any ideas
or suggestions is greatly appreciated.
The thread dump does not include what thread has the lock for < 0x94f12f98
>.
"pool-2-thread-9" prio=10 tid=0x083f6c00 nid=0x2b7a runnable
[0x8f445000..0x8f4461b0]
java.lang.Thread.State: RUNNABLE
at
com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:667)
- waiting to lock <0x94f12f98> (a java.lang.Object)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
at org.apache.mina.filter.support.SSLHandler.unwrap(SSLHandler.java:659)
at org.apache.mina.filter.support.SSLHandler.decrypt(SSLHandler.java:465)
at
org.apache.mina.filter.support.SSLHandler.messageReceived(SSLHandler.java:334)
at org.apache.mina.filter.SSLFilter.messageReceived(SSLFilter.java:411)
- locked <0x94f12c10> (a org.apache.mina.filter.support.SSLHandler)
at
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:362)
at
org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:54)
at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:800)
at
org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:83)
at
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:362)
at
org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:54)
at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:800)
at
org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java:247)
at
org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(ExecutorFilter.java:307)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"pool-4-thread-33" prio=10 tid=0x083c3800 nid=0x2bef waiting for monitor
entry [0x8e056000..0x8e0570b0]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.mina.filter.SSLFilter.filterWrite(SSLFilter.java:478)
- waiting to lock <0x94f12c10> (a
org.apache.mina.filter.support.SSLHandler)
at
org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite(AbstractIoFilterChain.java:445)
at
org.apache.mina.common.support.AbstractIoFilterChain.access$1400(AbstractIoFilterChain.java:54)
at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterWrite(AbstractIoFilterChain.java:824)
at
com.verisign.mina.filter.SetSessionIdInLogFilter.filterWrite(SetSessionIdInLogFilter.java:102)
at
org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite(AbstractIoFilterChain.java:445)
at
org.apache.mina.common.support.AbstractIoFilterChain.access$1400(AbstractIoFilterChain.java:54)
at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterWrite(AbstractIoFilterChain.java:824)
at
org.apache.mina.filter.codec.ProtocolCodecFilter.filterWrite(ProtocolCodecFilter.java:227)
at
org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite(AbstractIoFilterChain.java:445)
at
org.apache.mina.common.support.AbstractIoFilterChain.access$1400(AbstractIoFilterChain.java:54)
at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterWrite(AbstractIoFilterChain.java:824)
at
org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.filterWrite(AbstractIoFilterChain.java:727)
at
org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite(AbstractIoFilterChain.java:445)
at
org.apache.mina.common.support.AbstractIoFilterChain.fireFilterWrite(AbstractIoFilterChain.java:436)
at
org.apache.mina.transport.socket.nio.SocketSessionImpl.write0(SocketSessionImpl.java:196)
at
org.apache.mina.common.support.BaseIoSession.write(BaseIoSession.java:149)
at
org.apache.mina.common.support.BaseIoSession.write(BaseIoSession.java:135)
at
com.verisign.cag.protocol.common.PipedSession.write(PipedSession.java:78)
at
com.verisign.cag.protocol.common.SourceSession.write(SourceSession.java:371)
at
com.verisign.cag.protocol.common.SourceSession.messageReceived(SourceSession.java:245)
at
com.verisign.cag.protocol.srsepp.SRSEPPDestinationHandler.messageReceived(SRSEPPDestinationHandler.java:68)
at
com.verisign.cag.protocol.srsepp.SRSEPPDestinationHandler.messageReceived(SRSEPPDestinationHandler.java:48)
at
org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived(AbstractIoFilterChain.java:703)
at
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:362)
at
org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:54)
at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:800)
at
org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimpleProtocolDecoderOutput.java:62)
at
org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:200)
at
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:362)
at
org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:54)
at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:800)
at
com.verisign.mina.filter.SetSessionIdInLogFilter.messageReceived(SetSessionIdInLogFilter.java:84)
at
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:362)
at
org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:54)
at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:800)
at
org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java:247)
at
org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(ExecutorFilter.java:307)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
--
View this message in context: http://www.nabble.com/Deadlock-with-Mina-based-SSL-Gateway-tf4159854s16868.html#a11835745
Sent from the Apache MINA Support Forum mailing list archive at Nabble.com.
Re: Deadlock with Mina-based SSL Gateway
Posted by Maarten Bosteels <mb...@gmail.com>.
On 7/28/07, James Gould <jg...@verisign.com> wrote:
>
>
> Thank you for the quick response. We upgraded the
> mina-filter-ssl-1.1.0.jar
> to mina-filter-ssl-1.1.1-20070611.145558-5.jar to see if it helped a day
> ago. Upgrading to mina-filter-ssl-1.1.1-20070611.145558-5.jar did address
> our SSL Handshake hang issue, but than this issue happened. Do you have
> clients that pipeline requests, which could expose a similar concurrency
> problem? I know that this problem does not occur with non-pipelining
> clients.
James,
What exactly do you mean by "clients that pipeline requests" ?
Do you mean clients that send multiple requests at once without waiting for
their respective responses ?
If that's what you mean, then yes, we most probably have clients doing this.
But we don't process these requests concurrently.
We are running a mina-based EPP server for some time now. I guess you are
familiar with EPP :-)
Maarten
I'm taking your advice and upgrading to the official mina-core-1.1.1.jar and
> mina-filter-ssl-1.1.1.jar to see it that makes a difference. I'll post my
> findings after trying it out.
>
>
> Maarten Bosteels-4 wrote:
> >
> > James,
> >
> > People have reported deadlocks in SSLFilter before.
> > We are using the SSLFilter for one and a half year without issues, but
> > that's probably because
> > we're not writing to the IoSession from multiple threads simultaneously.
> >
> > Trustin has fixed at least two bugs in SSLFilter since 2007-06-11
> > therefor I suggest you try with the latest release mina-1.1.1 which was
> > released on July 18th, 2007.
> >
> > https://issues.apache.org/jira/browse/DIRMINA-390
> > https://issues.apache.org/jira/browse/DIRMINA-402
> >
> > By the way, the version of jconsole that comes with jdk1.6 has a "Detect
> > Deadlock" button, have you tried that ?
> >
> > Maarten
> >
> > On 7/27/07, James Gould <jg...@verisign.com> wrote:
> >>
> >>
> >> I've deployed a Mina-based SSL Gateway that uses a remote client
> >> IoAcceptor
> >> with the SSLFilter, called the source channel, linked to an IoConnector
> >> that
> >> does not include the SSLFilter, called the destination channel, to TCP
> >> communicate to a back-end Application Server. The IoSession of the
> >> source
> >> and destination channels are held in a PipedSession where the source
> can
> >> push packets to the destination and vice-versa. We're coming across a
> >> synchronization issue, where the Gateway will run for over an hour and
> >> than
> >> will lock up. We incorporated the ExecutorFilter for both the source
> and
> >> destination to reduce the likelihood of the lock up, but it eventually
> >> will
> >> lock up the threads in the source and destination thread pools.
> >>
> >> We've taken a thread dump of the Gateway in the bad state and the
> >> following
> >> is a typical scenario, where the client might be pipelining. In
> >> pipelining,
> >> the client could send packets without waiting for the response of each
> >> packet. What we see is the source decrypting the request packet and
> the
> >> destination attempting to get a lock on the SSLHandler to encrypt the
> >> response back to the client. The issue is that the source is waiting
> on
> >> a
> >> lock in the SSLEngineImpl.unwrap of the Sun JDK 6.0 U2. The thread
> dump
> >> does not show who has the SSLEngineImpl.unwrap lock, but I believe that
> >> whoever has the lock can't release it due to a deadlock scenario. An
> >> example of the thread dump entries of the scenario are included. I'm
> >> reviewing the SSLFilter and SSLHandler, I'm not sure what code path
> could
> >> cause such a scenario.
> >>
> >> To note, we're running Mina 1.1.0 with
> >> mina-filter-ssl-1.1.1-20070611.145558-5.jar, since using the
> >> mina-filter-1.1.0.jar caused lockup issues with the SSL handshake. We
> >> deployed mina-filter-ssl-1.1.1-20070611.145558-5.jar to address the SSL
> >> handshake induced lockup.
> >>
> >> Is anyone else have similar SSL synchronization/deadlock issues? Any
> >> ideas
> >> or suggestions is greatly appreciated.
> >>
> >> The thread dump does not include what thread has the lock for <
> >> 0x94f12f98
> >> >.
> >>
> >> "pool-2-thread-9" prio=10 tid=0x083f6c00 nid=0x2b7a runnable
> >> [0x8f445000..0x8f4461b0]
> >> java.lang.Thread.State: RUNNABLE
> >> at
> >> com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java
> :667)
> >> - waiting to lock <0x94f12f98> (a java.lang.Object)
> >> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
> >> at org.apache.mina.filter.support.SSLHandler.unwrap(
> >> SSLHandler.java:659)
> >> at org.apache.mina.filter.support.SSLHandler.decrypt(
> >> SSLHandler.java:465)
> >> at
> >> org.apache.mina.filter.support.SSLHandler.messageReceived(
> SSLHandler.java
> >> :334)
> >> at
> >> org.apache.mina.filter.SSLFilter.messageReceived(SSLFilter.java
> >> :411)
> >> - locked <0x94f12c10> (a org.apache.mina.filter.support.SSLHandler)
> >> at
> >>
> >>
> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived
> >> (AbstractIoFilterChain.java:362)
> >> at
> >> org.apache.mina.common.support.AbstractIoFilterChain.access$1100(
> >> AbstractIoFilterChain.java:54)
> >> at
> >>
> >>
> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived
> >> (AbstractIoFilterChain.java:800)
> >> at
> >> org.apache.mina.common.IoFilterAdapter.messageReceived(
> >> IoFilterAdapter.java:83)
> >> at
> >>
> >>
> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived
> >> (AbstractIoFilterChain.java:362)
> >> at
> >> org.apache.mina.common.support.AbstractIoFilterChain.access$1100(
> >> AbstractIoFilterChain.java:54)
> >> at
> >>
> >>
> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived
> >> (AbstractIoFilterChain.java:800)
> >> at
> >> org.apache.mina.filter.executor.ExecutorFilter.processEvent(
> >> ExecutorFilter.java:247)
> >> at
> >>
> org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(
> >> ExecutorFilter.java:307)
> >> at
> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
> >> ThreadPoolExecutor.java:885)
> >> at
> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java
> >> :907)
> >> at java.lang.Thread.run(Thread.java:619)
> >>
> >> "pool-4-thread-33" prio=10 tid=0x083c3800 nid=0x2bef waiting for
> monitor
> >> entry [0x8e056000..0x8e0570b0]
> >> java.lang.Thread.State: BLOCKED (on object monitor)
> >> at org.apache.mina.filter.SSLFilter.filterWrite(SSLFilter.java
> >> :478)
> >> - waiting to lock <0x94f12c10> (a
> >> org.apache.mina.filter.support.SSLHandler)
> >> at
> >>
> >>
> org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite
> >> (AbstractIoFilterChain.java:445)
> >> at
> >> org.apache.mina.common.support.AbstractIoFilterChain.access$1400(
> >> AbstractIoFilterChain.java:54)
> >> at
> >>
> >>
> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterWrite
> >> (AbstractIoFilterChain.java:824)
> >> at
> >> com.verisign.mina.filter.SetSessionIdInLogFilter.filterWrite(
> >> SetSessionIdInLogFilter.java:102)
> >> at
> >>
> >>
> org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite
> >> (AbstractIoFilterChain.java:445)
> >> at
> >> org.apache.mina.common.support.AbstractIoFilterChain.access$1400(
> >> AbstractIoFilterChain.java:54)
> >> at
> >>
> >>
> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterWrite
> >> (AbstractIoFilterChain.java:824)
> >> at
> >> org.apache.mina.filter.codec.ProtocolCodecFilter.filterWrite(
> >> ProtocolCodecFilter.java:227)
> >> at
> >>
> >>
> org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite
> >> (AbstractIoFilterChain.java:445)
> >> at
> >> org.apache.mina.common.support.AbstractIoFilterChain.access$1400(
> >> AbstractIoFilterChain.java:54)
> >> at
> >>
> >>
> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterWrite
> >> (AbstractIoFilterChain.java:824)
> >> at
> >>
> >>
> org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.filterWrite
> >> (AbstractIoFilterChain.java:727)
> >> at
> >>
> >>
> org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite
> >> (AbstractIoFilterChain.java:445)
> >> at
> >> org.apache.mina.common.support.AbstractIoFilterChain.fireFilterWrite(
> >> AbstractIoFilterChain.java:436)
> >> at
> >> org.apache.mina.transport.socket.nio.SocketSessionImpl.write0(
> >> SocketSessionImpl.java:196)
> >> at
> >> org.apache.mina.common.support.BaseIoSession.write(BaseIoSession.java
> :149)
> >> at
> >> org.apache.mina.common.support.BaseIoSession.write(BaseIoSession.java
> :135)
> >> at
> >> com.verisign.cag.protocol.common.PipedSession.write(PipedSession.java
> :78)
> >> at
> >> com.verisign.cag.protocol.common.SourceSession.write(SourceSession.java
> >> :371)
> >> at
> >> com.verisign.cag.protocol.common.SourceSession.messageReceived(
> >> SourceSession.java:245)
> >> at
> >>
> com.verisign.cag.protocol.srsepp.SRSEPPDestinationHandler.messageReceived(
> >> SRSEPPDestinationHandler.java:68)
> >> at
> >>
> com.verisign.cag.protocol.srsepp.SRSEPPDestinationHandler.messageReceived(
> >> SRSEPPDestinationHandler.java:48)
> >> at
> >>
> >>
> org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived
> >> (AbstractIoFilterChain.java:703)
> >> at
> >>
> >>
> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived
> >> (AbstractIoFilterChain.java:362)
> >> at
> >> org.apache.mina.common.support.AbstractIoFilterChain.access$1100(
> >> AbstractIoFilterChain.java:54)
> >> at
> >>
> >>
> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived
> >> (AbstractIoFilterChain.java:800)
> >> at
> >> org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(
> >> SimpleProtocolDecoderOutput.java:62)
> >> at
> >> org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(
> >> ProtocolCodecFilter.java:200)
> >> at
> >>
> >>
> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived
> >> (AbstractIoFilterChain.java:362)
> >> at
> >> org.apache.mina.common.support.AbstractIoFilterChain.access$1100(
> >> AbstractIoFilterChain.java:54)
> >> at
> >>
> >>
> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived
> >> (AbstractIoFilterChain.java:800)
> >> at
> >> com.verisign.mina.filter.SetSessionIdInLogFilter.messageReceived(
> >> SetSessionIdInLogFilter.java:84)
> >> at
> >>
> >>
> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived
> >> (AbstractIoFilterChain.java:362)
> >> at
> >> org.apache.mina.common.support.AbstractIoFilterChain.access$1100(
> >> AbstractIoFilterChain.java:54)
> >> at
> >>
> >>
> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived
> >> (AbstractIoFilterChain.java:800)
> >> at
> >> org.apache.mina.filter.executor.ExecutorFilter.processEvent(
> >> ExecutorFilter.java:247)
> >> at
> >>
> org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(
> >> ExecutorFilter.java:307)
> >> at
> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
> >> ThreadPoolExecutor.java:885)
> >> at
> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java
> >> :907)
> >> at java.lang.Thread.run(Thread.java:619)
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Deadlock-with-Mina-based-SSL-Gateway-tf4159854s16868.html#a11835745
> >> Sent from the Apache MINA Support Forum mailing list archive at
> >> Nabble.com
> >> .
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Deadlock-with-Mina-based-SSL-Gateway-tf4159854s16868.html#a11837353
> Sent from the Apache MINA Support Forum mailing list archive at Nabble.com
> .
>
>
Re: Deadlock with Mina-based SSL Gateway
Posted by Trustin Lee <tr...@gmail.com>.
On 7/28/07, Trustin Lee <tr...@gmail.com> wrote:
> On 7/28/07, James Gould <jg...@verisign.com> wrote:
> >
> > Thank you for the quick response. We upgraded the mina-filter-ssl-1.1.0.jar
> > to mina-filter-ssl-1.1.1-20070611.145558-5.jar to see if it helped a day
> > ago. Upgrading to mina-filter-ssl-1.1.1-20070611.145558-5.jar did address
> > our SSL Handshake hang issue, but than this issue happened. Do you have
> > clients that pipeline requests, which could expose a similar concurrency
> > problem? I know that this problem does not occur with non-pipelining
> > clients.
> >
> > I'm taking your advice and upgrading to the official mina-core-1.1.1.jar and
> > mina-filter-ssl-1.1.1.jar to see it that makes a difference. I'll post my
> > findings after trying it out.
>
> I think upgrading mina-core-1.1.1 won't make a difference IMO.
Ahh, please upgrade to mina-filter-ssl-1.1.1.jar. It might make a difference.
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
Re: Deadlock with Mina-based SSL Gateway
Posted by Trustin Lee <tr...@gmail.com>.
On 7/28/07, James Gould <jg...@verisign.com> wrote:
>
> Thank you for the quick response. We upgraded the mina-filter-ssl-1.1.0.jar
> to mina-filter-ssl-1.1.1-20070611.145558-5.jar to see if it helped a day
> ago. Upgrading to mina-filter-ssl-1.1.1-20070611.145558-5.jar did address
> our SSL Handshake hang issue, but than this issue happened. Do you have
> clients that pipeline requests, which could expose a similar concurrency
> problem? I know that this problem does not occur with non-pipelining
> clients.
>
> I'm taking your advice and upgrading to the official mina-core-1.1.1.jar and
> mina-filter-ssl-1.1.1.jar to see it that makes a difference. I'll post my
> findings after trying it out.
I think upgrading mina-core-1.1.1 won't make a difference IMO.
I can't find any SSLFilter problem from the thread dump either.
SSLEngine seems to acquire lock internally, but it's SUN
implementation so we don't have any control over it. I think it
doesn't matter because we always acquire lock for SSLHandler before
accessing SSLEngine.
What's weird is that SSLEngine is blocked by something. If we always
acquire a lock for SSLHandler before accessing SSLEngine, SSLEngine
should never wait for a lock. Let us review SSLFilter and
SSLHandler...
Thanks,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
Re: Deadlock with Mina-based SSL Gateway
Posted by James Gould <jg...@verisign.com>.
Thank you for the quick response. We upgraded the mina-filter-ssl-1.1.0.jar
to mina-filter-ssl-1.1.1-20070611.145558-5.jar to see if it helped a day
ago. Upgrading to mina-filter-ssl-1.1.1-20070611.145558-5.jar did address
our SSL Handshake hang issue, but than this issue happened. Do you have
clients that pipeline requests, which could expose a similar concurrency
problem? I know that this problem does not occur with non-pipelining
clients.
I'm taking your advice and upgrading to the official mina-core-1.1.1.jar and
mina-filter-ssl-1.1.1.jar to see it that makes a difference. I'll post my
findings after trying it out.
Maarten Bosteels-4 wrote:
>
> James,
>
> People have reported deadlocks in SSLFilter before.
> We are using the SSLFilter for one and a half year without issues, but
> that's probably because
> we're not writing to the IoSession from multiple threads simultaneously.
>
> Trustin has fixed at least two bugs in SSLFilter since 2007-06-11
> therefor I suggest you try with the latest release mina-1.1.1 which was
> released on July 18th, 2007.
>
> https://issues.apache.org/jira/browse/DIRMINA-390
> https://issues.apache.org/jira/browse/DIRMINA-402
>
> By the way, the version of jconsole that comes with jdk1.6 has a "Detect
> Deadlock" button, have you tried that ?
>
> Maarten
>
> On 7/27/07, James Gould <jg...@verisign.com> wrote:
>>
>>
>> I've deployed a Mina-based SSL Gateway that uses a remote client
>> IoAcceptor
>> with the SSLFilter, called the source channel, linked to an IoConnector
>> that
>> does not include the SSLFilter, called the destination channel, to TCP
>> communicate to a back-end Application Server. The IoSession of the
>> source
>> and destination channels are held in a PipedSession where the source can
>> push packets to the destination and vice-versa. We're coming across a
>> synchronization issue, where the Gateway will run for over an hour and
>> than
>> will lock up. We incorporated the ExecutorFilter for both the source and
>> destination to reduce the likelihood of the lock up, but it eventually
>> will
>> lock up the threads in the source and destination thread pools.
>>
>> We've taken a thread dump of the Gateway in the bad state and the
>> following
>> is a typical scenario, where the client might be pipelining. In
>> pipelining,
>> the client could send packets without waiting for the response of each
>> packet. What we see is the source decrypting the request packet and the
>> destination attempting to get a lock on the SSLHandler to encrypt the
>> response back to the client. The issue is that the source is waiting on
>> a
>> lock in the SSLEngineImpl.unwrap of the Sun JDK 6.0 U2. The thread dump
>> does not show who has the SSLEngineImpl.unwrap lock, but I believe that
>> whoever has the lock can't release it due to a deadlock scenario. An
>> example of the thread dump entries of the scenario are included. I'm
>> reviewing the SSLFilter and SSLHandler, I'm not sure what code path could
>> cause such a scenario.
>>
>> To note, we're running Mina 1.1.0 with
>> mina-filter-ssl-1.1.1-20070611.145558-5.jar, since using the
>> mina-filter-1.1.0.jar caused lockup issues with the SSL handshake. We
>> deployed mina-filter-ssl-1.1.1-20070611.145558-5.jar to address the SSL
>> handshake induced lockup.
>>
>> Is anyone else have similar SSL synchronization/deadlock issues? Any
>> ideas
>> or suggestions is greatly appreciated.
>>
>> The thread dump does not include what thread has the lock for <
>> 0x94f12f98
>> >.
>>
>> "pool-2-thread-9" prio=10 tid=0x083f6c00 nid=0x2b7a runnable
>> [0x8f445000..0x8f4461b0]
>> java.lang.Thread.State: RUNNABLE
>> at
>> com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:667)
>> - waiting to lock <0x94f12f98> (a java.lang.Object)
>> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
>> at org.apache.mina.filter.support.SSLHandler.unwrap(
>> SSLHandler.java:659)
>> at org.apache.mina.filter.support.SSLHandler.decrypt(
>> SSLHandler.java:465)
>> at
>> org.apache.mina.filter.support.SSLHandler.messageReceived(SSLHandler.java
>> :334)
>> at
>> org.apache.mina.filter.SSLFilter.messageReceived(SSLFilter.java
>> :411)
>> - locked <0x94f12c10> (a org.apache.mina.filter.support.SSLHandler)
>> at
>>
>> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived
>> (AbstractIoFilterChain.java:362)
>> at
>> org.apache.mina.common.support.AbstractIoFilterChain.access$1100(
>> AbstractIoFilterChain.java:54)
>> at
>>
>> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived
>> (AbstractIoFilterChain.java:800)
>> at
>> org.apache.mina.common.IoFilterAdapter.messageReceived(
>> IoFilterAdapter.java:83)
>> at
>>
>> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived
>> (AbstractIoFilterChain.java:362)
>> at
>> org.apache.mina.common.support.AbstractIoFilterChain.access$1100(
>> AbstractIoFilterChain.java:54)
>> at
>>
>> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived
>> (AbstractIoFilterChain.java:800)
>> at
>> org.apache.mina.filter.executor.ExecutorFilter.processEvent(
>> ExecutorFilter.java:247)
>> at
>> org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(
>> ExecutorFilter.java:307)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
>> ThreadPoolExecutor.java:885)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
>> :907)
>> at java.lang.Thread.run(Thread.java:619)
>>
>> "pool-4-thread-33" prio=10 tid=0x083c3800 nid=0x2bef waiting for monitor
>> entry [0x8e056000..0x8e0570b0]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at org.apache.mina.filter.SSLFilter.filterWrite(SSLFilter.java
>> :478)
>> - waiting to lock <0x94f12c10> (a
>> org.apache.mina.filter.support.SSLHandler)
>> at
>>
>> org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite
>> (AbstractIoFilterChain.java:445)
>> at
>> org.apache.mina.common.support.AbstractIoFilterChain.access$1400(
>> AbstractIoFilterChain.java:54)
>> at
>>
>> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterWrite
>> (AbstractIoFilterChain.java:824)
>> at
>> com.verisign.mina.filter.SetSessionIdInLogFilter.filterWrite(
>> SetSessionIdInLogFilter.java:102)
>> at
>>
>> org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite
>> (AbstractIoFilterChain.java:445)
>> at
>> org.apache.mina.common.support.AbstractIoFilterChain.access$1400(
>> AbstractIoFilterChain.java:54)
>> at
>>
>> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterWrite
>> (AbstractIoFilterChain.java:824)
>> at
>> org.apache.mina.filter.codec.ProtocolCodecFilter.filterWrite(
>> ProtocolCodecFilter.java:227)
>> at
>>
>> org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite
>> (AbstractIoFilterChain.java:445)
>> at
>> org.apache.mina.common.support.AbstractIoFilterChain.access$1400(
>> AbstractIoFilterChain.java:54)
>> at
>>
>> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterWrite
>> (AbstractIoFilterChain.java:824)
>> at
>>
>> org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.filterWrite
>> (AbstractIoFilterChain.java:727)
>> at
>>
>> org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite
>> (AbstractIoFilterChain.java:445)
>> at
>> org.apache.mina.common.support.AbstractIoFilterChain.fireFilterWrite(
>> AbstractIoFilterChain.java:436)
>> at
>> org.apache.mina.transport.socket.nio.SocketSessionImpl.write0(
>> SocketSessionImpl.java:196)
>> at
>> org.apache.mina.common.support.BaseIoSession.write(BaseIoSession.java:149)
>> at
>> org.apache.mina.common.support.BaseIoSession.write(BaseIoSession.java:135)
>> at
>> com.verisign.cag.protocol.common.PipedSession.write(PipedSession.java:78)
>> at
>> com.verisign.cag.protocol.common.SourceSession.write(SourceSession.java
>> :371)
>> at
>> com.verisign.cag.protocol.common.SourceSession.messageReceived(
>> SourceSession.java:245)
>> at
>> com.verisign.cag.protocol.srsepp.SRSEPPDestinationHandler.messageReceived(
>> SRSEPPDestinationHandler.java:68)
>> at
>> com.verisign.cag.protocol.srsepp.SRSEPPDestinationHandler.messageReceived(
>> SRSEPPDestinationHandler.java:48)
>> at
>>
>> org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived
>> (AbstractIoFilterChain.java:703)
>> at
>>
>> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived
>> (AbstractIoFilterChain.java:362)
>> at
>> org.apache.mina.common.support.AbstractIoFilterChain.access$1100(
>> AbstractIoFilterChain.java:54)
>> at
>>
>> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived
>> (AbstractIoFilterChain.java:800)
>> at
>> org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(
>> SimpleProtocolDecoderOutput.java:62)
>> at
>> org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(
>> ProtocolCodecFilter.java:200)
>> at
>>
>> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived
>> (AbstractIoFilterChain.java:362)
>> at
>> org.apache.mina.common.support.AbstractIoFilterChain.access$1100(
>> AbstractIoFilterChain.java:54)
>> at
>>
>> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived
>> (AbstractIoFilterChain.java:800)
>> at
>> com.verisign.mina.filter.SetSessionIdInLogFilter.messageReceived(
>> SetSessionIdInLogFilter.java:84)
>> at
>>
>> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived
>> (AbstractIoFilterChain.java:362)
>> at
>> org.apache.mina.common.support.AbstractIoFilterChain.access$1100(
>> AbstractIoFilterChain.java:54)
>> at
>>
>> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived
>> (AbstractIoFilterChain.java:800)
>> at
>> org.apache.mina.filter.executor.ExecutorFilter.processEvent(
>> ExecutorFilter.java:247)
>> at
>> org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(
>> ExecutorFilter.java:307)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
>> ThreadPoolExecutor.java:885)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
>> :907)
>> at java.lang.Thread.run(Thread.java:619)
>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Deadlock-with-Mina-based-SSL-Gateway-tf4159854s16868.html#a11835745
>> Sent from the Apache MINA Support Forum mailing list archive at
>> Nabble.com
>> .
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Deadlock-with-Mina-based-SSL-Gateway-tf4159854s16868.html#a11837353
Sent from the Apache MINA Support Forum mailing list archive at Nabble.com.
Re: Deadlock with Mina-based SSL Gateway
Posted by Maarten Bosteels <mb...@gmail.com>.
James,
People have reported deadlocks in SSLFilter before.
We are using the SSLFilter for one and a half year without issues, but
that's probably because
we're not writing to the IoSession from multiple threads simultaneously.
Trustin has fixed at least two bugs in SSLFilter since 2007-06-11
therefor I suggest you try with the latest release mina-1.1.1 which was
released on July 18th, 2007.
https://issues.apache.org/jira/browse/DIRMINA-390
https://issues.apache.org/jira/browse/DIRMINA-402
By the way, the version of jconsole that comes with jdk1.6 has a "Detect
Deadlock" button, have you tried that ?
Maarten
On 7/27/07, James Gould <jg...@verisign.com> wrote:
>
>
> I've deployed a Mina-based SSL Gateway that uses a remote client
> IoAcceptor
> with the SSLFilter, called the source channel, linked to an IoConnector
> that
> does not include the SSLFilter, called the destination channel, to TCP
> communicate to a back-end Application Server. The IoSession of the source
> and destination channels are held in a PipedSession where the source can
> push packets to the destination and vice-versa. We're coming across a
> synchronization issue, where the Gateway will run for over an hour and
> than
> will lock up. We incorporated the ExecutorFilter for both the source and
> destination to reduce the likelihood of the lock up, but it eventually
> will
> lock up the threads in the source and destination thread pools.
>
> We've taken a thread dump of the Gateway in the bad state and the
> following
> is a typical scenario, where the client might be pipelining. In
> pipelining,
> the client could send packets without waiting for the response of each
> packet. What we see is the source decrypting the request packet and the
> destination attempting to get a lock on the SSLHandler to encrypt the
> response back to the client. The issue is that the source is waiting on a
> lock in the SSLEngineImpl.unwrap of the Sun JDK 6.0 U2. The thread dump
> does not show who has the SSLEngineImpl.unwrap lock, but I believe that
> whoever has the lock can't release it due to a deadlock scenario. An
> example of the thread dump entries of the scenario are included. I'm
> reviewing the SSLFilter and SSLHandler, I'm not sure what code path could
> cause such a scenario.
>
> To note, we're running Mina 1.1.0 with
> mina-filter-ssl-1.1.1-20070611.145558-5.jar, since using the
> mina-filter-1.1.0.jar caused lockup issues with the SSL handshake. We
> deployed mina-filter-ssl-1.1.1-20070611.145558-5.jar to address the SSL
> handshake induced lockup.
>
> Is anyone else have similar SSL synchronization/deadlock issues? Any
> ideas
> or suggestions is greatly appreciated.
>
> The thread dump does not include what thread has the lock for < 0x94f12f98
> >.
>
> "pool-2-thread-9" prio=10 tid=0x083f6c00 nid=0x2b7a runnable
> [0x8f445000..0x8f4461b0]
> java.lang.Thread.State: RUNNABLE
> at
> com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:667)
> - waiting to lock <0x94f12f98> (a java.lang.Object)
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
> at org.apache.mina.filter.support.SSLHandler.unwrap(
> SSLHandler.java:659)
> at org.apache.mina.filter.support.SSLHandler.decrypt(
> SSLHandler.java:465)
> at
> org.apache.mina.filter.support.SSLHandler.messageReceived(SSLHandler.java
> :334)
> at org.apache.mina.filter.SSLFilter.messageReceived(SSLFilter.java
> :411)
> - locked <0x94f12c10> (a org.apache.mina.filter.support.SSLHandler)
> at
>
> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived
> (AbstractIoFilterChain.java:362)
> at
> org.apache.mina.common.support.AbstractIoFilterChain.access$1100(
> AbstractIoFilterChain.java:54)
> at
>
> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived
> (AbstractIoFilterChain.java:800)
> at
> org.apache.mina.common.IoFilterAdapter.messageReceived(
> IoFilterAdapter.java:83)
> at
>
> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived
> (AbstractIoFilterChain.java:362)
> at
> org.apache.mina.common.support.AbstractIoFilterChain.access$1100(
> AbstractIoFilterChain.java:54)
> at
>
> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived
> (AbstractIoFilterChain.java:800)
> at
> org.apache.mina.filter.executor.ExecutorFilter.processEvent(
> ExecutorFilter.java:247)
> at
> org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(
> ExecutorFilter.java:307)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
> ThreadPoolExecutor.java:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
> :907)
> at java.lang.Thread.run(Thread.java:619)
>
> "pool-4-thread-33" prio=10 tid=0x083c3800 nid=0x2bef waiting for monitor
> entry [0x8e056000..0x8e0570b0]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.mina.filter.SSLFilter.filterWrite(SSLFilter.java
> :478)
> - waiting to lock <0x94f12c10> (a
> org.apache.mina.filter.support.SSLHandler)
> at
>
> org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite
> (AbstractIoFilterChain.java:445)
> at
> org.apache.mina.common.support.AbstractIoFilterChain.access$1400(
> AbstractIoFilterChain.java:54)
> at
>
> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterWrite
> (AbstractIoFilterChain.java:824)
> at
> com.verisign.mina.filter.SetSessionIdInLogFilter.filterWrite(
> SetSessionIdInLogFilter.java:102)
> at
>
> org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite
> (AbstractIoFilterChain.java:445)
> at
> org.apache.mina.common.support.AbstractIoFilterChain.access$1400(
> AbstractIoFilterChain.java:54)
> at
>
> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterWrite
> (AbstractIoFilterChain.java:824)
> at
> org.apache.mina.filter.codec.ProtocolCodecFilter.filterWrite(
> ProtocolCodecFilter.java:227)
> at
>
> org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite
> (AbstractIoFilterChain.java:445)
> at
> org.apache.mina.common.support.AbstractIoFilterChain.access$1400(
> AbstractIoFilterChain.java:54)
> at
>
> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterWrite
> (AbstractIoFilterChain.java:824)
> at
>
> org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.filterWrite
> (AbstractIoFilterChain.java:727)
> at
>
> org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite
> (AbstractIoFilterChain.java:445)
> at
> org.apache.mina.common.support.AbstractIoFilterChain.fireFilterWrite(
> AbstractIoFilterChain.java:436)
> at
> org.apache.mina.transport.socket.nio.SocketSessionImpl.write0(
> SocketSessionImpl.java:196)
> at
> org.apache.mina.common.support.BaseIoSession.write(BaseIoSession.java:149)
> at
> org.apache.mina.common.support.BaseIoSession.write(BaseIoSession.java:135)
> at
> com.verisign.cag.protocol.common.PipedSession.write(PipedSession.java:78)
> at
> com.verisign.cag.protocol.common.SourceSession.write(SourceSession.java
> :371)
> at
> com.verisign.cag.protocol.common.SourceSession.messageReceived(
> SourceSession.java:245)
> at
> com.verisign.cag.protocol.srsepp.SRSEPPDestinationHandler.messageReceived(
> SRSEPPDestinationHandler.java:68)
> at
> com.verisign.cag.protocol.srsepp.SRSEPPDestinationHandler.messageReceived(
> SRSEPPDestinationHandler.java:48)
> at
>
> org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived
> (AbstractIoFilterChain.java:703)
> at
>
> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived
> (AbstractIoFilterChain.java:362)
> at
> org.apache.mina.common.support.AbstractIoFilterChain.access$1100(
> AbstractIoFilterChain.java:54)
> at
>
> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived
> (AbstractIoFilterChain.java:800)
> at
> org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(
> SimpleProtocolDecoderOutput.java:62)
> at
> org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(
> ProtocolCodecFilter.java:200)
> at
>
> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived
> (AbstractIoFilterChain.java:362)
> at
> org.apache.mina.common.support.AbstractIoFilterChain.access$1100(
> AbstractIoFilterChain.java:54)
> at
>
> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived
> (AbstractIoFilterChain.java:800)
> at
> com.verisign.mina.filter.SetSessionIdInLogFilter.messageReceived(
> SetSessionIdInLogFilter.java:84)
> at
>
> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived
> (AbstractIoFilterChain.java:362)
> at
> org.apache.mina.common.support.AbstractIoFilterChain.access$1100(
> AbstractIoFilterChain.java:54)
> at
>
> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived
> (AbstractIoFilterChain.java:800)
> at
> org.apache.mina.filter.executor.ExecutorFilter.processEvent(
> ExecutorFilter.java:247)
> at
> org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(
> ExecutorFilter.java:307)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
> ThreadPoolExecutor.java:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
> :907)
> at java.lang.Thread.run(Thread.java:619)
>
>
> --
> View this message in context:
> http://www.nabble.com/Deadlock-with-Mina-based-SSL-Gateway-tf4159854s16868.html#a11835745
> Sent from the Apache MINA Support Forum mailing list archive at Nabble.com
> .
>
>