You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by Christoph John <ch...@macd.com> on 2017/10/09 09:08:00 UTC
Re: leaking NioProcessors/NioSocketConnectors hanging in call to
dispose
Hi,
I have another question regarding this one. There is
https://issues.apache.org/jira/browse/DIRMINA-1060 which also sounds a little like the problem I'm
having. When the connectors are hanging in the call to dispose() then there always is an
accompanying NioProcessor which is hanging in select().
Example:
"NioProcessor-60" #100328 prio=5 os_prio=0 tid=0x00007f2a10003000 nid=0x2e71 runnable
[0x00007f2a388b1000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e239c118> (a sun.nio.ch.Util$3)
- locked <0x00000000e239c108> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e239bed0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:98)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1075)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"NioSocketConnector-38" #100326 prio=5 os_prio=0 tid=0x00007f2a3001d800 nid=0x2e6f in Object.wait()
[0x00007f2a1f2d3000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.mina.core.future.DefaultIoFuture.await0(DefaultIoFuture.java:209)
- locked <0x00000000e246ae08> (a org.apache.mina.core.future.DefaultIoFuture)
at org.apache.mina.core.future.DefaultIoFuture.awaitUninterruptibly(DefaultIoFuture.java:141)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.dispose(AbstractPollingIoProcessor.java:188)
at org.apache.mina.core.service.SimpleIoProcessorPool.dispose(SimpleIoProcessorPool.java:329)
- locked <0x00000000e246ae40> (a java.lang.Object)
at
org.apache.mina.core.polling.AbstractPollingIoConnector$Connector.run(AbstractPollingIoConnector.java:582)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
At first I thought that this was related to https://issues.apache.org/jira/browse/DIRMINA-1059. In
that ticket the synchronization was improved. However, I am also running into the problem with a
build of 2.0.17-SNAPSHOT where DIRMINA-1059 was solved.
So my only hope was DIRMINA-1060 ;) Could this improve the situation?
Thanks,
Chris.
--
Christoph John
Development & Support
Direct: +49 241 557080-28
Mailto:Christoph.John@macd.com
http://www.macd.com <http://www.macd.com/>
----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
MACD GmbH
Oppenhoffallee 103
D-52066 Aachen
Tel: +49 241 557080-0 | Fax: +49 241 557080-10
Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663
Geschäftsführer: George Macdonald
----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
take care of the environment - print only if necessary
Re: leaking NioProcessors/NioSocketConnectors hanging in call to
dispose
Posted by Emmanuel Lécharny <el...@gmail.com>.
Hi Christophe, Jonathan,
when dispose() is called, is first set the 'disposing' flag, which will
be checked in the IoProcessor select loop. The disposalFuture is
signaled after the select loop, in a try... catch... finally.
That supposes the loop ends at some point, and the only reason for the
infinite loop to exit is that we have no session associated with the
processor. Session are removed from the IoProcessor by this piece of code :
if (isDisposing()) { // Clearly, the
'disposing' flag is set
boolean hasKeys = false;
for (Iterator<S> i = allSessions(); i.hasNext();) {
IoSession session = i.next();
if (session.isActive()) { // A
session is active if its selectionKey is accepting OP_READ ror OP_WRITE
scheduleRemove((S) session);
hasKeys = true;
}
}
if (hasKeys) {
wakeup();
}
}
The only reason I see for a session to not be removed from the
IoProcessor is that it's not active anymore. That may be because its
channel has been closed, but the session has not been removed from the
IoProcessor. This is soemthing to investigate (typically, what happens
when teh HS fails...)
Le 10/10/2017 à 10:49, Christoph John a écrit :
> Hi,
>
> thanks for your reply.
> In fact it is hanging forever, i.e. until the process stops. I have
> attached the original message I've sent to the mailing list. It only
> does occur sometimes for SSL connections with a failing handshake.
> Unfortunately I have no reproducable example for MINA itself. I could
> probably put something together for QuickFIX/J (the open source
> project I am working on).
>
> My OS is Ubuntu 14.04.5, JDK1.8_144 and the problem appears not so
> often on my machine but almost every time on the TravisCI build server
> (https://travis-ci.org/quickfix-j/quickfixj/builds/283210509). As a
> result, some of the SSL related tests are failing. TravisCI has almost
> similar setup with JDK1.8_144 and Debian Linux.
>
> What would be a good starting point to create a test? I see that there
> is an SslTest in the mina-core module. So I probably have to change
> that test to repeatedly connect and get a handshake exception
> everytime and then take a number of stack traces.
>
> Thanks,
> Chris.
>
>
>
>
> On 09/10/17 14:51, Jonathan Valliere wrote:
>> What OS / Java Version / etc; Do you have a reproducible example?
>>
>> On Mon, Oct 9, 2017 at 8:34 AM, Jonathan Valliere
>> <jon.valliere@emoten.com <ma...@emoten.com>> wrote:
>>
>> Let me know if its hanging more than 1s
>>
>> On Mon, Oct 9, 2017 at 5:08 AM, Christoph John
>> <christoph.john@macd.com
>> <ma...@macd.com>> wrote:
>>
>> Hi,
>>
>> I have another question regarding this one. There is
>> https://issues.apache.org/jira/browse/DIRMINA-1060
>> <https://issues.apache.org/jira/browse/DIRMINA-1060> which
>> also sounds a little like the
>> problem I'm having. When the connectors are hanging in the
>> call to dispose() then there
>> always is an accompanying NioProcessor which is hanging in
>> select().
>>
>> Example:
>> "NioProcessor-60" #100328 prio=5 os_prio=0
>> tid=0x00007f2a10003000 nid=0x2e71 runnable
>> [0x00007f2a388b1000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x00000000e239c118> (a sun.nio.ch.Util$3)
>> - locked <0x00000000e239c108> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked <0x00000000e239bed0> (a
>> sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:98)
>> at
>>
>> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1075)
>> at
>> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>> "NioSocketConnector-38" #100326 prio=5 os_prio=0
>> tid=0x00007f2a3001d800 nid=0x2e6f in
>> Object.wait() [0x00007f2a1f2d3000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at org.apache.mina.core.future.De
>>
>> <http://org.apache.mina.core.future.De>faultIoFuture.await0(DefaultIoFuture.java:209)
>> - locked <0x00000000e246ae08> (a
>> org.apache.mina.core.future.De
>> <http://org.apache.mina.core.future.De>faultIoFuture)
>> at org.apache.mina.core.future.De
>>
>> <http://org.apache.mina.core.future.De>faultIoFuture.awaitUninterruptibly(DefaultIoFuture.java:141)
>> at
>>
>> org.apache.mina.core.polling.AbstractPollingIoProcessor.dispose(AbstractPollingIoProcessor.java:188)
>> at
>>
>> org.apache.mina.core.service.SimpleIoProcessorPool.dispose(SimpleIoProcessorPool.java:329)
>> - locked <0x00000000e246ae40> (a java.lang.Object)
>> at
>>
>> org.apache.mina.core.polling.AbstractPollingIoConnector$Connector.run(AbstractPollingIoConnector.java:582)
>> at
>> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>> At first I thought that this was related to
>> https://issues.apache.org/jira/browse/DIRMINA-1059
>> <https://issues.apache.org/jira/browse/DIRMINA-1059>. In that
>> ticket the synchronization
>> was improved. However, I am also running into the problem
>> with a build of 2.0.17-SNAPSHOT
>> where DIRMINA-1059 was solved.
>>
>> So my only hope was DIRMINA-1060 ;) Could this improve the
>> situation?
>>
>> Thanks,
>> Chris.
>>
>>
>> -- Christoph John
>> Development & Support
>> Direct: +49 241 557080-28 <tel:%2B49%20241%20557080-28>
>> Mailto:Christoph.John@macd.com <ma...@macd.com>
>>
>>
>>
>> http://www.macd.com <http://www.macd.com/>
>>
>> ----------------------------------------------------------------------------------------------------
>>
>>
>> ----------------------------------------------------------------------------------------------------
>> MACD GmbH
>> Oppenhoffallee 103
>> <https://maps.google.com/?q=Oppenhoffallee+103&entry=gmail&source=g>
>> D-52066 Aachen
>> Tel: +49 241 557080-0 <tel:%2B49%20241%20557080-0> | Fax: +49
>> 241 557080-10
>> <tel:%2B49%20241%20557080-10>
>> Amtsgericht Aachen: HRB 8151
>> Ust.-Id: DE 813021663
>>
>> Geschäftsführer: George Macdonald
>>
>> ----------------------------------------------------------------------------------------------------
>>
>>
>> ----------------------------------------------------------------------------------------------------
>>
>> take care of the environment - print only if necessary
>>
>>
>>
>
--
Emmanuel Lecharny
Symas.com
directory.apache.org
Re: leaking NioProcessors/NioSocketConnectors hanging in call to
dispose - DIRMINA-1076
Posted by Christoph John <ch...@macd.com>.
Of course, no problem. Thanks and cheers! ;)
Chris.
On 12/02/18 16:24, Jonathan Valliere wrote:
> Brilliant, that session counter assertion is already paying off. I won’t be able to look at it
> before next weekend. Regular work-work during the week and all. Cheers!
>
> On Mon, Feb 12, 2018 at 8:02 AM, Christoph John <christoph.john@macd.com
> <ma...@macd.com>> wrote:
>
> Hi Jonathan,
>
> thanks again for your work on DIRMINA-1076. I think I might have encountered a related case
> and openend DIRMINA-1077for it.
>
> Thanks in advance and best regards,
> Christoph.
>
>
>
--
Christoph John
Development & Support
T +49 241 557080-28
christoph.john@macd.com
MACD GmbH
Oppenhoffallee 103
D-52066 Aachen
www.macd.com
Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663
Geschäftsführer: George Macdonald
Re: leaking NioProcessors/NioSocketConnectors hanging in call to
dispose - DIRMINA-1076
Posted by Jonathan Valliere <jo...@emoten.com>.
Brilliant, that session counter assertion is already paying off. I won’t
be able to look at it before next weekend. Regular work-work during the
week and all. Cheers!
On Mon, Feb 12, 2018 at 8:02 AM, Christoph John <ch...@macd.com>
wrote:
> Hi Jonathan,
>
> thanks again for your work on DIRMINA-1076. I think I might have
> encountered a related case and openend DIRMINA-1077for it.
>
> Thanks in advance and best regards,
> Christoph.
>
>
>
> On 09/02/18 15:22, Christoph John wrote:
>
>> OK, no problem. Just created https://issues.apache.org/jira
>> /browse/DIRMINA-1076
>>
>> Thanks in advanceand best regards,
>> Chris.
>>
>>
>> On 09/02/18 15:11, Jonathan Valliere wrote:
>>
>>> If there isn’t a ticket in JIRA, please create one. I’d prefer to work
>>> there instead of email.
>>>
>>>
>>> On Fri, Feb 9, 2018 at 8:56 AM Christoph John <christoph.john@macd.com
>>> <ma...@macd.com>> wrote:
>>>
>>> Yes, that is the patch that I attached. It is simply executing the
>>> test in
>>> AbstractIoServiceTest in
>>> a loop.
>>>
>>> Thanks,
>>> Chris.
>>>
>>> On 09/02/18 14:47, Jonathan Valliere wrote:
>>> > It’s been a while, I forget, is there code to run the brute force
>>> test available?
>>> >
>>> > On Fri, Feb 9, 2018 at 7:56 AM Christoph John <
>>> christoph.john@macd.com
>>> <ma...@macd.com>
>>> > <mailto:christoph.john@macd.com <ma...@macd.com>>>
>>> wrote:
>>> >
>>> > Hi againafter some time. ;)
>>> >
>>> > I was now able to reproduce the problem with a MINA test. Or
>>> let's say I did the brute-force
>>> > approach by re-running one test in an endless loop.
>>> > I have attached apatch of the test (against
>>> https://github.com/apache/mina/tree/2.0
>>> <https://github.com/apache/mina/tree/2.0>) and a
>>> > stack trace. After a few loops the test is stuck. You can see
>>> a lot of threads hanging in
>>> > dispose() and the test is stuck when it tries to dispose the
>>> acceptor.
>>> >
>>> > What is a little strange is that the javadoc says that
>>> connector.dispose(TRUE) should not be
>>> > called from an IoFutureListener, but in the test it is done
>>> anyway. However, changing the
>>> > parameter to FALSE does not help either.
>>> >
>>> > Is there anything that can be done to prevent this hang?
>>> >
>>> > Many thanks in advance for your help and best regards,
>>> > Chris.
>>> >
>>> >
>>>
>>>
>>
> --
> Christoph John
> Development & Support
> T +49 241 557080-28
> christoph.john@macd.com
>
> MACD GmbH
> Oppenhoffallee 103
> D-52066 Aachen
> www.macd.com
>
> Amtsgericht Aachen: HRB 8151
> Ust.-Id: DE 813021663
> Geschäftsführer: George Macdonald
>
>
Re: leaking NioProcessors/NioSocketConnectors hanging in call to
dispose - DIRMINA-1076
Posted by Christoph John <ch...@macd.com>.
Hi Jonathan,
thanks again for your work on DIRMINA-1076. I think I might have encountered a related case and
openend DIRMINA-1077for it.
Thanks in advance and best regards,
Christoph.
On 09/02/18 15:22, Christoph John wrote:
> OK, no problem. Just created https://issues.apache.org/jira/browse/DIRMINA-1076
>
> Thanks in advanceand best regards,
> Chris.
>
>
> On 09/02/18 15:11, Jonathan Valliere wrote:
>> If there isn’t a ticket in JIRA, please create one. I’d prefer to work there instead of email.
>>
>>
>> On Fri, Feb 9, 2018 at 8:56 AM Christoph John <christoph.john@macd.com
>> <ma...@macd.com>> wrote:
>>
>> Yes, that is the patch that I attached. It is simply executing the test in
>> AbstractIoServiceTest in
>> a loop.
>>
>> Thanks,
>> Chris.
>>
>> On 09/02/18 14:47, Jonathan Valliere wrote:
>> > It’s been a while, I forget, is there code to run the brute force test available?
>> >
>> > On Fri, Feb 9, 2018 at 7:56 AM Christoph John <christoph.john@macd.com
>> <ma...@macd.com>
>> > <mailto:christoph.john@macd.com <ma...@macd.com>>> wrote:
>> >
>> > Hi againafter some time. ;)
>> >
>> > I was now able to reproduce the problem with a MINA test. Or let's say I did the
>> brute-force
>> > approach by re-running one test in an endless loop.
>> > I have attached apatch of the test (against https://github.com/apache/mina/tree/2.0
>> <https://github.com/apache/mina/tree/2.0>) and a
>> > stack trace. After a few loops the test is stuck. You can see a lot of threads hanging in
>> > dispose() and the test is stuck when it tries to dispose the acceptor.
>> >
>> > What is a little strange is that the javadoc says that connector.dispose(TRUE) should
>> not be
>> > called from an IoFutureListener, but in the test it is done anyway. However, changing the
>> > parameter to FALSE does not help either.
>> >
>> > Is there anything that can be done to prevent this hang?
>> >
>> > Many thanks in advance for your help and best regards,
>> > Chris.
>> >
>> >
>>
>
--
Christoph John
Development & Support
T +49 241 557080-28
christoph.john@macd.com
MACD GmbH
Oppenhoffallee 103
D-52066 Aachen
www.macd.com
Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663
Geschäftsführer: George Macdonald
Re: leaking NioProcessors/NioSocketConnectors hanging in call to
dispose - DIRMINA-1076
Posted by Christoph John <ch...@macd.com>.
OK, no problem. Just created https://issues.apache.org/jira/browse/DIRMINA-1076
Thanks in advanceand best regards,
Chris.
On 09/02/18 15:11, Jonathan Valliere wrote:
> If there isn’t a ticket in JIRA, please create one. I’d prefer to work there instead of email.
>
>
> On Fri, Feb 9, 2018 at 8:56 AM Christoph John <christoph.john@macd.com
> <ma...@macd.com>> wrote:
>
> Yes, that is the patch that I attached. It is simply executing the test in
> AbstractIoServiceTest in
> a loop.
>
> Thanks,
> Chris.
>
> On 09/02/18 14:47, Jonathan Valliere wrote:
> > It’s been a while, I forget, is there code to run the brute force test available?
> >
> > On Fri, Feb 9, 2018 at 7:56 AM Christoph John <christoph.john@macd.com
> <ma...@macd.com>
> > <mailto:christoph.john@macd.com <ma...@macd.com>>> wrote:
> >
> > Hi againafter some time. ;)
> >
> > I was now able to reproduce the problem with a MINA test. Or let's say I did the brute-force
> > approach by re-running one test in an endless loop.
> > I have attached apatch of the test (against https://github.com/apache/mina/tree/2.0
> <https://github.com/apache/mina/tree/2.0>) and a
> > stack trace. After a few loops the test is stuck. You can see a lot of threads hanging in
> > dispose() and the test is stuck when it tries to dispose the acceptor.
> >
> > What is a little strange is that the javadoc says that connector.dispose(TRUE) should not be
> > called from an IoFutureListener, but in the test it is done anyway. However, changing the
> > parameter to FALSE does not help either.
> >
> > Is there anything that can be done to prevent this hang?
> >
> > Many thanks in advance for your help and best regards,
> > Chris.
> >
> >
>
--
Christoph John
Development & Support
T +49 241 557080-28
christoph.john@macd.com
MACD GmbH
Oppenhoffallee 103
D-52066 Aachen
www.macd.com
Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663
Geschäftsführer: George Macdonald
Re: leaking NioProcessors/NioSocketConnectors hanging in call to dispose
Posted by Jonathan Valliere <jo...@emoten.com>.
If there isn’t a ticket in JIRA, please create one. I’d prefer to work
there instead of email.
On Fri, Feb 9, 2018 at 8:56 AM Christoph John <ch...@macd.com>
wrote:
> Yes, that is the patch that I attached. It is simply executing the test in
> AbstractIoServiceTest in
> a loop.
>
> Thanks,
> Chris.
>
> On 09/02/18 14:47, Jonathan Valliere wrote:
> > It’s been a while, I forget, is there code to run the brute force test
> available?
> >
> > On Fri, Feb 9, 2018 at 7:56 AM Christoph John <christoph.john@macd.com
> > <ma...@macd.com>> wrote:
> >
> > Hi againafter some time. ;)
> >
> > I was now able to reproduce the problem with a MINA test. Or let's
> say I did the brute-force
> > approach by re-running one test in an endless loop.
> > I have attached apatch of the test (against
> https://github.com/apache/mina/tree/2.0) and a
> > stack trace. After a few loops the test is stuck. You can see a lot
> of threads hanging in
> > dispose() and the test is stuck when it tries to dispose the
> acceptor.
> >
> > What is a little strange is that the javadoc says that
> connector.dispose(TRUE) should not be
> > called from an IoFutureListener, but in the test it is done anyway.
> However, changing the
> > parameter to FALSE does not help either.
> >
> > Is there anything that can be done to prevent this hang?
> >
> > Many thanks in advance for your help and best regards,
> > Chris.
> >
> >
> > On 15/12/17 13:04, Christoph John wrote:
> >> Hi again,
> >>
> >> in the meantime I probably figured out where the problem is, but
> still have to do some more
> >> teststo be sure.
> >> I now have added logic to iterate over the managed sessions of an
> IoConnector and close them
> >> before calling IoConnector.dispose().
> >>
> >> I will let you know if this solved the problem.
> >>
> >> Thanks,
> >> Chris.
> >>
> >> On 16/10/17 16:20, Jonathan Valliere wrote:
> >>> If you can figure out a reproducible test case then I'm sure that
> we can fix it.
> >>>
> >>> On Mon, Oct 16, 2017 at 10:01 AM Christoph John <
> christoph.john@macd.com
> >>> <ma...@macd.com> <ma...@macd.com>
> >>> <ma...@macd.com>> wrote:
> >>>
> >>> Hi Emmanuel,
> >>>
> >>> yes, but unfortunately it did not help.
> >>>
> >>> Thanks,
> >>> Chris.
> >>>
> >>>
> >>> On 16/10/17 15:41, Emmanuel Lécharny wrote:
> >>> > Hi Christoph,
> >>> >
> >>> >
> >>> > have you tried with this patch ? :
> >>> >
> >>> >
> >>> > diff --git
> >>> > a/mina-core/src/main/java/org/apache/mina/core/polling/
> AbstractPollingIoProcessor.java
> >>> > b/mina-core/src/main/java/org/apache/mina/core/polling/
> AbstractPollingIoProcessor.java
> >>> > index 50ebd4e..575b2f4 100644
> >>> > ---
> >>> > a/mina-core/src/main/java/org/apache/mina/core/polling/
> AbstractPollingIoProcessor.java
> >>> > +++
> >>> > b/mina-core/src/main/java/org/apache/mina/core/polling/
> AbstractPollingIoProcessor.java
> >>> > @@ -695,8 +695,9 @@
> >>> > for (Iterator<S> i =
> allSessions(); i.hasNext();) {
> >>> > IoSession session = i.next();
> >>> >
> >>> > + scheduleRemove((S) session);
> >>> > +
> >>> > if (session.isActive()) {
> >>> > - scheduleRemove((S) session);
> >>> > hasKeys = true;
> >>> > }
> >>> > }
> >>> >
> >>> >
> >>>
> >>
> >
> > --
> > Christoph John
> > Development & Support
> > T +49 241 557080-28 <+49%20241%2055708028>
> > christoph.john@macd.com <ma...@macd.com>
> >
> > MACD GmbH
> > Oppenhoffallee 103
> > D-52066 Aachen
> > www.macd.com <http://www.macd.com>
> >
> > Amtsgericht Aachen: HRB 8151
> > Ust.-Id: DE 813021663
> > Geschäftsführer: George Macdonald
> >
>
> --
> Christoph John
> Development & Support
> T +49 241 557080-28 <+49%20241%2055708028>
> christoph.john@macd.com
>
> MACD GmbH
> Oppenhoffallee 103
> D-52066 Aachen
> www.macd.com
>
> Amtsgericht Aachen: HRB 8151
> Ust.-Id: DE 813021663
> Geschäftsführer: George Macdonald
>
>
Re: leaking NioProcessors/NioSocketConnectors hanging in call to
dispose
Posted by Christoph John <ch...@macd.com>.
Yes, that is the patch that I attached. It is simply executing the test in AbstractIoServiceTest in
a loop.
Thanks,
Chris.
On 09/02/18 14:47, Jonathan Valliere wrote:
> It’s been a while, I forget, is there code to run the brute force test available?
>
> On Fri, Feb 9, 2018 at 7:56 AM Christoph John <christoph.john@macd.com
> <ma...@macd.com>> wrote:
>
> Hi againafter some time. ;)
>
> I was now able to reproduce the problem with a MINA test. Or let's say I did the brute-force
> approach by re-running one test in an endless loop.
> I have attached apatch of the test (against https://github.com/apache/mina/tree/2.0) and a
> stack trace. After a few loops the test is stuck. You can see a lot of threads hanging in
> dispose() and the test is stuck when it tries to dispose the acceptor.
>
> What is a little strange is that the javadoc says that connector.dispose(TRUE) should not be
> called from an IoFutureListener, but in the test it is done anyway. However, changing the
> parameter to FALSE does not help either.
>
> Is there anything that can be done to prevent this hang?
>
> Many thanks in advance for your help and best regards,
> Chris.
>
>
> On 15/12/17 13:04, Christoph John wrote:
>> Hi again,
>>
>> in the meantime I probably figured out where the problem is, but still have to do some more
>> teststo be sure.
>> I now have added logic to iterate over the managed sessions of an IoConnector and close them
>> before calling IoConnector.dispose().
>>
>> I will let you know if this solved the problem.
>>
>> Thanks,
>> Chris.
>>
>> On 16/10/17 16:20, Jonathan Valliere wrote:
>>> If you can figure out a reproducible test case then I'm sure that we can fix it.
>>>
>>> On Mon, Oct 16, 2017 at 10:01 AM Christoph John <christoph.john@macd.com
>>> <ma...@macd.com> <ma...@macd.com>
>>> <ma...@macd.com>> wrote:
>>>
>>> Hi Emmanuel,
>>>
>>> yes, but unfortunately it did not help.
>>>
>>> Thanks,
>>> Chris.
>>>
>>>
>>> On 16/10/17 15:41, Emmanuel Lécharny wrote:
>>> > Hi Christoph,
>>> >
>>> >
>>> > have you tried with this patch ? :
>>> >
>>> >
>>> > diff --git
>>> > a/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
>>> > b/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
>>> > index 50ebd4e..575b2f4 100644
>>> > ---
>>> > a/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
>>> > +++
>>> > b/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
>>> > @@ -695,8 +695,9 @@
>>> > for (Iterator<S> i = allSessions(); i.hasNext();) {
>>> > IoSession session = i.next();
>>> >
>>> > + scheduleRemove((S) session);
>>> > +
>>> > if (session.isActive()) {
>>> > - scheduleRemove((S) session);
>>> > hasKeys = true;
>>> > }
>>> > }
>>> >
>>> >
>>>
>>
>
> --
> Christoph John
> Development & Support
> T +49 241 557080-28
> christoph.john@macd.com <ma...@macd.com>
>
> MACD GmbH
> Oppenhoffallee 103
> D-52066 Aachen
> www.macd.com <http://www.macd.com>
>
> Amtsgericht Aachen: HRB 8151
> Ust.-Id: DE 813021663
> Geschäftsführer: George Macdonald
>
--
Christoph John
Development & Support
T +49 241 557080-28
christoph.john@macd.com
MACD GmbH
Oppenhoffallee 103
D-52066 Aachen
www.macd.com
Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663
Geschäftsführer: George Macdonald
Re: leaking NioProcessors/NioSocketConnectors hanging in call to dispose
Posted by Jonathan Valliere <jo...@emoten.com>.
It’s been a while, I forget, is there code to run the brute force test
available?
On Fri, Feb 9, 2018 at 7:56 AM Christoph John <ch...@macd.com>
wrote:
> Hi again after some time. ;)
>
> I was now able to reproduce the problem with a MINA test. Or let's say I did
> the brute-force approach by re-running one test in an endless loop.
> I have attached a patch of the test (against
> https://github.com/apache/mina/tree/2.0) and a stack trace. After a few
> loops the test is stuck. You can see a lot of threads hanging in dispose()
> and the test is stuck when it tries to dispose the acceptor.
>
> What is a little strange is that the javadoc says that connector.dispose(
> TRUE) should not be called from an IoFutureListener, but in the test it
> is done anyway. However, changing the parameter to FALSE does not help
> either.
>
> Is there anything that can be done to prevent this hang?
>
> Many thanks in advance for your help and best regards,
> Chris.
>
>
> On 15/12/17 13:04, Christoph John wrote:
>
> Hi again,
>
> in the meantime I probably figured out where the problem is, but still
> have to do some more teststo be sure.
> I now have added logic to iterate over the managed sessions of an
> IoConnector and close them before calling IoConnector.dispose().
>
> I will let you know if this solved the problem.
>
> Thanks,
> Chris.
>
> On 16/10/17 16:20, Jonathan Valliere wrote:
>
> If you can figure out a reproducible test case then I'm sure that we can
> fix it.
>
> On Mon, Oct 16, 2017 at 10:01 AM Christoph John <christoph.john@macd.com
> <ma...@macd.com> <ch...@macd.com>> wrote:
>
> Hi Emmanuel,
>
> yes, but unfortunately it did not help.
>
> Thanks,
> Chris.
>
>
> On 16/10/17 15:41, Emmanuel Lécharny wrote:
> > Hi Christoph,
> >
> >
> > have you tried with this patch ? :
> >
> >
> > diff --git
> >
> a/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
> >
> b/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
> > index 50ebd4e..575b2f4 100644
> > ---
> >
> a/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
> > +++
> >
> b/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
> > @@ -695,8 +695,9 @@
> > for (Iterator<S> i = allSessions();
> i.hasNext();) {
> > IoSession session = i.next();
> >
> > + scheduleRemove((S) session);
> > +
> > if (session.isActive()) {
> > - scheduleRemove((S) session);
> > hasKeys = true;
> > }
> > }
> >
> >
>
>
>
> --
> Christoph John
> Development & Support
> T +49 241 557080-28christoph.john@macd.com
>
> MACD GmbH
> Oppenhoffallee 103
> D-52066 Aachenwww.macd.com
>
> Amtsgericht Aachen: HRB 8151
> Ust.-Id: DE 813021663
> Geschäftsführer: George Macdonald
>
>
Re: leaking NioProcessors/NioSocketConnectors hanging in call to
dispose
Posted by Christoph John <ch...@macd.com>.
Hi againafter some time. ;)
I was now able to reproduce the problem with a MINA test. Or let's say I did the brute-force
approach by re-running one test in an endless loop.
I have attached apatch of the test (against https://github.com/apache/mina/tree/2.0) and a stack
trace. After a few loops the test is stuck. You can see a lot of threads hanging in dispose() and
the test is stuck when it tries to dispose the acceptor.
What is a little strange is that the javadoc says that connector.dispose(TRUE) should not be called
from an IoFutureListener, but in the test it is done anyway. However, changing the parameter to
FALSE does not help either.
Is there anything that can be done to prevent this hang?
Many thanks in advance for your help and best regards,
Chris.
On 15/12/17 13:04, Christoph John wrote:
> Hi again,
>
> in the meantime I probably figured out where the problem is, but still have to do some more
> teststo be sure.
> I now have added logic to iterate over the managed sessions of an IoConnector and close them
> before calling IoConnector.dispose().
>
> I will let you know if this solved the problem.
>
> Thanks,
> Chris.
>
> On 16/10/17 16:20, Jonathan Valliere wrote:
>> If you can figure out a reproducible test case then I'm sure that we can fix it.
>>
>> On Mon, Oct 16, 2017 at 10:01 AM Christoph John <christoph.john@macd.com
>> <ma...@macd.com>> wrote:
>>
>> Hi Emmanuel,
>>
>> yes, but unfortunately it did not help.
>>
>> Thanks,
>> Chris.
>>
>>
>> On 16/10/17 15:41, Emmanuel Lécharny wrote:
>> > Hi Christoph,
>> >
>> >
>> > have you tried with this patch ? :
>> >
>> >
>> > diff --git
>> > a/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
>> > b/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
>> > index 50ebd4e..575b2f4 100644
>> > ---
>> > a/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
>> > +++
>> > b/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
>> > @@ -695,8 +695,9 @@
>> > for (Iterator<S> i = allSessions(); i.hasNext();) {
>> > IoSession session = i.next();
>> >
>> > + scheduleRemove((S) session);
>> > +
>> > if (session.isActive()) {
>> > - scheduleRemove((S) session);
>> > hasKeys = true;
>> > }
>> > }
>> >
>> >
>>
>
--
Christoph John
Development & Support
T +49 241 557080-28
christoph.john@macd.com
MACD GmbH
Oppenhoffallee 103
D-52066 Aachen
www.macd.com
Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663
Geschäftsführer: George Macdonald
Re: leaking NioProcessors/NioSocketConnectors hanging in call to
dispose
Posted by Christoph John <ch...@macd.com>.
Hi again,
in the meantime I probably figured out where the problem is, but still have to do some more teststo
be sure.
I now have added logic to iterate over the managed sessions of an IoConnector and close them before
calling IoConnector.dispose().
I will let you know if this solved the problem.
Thanks,
Chris.
On 16/10/17 16:20, Jonathan Valliere wrote:
> If you can figure out a reproducible test case then I'm sure that we can fix it.
>
> On Mon, Oct 16, 2017 at 10:01 AM Christoph John <christoph.john@macd.com
> <ma...@macd.com>> wrote:
>
> Hi Emmanuel,
>
> yes, but unfortunately it did not help.
>
> Thanks,
> Chris.
>
>
> On 16/10/17 15:41, Emmanuel Lécharny wrote:
> > Hi Christoph,
> >
> >
> > have you tried with this patch ? :
> >
> >
> > diff --git
> > a/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
> > b/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
> > index 50ebd4e..575b2f4 100644
> > ---
> > a/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
> > +++
> > b/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
> > @@ -695,8 +695,9 @@
> > for (Iterator<S> i = allSessions(); i.hasNext();) {
> > IoSession session = i.next();
> >
> > + scheduleRemove((S) session);
> > +
> > if (session.isActive()) {
> > - scheduleRemove((S) session);
> > hasKeys = true;
> > }
> > }
> >
> >
>
--
Christoph John
Development & Support
T +49 241 557080-28
christoph.john@macd.com
MACD GmbH
Oppenhoffallee 103
D-52066 Aachen
www.macd.com
Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663
Geschäftsführer: George Macdonald
Re: leaking NioProcessors/NioSocketConnectors hanging in call to dispose
Posted by Jonathan Valliere <jo...@emoten.com>.
If you can figure out a reproducible test case then I'm sure that we can
fix it.
On Mon, Oct 16, 2017 at 10:01 AM Christoph John <ch...@macd.com>
wrote:
> Hi Emmanuel,
>
> yes, but unfortunately it did not help.
>
> Thanks,
> Chris.
>
>
> On 16/10/17 15:41, Emmanuel Lécharny wrote:
> > Hi Christoph,
> >
> >
> > have you tried with this patch ? :
> >
> >
> > diff --git
> >
> a/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
> >
> b/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
> > index 50ebd4e..575b2f4 100644
> > ---
> >
> a/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
> > +++
> >
> b/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
> > @@ -695,8 +695,9 @@
> > for (Iterator<S> i = allSessions();
> i.hasNext();) {
> > IoSession session = i.next();
> >
> > + scheduleRemove((S) session);
> > +
> > if (session.isActive()) {
> > - scheduleRemove((S) session);
> > hasKeys = true;
> > }
> > }
> >
> >
> > Le 16/10/2017 à 14:41, Christoph John a écrit :
> >> Hi Emmanuel, Jonathan,
> >>
> >> thanks for your patches. Unfortunately neither of them does correct
> >> the problem I am facing.
> >> I'd suggest that I'll try to reproduce this with a MINA-based test
> >> first. Otherwise it is probably just guess-work.
> >>
> >> Thanks for your help and best regards,
> >> Chris.
> >>
> >>
> >> On 11/10/17 16:14, Jonathan Valliere wrote:
> >>> Basically my proposed patch moves the check dispose conditional into a
> >>> finally loop forcing it to run even if exceptions were caught.
> >>>
> >>> If the SSLFilter did something weird then it would be blocked on that
> >>> filter. What is happening is that the Processor is not dying as
> >>> requested.
> >>> That's why I think my patch ought to resolve the problem.
> >>>
> >>> On Wed, Oct 11, 2017 at 10:05 AM Emmanuel Lécharny <
> elecharny@gmail.com>
> >>> wrote:
> >>>
> >>>> Le 11/10/2017 à 14:51, Christoph John a écrit :
> >>>>> Thanks, will try. Did this patch apply to 2.0.16? The line numbers do
> >>>>> not match. I have added it manually in line 1162 and removed the
> >>>>> statement from line 1164.
> >>>> It's for 2.0.17-SNAPSHOT, so you may have differences with 2.0.16, but
> >>>> minor.
> >>>>
> >>>> The idea here is to remove a session when it's i teh list of managed
> >>>> version, even if it's not valid (that might be a corner case when teh
> >>>> SSLFilter does not properly handle an exception and close the channel.
> >>>> Call it a workaround atm)
> >>>>
> >>>> --
> >>>> Emmanuel Lecharny
> >>>>
> >>>> Symas.com
> >>>> directory.apache.org
> >>>>
> >>>>
>
> --
> Christoph John
> Development & Support
> Direct: +49 241 557080-28
> Mailto:Christoph.John@macd.com
>
>
>
> http://www.macd.com <http://www.macd.com/>
>
> ----------------------------------------------------------------------------------------------------
>
>
> ----------------------------------------------------------------------------------------------------
> MACD GmbH
> Oppenhoffallee 103
> D-52066 Aachen
> Tel: +49 241 557080-0 | Fax: +49 241 557080-10
> Amtsgericht Aachen: HRB 8151
> Ust.-Id: DE 813021663
>
> Geschäftsführer: George Macdonald
>
> ----------------------------------------------------------------------------------------------------
>
>
> ----------------------------------------------------------------------------------------------------
>
> take care of the environment - print only if necessary
>
Re: leaking NioProcessors/NioSocketConnectors hanging in call to
dispose
Posted by Christoph John <ch...@macd.com>.
Hi Emmanuel,
yes, but unfortunately it did not help.
Thanks,
Chris.
On 16/10/17 15:41, Emmanuel Lécharny wrote:
> Hi Christoph,
>
>
> have you tried with this patch ? :
>
>
> diff --git
> a/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
> b/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
> index 50ebd4e..575b2f4 100644
> ---
> a/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
> +++
> b/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
> @@ -695,8 +695,9 @@
> for (Iterator<S> i = allSessions(); i.hasNext();) {
> IoSession session = i.next();
>
> + scheduleRemove((S) session);
> +
> if (session.isActive()) {
> - scheduleRemove((S) session);
> hasKeys = true;
> }
> }
>
>
> Le 16/10/2017 à 14:41, Christoph John a écrit :
>> Hi Emmanuel, Jonathan,
>>
>> thanks for your patches. Unfortunately neither of them does correct
>> the problem I am facing.
>> I'd suggest that I'll try to reproduce this with a MINA-based test
>> first. Otherwise it is probably just guess-work.
>>
>> Thanks for your help and best regards,
>> Chris.
>>
>>
>> On 11/10/17 16:14, Jonathan Valliere wrote:
>>> Basically my proposed patch moves the check dispose conditional into a
>>> finally loop forcing it to run even if exceptions were caught.
>>>
>>> If the SSLFilter did something weird then it would be blocked on that
>>> filter. What is happening is that the Processor is not dying as
>>> requested.
>>> That's why I think my patch ought to resolve the problem.
>>>
>>> On Wed, Oct 11, 2017 at 10:05 AM Emmanuel Lécharny <el...@gmail.com>
>>> wrote:
>>>
>>>> Le 11/10/2017 à 14:51, Christoph John a écrit :
>>>>> Thanks, will try. Did this patch apply to 2.0.16? The line numbers do
>>>>> not match. I have added it manually in line 1162 and removed the
>>>>> statement from line 1164.
>>>> It's for 2.0.17-SNAPSHOT, so you may have differences with 2.0.16, but
>>>> minor.
>>>>
>>>> The idea here is to remove a session when it's i teh list of managed
>>>> version, even if it's not valid (that might be a corner case when teh
>>>> SSLFilter does not properly handle an exception and close the channel.
>>>> Call it a workaround atm)
>>>>
>>>> --
>>>> Emmanuel Lecharny
>>>>
>>>> Symas.com
>>>> directory.apache.org
>>>>
>>>>
--
Christoph John
Development & Support
Direct: +49 241 557080-28
Mailto:Christoph.John@macd.com
http://www.macd.com <http://www.macd.com/>
----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
MACD GmbH
Oppenhoffallee 103
D-52066 Aachen
Tel: +49 241 557080-0 | Fax: +49 241 557080-10
Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663
Geschäftsführer: George Macdonald
----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
take care of the environment - print only if necessary
Re: leaking NioProcessors/NioSocketConnectors hanging in call to
dispose
Posted by Emmanuel Lécharny <el...@gmail.com>.
Hi Christoph,
have you tried with this patch ? :
diff --git
a/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
b/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
index 50ebd4e..575b2f4 100644
---
a/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
+++
b/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
@@ -695,8 +695,9 @@
for (Iterator<S> i = allSessions(); i.hasNext();) {
IoSession session = i.next();
+ scheduleRemove((S) session);
+
if (session.isActive()) {
- scheduleRemove((S) session);
hasKeys = true;
}
}
Le 16/10/2017 à 14:41, Christoph John a écrit :
> Hi Emmanuel, Jonathan,
>
> thanks for your patches. Unfortunately neither of them does correct
> the problem I am facing.
> I'd suggest that I'll try to reproduce this with a MINA-based test
> first. Otherwise it is probably just guess-work.
>
> Thanks for your help and best regards,
> Chris.
>
>
> On 11/10/17 16:14, Jonathan Valliere wrote:
>> Basically my proposed patch moves the check dispose conditional into a
>> finally loop forcing it to run even if exceptions were caught.
>>
>> If the SSLFilter did something weird then it would be blocked on that
>> filter. What is happening is that the Processor is not dying as
>> requested.
>> That's why I think my patch ought to resolve the problem.
>>
>> On Wed, Oct 11, 2017 at 10:05 AM Emmanuel Lécharny <el...@gmail.com>
>> wrote:
>>
>>>
>>> Le 11/10/2017 à 14:51, Christoph John a écrit :
>>>> Thanks, will try. Did this patch apply to 2.0.16? The line numbers do
>>>> not match. I have added it manually in line 1162 and removed the
>>>> statement from line 1164.
>>> It's for 2.0.17-SNAPSHOT, so you may have differences with 2.0.16, but
>>> minor.
>>>
>>> The idea here is to remove a session when it's i teh list of managed
>>> version, even if it's not valid (that might be a corner case when teh
>>> SSLFilter does not properly handle an exception and close the channel.
>>> Call it a workaround atm)
>>>
>>> --
>>> Emmanuel Lecharny
>>>
>>> Symas.com
>>> directory.apache.org
>>>
>>>
>
--
Emmanuel Lecharny
Symas.com
directory.apache.org
Re: leaking NioProcessors/NioSocketConnectors hanging in call to
dispose
Posted by Christoph John <ch...@macd.com>.
Hi Emmanuel, Jonathan,
thanks for your patches. Unfortunately neither of them does correct the problem I am facing.
I'd suggest that I'll try to reproduce this with a MINA-based test first. Otherwise it is probably
just guess-work.
Thanks for your help and best regards,
Chris.
On 11/10/17 16:14, Jonathan Valliere wrote:
> Basically my proposed patch moves the check dispose conditional into a
> finally loop forcing it to run even if exceptions were caught.
>
> If the SSLFilter did something weird then it would be blocked on that
> filter. What is happening is that the Processor is not dying as requested.
> That's why I think my patch ought to resolve the problem.
>
> On Wed, Oct 11, 2017 at 10:05 AM Emmanuel Lécharny <el...@gmail.com>
> wrote:
>
>>
>> Le 11/10/2017 à 14:51, Christoph John a écrit :
>>> Thanks, will try. Did this patch apply to 2.0.16? The line numbers do
>>> not match. I have added it manually in line 1162 and removed the
>>> statement from line 1164.
>> It's for 2.0.17-SNAPSHOT, so you may have differences with 2.0.16, but
>> minor.
>>
>> The idea here is to remove a session when it's i teh list of managed
>> version, even if it's not valid (that might be a corner case when teh
>> SSLFilter does not properly handle an exception and close the channel.
>> Call it a workaround atm)
>>
>> --
>> Emmanuel Lecharny
>>
>> Symas.com
>> directory.apache.org
>>
>>
--
Christoph John
Development & Support
Direct: +49 241 557080-28
Mailto:Christoph.John@macd.com
http://www.macd.com <http://www.macd.com/>
----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
MACD GmbH
Oppenhoffallee 103
D-52066 Aachen
Tel: +49 241 557080-0 | Fax: +49 241 557080-10
Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663
Geschäftsführer: George Macdonald
----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
take care of the environment - print only if necessary
Re: leaking NioProcessors/NioSocketConnectors hanging in call to dispose
Posted by Jonathan Valliere <jo...@emoten.com>.
Basically my proposed patch moves the check dispose conditional into a
finally loop forcing it to run even if exceptions were caught.
If the SSLFilter did something weird then it would be blocked on that
filter. What is happening is that the Processor is not dying as requested.
That's why I think my patch ought to resolve the problem.
On Wed, Oct 11, 2017 at 10:05 AM Emmanuel Lécharny <el...@gmail.com>
wrote:
>
>
> Le 11/10/2017 à 14:51, Christoph John a écrit :
> > Thanks, will try. Did this patch apply to 2.0.16? The line numbers do
> > not match. I have added it manually in line 1162 and removed the
> > statement from line 1164.
>
> It's for 2.0.17-SNAPSHOT, so you may have differences with 2.0.16, but
> minor.
>
> The idea here is to remove a session when it's i teh list of managed
> version, even if it's not valid (that might be a corner case when teh
> SSLFilter does not properly handle an exception and close the channel.
> Call it a workaround atm)
>
> --
> Emmanuel Lecharny
>
> Symas.com
> directory.apache.org
>
>
Re: leaking NioProcessors/NioSocketConnectors hanging in call to
dispose
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 11/10/2017 à 14:51, Christoph John a écrit :
> Thanks, will try. Did this patch apply to 2.0.16? The line numbers do
> not match. I have added it manually in line 1162 and removed the
> statement from line 1164.
It's for 2.0.17-SNAPSHOT, so you may have differences with 2.0.16, but
minor.
The idea here is to remove a session when it's i teh list of managed
version, even if it's not valid (that might be a corner case when teh
SSLFilter does not properly handle an exception and close the channel.
Call it a workaround atm)
--
Emmanuel Lecharny
Symas.com
directory.apache.org
Re: leaking NioProcessors/NioSocketConnectors hanging in call to
dispose
Posted by Christoph John <ch...@macd.com>.
Thanks, will try. Did this patch apply to 2.0.16? The line numbers do not match. I have added it
manually in line 1162 and removed the statement from line 1164.
Cheers,
Chris.
On 11/10/17 14:19, Emmanuel Lécharny wrote:
> Hi,
>
>
> can you test with this patch ?
>
>
> diff --git
> a/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
> b/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
> index 50ebd4e..575b2f4 100644
> ---
> a/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
> +++
> b/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
> @@ -695,8 +695,9 @@
> for (Iterator<S> i = allSessions(); i.hasNext();) {
> IoSession session = i.next();
>
> + scheduleRemove((S) session);
> +
> if (session.isActive()) {
> - scheduleRemove((S) session);
> hasKeys = true;
> }
> }
>
>
> Le 11/10/2017 à 12:04, Christoph John a écrit :
>> Hi,
>>
>> thanks for the patch. I am using 2.0.16.
>> Oddly enough when I run all MINA tests then the ConnectorTest is
>> hanging on my machine in the testTCPWithSSL method. But I don't know
>> if this is related. (stack trace attached)
>> However, I will try out your patch and let you know.
>>
>> Thanks again,
>> Chris.
>>
>>
>> On 10/10/17 20:47, Jonathan Valliere wrote:
>>> Which version of Mina are you using or are you building from Git?
>>>
>>> Please pull tag/2.0.16 from GIT and apply the attached patch. Let me
>>> know if that fixes your problem. Sorry about the excess changes in
>>> the patch; the java code formatter made a lot of changes. If this
>>> works then we can create a JIRA bug.
>>>
>>> On Tue, Oct 10, 2017 at 4:49 AM, Christoph John
>>> <christoph.john@macd.com <ma...@macd.com>> wrote:
>>>
>>> Hi,
>>>
>>> thanks for your reply.
>>> In fact it is hanging forever, i.e. until the process stops. I
>>> have attached the original
>>> message I've sent to the mailing list. It only does occur
>>> sometimes for SSL connections with a
>>> failing handshake.
>>> Unfortunately I have no reproducable example for MINA itself. I
>>> could probably put something
>>> together for QuickFIX/J (the open source project I am working on).
>>>
>>> My OS is Ubuntu 14.04.5, JDK1.8_144 and the problem appears not
>>> so often on my machine but
>>> almost every time on the TravisCI build server
>>> (https://travis-ci.org/quickfix-j/quickfixj/builds/283210509
>>> <https://travis-ci.org/quickfix-j/quickfixj/builds/283210509>).
>>> As a result, some of the SSL
>>> related tests are failing. TravisCI has almost similar setup with
>>> JDK1.8_144 and Debian Linux.
>>>
>>> What would be a good starting point to create a test? I see that
>>> there is an SslTest in the
>>> mina-core module. So I probably have to change that test to
>>> repeatedly connect and get a
>>> handshake exception everytime and then take a number of stack
>>> traces.
>>>
>>> Thanks,
>>> Chris.
>>>
>>>
>>>
>>>
>>>
>>> On 09/10/17 14:51, Jonathan Valliere wrote:
>>>> What OS / Java Version / etc; Do you have a reproducible example?
>>>>
>>>> On Mon, Oct 9, 2017 at 8:34 AM, Jonathan Valliere
>>>> <jon.valliere@emoten.com
>>>> <ma...@emoten.com>> wrote:
>>>>
>>>> Let me know if its hanging more than 1s
>>>>
>>>> On Mon, Oct 9, 2017 at 5:08 AM, Christoph John
>>>> <christoph.john@macd.com
>>>> <ma...@macd.com>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I have another question regarding this one. There is
>>>> https://issues.apache.org/jira/browse/DIRMINA-1060
>>>> <https://issues.apache.org/jira/browse/DIRMINA-1060>
>>>> which also sounds a little like
>>>> the problem I'm having. When the connectors are hanging
>>>> in the call to dispose() then
>>>> there always is an accompanying NioProcessor which is
>>>> hanging in select().
>>>>
>>>> Example:
>>>> "NioProcessor-60" #100328 prio=5 os_prio=0
>>>> tid=0x00007f2a10003000 nid=0x2e71 runnable
>>>> [0x00007f2a388b1000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native
>>>> Method)
>>>> at
>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>>>> at
>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x00000000e239c118> (a sun.nio.ch.Util$3)
>>>> - locked <0x00000000e239c108> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000000e239bed0> (a
>>>> sun.nio.ch.EPollSelectorImpl)
>>>> at
>>>> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:98)
>>>> at
>>>>
>>>> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1075)
>>>> at
>>>>
>>>> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>> at java.lang.Thread.run(Thread.java:748)
>>>>
>>>>
>>>> "NioSocketConnector-38" #100326 prio=5 os_prio=0
>>>> tid=0x00007f2a3001d800 nid=0x2e6f in
>>>> Object.wait() [0x00007f2a1f2d3000]
>>>> java.lang.Thread.State: TIMED_WAITING (on object
>>>> monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at org.apache.mina.core.future.De
>>>>
>>>> <http://org.apache.mina.core.future.De>faultIoFuture.await0(DefaultIoFuture.java:209)
>>>> - locked <0x00000000e246ae08> (a
>>>> org.apache.mina.core.future.De
>>>> <http://org.apache.mina.core.future.De>faultIoFuture)
>>>> at org.apache.mina.core.future.De
>>>>
>>>> <http://org.apache.mina.core.future.De>faultIoFuture.awaitUninterruptibly(DefaultIoFuture.java:141)
>>>> at
>>>>
>>>> org.apache.mina.core.polling.AbstractPollingIoProcessor.dispose(AbstractPollingIoProcessor.java:188)
>>>> at
>>>>
>>>> org.apache.mina.core.service.SimpleIoProcessorPool.dispose(SimpleIoProcessorPool.java:329)
>>>> - locked <0x00000000e246ae40> (a java.lang.Object)
>>>> at
>>>>
>>>> org.apache.mina.core.polling.AbstractPollingIoConnector$Connector.run(AbstractPollingIoConnector.java:582)
>>>> at
>>>>
>>>> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>> at java.lang.Thread.run(Thread.java:748)
>>>>
>>>>
>>>> At first I thought that this was related to
>>>> https://issues.apache.org/jira/browse/DIRMINA-1059
>>>> <https://issues.apache.org/jira/browse/DIRMINA-1059>. In
>>>> that ticket the
>>>> synchronization was improved. However, I am also running
>>>> into the problem with a
>>>> build of 2.0.17-SNAPSHOT where DIRMINA-1059 was solved.
>>>>
>>>> So my only hope was DIRMINA-1060 ;) Could this improve
>>>> the situation?
>>>>
>>>> Thanks,
>>>> Chris.
>>>>
>>>>
>>>> -- Christoph John
>>>> Development & Support
>>>> Direct: +49 241 557080-28 <tel:%2B49%20241%20557080-28>
>>>> Mailto:Christoph.John@macd.com
>>>> <ma...@macd.com>
>>>>
>>>>
>>>>
>>>> http://www.macd.com <http://www.macd.com/>
>>>>
>>>> ----------------------------------------------------------------------------------------------------
>>>>
>>>>
>>>> ----------------------------------------------------------------------------------------------------
>>>> MACD GmbH
>>>> Oppenhoffallee 103
>>>> <https://maps.google.com/?q=Oppenhoffallee+103&entry=gmail&source=g>
>>>> D-52066 Aachen
>>>> Tel: +49 241 557080-0 <tel:%2B49%20241%20557080-0> |
>>>> Fax: +49 241 557080-10
>>>> <tel:%2B49%20241%20557080-10>
>>>> Amtsgericht Aachen: HRB 8151
>>>> Ust.-Id: DE 813021663
>>>>
>>>> Geschäftsführer: George Macdonald
>>>>
>>>> ----------------------------------------------------------------------------------------------------
>>>>
>>>>
>>>> ----------------------------------------------------------------------------------------------------
>>>>
>>>> take care of the environment - print only if necessary
>>>>
>>>>
>>>>
>>> -- Christoph John
>>> Development & Support
>>> Direct: +49 241 557080-28 <tel:+49%20241%2055708028>
>>> Mailto:Christoph.John@macd.com
>>>
>>>
>>>
>>> http://www.macd.com <http://www.macd.com/>
>>>
>>> ----------------------------------------------------------------------------------------------------
>>>
>>>
>>> ----------------------------------------------------------------------------------------------------
>>> MACD GmbH
>>> Oppenhoffallee 103
>>> D-52066 Aachen
>>> Tel: +49 241 557080-0 | Fax: +49 241 557080-10
>>> Amtsgericht Aachen: HRB 8151
>>> Ust.-Id: DE 813021663
>>>
>>> Geschäftsführer: George Macdonald
>>>
>>> ----------------------------------------------------------------------------------------------------
>>>
>>>
>>> ----------------------------------------------------------------------------------------------------
>>>
>>> take care of the environment - print only if necessary
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Christoph John <christoph.john@macd.com
>>> <ma...@macd.com>>
>>> To: dev@mina.apache.org <ma...@mina.apache.org>
>>> Cc:
>>> Bcc:
>>> Date: Wed, 26 Jul 2017 13:59:58 +0200
>>> Subject: leaking NioProcessors/NioSocketConnectors hanging in
>>> call to dispose
>>> Hi,
>>>
>>> I am a developer and maintainer of the QuickFIX/J project
>>> (https://github.com/quickfix-j/quickfixj
>>> <https://github.com/quickfix-j/quickfixj>) and I have
>>> a question regarding NioSocketConnectors.
>>>
>>> We are facing a problem when there is a process that constantly
>>> (every 30 seconds) tries to
>>> connect to a counterparty and the connection is established but
>>> dropped shortly after. Then
>>> sometimes the NioProcessors/NioSocketConnectors are not cleaned
>>> up properly. In the stack
>>> trace we see them hanging in a call to dispose:
>>>
>>> "NioProcessor-1140" #239 prio=5 os_prio=0 tid=0x0000000001fe1800
>>> nid=0x2523 runnable
>>> [0x00007f9c67e8f000]
>>> java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at
>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>>> at
>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> - locked <0x00000000f6699e60> (a sun.nio.ch.Util$3)
>>> - locked <0x00000000f6699e50> (a
>>> java.util.Collections$UnmodifiableSet)
>>> - locked <0x00000000f6699c18> (a
>>> sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> at
>>> org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:98)
>>> at
>>>
>>> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1075)
>>> at
>>> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:748)
>>>
>>> "NioSocketConnector-68" #238 prio=5 os_prio=0
>>> tid=0x00007f9c70caf000 nid=0x2522 in
>>> Object.wait() [0x00007f9c6af9f000]
>>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> at java.lang.Object.wait(Native Method)
>>> at org.apache.mina.core.future.De
>>>
>>> <http://org.apache.mina.core.future.De>faultIoFuture.await0(DefaultIoFuture.java:209)
>>> - locked <0x00000000f66ac718> (a
>>> org.apache.mina.core.future.De
>>> <http://org.apache.mina.core.future.De>faultIoFuture)
>>> at org.apache.mina.core.future.De
>>>
>>> <http://org.apache.mina.core.future.De>faultIoFuture.awaitUninterruptibly(DefaultIoFuture.java:141)
>>> at
>>>
>>> org.apache.mina.core.polling.AbstractPollingIoProcessor.dispose(AbstractPollingIoProcessor.java:188)
>>> at
>>>
>>> org.apache.mina.core.service.SimpleIoProcessorPool.dispose(SimpleIoProcessorPool.java:329)
>>> - locked <0x00000000f66ac750> (a java.lang.Object)
>>> at
>>>
>>> org.apache.mina.core.polling.AbstractPollingIoConnector$Connector.run(AbstractPollingIoConnector.java:582)
>>> at
>>> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:748)
>>>
>>> It does not happen very often: about 5% of the connection
>>> attempts leave a NioSocketConnector
>>> hanging.
>>> It only seems to happen though when the connection is
>>> disconnected by
>>> "javax.net.ssl.SSLHandshakeException: SSL handshake failed".
>>> Although there are cases when
>>> there is no leak even on an SSLHandshakeException.
>>> If the connection was reset "normally" by "java.io.IOException:
>>> Connection reset by peer" then
>>> the leak does not seem to occur. It also does not occur when the
>>> connection is refused right away.
>>>
>>> Since this seems to be related to SSL connections: is there
>>> something that we need to take
>>> care of when using the SSL filter?
>>>
>>> The code for the IoSessionInitiator can be found here:
>>>
>>> https://github.com/quickfix-j/quickfixj/blob/master/quickfixj-core/src/main/java/quickfix/mina/initiator/IoSessionInitiator.java
>>>
>>> <https://github.com/quickfix-j/quickfixj/blob/master/quickfixj-core/src/main/java/quickfix/mina/initiator/IoSessionInitiator.java>
>>> I have added some comments in this gist (starting with "chrjohn"):
>>> https://gist.github.com/chrjohn/2671f06d80e8d917d9061b573477ec5b
>>> <https://gist.github.com/chrjohn/2671f06d80e8d917d9061b573477ec5b>
>>>
>>> I cannot rule out that we might be doing something wrong here, so
>>> any pointer is appreciated. :)
>>>
>>> Thanks in advance for your help and best regards,
>>> Chris.
>>>
>>> -- Christoph John
>>> Development & Support
>>> Direct: +49 241 557080-28 <tel:%2B49%20241%20557080-28>
>>> Mailto:Christoph.John@macd.com <ma...@macd.com>
>>>
>>>
>>>
>>> http://www.macd.com <http://www.macd.com/>
>>>
>>> ----------------------------------------------------------------------------------------------------
>>>
>>>
>>> ----------------------------------------------------------------------------------------------------
>>> MACD GmbH
>>> Oppenhoffallee 103
>>> D-52066 Aachen
>>> Tel: +49 241 557080-0 <tel:%2B49%20241%20557080-0> | Fax: +49 241
>>> 557080-10
>>> <tel:%2B49%20241%20557080-10>
>>> Amtsgericht Aachen: HRB 8151
>>> Ust.-Id: DE 813021663
>>>
>>> Geschäftsführer: George Macdonald
>>>
>>> ----------------------------------------------------------------------------------------------------
>>>
>>>
>>> ----------------------------------------------------------------------------------------------------
>>>
>>> take care of the environment - print only if necessary
>>>
>>>
--
Christoph John
Development & Support
Direct: +49 241 557080-28
Mailto:Christoph.John@macd.com
http://www.macd.com <http://www.macd.com/>
----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
MACD GmbH
Oppenhoffallee 103
D-52066 Aachen
Tel: +49 241 557080-0 | Fax: +49 241 557080-10
Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663
Geschäftsführer: George Macdonald
----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
take care of the environment - print only if necessary
Re: leaking NioProcessors/NioSocketConnectors hanging in call to
dispose
Posted by Emmanuel Lécharny <el...@gmail.com>.
Hi,
can you test with this patch ?
diff --git
a/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
b/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
index 50ebd4e..575b2f4 100644
---
a/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
+++
b/mina-core/src/main/java/org/apache/mina/core/polling/AbstractPollingIoProcessor.java
@@ -695,8 +695,9 @@
for (Iterator<S> i = allSessions(); i.hasNext();) {
IoSession session = i.next();
+ scheduleRemove((S) session);
+
if (session.isActive()) {
- scheduleRemove((S) session);
hasKeys = true;
}
}
Le 11/10/2017 à 12:04, Christoph John a écrit :
> Hi,
>
> thanks for the patch. I am using 2.0.16.
> Oddly enough when I run all MINA tests then the ConnectorTest is
> hanging on my machine in the testTCPWithSSL method. But I don't know
> if this is related. (stack trace attached)
> However, I will try out your patch and let you know.
>
> Thanks again,
> Chris.
>
>
> On 10/10/17 20:47, Jonathan Valliere wrote:
>> Which version of Mina are you using or are you building from Git?
>>
>> Please pull tag/2.0.16 from GIT and apply the attached patch. Let me
>> know if that fixes your problem. Sorry about the excess changes in
>> the patch; the java code formatter made a lot of changes. If this
>> works then we can create a JIRA bug.
>>
>> On Tue, Oct 10, 2017 at 4:49 AM, Christoph John
>> <christoph.john@macd.com <ma...@macd.com>> wrote:
>>
>> Hi,
>>
>> thanks for your reply.
>> In fact it is hanging forever, i.e. until the process stops. I
>> have attached the original
>> message I've sent to the mailing list. It only does occur
>> sometimes for SSL connections with a
>> failing handshake.
>> Unfortunately I have no reproducable example for MINA itself. I
>> could probably put something
>> together for QuickFIX/J (the open source project I am working on).
>>
>> My OS is Ubuntu 14.04.5, JDK1.8_144 and the problem appears not
>> so often on my machine but
>> almost every time on the TravisCI build server
>> (https://travis-ci.org/quickfix-j/quickfixj/builds/283210509
>> <https://travis-ci.org/quickfix-j/quickfixj/builds/283210509>).
>> As a result, some of the SSL
>> related tests are failing. TravisCI has almost similar setup with
>> JDK1.8_144 and Debian Linux.
>>
>> What would be a good starting point to create a test? I see that
>> there is an SslTest in the
>> mina-core module. So I probably have to change that test to
>> repeatedly connect and get a
>> handshake exception everytime and then take a number of stack
>> traces.
>>
>> Thanks,
>> Chris.
>>
>>
>>
>>
>>
>> On 09/10/17 14:51, Jonathan Valliere wrote:
>>> What OS / Java Version / etc; Do you have a reproducible example?
>>>
>>> On Mon, Oct 9, 2017 at 8:34 AM, Jonathan Valliere
>>> <jon.valliere@emoten.com
>>> <ma...@emoten.com>> wrote:
>>>
>>> Let me know if its hanging more than 1s
>>>
>>> On Mon, Oct 9, 2017 at 5:08 AM, Christoph John
>>> <christoph.john@macd.com
>>> <ma...@macd.com>> wrote:
>>>
>>> Hi,
>>>
>>> I have another question regarding this one. There is
>>> https://issues.apache.org/jira/browse/DIRMINA-1060
>>> <https://issues.apache.org/jira/browse/DIRMINA-1060>
>>> which also sounds a little like
>>> the problem I'm having. When the connectors are hanging
>>> in the call to dispose() then
>>> there always is an accompanying NioProcessor which is
>>> hanging in select().
>>>
>>> Example:
>>> "NioProcessor-60" #100328 prio=5 os_prio=0
>>> tid=0x00007f2a10003000 nid=0x2e71 runnable
>>> [0x00007f2a388b1000]
>>> java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native
>>> Method)
>>> at
>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>>> at
>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> - locked <0x00000000e239c118> (a sun.nio.ch.Util$3)
>>> - locked <0x00000000e239c108> (a
>>> java.util.Collections$UnmodifiableSet)
>>> - locked <0x00000000e239bed0> (a
>>> sun.nio.ch.EPollSelectorImpl)
>>> at
>>> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> at
>>> org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:98)
>>> at
>>>
>>> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1075)
>>> at
>>>
>>> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> at java.lang.Thread.run(Thread.java:748)
>>>
>>>
>>> "NioSocketConnector-38" #100326 prio=5 os_prio=0
>>> tid=0x00007f2a3001d800 nid=0x2e6f in
>>> Object.wait() [0x00007f2a1f2d3000]
>>> java.lang.Thread.State: TIMED_WAITING (on object
>>> monitor)
>>> at java.lang.Object.wait(Native Method)
>>> at org.apache.mina.core.future.De
>>>
>>> <http://org.apache.mina.core.future.De>faultIoFuture.await0(DefaultIoFuture.java:209)
>>> - locked <0x00000000e246ae08> (a
>>> org.apache.mina.core.future.De
>>> <http://org.apache.mina.core.future.De>faultIoFuture)
>>> at org.apache.mina.core.future.De
>>>
>>> <http://org.apache.mina.core.future.De>faultIoFuture.awaitUninterruptibly(DefaultIoFuture.java:141)
>>> at
>>>
>>> org.apache.mina.core.polling.AbstractPollingIoProcessor.dispose(AbstractPollingIoProcessor.java:188)
>>> at
>>>
>>> org.apache.mina.core.service.SimpleIoProcessorPool.dispose(SimpleIoProcessorPool.java:329)
>>> - locked <0x00000000e246ae40> (a java.lang.Object)
>>> at
>>>
>>> org.apache.mina.core.polling.AbstractPollingIoConnector$Connector.run(AbstractPollingIoConnector.java:582)
>>> at
>>>
>>> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> at java.lang.Thread.run(Thread.java:748)
>>>
>>>
>>> At first I thought that this was related to
>>> https://issues.apache.org/jira/browse/DIRMINA-1059
>>> <https://issues.apache.org/jira/browse/DIRMINA-1059>. In
>>> that ticket the
>>> synchronization was improved. However, I am also running
>>> into the problem with a
>>> build of 2.0.17-SNAPSHOT where DIRMINA-1059 was solved.
>>>
>>> So my only hope was DIRMINA-1060 ;) Could this improve
>>> the situation?
>>>
>>> Thanks,
>>> Chris.
>>>
>>>
>>> -- Christoph John
>>> Development & Support
>>> Direct: +49 241 557080-28 <tel:%2B49%20241%20557080-28>
>>> Mailto:Christoph.John@macd.com
>>> <ma...@macd.com>
>>>
>>>
>>>
>>> http://www.macd.com <http://www.macd.com/>
>>>
>>> ----------------------------------------------------------------------------------------------------
>>>
>>>
>>> ----------------------------------------------------------------------------------------------------
>>> MACD GmbH
>>> Oppenhoffallee 103
>>> <https://maps.google.com/?q=Oppenhoffallee+103&entry=gmail&source=g>
>>> D-52066 Aachen
>>> Tel: +49 241 557080-0 <tel:%2B49%20241%20557080-0> |
>>> Fax: +49 241 557080-10
>>> <tel:%2B49%20241%20557080-10>
>>> Amtsgericht Aachen: HRB 8151
>>> Ust.-Id: DE 813021663
>>>
>>> Geschäftsführer: George Macdonald
>>>
>>> ----------------------------------------------------------------------------------------------------
>>>
>>>
>>> ----------------------------------------------------------------------------------------------------
>>>
>>> take care of the environment - print only if necessary
>>>
>>>
>>>
>>
>> -- Christoph John
>> Development & Support
>> Direct: +49 241 557080-28 <tel:+49%20241%2055708028>
>> Mailto:Christoph.John@macd.com
>>
>>
>>
>> http://www.macd.com <http://www.macd.com/>
>>
>> ----------------------------------------------------------------------------------------------------
>>
>>
>> ----------------------------------------------------------------------------------------------------
>> MACD GmbH
>> Oppenhoffallee 103
>> D-52066 Aachen
>> Tel: +49 241 557080-0 | Fax: +49 241 557080-10
>> Amtsgericht Aachen: HRB 8151
>> Ust.-Id: DE 813021663
>>
>> Geschäftsführer: George Macdonald
>>
>> ----------------------------------------------------------------------------------------------------
>>
>>
>> ----------------------------------------------------------------------------------------------------
>>
>> take care of the environment - print only if necessary
>>
>>
>> ---------- Forwarded message ----------
>> From: Christoph John <christoph.john@macd.com
>> <ma...@macd.com>>
>> To: dev@mina.apache.org <ma...@mina.apache.org>
>> Cc:
>> Bcc:
>> Date: Wed, 26 Jul 2017 13:59:58 +0200
>> Subject: leaking NioProcessors/NioSocketConnectors hanging in
>> call to dispose
>> Hi,
>>
>> I am a developer and maintainer of the QuickFIX/J project
>> (https://github.com/quickfix-j/quickfixj
>> <https://github.com/quickfix-j/quickfixj>) and I have
>> a question regarding NioSocketConnectors.
>>
>> We are facing a problem when there is a process that constantly
>> (every 30 seconds) tries to
>> connect to a counterparty and the connection is established but
>> dropped shortly after. Then
>> sometimes the NioProcessors/NioSocketConnectors are not cleaned
>> up properly. In the stack
>> trace we see them hanging in a call to dispose:
>>
>> "NioProcessor-1140" #239 prio=5 os_prio=0 tid=0x0000000001fe1800
>> nid=0x2523 runnable
>> [0x00007f9c67e8f000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x00000000f6699e60> (a sun.nio.ch.Util$3)
>> - locked <0x00000000f6699e50> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked <0x00000000f6699c18> (a
>> sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:98)
>> at
>>
>> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1075)
>> at
>> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:748)
>>
>> "NioSocketConnector-68" #238 prio=5 os_prio=0
>> tid=0x00007f9c70caf000 nid=0x2522 in
>> Object.wait() [0x00007f9c6af9f000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at org.apache.mina.core.future.De
>>
>> <http://org.apache.mina.core.future.De>faultIoFuture.await0(DefaultIoFuture.java:209)
>> - locked <0x00000000f66ac718> (a
>> org.apache.mina.core.future.De
>> <http://org.apache.mina.core.future.De>faultIoFuture)
>> at org.apache.mina.core.future.De
>>
>> <http://org.apache.mina.core.future.De>faultIoFuture.awaitUninterruptibly(DefaultIoFuture.java:141)
>> at
>>
>> org.apache.mina.core.polling.AbstractPollingIoProcessor.dispose(AbstractPollingIoProcessor.java:188)
>> at
>>
>> org.apache.mina.core.service.SimpleIoProcessorPool.dispose(SimpleIoProcessorPool.java:329)
>> - locked <0x00000000f66ac750> (a java.lang.Object)
>> at
>>
>> org.apache.mina.core.polling.AbstractPollingIoConnector$Connector.run(AbstractPollingIoConnector.java:582)
>> at
>> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:748)
>>
>> It does not happen very often: about 5% of the connection
>> attempts leave a NioSocketConnector
>> hanging.
>> It only seems to happen though when the connection is
>> disconnected by
>> "javax.net.ssl.SSLHandshakeException: SSL handshake failed".
>> Although there are cases when
>> there is no leak even on an SSLHandshakeException.
>> If the connection was reset "normally" by "java.io.IOException:
>> Connection reset by peer" then
>> the leak does not seem to occur. It also does not occur when the
>> connection is refused right away.
>>
>> Since this seems to be related to SSL connections: is there
>> something that we need to take
>> care of when using the SSL filter?
>>
>> The code for the IoSessionInitiator can be found here:
>>
>> https://github.com/quickfix-j/quickfixj/blob/master/quickfixj-core/src/main/java/quickfix/mina/initiator/IoSessionInitiator.java
>>
>> <https://github.com/quickfix-j/quickfixj/blob/master/quickfixj-core/src/main/java/quickfix/mina/initiator/IoSessionInitiator.java>
>> I have added some comments in this gist (starting with "chrjohn"):
>> https://gist.github.com/chrjohn/2671f06d80e8d917d9061b573477ec5b
>> <https://gist.github.com/chrjohn/2671f06d80e8d917d9061b573477ec5b>
>>
>> I cannot rule out that we might be doing something wrong here, so
>> any pointer is appreciated. :)
>>
>> Thanks in advance for your help and best regards,
>> Chris.
>>
>> -- Christoph John
>> Development & Support
>> Direct: +49 241 557080-28 <tel:%2B49%20241%20557080-28>
>> Mailto:Christoph.John@macd.com <ma...@macd.com>
>>
>>
>>
>> http://www.macd.com <http://www.macd.com/>
>>
>> ----------------------------------------------------------------------------------------------------
>>
>>
>> ----------------------------------------------------------------------------------------------------
>> MACD GmbH
>> Oppenhoffallee 103
>> D-52066 Aachen
>> Tel: +49 241 557080-0 <tel:%2B49%20241%20557080-0> | Fax: +49 241
>> 557080-10
>> <tel:%2B49%20241%20557080-10>
>> Amtsgericht Aachen: HRB 8151
>> Ust.-Id: DE 813021663
>>
>> Geschäftsführer: George Macdonald
>>
>> ----------------------------------------------------------------------------------------------------
>>
>>
>> ----------------------------------------------------------------------------------------------------
>>
>> take care of the environment - print only if necessary
>>
>>
>
--
Emmanuel Lecharny
Symas.com
directory.apache.org
Re: leaking NioProcessors/NioSocketConnectors hanging in call to
dispose
Posted by Christoph John <ch...@macd.com>.
Hi,
thanks for the patch. I am using 2.0.16.
Oddly enough when I run all MINA tests then the ConnectorTest is hanging on my machine in the
testTCPWithSSL method. But I don't know if this is related. (stack trace attached)
However, I will try out your patch and let you know.
Thanks again,
Chris.
On 10/10/17 20:47, Jonathan Valliere wrote:
> Which version of Mina are you using or are you building from Git?
>
> Please pull tag/2.0.16 from GIT and apply the attached patch. Let me know if that fixes your
> problem. Sorry about the excess changes in the patch; the java code formatter made a lot of
> changes. If this works then we can create a JIRA bug.
>
> On Tue, Oct 10, 2017 at 4:49 AM, Christoph John <christoph.john@macd.com
> <ma...@macd.com>> wrote:
>
> Hi,
>
> thanks for your reply.
> In fact it is hanging forever, i.e. until the process stops. I have attached the original
> message I've sent to the mailing list. It only does occur sometimes for SSL connections with a
> failing handshake.
> Unfortunately I have no reproducable example for MINA itself. I could probably put something
> together for QuickFIX/J (the open source project I am working on).
>
> My OS is Ubuntu 14.04.5, JDK1.8_144 and the problem appears not so often on my machine but
> almost every time on the TravisCI build server
> (https://travis-ci.org/quickfix-j/quickfixj/builds/283210509
> <https://travis-ci.org/quickfix-j/quickfixj/builds/283210509>). As a result, some of the SSL
> related tests are failing. TravisCI has almost similar setup with JDK1.8_144 and Debian Linux.
>
> What would be a good starting point to create a test? I see that there is an SslTest in the
> mina-core module. So I probably have to change that test to repeatedly connect and get a
> handshake exception everytime and then take a number of stack traces.
>
> Thanks,
> Chris.
>
>
>
>
>
> On 09/10/17 14:51, Jonathan Valliere wrote:
>> What OS / Java Version / etc; Do you have a reproducible example?
>>
>> On Mon, Oct 9, 2017 at 8:34 AM, Jonathan Valliere <jon.valliere@emoten.com
>> <ma...@emoten.com>> wrote:
>>
>> Let me know if its hanging more than 1s
>>
>> On Mon, Oct 9, 2017 at 5:08 AM, Christoph John <christoph.john@macd.com
>> <ma...@macd.com>> wrote:
>>
>> Hi,
>>
>> I have another question regarding this one. There is
>> https://issues.apache.org/jira/browse/DIRMINA-1060
>> <https://issues.apache.org/jira/browse/DIRMINA-1060> which also sounds a little like
>> the problem I'm having. When the connectors are hanging in the call to dispose() then
>> there always is an accompanying NioProcessor which is hanging in select().
>>
>> Example:
>> "NioProcessor-60" #100328 prio=5 os_prio=0 tid=0x00007f2a10003000 nid=0x2e71 runnable
>> [0x00007f2a388b1000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x00000000e239c118> (a sun.nio.ch.Util$3)
>> - locked <0x00000000e239c108> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000000e239bed0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:98)
>> at
>> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1075)
>> at
>> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>> "NioSocketConnector-38" #100326 prio=5 os_prio=0 tid=0x00007f2a3001d800 nid=0x2e6f in
>> Object.wait() [0x00007f2a1f2d3000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at org.apache.mina.core.future.De
>> <http://org.apache.mina.core.future.De>faultIoFuture.await0(DefaultIoFuture.java:209)
>> - locked <0x00000000e246ae08> (a org.apache.mina.core.future.De
>> <http://org.apache.mina.core.future.De>faultIoFuture)
>> at org.apache.mina.core.future.De
>> <http://org.apache.mina.core.future.De>faultIoFuture.awaitUninterruptibly(DefaultIoFuture.java:141)
>> at
>> org.apache.mina.core.polling.AbstractPollingIoProcessor.dispose(AbstractPollingIoProcessor.java:188)
>> at
>> org.apache.mina.core.service.SimpleIoProcessorPool.dispose(SimpleIoProcessorPool.java:329)
>> - locked <0x00000000e246ae40> (a java.lang.Object)
>> at
>> org.apache.mina.core.polling.AbstractPollingIoConnector$Connector.run(AbstractPollingIoConnector.java:582)
>> at
>> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>> At first I thought that this was related to
>> https://issues.apache.org/jira/browse/DIRMINA-1059
>> <https://issues.apache.org/jira/browse/DIRMINA-1059>. In that ticket the
>> synchronization was improved. However, I am also running into the problem with a
>> build of 2.0.17-SNAPSHOT where DIRMINA-1059 was solved.
>>
>> So my only hope was DIRMINA-1060 ;) Could this improve the situation?
>>
>> Thanks,
>> Chris.
>>
>>
>> --
>> Christoph John
>> Development & Support
>> Direct: +49 241 557080-28 <tel:%2B49%20241%20557080-28>
>> Mailto:Christoph.John@macd.com <ma...@macd.com>
>>
>>
>>
>> http://www.macd.com <http://www.macd.com/>
>> ----------------------------------------------------------------------------------------------------
>>
>> ----------------------------------------------------------------------------------------------------
>> MACD GmbH
>> Oppenhoffallee 103 <https://maps.google.com/?q=Oppenhoffallee+103&entry=gmail&source=g>
>> D-52066 Aachen
>> Tel: +49 241 557080-0 <tel:%2B49%20241%20557080-0> | Fax: +49 241 557080-10
>> <tel:%2B49%20241%20557080-10>
>> Amtsgericht Aachen: HRB 8151
>> Ust.-Id: DE 813021663
>>
>> Geschäftsführer: George Macdonald
>> ----------------------------------------------------------------------------------------------------
>>
>> ----------------------------------------------------------------------------------------------------
>>
>> take care of the environment - print only if necessary
>>
>>
>>
>
> --
> Christoph John
> Development & Support
> Direct: +49 241 557080-28 <tel:+49%20241%2055708028>
> Mailto:Christoph.John@macd.com
>
>
>
> http://www.macd.com <http://www.macd.com/>
> ----------------------------------------------------------------------------------------------------
>
> ----------------------------------------------------------------------------------------------------
> MACD GmbH
> Oppenhoffallee 103
> D-52066 Aachen
> Tel: +49 241 557080-0 | Fax: +49 241 557080-10
> Amtsgericht Aachen: HRB 8151
> Ust.-Id: DE 813021663
>
> Geschäftsführer: George Macdonald
> ----------------------------------------------------------------------------------------------------
>
> ----------------------------------------------------------------------------------------------------
>
> take care of the environment - print only if necessary
>
>
> ---------- Forwarded message ----------
> From: Christoph John <christoph.john@macd.com <ma...@macd.com>>
> To: dev@mina.apache.org <ma...@mina.apache.org>
> Cc:
> Bcc:
> Date: Wed, 26 Jul 2017 13:59:58 +0200
> Subject: leaking NioProcessors/NioSocketConnectors hanging in call to dispose
> Hi,
>
> I am a developer and maintainer of the QuickFIX/J project
> (https://github.com/quickfix-j/quickfixj <https://github.com/quickfix-j/quickfixj>) and I have
> a question regarding NioSocketConnectors.
>
> We are facing a problem when there is a process that constantly (every 30 seconds) tries to
> connect to a counterparty and the connection is established but dropped shortly after. Then
> sometimes the NioProcessors/NioSocketConnectors are not cleaned up properly. In the stack
> trace we see them hanging in a call to dispose:
>
> "NioProcessor-1140" #239 prio=5 os_prio=0 tid=0x0000000001fe1800 nid=0x2523 runnable
> [0x00007f9c67e8f000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000f6699e60> (a sun.nio.ch.Util$3)
> - locked <0x00000000f6699e50> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000f6699c18> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:98)
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1075)
> at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
>
> "NioSocketConnector-68" #238 prio=5 os_prio=0 tid=0x00007f9c70caf000 nid=0x2522 in
> Object.wait() [0x00007f9c6af9f000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at org.apache.mina.core.future.De
> <http://org.apache.mina.core.future.De>faultIoFuture.await0(DefaultIoFuture.java:209)
> - locked <0x00000000f66ac718> (a org.apache.mina.core.future.De
> <http://org.apache.mina.core.future.De>faultIoFuture)
> at org.apache.mina.core.future.De
> <http://org.apache.mina.core.future.De>faultIoFuture.awaitUninterruptibly(DefaultIoFuture.java:141)
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.dispose(AbstractPollingIoProcessor.java:188)
> at
> org.apache.mina.core.service.SimpleIoProcessorPool.dispose(SimpleIoProcessorPool.java:329)
> - locked <0x00000000f66ac750> (a java.lang.Object)
> at
> org.apache.mina.core.polling.AbstractPollingIoConnector$Connector.run(AbstractPollingIoConnector.java:582)
> at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
>
> It does not happen very often: about 5% of the connection attempts leave a NioSocketConnector
> hanging.
> It only seems to happen though when the connection is disconnected by
> "javax.net.ssl.SSLHandshakeException: SSL handshake failed". Although there are cases when
> there is no leak even on an SSLHandshakeException.
> If the connection was reset "normally" by "java.io.IOException: Connection reset by peer" then
> the leak does not seem to occur. It also does not occur when the connection is refused right away.
>
> Since this seems to be related to SSL connections: is there something that we need to take
> care of when using the SSL filter?
>
> The code for the IoSessionInitiator can be found here:
> https://github.com/quickfix-j/quickfixj/blob/master/quickfixj-core/src/main/java/quickfix/mina/initiator/IoSessionInitiator.java
> <https://github.com/quickfix-j/quickfixj/blob/master/quickfixj-core/src/main/java/quickfix/mina/initiator/IoSessionInitiator.java>
> I have added some comments in this gist (starting with "chrjohn"):
> https://gist.github.com/chrjohn/2671f06d80e8d917d9061b573477ec5b
> <https://gist.github.com/chrjohn/2671f06d80e8d917d9061b573477ec5b>
>
> I cannot rule out that we might be doing something wrong here, so any pointer is appreciated. :)
>
> Thanks in advance for your help and best regards,
> Chris.
>
> --
> Christoph John
> Development & Support
> Direct: +49 241 557080-28 <tel:%2B49%20241%20557080-28>
> Mailto:Christoph.John@macd.com <ma...@macd.com>
>
>
>
> http://www.macd.com <http://www.macd.com/>
> ----------------------------------------------------------------------------------------------------
>
> ----------------------------------------------------------------------------------------------------
> MACD GmbH
> Oppenhoffallee 103
> D-52066 Aachen
> Tel: +49 241 557080-0 <tel:%2B49%20241%20557080-0> | Fax: +49 241 557080-10
> <tel:%2B49%20241%20557080-10>
> Amtsgericht Aachen: HRB 8151
> Ust.-Id: DE 813021663
>
> Geschäftsführer: George Macdonald
> ----------------------------------------------------------------------------------------------------
>
> ----------------------------------------------------------------------------------------------------
>
> take care of the environment - print only if necessary
>
>
--
Christoph John
Development & Support
Direct: +49 241 557080-28
Mailto:Christoph.John@macd.com
http://www.macd.com <http://www.macd.com/>
----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
MACD GmbH
Oppenhoffallee 103
D-52066 Aachen
Tel: +49 241 557080-0 | Fax: +49 241 557080-10
Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663
Geschäftsführer: George Macdonald
----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
take care of the environment - print only if necessary
Re: leaking NioProcessors/NioSocketConnectors hanging in call to dispose
Posted by Jonathan Valliere <jo...@emoten.com>.
Which version of Mina are you using or are you building from Git?
Please pull tag/2.0.16 from GIT and apply the attached patch. Let me know
if that fixes your problem. Sorry about the excess changes in the patch;
the java code formatter made a lot of changes. If this works then we can
create a JIRA bug.
On Tue, Oct 10, 2017 at 4:49 AM, Christoph John <ch...@macd.com>
wrote:
> Hi,
>
> thanks for your reply.
> In fact it is hanging forever, i.e. until the process stops. I have
> attached the original message I've sent to the mailing list. It only does
> occur sometimes for SSL connections with a failing handshake.
> Unfortunately I have no reproducable example for MINA itself. I could
> probably put something together for QuickFIX/J (the open source project I
> am working on).
>
> My OS is Ubuntu 14.04.5, JDK1.8_144 and the problem appears not so often
> on my machine but almost every time on the TravisCI build server (
> https://travis-ci.org/quickfix-j/quickfixj/builds/283210509). As a
> result, some of the SSL related tests are failing. TravisCI has almost
> similar setup with JDK1.8_144 and Debian Linux.
>
> What would be a good starting point to create a test? I see that there is
> an SslTest in the mina-core module. So I probably have to change that test
> to repeatedly connect and get a handshake exception everytime and then take
> a number of stack traces.
>
> Thanks,
> Chris.
>
>
>
>
>
> On 09/10/17 14:51, Jonathan Valliere wrote:
>
> What OS / Java Version / etc; Do you have a reproducible example?
>
> On Mon, Oct 9, 2017 at 8:34 AM, Jonathan Valliere <jon.valliere@emoten.com
> > wrote:
>
>> Let me know if its hanging more than 1s
>>
>> On Mon, Oct 9, 2017 at 5:08 AM, Christoph John <ch...@macd.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I have another question regarding this one. There is
>>> https://issues.apache.org/jira/browse/DIRMINA-1060 which also sounds a
>>> little like the problem I'm having. When the connectors are hanging in the
>>> call to dispose() then there always is an accompanying NioProcessor which
>>> is hanging in select().
>>>
>>> Example:
>>> "NioProcessor-60" #100328 prio=5 os_prio=0 tid=0x00007f2a10003000
>>> nid=0x2e71 runnable [0x00007f2a388b1000]
>>> java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java
>>> :93)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> - locked <0x00000000e239c118> (a sun.nio.ch.Util$3)
>>> - locked <0x00000000e239c108> (a java.util.Collections$Unmodifi
>>> ableSet)
>>> - locked <0x00000000e239bed0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> at org.apache.mina.transport.socket.nio.NioProcessor.select(Nio
>>> Processor.java:98)
>>> at org.apache.mina.core.polling.AbstractPollingIoProcessor$Proc
>>> essor.run(AbstractPollingIoProcessor.java:1075)
>>> at org.apache.mina.util.NamePreservingRunnable.run(NamePreservi
>>> ngRunnable.java:64)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>> Executor.java:1149)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>> lExecutor.java:624)
>>> at java.lang.Thread.run(Thread.java:748)
>>>
>>>
>>> "NioSocketConnector-38" #100326 prio=5 os_prio=0 tid=0x00007f2a3001d800
>>> nid=0x2e6f in Object.wait() [0x00007f2a1f2d3000]
>>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> at java.lang.Object.wait(Native Method)
>>> at org.apache.mina.core.future.DefaultIoFuture.await0(DefaultIo
>>> Future.java:209)
>>> - locked <0x00000000e246ae08> (a org.apache.mina.core.future.De
>>> faultIoFuture)
>>> at org.apache.mina.core.future.DefaultIoFuture.awaitUninterrupt
>>> ibly(DefaultIoFuture.java:141)
>>> at org.apache.mina.core.polling.AbstractPollingIoProcessor.disp
>>> ose(AbstractPollingIoProcessor.java:188)
>>> at org.apache.mina.core.service.SimpleIoProcessorPool.dispose(S
>>> impleIoProcessorPool.java:329)
>>> - locked <0x00000000e246ae40> (a java.lang.Object)
>>> at org.apache.mina.core.polling.AbstractPollingIoConnector$Conn
>>> ector.run(AbstractPollingIoConnector.java:582)
>>> at org.apache.mina.util.NamePreservingRunnable.run(NamePreservi
>>> ngRunnable.java:64)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>> Executor.java:1149)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>> lExecutor.java:624)
>>> at java.lang.Thread.run(Thread.java:748)
>>>
>>>
>>> At first I thought that this was related to
>>> https://issues.apache.org/jira/browse/DIRMINA-1059. In that ticket the
>>> synchronization was improved. However, I am also running into the problem
>>> with a build of 2.0.17-SNAPSHOT where DIRMINA-1059 was solved.
>>>
>>> So my only hope was DIRMINA-1060 ;) Could this improve the situation?
>>>
>>> Thanks,
>>> Chris.
>>>
>>>
>>> --
>>> Christoph John
>>> Development & Support
>>> Direct: +49 241 557080-28
>>> Mailto:Christoph.John@macd.com
>>>
>>>
>>>
>>> http://www.macd.com <http://www.macd.com/>
>>> ------------------------------------------------------------
>>> ----------------------------------------
>>>
>>> ------------------------------------------------------------
>>> ----------------------------------------
>>> MACD GmbH
>>> Oppenhoffallee 103
>>> <https://maps.google.com/?q=Oppenhoffallee+103&entry=gmail&source=g>
>>> D-52066 Aachen
>>> Tel: +49 241 557080-0 | Fax: +49 241 557080-10
>>> Amtsgericht Aachen: HRB 8151
>>> Ust.-Id: DE 813021663
>>>
>>> Geschäftsführer: George Macdonald
>>> ------------------------------------------------------------
>>> ----------------------------------------
>>>
>>> ------------------------------------------------------------
>>> ----------------------------------------
>>>
>>> take care of the environment - print only if necessary
>>>
>>
>>
>
> --
> Christoph John
> Development & Support
> Direct: +49 241 557080-28 <+49%20241%2055708028>
> Mailto:Christoph.John@macd.com <Ch...@macd.com>
>
>
>
> http://www.macd.com
> ------------------------------
> ------------------------------
> MACD GmbH
> Oppenhoffallee 103
> D-52066 Aachen
> Tel: +49 241 557080-0 | Fax: +49 241 557080-10
> Amtsgericht Aachen: HRB 8151
> Ust.-Id: DE 813021663
>
> Geschäftsführer: George Macdonald
> ------------------------------
> ------------------------------
> take care of the environment - print only if necessary
>
>
> ---------- Forwarded message ----------
> From: Christoph John <ch...@macd.com>
> To: dev@mina.apache.org
> Cc:
> Bcc:
> Date: Wed, 26 Jul 2017 13:59:58 +0200
> Subject: leaking NioProcessors/NioSocketConnectors hanging in call to
> dispose
> Hi,
>
> I am a developer and maintainer of the QuickFIX/J project (
> https://github.com/quickfix-j/quickfixj) and I have a question regarding
> NioSocketConnectors.
>
> We are facing a problem when there is a process that constantly (every 30
> seconds) tries to connect to a counterparty and the connection is
> established but dropped shortly after. Then sometimes the
> NioProcessors/NioSocketConnectors are not cleaned up properly. In the
> stack trace we see them hanging in a call to dispose:
>
> "NioProcessor-1140" #239 prio=5 os_prio=0 tid=0x0000000001fe1800
> nid=0x2523 runnable [0x00007f9c67e8f000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java
> :93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000f6699e60> (a sun.nio.ch.Util$3)
> - locked <0x00000000f6699e50> (a java.util.Collections$Unmodifi
> ableSet)
> - locked <0x00000000f6699c18> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at org.apache.mina.transport.socket.nio.NioProcessor.select(Nio
> Processor.java:98)
> at org.apache.mina.core.polling.AbstractPollingIoProcessor$Proc
> essor.run(AbstractPollingIoProcessor.java:1075)
> at org.apache.mina.util.NamePreservingRunnable.run(NamePreservi
> ngRunnable.java:64)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
>
> "NioSocketConnector-68" #238 prio=5 os_prio=0 tid=0x00007f9c70caf000
> nid=0x2522 in Object.wait() [0x00007f9c6af9f000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at org.apache.mina.core.future.DefaultIoFuture.await0(DefaultIo
> Future.java:209)
> - locked <0x00000000f66ac718> (a org.apache.mina.core.future.De
> faultIoFuture)
> at org.apache.mina.core.future.DefaultIoFuture.awaitUninterrupt
> ibly(DefaultIoFuture.java:141)
> at org.apache.mina.core.polling.AbstractPollingIoProcessor.disp
> ose(AbstractPollingIoProcessor.java:188)
> at org.apache.mina.core.service.SimpleIoProcessorPool.dispose(S
> impleIoProcessorPool.java:329)
> - locked <0x00000000f66ac750> (a java.lang.Object)
> at org.apache.mina.core.polling.AbstractPollingIoConnector$Conn
> ector.run(AbstractPollingIoConnector.java:582)
> at org.apache.mina.util.NamePreservingRunnable.run(NamePreservi
> ngRunnable.java:64)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
>
> It does not happen very often: about 5% of the connection attempts leave a
> NioSocketConnector hanging.
> It only seems to happen though when the connection is disconnected by
> "javax.net.ssl.SSLHandshakeException: SSL handshake failed". Although
> there are cases when there is no leak even on an SSLHandshakeException.
> If the connection was reset "normally" by "java.io.IOException: Connection
> reset by peer" then the leak does not seem to occur. It also does not occur
> when the connection is refused right away.
>
> Since this seems to be related to SSL connections: is there something that
> we need to take care of when using the SSL filter?
>
> The code for the IoSessionInitiator can be found here:
> https://github.com/quickfix-j/quickfixj/blob/master/quickfix
> j-core/src/main/java/quickfix/mina/initiator/IoSessionInitiator.java
> I have added some comments in this gist (starting with "chrjohn"):
> https://gist.github.com/chrjohn/2671f06d80e8d917d9061b573477ec5b
>
> I cannot rule out that we might be doing something wrong here, so any
> pointer is appreciated. :)
>
> Thanks in advance for your help and best regards,
> Chris.
>
> --
> Christoph John
> Development & Support
> Direct: +49 241 557080-28
> Mailto:Christoph.John@macd.com
>
>
>
> http://www.macd.com <http://www.macd.com/>
> ------------------------------------------------------------
> ----------------------------------------
>
> ------------------------------------------------------------
> ----------------------------------------
> MACD GmbH
> Oppenhoffallee 103
> D-52066 Aachen
> Tel: +49 241 557080-0 | Fax: +49 241 557080-10
> Amtsgericht Aachen: HRB 8151
> Ust.-Id: DE 813021663
>
> Geschäftsführer: George Macdonald
> ------------------------------------------------------------
> ----------------------------------------
>
> ------------------------------------------------------------
> ----------------------------------------
>
> take care of the environment - print only if necessary
>
>
Re: leaking NioProcessors/NioSocketConnectors hanging in call to
dispose
Posted by Christoph John <ch...@macd.com>.
Hi,
thanks for your reply.
In fact it is hanging forever, i.e. until the process stops. I have attached the original message
I've sent to the mailing list. It only does occur sometimes for SSL connections with a failing
handshake.
Unfortunately I have no reproducable example for MINA itself. I could probably put something
together for QuickFIX/J (the open source project I am working on).
My OS is Ubuntu 14.04.5, JDK1.8_144 and the problem appears not so often on my machine but almost
every time on the TravisCI build server
(https://travis-ci.org/quickfix-j/quickfixj/builds/283210509). As a result, some of the SSL related
tests are failing. TravisCI has almost similar setup with JDK1.8_144 and Debian Linux.
What would be a good starting point to create a test? I see that there is an SslTest in the
mina-core module. So I probably have to change that test to repeatedly connect and get a handshake
exception everytime and then take a number of stack traces.
Thanks,
Chris.
On 09/10/17 14:51, Jonathan Valliere wrote:
> What OS / Java Version / etc; Do you have a reproducible example?
>
> On Mon, Oct 9, 2017 at 8:34 AM, Jonathan Valliere <jon.valliere@emoten.com
> <ma...@emoten.com>> wrote:
>
> Let me know if its hanging more than 1s
>
> On Mon, Oct 9, 2017 at 5:08 AM, Christoph John <christoph.john@macd.com
> <ma...@macd.com>> wrote:
>
> Hi,
>
> I have another question regarding this one. There is
> https://issues.apache.org/jira/browse/DIRMINA-1060
> <https://issues.apache.org/jira/browse/DIRMINA-1060> which also sounds a little like the
> problem I'm having. When the connectors are hanging in the call to dispose() then there
> always is an accompanying NioProcessor which is hanging in select().
>
> Example:
> "NioProcessor-60" #100328 prio=5 os_prio=0 tid=0x00007f2a10003000 nid=0x2e71 runnable
> [0x00007f2a388b1000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e239c118> (a sun.nio.ch.Util$3)
> - locked <0x00000000e239c108> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e239bed0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:98)
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1075)
> at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>
> "NioSocketConnector-38" #100326 prio=5 os_prio=0 tid=0x00007f2a3001d800 nid=0x2e6f in
> Object.wait() [0x00007f2a1f2d3000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at org.apache.mina.core.future.De
> <http://org.apache.mina.core.future.De>faultIoFuture.await0(DefaultIoFuture.java:209)
> - locked <0x00000000e246ae08> (a org.apache.mina.core.future.De
> <http://org.apache.mina.core.future.De>faultIoFuture)
> at org.apache.mina.core.future.De
> <http://org.apache.mina.core.future.De>faultIoFuture.awaitUninterruptibly(DefaultIoFuture.java:141)
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.dispose(AbstractPollingIoProcessor.java:188)
> at
> org.apache.mina.core.service.SimpleIoProcessorPool.dispose(SimpleIoProcessorPool.java:329)
> - locked <0x00000000e246ae40> (a java.lang.Object)
> at
> org.apache.mina.core.polling.AbstractPollingIoConnector$Connector.run(AbstractPollingIoConnector.java:582)
> at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>
> At first I thought that this was related to
> https://issues.apache.org/jira/browse/DIRMINA-1059
> <https://issues.apache.org/jira/browse/DIRMINA-1059>. In that ticket the synchronization
> was improved. However, I am also running into the problem with a build of 2.0.17-SNAPSHOT
> where DIRMINA-1059 was solved.
>
> So my only hope was DIRMINA-1060 ;) Could this improve the situation?
>
> Thanks,
> Chris.
>
>
> --
> Christoph John
> Development & Support
> Direct: +49 241 557080-28 <tel:%2B49%20241%20557080-28>
> Mailto:Christoph.John@macd.com <ma...@macd.com>
>
>
>
> http://www.macd.com <http://www.macd.com/>
> ----------------------------------------------------------------------------------------------------
>
> ----------------------------------------------------------------------------------------------------
> MACD GmbH
> Oppenhoffallee 103 <https://maps.google.com/?q=Oppenhoffallee+103&entry=gmail&source=g>
> D-52066 Aachen
> Tel: +49 241 557080-0 <tel:%2B49%20241%20557080-0> | Fax: +49 241 557080-10
> <tel:%2B49%20241%20557080-10>
> Amtsgericht Aachen: HRB 8151
> Ust.-Id: DE 813021663
>
> Geschäftsführer: George Macdonald
> ----------------------------------------------------------------------------------------------------
>
> ----------------------------------------------------------------------------------------------------
>
> take care of the environment - print only if necessary
>
>
>
--
Christoph John
Development & Support
Direct: +49 241 557080-28
Mailto:Christoph.John@macd.com
http://www.macd.com <http://www.macd.com/>
----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
MACD GmbH
Oppenhoffallee 103
D-52066 Aachen
Tel: +49 241 557080-0 | Fax: +49 241 557080-10
Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663
Geschäftsführer: George Macdonald
----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
take care of the environment - print only if necessary
Re: leaking NioProcessors/NioSocketConnectors hanging in call to dispose
Posted by Jonathan Valliere <jo...@emoten.com>.
What OS / Java Version / etc; Do you have a reproducible example?
On Mon, Oct 9, 2017 at 8:34 AM, Jonathan Valliere <jo...@emoten.com>
wrote:
> Let me know if its hanging more than 1s
>
> On Mon, Oct 9, 2017 at 5:08 AM, Christoph John <ch...@macd.com>
> wrote:
>
>> Hi,
>>
>> I have another question regarding this one. There is
>> https://issues.apache.org/jira/browse/DIRMINA-1060 which also sounds a
>> little like the problem I'm having. When the connectors are hanging in the
>> call to dispose() then there always is an accompanying NioProcessor which
>> is hanging in select().
>>
>> Example:
>> "NioProcessor-60" #100328 prio=5 os_prio=0 tid=0x00007f2a10003000
>> nid=0x2e71 runnable [0x00007f2a388b1000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java
>> :93)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x00000000e239c118> (a sun.nio.ch.Util$3)
>> - locked <0x00000000e239c108> (a java.util.Collections$Unmodifi
>> ableSet)
>> - locked <0x00000000e239bed0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at org.apache.mina.transport.socket.nio.NioProcessor.select(Nio
>> Processor.java:98)
>> at org.apache.mina.core.polling.AbstractPollingIoProcessor$Proc
>> essor.run(AbstractPollingIoProcessor.java:1075)
>> at org.apache.mina.util.NamePreservingRunnable.run(NamePreservi
>> ngRunnable.java:64)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1149)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>> "NioSocketConnector-38" #100326 prio=5 os_prio=0 tid=0x00007f2a3001d800
>> nid=0x2e6f in Object.wait() [0x00007f2a1f2d3000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at org.apache.mina.core.future.DefaultIoFuture.await0(DefaultIo
>> Future.java:209)
>> - locked <0x00000000e246ae08> (a org.apache.mina.core.future.De
>> faultIoFuture)
>> at org.apache.mina.core.future.DefaultIoFuture.awaitUninterrupt
>> ibly(DefaultIoFuture.java:141)
>> at org.apache.mina.core.polling.AbstractPollingIoProcessor.disp
>> ose(AbstractPollingIoProcessor.java:188)
>> at org.apache.mina.core.service.SimpleIoProcessorPool.dispose(S
>> impleIoProcessorPool.java:329)
>> - locked <0x00000000e246ae40> (a java.lang.Object)
>> at org.apache.mina.core.polling.AbstractPollingIoConnector$Conn
>> ector.run(AbstractPollingIoConnector.java:582)
>> at org.apache.mina.util.NamePreservingRunnable.run(NamePreservi
>> ngRunnable.java:64)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1149)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>> At first I thought that this was related to
>> https://issues.apache.org/jira/browse/DIRMINA-1059. In that ticket the
>> synchronization was improved. However, I am also running into the problem
>> with a build of 2.0.17-SNAPSHOT where DIRMINA-1059 was solved.
>>
>> So my only hope was DIRMINA-1060 ;) Could this improve the situation?
>>
>> Thanks,
>> Chris.
>>
>>
>> --
>> Christoph John
>> Development & Support
>> Direct: +49 241 557080-28
>> Mailto:Christoph.John@macd.com
>>
>>
>>
>> http://www.macd.com <http://www.macd.com/>
>> ------------------------------------------------------------
>> ----------------------------------------
>>
>> ------------------------------------------------------------
>> ----------------------------------------
>> MACD GmbH
>> Oppenhoffallee 103
>> <https://maps.google.com/?q=Oppenhoffallee+103&entry=gmail&source=g>
>> D-52066 Aachen
>> Tel: +49 241 557080-0 | Fax: +49 241 557080-10
>> Amtsgericht Aachen: HRB 8151
>> Ust.-Id: DE 813021663
>>
>> Geschäftsführer: George Macdonald
>> ------------------------------------------------------------
>> ----------------------------------------
>>
>> ------------------------------------------------------------
>> ----------------------------------------
>>
>> take care of the environment - print only if necessary
>>
>
>
Re: leaking NioProcessors/NioSocketConnectors hanging in call to dispose
Posted by Jonathan Valliere <jo...@emoten.com>.
Let me know if its hanging more than 1s
On Mon, Oct 9, 2017 at 5:08 AM, Christoph John <ch...@macd.com>
wrote:
> Hi,
>
> I have another question regarding this one. There is
> https://issues.apache.org/jira/browse/DIRMINA-1060 which also sounds a
> little like the problem I'm having. When the connectors are hanging in the
> call to dispose() then there always is an accompanying NioProcessor which
> is hanging in select().
>
> Example:
> "NioProcessor-60" #100328 prio=5 os_prio=0 tid=0x00007f2a10003000
> nid=0x2e71 runnable [0x00007f2a388b1000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java
> :93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e239c118> (a sun.nio.ch.Util$3)
> - locked <0x00000000e239c108> (a java.util.Collections$Unmodifi
> ableSet)
> - locked <0x00000000e239bed0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at org.apache.mina.transport.socket.nio.NioProcessor.select(
> NioProcessor.java:98)
> at org.apache.mina.core.polling.AbstractPollingIoProcessor$Proc
> essor.run(AbstractPollingIoProcessor.java:1075)
> at org.apache.mina.util.NamePreservingRunnable.run(NamePreservi
> ngRunnable.java:64)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>
> "NioSocketConnector-38" #100326 prio=5 os_prio=0 tid=0x00007f2a3001d800
> nid=0x2e6f in Object.wait() [0x00007f2a1f2d3000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at org.apache.mina.core.future.DefaultIoFuture.await0(DefaultIo
> Future.java:209)
> - locked <0x00000000e246ae08> (a org.apache.mina.core.future.De
> faultIoFuture)
> at org.apache.mina.core.future.DefaultIoFuture.awaitUninterrupt
> ibly(DefaultIoFuture.java:141)
> at org.apache.mina.core.polling.AbstractPollingIoProcessor.disp
> ose(AbstractPollingIoProcessor.java:188)
> at org.apache.mina.core.service.SimpleIoProcessorPool.dispose(S
> impleIoProcessorPool.java:329)
> - locked <0x00000000e246ae40> (a java.lang.Object)
> at org.apache.mina.core.polling.AbstractPollingIoConnector$Conn
> ector.run(AbstractPollingIoConnector.java:582)
> at org.apache.mina.util.NamePreservingRunnable.run(NamePreservi
> ngRunnable.java:64)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>
> At first I thought that this was related to https://issues.apache.org/jira
> /browse/DIRMINA-1059. In that ticket the synchronization was improved.
> However, I am also running into the problem with a build of 2.0.17-SNAPSHOT
> where DIRMINA-1059 was solved.
>
> So my only hope was DIRMINA-1060 ;) Could this improve the situation?
>
> Thanks,
> Chris.
>
>
> --
> Christoph John
> Development & Support
> Direct: +49 241 557080-28
> Mailto:Christoph.John@macd.com
>
>
>
> http://www.macd.com <http://www.macd.com/>
> ------------------------------------------------------------
> ----------------------------------------
>
> ------------------------------------------------------------
> ----------------------------------------
> MACD GmbH
> Oppenhoffallee 103
> <https://maps.google.com/?q=Oppenhoffallee+103&entry=gmail&source=g>
> D-52066 Aachen
> Tel: +49 241 557080-0 | Fax: +49 241 557080-10
> Amtsgericht Aachen: HRB 8151
> Ust.-Id: DE 813021663
>
> Geschäftsführer: George Macdonald
> ------------------------------------------------------------
> ----------------------------------------
>
> ------------------------------------------------------------
> ----------------------------------------
>
> take care of the environment - print only if necessary
>